Skip to main content

pySpark Developer with AWS EMR Experience Job in Portland, Oregon United States

Mandatory skills:
  • 5 years of experience in programming with python. Strong proficiency in python
  • Familiarity with functional programming concepts
  • 3 years of Hands on experience in developing ETL data pipelines using pyspark on AWS EMR
  • Hands on experience of XML processing using python
  • Good understanding  of Spark’s RDD API
  • Good understanding of Spark’s Dataframe and API
  • Experience in configuring EMR clusters on AWS
  • Experience and good understanding of Apache Spark Data sources API.
  • Experience of dealing with AWS S3 object storage from Spark. 
  • Experience in trouble shooting spark jobs. Knowledge of monitoring spark jobs using Spark UI
  • Performance tuning of Spark jobs.
  • Understanding fundamental design principles behind a business processes
Nice to have skills:
  • Knowledge of AWS SDK CLI
  • Experience of setting up continuous integration/deployment of spark jobs to EMR clusters
  • Knowledge of scheduling spark applications in AWS EMR cluster.
  • Understanding the differences between Hadoop Mapreduce and Apache Spark
  • Proficient understanding of code versioning tools as Git, SVN
  • Design, development and implementation of  performant ETL pipelines using python API (pySpark) of Apache Spark on AWS EMR
  • Writing reusable, testable, and efficient code
  • Integration of data storage solutions in spark – especially with AWS S3 object storage.
  • Performance tuning of pySpark scripts.
  • Need to ensure overall build delivery quality is good and on time delivery is done at all times.
  • Should be able to handle meetings with customers with ease.
  • Need to have excellent communication skills to interact with customer.
  • Be a team player and willing to work in an onsite offshore model, mentor other folks in the team (onsite as well as offshore)
Send resume to


Popular posts from this blog

HR and Admin Officer Job in Sharjah United Arab Emirates

We have been appointed to recruit an HR and Admin Officer for our Sharjah based client. The position requires the candidate to provide general day-to-day HR support across the Group as part of the Group HR function in relation to all HR, Employee Benefits, Training and Development and other administration for Group employees all in compliance with statutory obligations and best practice. • Working closely with departments, assisting line managers to understand and implement policies and procedures that are set forth in the company. • Promoting equality and diversity as part of the culture of the organization through monitoring that regular appraisals & warnings are carried out by Line Managers before action of any kind is taken. • Liaising with a wide range of people involved in policy areas such as staff performance and health and safety, using the guidelines provided by company. • Assist in Recruiting staff – by liaising with Outlet Managers & advertising their needs after CEO a…

Operations and Marketing Officer - Catering Job in Sharjah United Arab Emirates

Duties and Responsibilities Work, and develop relationships, with external suppliers to ensure the very best reputation within the industry, and receives the service required to ensure that the operational Food and Beverage team can deliver the highest quality product, and the highest financial return Work closely with the Manager to ensure correct stock levels are available from central distribution area, to assist the operational Food and Beverage team Ensure strict compliance with all relevant Hygiene and Safety legislation and requirements Ensure that the industry standard with regard to safety and hygiene Champion a training culture within the Food and Beverage team to ensure succession planning, and a culture that exceeds the very best the industry has to offer The Operations Manager will constantly review the product range to ensure that all key quality standards are maintained The Food & Beverage Department has a wide number of external contractors and the Operations Manager will…

GP Doctor - Functional Medicine Job in Abu Dhabi United Arab Emirates

A well-established well-being center in Dubai is looking for a Female GP Doctor and/or Functional Medicine Doctor to join their highly professional on a full-time basis. Responsibilities Providing skilled health assessment, diagnosis, and treatment services (including contraception) to patients Ordering diagnostic tests as needed, checking and informing patients of results in a timely manner Referring patients appropriately to other providers if their needs exceed the range of care you are able to provide Consulting and collaborating with colleagues to provide optimal care Documenting all care provided and education/information given to patients within their health record, as per professional and company standards Requirements Female is preferredArabic speaker mandatoryPossess a bachelor degree in medicineMinimum five years of relevant work experience post graduationMust have anti-aging and/or medical spa experienceSend resume to

For more jobs in Dubai and other UAE states…