Senior Cloud Data Pipeline Specialist at beBeeCloudDataPipeline

Job Description:

Latest job information from beBeeCloudDataPipeline for the position of Senior Cloud Data Pipeline Specialist. If the Senior Cloud Data Pipeline Specialist vacancy in Karnataka matches your qualifications, please submit your latest application or CV directly through the updated Jobkos job portal.

Please note that applying for a job may not always be easy, as new candidates must meet certain qualifications and requirements set by the company. We hope the career opportunity at beBeeCloudDataPipeline for the position of Senior Cloud Data Pipeline Specialist below matches your qualifications.

Job Overview

We are seeking a highly skilled and experienced Senior Cloud Data Pipeline Specialist to join our team. In this role, you will be responsible for designing, developing, and implementing performant ETL pipelines using Python API (PySpark) of Apache Spark on AWS EMR.

Key Responsibilities:

• Design and Development: Design, develop, and implement end-to-end data pipeline solutions using PySpark and Apache Spark on AWS EMR.
• Code Quality and Efficiency: Write reusable, testable, and efficient code that meets the highest standards of quality and performance.
• Data Storage and Integration: Integrate data storage solutions in Spark, with a focus on AWS S3 object storage. Perform performance tuning of PySpark scripts to ensure optimal data processing.
• Meetings and Communication: Handle meetings with customers confidently and effectively communicate technical information to non-technical stakeholders.

Requirements:

• Experience: 8+ years of experience in programming with Python, with strong proficiency in the language.
• Functional Programming Concepts: Familiarity with functional programming concepts and their application in data pipeline development.
• ETL Pipelines and Data Lakes: 3+ years of hands-on experience in developing ETL data pipelines using PySpark on AWS EMR and building data lakes for large enterprises on AWS.
• Spark Knowledge: Good understanding of Spark's DataFrame and API, as well as experience in configuring EMR clusters on AWS and dealing with AWS S3 object storage from Spark.
• Troubleshooting and Performance Tuning: Experience in troubleshooting spark jobs, knowledge of monitoring spark jobs using Spark UI, and performance tuning of spark jobs.

Databricks-Specific Skills:

• Proof of Concept Solutions: Experience in developing and delivering end-to-end Proof of Concept (POC) solutions, including creating jobs and configuring clusters in Databricks.
• Notebook Orchestration and Unity Catalog: Basic proficiency in Databricks, notebook orchestration, and implementing modular code structures to enhance scalability and maintainability, as well as understanding of Unity Catalog and its role in data governance.

Important Considerations:

• Hands-on Experience: Candidates must have actual hands-on work experience, not just home projects or academic exercises.
• Clear Communication: Profiles should clearly state how much experience they have in each skill area, and candidates must know their CV/profile inside out, including all projects and responsibilities listed.

Job Info:

• Company: beBeeCloudDataPipeline
• Position: Senior Cloud Data Pipeline Specialist
• Work Location: Karnataka
• Country: IN

How to Submit an Application:

After reading and understanding the criteria and minimum qualification requirements explained in the job information Senior Cloud Data Pipeline Specialist at the office Karnataka above, immediately complete the job application files such as a job application letter, CV, photocopy of diploma, transcript, and other supplements as explained above. Submit via the Next Page link below.