Junior Data Engineer, Product Development

THE ROLE:

The Junior Data Engineer role at our company presents an exciting opportunity to work on large-scale data infrastructure projects and contribute to the development and maintenance of our data platform. The ideal candidate will have a strong background in data engineering, data architecture, and cloud technologies. As a data engineer, you will collaborate with cross-functional teams including data scientists, analysts, and software engineers to build and optimize data pipelines, data integration processes, and data storage solutions.

KEY RESPONSIBILITIES AND ACCOUNTABILITIES:

• Document and maintain data engineering processes, including data pipeline configurations, job schedules, and monitoring and alerting mechanisms.
• Collaborate with software engineers to integrate data engineering solutions with other systems and applications.
• Provide mentorship and guidance to junior data engineers, promoting best practices in data engineering and ensuring the growth and development of the team.
• Optimize and tune performance of data processes and workflows to ensure efficient data processing and analysis at scale.
• Design, develop, and maintain data pipelines, ETL/ELT processes, and data integrations to support efficient and reliable data ingestion, transformation, and loading.
• Work closely with infrastructure and DevOps teams to ensure the scalability, security, and availability of the data platform and data storage systems.
• Continuously evaluate and recommend new technologies, tools, and frameworks to improve the efficiency and effectiveness of data engineering processes.
• Collaborate with data scientists, data analysts, and other stakeholders to understand data requirements and ensure the availability, reliability, and accuracy of the data.
• Implement data governance practices, including data quality monitoring, data lineage tracking, and metadata management.
• Stay up-to-date with industry trends and advancements in data engineering, cloud technologies, and data processing frameworks.

PROFESSIONAL EXPERIENCE/QUALIFICATIONS/SKILLS:

• Experience in building and optimizing ETL/ELT processes, data pipelines, and workflows, using tools like Apache Airflow, Apache Kafka, or AWS Glue.
• Experience with version control systems, such as Git, and knowledge of CI/CD practices in a data engineering context.
• Strong problem-solving and troubleshooting skills, with the ability to identify and resolve data quality and performance issues.
• Knowledge of data security and privacy principles, as well as experience implementing data access controls and managing sensitive data.
• Candidates should have at least 2-3 years of experience in data engineering or a related field, with a solid understanding of cloud platforms and big data technologies.
• Strong proficiency in Python or another programming language commonly used in data engineering, such as Scala or Java.
• Excellent communication and collaboration skills, with the ability to work effectively in cross-functional teams and translate business requirements into technical solutions.
• Familiarity with data modeling, data warehousing concepts, and data governance best practices.
• Understanding of distributed computing principles and experience with distributed data processing frameworks like Apache Spark or Hadoop.
• In-depth knowledge of cloud platforms such as AWS, Azure, or Google Cloud, and experience with the services and tools they offer for data storage, processing, and analytics (e.g., S3, Redshift, BigQuery, Dataflow).
• Familiarity with containerization technologies like Docker and orchestration tools like Kubernetes.

EDUCATION:

• Bachelor's degree required; master’s degree preferred