Data Engineer Jobs

Data Engineer with AWS Redshift

Location – San Francisco, CA

Experience – 5 to 8 years

Responsibilities:

• Collaborate with data scientists, analysts, and other stakeholders to understand their data needs and build solutions that enable them to extract insights from data.
• Monitor and troubleshoot data pipelines to ensure data availability and reliability.
• Implement best practices for data governance, data security, and data quality to ensure data integrity across all data sources.
• Create and maintain documentation related to data architecture, data pipelines, and data models.
• Develop and implement effective data architecture solutions using Databricks and Lakehouse
• Stay up to date with emerging technologies and best practices in data engineering and big data processing.
• Provide thought leadership in the Databricks and Lakehouse space, both within the organization and externally
• Optimize and tune data pipelines for performance and scalability.
• Mentor and train other data engineers on best practices for data engineering and Databricks usage

Qualifications:

• Strong communication and collaboration skills, with the ability to work effectively in a team environment.
• 8+ years of experience in data engineering
• Strong understanding of ETL and ELT data ingestion, acquisition, and data processing patterns
• Strong understanding of data engineering, data warehousing, data modeling, data governance, and data security best practices.
• Working knowledge of DLT (Delta Live Tables) and Unity Catalog a plus
• Experience within Programming languages Python, Scala and Java (Python preferred)
• Excellent problem-solving and troubleshooting skills.
• Experience with cloud-based data warehousing platforms such as Synapse, AWS Redshift, Google BigQuery, or Snowflake
• 3+ years of experience working with Databricks and PySpark
• 6-8+ years of experience with SQL
• Appreciation for the Lakehouse medallion data architecture – bronze, silver, gold – and how those data stages are used.
• Bachelor's or Master's degree in Computer Science, Computer Engineering, or a related field
• Experience mentoring and training other data engineers.