Role Overview
We are seeking a highly motivated Lead Data Pipeline Engineer to join our dynamic team. In this role, you will drive the design and implementation of robust data pipelines that support our ambitious analytics goals. You will work closely with data architects and analysts to build scalable data solutions, ensuring data integrity and availability across various platforms. Your experience will help us leverage data effectively, driving insights and innovation across our organization.
Key Responsibilities:
- Design and build data pipelines for large-scale processing of structured and unstructured data.
- Collaborate with data scientists to enhance data processing workflows and improve analytics capabilities.
- Implement best practices for data quality and governance across all data pipelines.
- Optimize existing ETL processes to improve performance and efficiency.
- Monitor and troubleshoot data pipeline issues as they arise, ensuring minimal downtime.
- Document technical specifications and architecture of data pipelines for future reference.
- Train and mentor team members on data pipeline best practices.
- Stay updated with the latest industry trends and technologies in data engineering.
- Participate in code reviews and maintain high standards of code quality.
Required Skills:
- Proficiency in Python, Java, or Scala for data manipulation and analysis.
- Experience with SQL databases and data warehousing solutions.
- Familiarity with Apache Kafka or similar streaming technologies.
- Strong understanding of ETL tools and processes.
- Knowledge of cloud platforms such as AWS or Azure for data storage and processing.
- Experience with data modeling and database design principles.
- Ability to work with large datasets using Big Data technologies like Hadoop, Spark, or similar.
- Excellent troubleshooting and problem-solving skills.
Preferred Skills:
- Experience with machine learning workflows and model deployment.
- Familiarity with Docker and Kubernetes for containerization.
- Knowledge of data visualization tools such as Tableau or Power BI.
