Role Overview
We are seeking a highly skilled Senior Data Pipeline Engineer to join our dynamic team. The ideal candidate will be responsible for designing, building, and optimizing data pipelines to ensure the efficient processing of data. You will work closely with data scientists and stakeholders to deliver robust solutions that support data-driven decision-making, while adhering to best practices in data governance and security.
Key Responsibilities:
- Design and implement scalable data pipelines for processing large datasets
- Optimize existing data workflows for performance and reliability
- Collaborate with data scientists to understand their data requirements
- Monitor and troubleshoot issues within data infrastructure
- Review and improve data integration tools and technologies
- Ensure compliance with data governance and security protocols
- Document data pipeline architecture and processes
- Mentor junior engineers in best practices for data engineering
- Participate in code reviews and contribute to the development of internal standards
Required Skills:
- Proficient in Python and SQL for data manipulation
- Experience with ETL processes and tools
- Familiarity with big data technologies like Hadoop or Spark
- Strong understanding of data warehousing concepts
- Ability to optimize database performance
- Experience with cloud data platforms (e.g., AWS, GCP)
- Knowledge of data modeling and schema design
- Ability to troubleshoot complex data issues
- Excellent problem-solving and analytical skills
Preferred Skills:
- Experience with Kafka or similar streaming solutions
- Familiarity with machine learning pipelines
- Knowledge of Docker and containerization
- Experience with data visualization tools
