Role Overview
We are seeking a highly skilled Senior Data Pipeline Engineer to join our data team. This role involves designing, building, and maintaining scalable data pipelines that process large volumes of data efficiently. The ideal candidate should have a strong background in data engineering, with experience in various technologies for data manipulation and storage. You will collaborate with data scientists and analysts to deliver impactful data solutions that drive business insights.
Key Responsibilities:
- Design and implement data pipelines to automate data movement and transformation.
- Optimize existing pipelines for performance and reliability.
- Collaborate with data teams to define data requirements and ensure alignment.
- Monitor and troubleshoot data flow issues in real-time.
- Develop documentation for pipeline processes and architecture.
- Ensure data quality and integrity throughout the data lifecycle.
- Evaluate and incorporate new data technologies and tools as needed.
- Establish best practices for data governance and data security.
- Work closely with stakeholders to support data analytics initiatives.
Required Skills:
- Proficient in Python or similar programming languages.
- Experience with Apache Kafka or similar messaging systems.
- Strong understanding of SQL and database technologies.
- Knowledge of ETL processes and data warehousing.
- Familiarity with cloud platforms such as AWS or GCP.
- Ability to work with data lakes and big data technologies.
- Experience with Docker and container orchestration tools.
- Strong problem-solving and analytical skills.
- Good communication and team collaboration skills.
Preferred Skills:
- Experience with Apache Airflow for workflow management.
- Familiarity with machine learning concepts.
- Knowledge of NoSQL databases.
- Experience in data visualization tools.
- Understanding of DevOps practices.
