Role Overview
We are seeking an experienced ETL Data Engineer who will be responsible for designing, developing, and maintaining data pipelines. The ideal candidate will have a strong background in data integration and an ability to manage large-scale datasets efficiently. You will collaborate with various stakeholders to ensure data quality and accessibility while utilizing best practices in data engineering to streamline processes and drive insights from our data assets.
Key Responsibilities:
- Design and implement robust ETL processes to extract, transform, and load data from various sources.
- Optimize data pipelines for performance and efficiency in loading large-scale datasets.
- Collaborate with data analysts and stakeholders to understand data requirements and provide necessary solutions.
- Maintain and improve existing data models and ETL workflows to ensure data accuracy and availability.
- Conduct data quality checks and validate data completeness throughout the processing stages.
- Monitor and troubleshoot data integration issues, and implement timely fixes as needed.
- Develop and maintain documentation for all ETL processes and data flows.
- Implement security measures for sensitive data processing in line with company policies.
- Stay current with emerging technologies and best practices in data engineering.
- Participate in code reviews to ensure quality and adherence to best practices.
Required Skills:
- Proven experience in Python or SQL for data manipulation.
- Hands-on knowledge of ETL tools such as Talend, Informatica, or Apache Nifi.
- Experience with data warehousing concepts and architecture.
- Familiarity with cloud platforms like AWS or Azure for data storage.
- Understanding of database technologies like PostgreSQL or MySQL.
- Strong analytical and problem-solving skills with data analysis techniques.
- Experience in performance tuning and optimization of data pipelines.
- Knowledge of API integrations for data ingestion.
- Familiarity with NoSQL databases like MongoDB or Cassandra.
Preferred Skills:
- Experience with Apache Spark or Hadoop frameworks.
- Knowledge of data modeling concepts and techniques.
- Familiarity with machine learning and its integration with data engineering.
- Proficient in data visualization tools such as Tableau or Power BI.
