We seek a skilled Senior Data Quality Engineer with extensive experience in ETL processes, Python testing, and data validation. This critical role involves designing, implementing, and executing comprehensive test strategies to ensure the integrity and reliability of our data pipelines and analytical models. The ideal candidate will have a passion for data quality, a strong background in ETL technologies, and expertise in cloud-based data platforms such as Azure Data Lake.
Test Strategy and Design:
Collaborate with data engineers, analysts, and business stakeholders to understand complex data requirements and workflows.
Develop comprehensive test strategies, plans, and cases for ETL processes, data pipelines, and analytical models.
Design and implement automated testing frameworks to improve efficiency and coverage.
ETL and Data Pipeline Testing:
Validate end-to-end data extraction, transformation, and loading processes to ensure data integrity, accuracy, and consistency.
Perform thorough data quality checks, including profiling, reconciliation, and validation against source systems.
Verify complex data transformations, aggregations, and business rules applied during ETL processes.
Conduct performance testing and optimization of data pipelines.
Python and Analytics Testing:
Develop and execute automated Python tests to validate data processing scripts, analytical models, and machine learning algorithms.
Create reusable test frameworks and libraries to streamline testing processes and promote best practices.
Identify, report, and troubleshoot bugs, inconsistencies, and performance issues in Python-based data processing and analytics workflows.
Develop and maintain utilities for data reconciliation and quality assurance.
Test Execution, Reporting, and Continuous Improvement:
Execute test cases and manage defects using modern test management and version control tools.
Generate comprehensive test reports and effectively communicate findings to development, data, and business teams.
Track and verify defect resolutions, ensuring they are addressed and retested appropriately.
Continuously improve testing methodologies and processes to enhance efficiency and effectiveness.
Cloud Data Platform Expertise:
Work with cloud-based data platforms, notably Azure Data Lake, to ensure data quality and integrity in cloud environments.
Develop and implement testing strategies for data lake ingestion, processing, and consumption patterns.
Leadership and Mentorship:
Provide technical leadership and mentorship to junior team members.
Contribute to developing best practices and standards for data quality assurance across the organization.
Bachelor's degree in Computer Science, Information Systems, or a related field.
5+ years of experience in data quality assurance, focusing on ETL and data pipeline testing.
Strong proficiency in Python programming and testing frameworks (e.g., pytest, unittest).
In-depth knowledge of ETL processes, data warehousing concepts, and SQL.
Experience with cloud-based data platforms, preferably Azure Data Lake.
Familiarity with Informatica PowerCenter or other ETL tools.
Excellent analytical and problem-solving skills.
Strong communication skills and ability to work effectively in cross-functional teams.
Master's degree in a relevant field.
Certifications in data engineering, cloud platforms, or quality assurance.
Experience with big data technologies (e.g., Hadoop, Spark).
Knowledge of data governance and compliance requirements.
Experience with agile methodologies and CI/CD practices.