Data Quality Engineer (Azure & Databricks)

About the role

We are looking for a Data Quality Engineer to ensure the reliability and consistency of data across our modern Azure‑based platform. You will work closely with data engineering and business teams to validate pipelines and deliver production‑ready data assets.

Key responsibilities

• Design and implement a data quality framework across Bronze, Silver, and Gold layers, defining validation rules, thresholds, and alerting standards.
• Build and maintain automated data quality checks within Databricks pipelines, including row counts, null checks, referential integrity, schema validation, and business rule assertions.
• Own reconciliation between source systems and Databricks layers to ensure accurate data landing and correct transformation outputs.
• Validate identity resolution outputs in the Silver layer, reviewing match rates and investigating false positives and negatives.
• Perform end‑to‑end pipeline testing from ingestion through the Gold layer and verify downstream reporting accuracy.
• Partner with Data Engineers to define acceptance criteria for each sprint’s pipeline and data model deliverables before production promotion.
• Support user acceptance testing with client stakeholders to confirm Gold layer outputs meet reporting requirements.
• Document QA processes, test results, and data quality findings for client handoff at engagement close.
• Monitor pipeline health post‑deployment, triage data quality incidents, and work with engineers to resolve root causes quickly.

Required profile

• Experience with Azure‑based data platforms, especially Databricks.
• Strong understanding of data quality frameworks and testing methodologies for data pipelines.
• Proven experience validating ETL/ELT processes and working with layered architectures (Bronze, Silver, Gold).
• Strong SQL skills and ability to analyze large datasets.

Required skills

• Azure
• Databricks
• SQL