Platform Architect (Databricks Lakehouse)

About the role

We are seeking a Platform Architect with deep expertise in the Databricks Lakehouse ecosystem to lead the design and implementation of end‑to‑end data solutions. This remote position will focus on building reliable, production‑grade pipelines and governance frameworks across modern cloud environments.

Key responsibilities

• Design and implement Lakehouse Medallion Architecture (Bronze, Silver, Gold) for scalable data processing.
• Develop, troubleshoot, and tune large‑scale Apache Spark jobs using caching, partitioning, and broadcast joins.
• Architect Delta Lake solutions, ensuring ACID compliance, schema enforcement, time‑travel, and performance optimizations such as Z‑ordering.
• Create unified data and AI governance models with Unity Catalog, including RBAC, row/column‑level security, and lineage tracking.
• Integrate and optimise cloud infrastructure on AWS, Azure, or GCP, managing services like ADLS, S3, IAM, and virtual networks.
• Monitor DBUs, right‑size serverless and multi‑node clusters, and implement cost‑control best practices.

Required profile

• Proven experience designing end‑to‑end Lakehouse architectures and production pipelines.
• Deep understanding of Spark internals and performance tuning.
• Strong knowledge of Delta Lake and Unity Catalog governance features.
• Hands‑on experience with major cloud platforms (AWS, Azure, GCP) and their native storage and security services.
• Advanced SQL proficiency and fluency in Python or Scala.
• Ability to monitor and optimise Databricks usage to avoid unexpected cloud costs.

Required skills

• Databricks Lakehouse Platform
• Apache Spark
• Delta Lake
• Unity Catalog
• AWS (S3, IAM, VNet)
• Azure (ADLS, Entra)
• Google Cloud Platform
• Advanced SQL
• Python
• Scala