Data Engineer – Regulatory Index Platform

About the role

Labrynth is building AI‑powered platforms that simplify regulatory complexity for heavily regulated industries. We are looking for a Data Engineer to design and operate the data platform that powers our Red Tape Index, turning fragmented public data into transparent, auditable rankings.

Key responsibilities

• Develop and maintain resilient scrapers and ingestion pipelines for diverse sources such as open‑data portals, APIs, HTML pages, PDFs, Excel files, and protected commercial sites.
• Design, evolve, and migrate PostgreSQL schemas across multiple jurisdictions and domains.
• Build medallion (bronze → silver → gold) data transforms that are idempotent, content‑hashed, and lineage‑tracked.
• Implement and defend index methodology, including winsorization, percentile ranking, weighting, composite construction, and sensitivity testing.
• Assess data feasibility early, clarify requirements with partners, and translate ambiguous index concepts into executable plans.
• Operate pipelines on Prefect‑driven orchestration with ECS Fargate tasks, ensuring full observability.

Required profile

• Strong Python and SQL expertise with production experience in PostgreSQL schema design and migrations (SQLAlchemy, Alembic or equivalents).
• Hands‑on experience building data pipelines using a lakehouse/medallion approach.
• Proven web‑scraping skills beyond basic requests, including anti‑bot evasion and browser automation.
• Ability to work autonomously in a fast‑moving, client‑embedded engineering team.

Required skills

• Python (3.14 or newer)
• SQL / PostgreSQL
• SQLAlchemy, Alembic
• Prefect (v3)
• ECS Fargate
• HTTP/2 client libraries
• TLS fingerprint evasion techniques
• Browser automation tools (e.g., Selenium, Playwright)
• Polars, uv, ruff, ty, marimo