Data platforms where pipelines fail loudly — not silently
I design and build cloud data platforms that survive partial failures, schema drift, and operational scale — incremental ingestion with checkpoint recovery, contract-driven data quality, medallion lakehouse patterns, and CI-validated platform automation on AWS, Azure, Databricks, and Snowflake.
Senior data engineering, to me, is judgment under constraint: intentional trade-offs, failure-aware design, and systems the next team can actually operate.
Design for failure.
Automate repeatable work.
Measure data quality.
Document decisions.
Keep systems understandable.
| Signal | How it shows up |
|---|---|
| System design | Architecture docs, ADRs, and clear boundaries between ingestion, transform, quality, and platform layers |
| Failure handling | Checkpoint recovery, idempotent loads, quality gates before bronze, operations runbooks |
| Operational proof | pytest, GitHub Actions CI, smoke tests, alert routing, and honest trade-off documentation |
| ⭐ Lakehouse |
lakehouse-platform-starter — Runnable reference architecture: Airflow + Cosmos · dbt · Iceberg · Trino · OpenLineage · Great Expectations
|
Connected layers — not isolated demo repos:
| Layer | Project | Focus |
|---|---|---|
| Flagship · Lakehouse | lakehouse-platform-starter | Iceberg + Trino + Cosmos dbt + Airflow + Marquez + GE · Docker stack · CI · hosted docs |
| Ingest & transform | production-data-pipeline | Incremental API · PostgreSQL bronze · dbt · Airflow · v0.1.0 |
| Quality & observability | data-quality-observability | YAML contracts · schema/freshness checks · run history · alerts |
| Platform & governance | cloud-lakehouse-blueprint | Medallion manifests · Terraform · IAM · lineage · CI validation |
| Domain | Technologies & practices |
|---|---|
| Data engineering | Python · SQL · incremental ingestion · ETL/ELT · API pipelines · checkpointing · idempotent loads |
| Data architecture | Medallion lakehouse · Iceberg · bronze/silver/gold · lineage · governance · cost modeling |
| Orchestration | Apache Airflow · Cosmos · dbt · Prefect · Spark |
| Cloud platforms | AWS · Azure · Databricks · Snowflake |
| Quality & observability | Data contracts · Great Expectations · OpenLineage · schema validation · CI/CD |
Production operations knowledge contributed upstream:
| Project | PR | Change |
|---|---|---|
| Airflow | #70185 open | dbt Cloud job metadata on OpenLineage events (#68661) |
| Prefect | #22500 ✓ merged | Kubernetes readiness vs liveness probes |
| Prefect | #22533 | Global concurrency limit setup docs |
| dbt docs | #9606 | Prefixed custom schema troubleshooting |
| Article | Topic |
|---|---|
| Building a Production Data Pipeline with Incremental Loading and dbt | Incremental checkpoints, idempotent loads, medallion layering, Airflow orchestration, failure modes |
More at br413.github.io
Open to senior data engineering roles, data platform architecture discussions, and technical collaboration.
Website: br413.github.io · Flagship: lakehouse-platform-starter · dbt docs: live
data engineering · lakehouse · dbt · Airflow · Iceberg · Trino · OpenLineage · Terraform