What problem does this system address?
Most data infrastructure guidance assumes teams of 5 to 15 engineers, making technology choices (Kubernetes, Spark, Airflow, complex lakehouse architectures) that a single person cannot operate, debug, and maintain simultaneously, leaving solo data engineers without practical guidance for their actual constraint.
I was the sole data engineer for a 120-person company. The “modern data stack” recommendations I found assumed I could maintain a separate ingestion tool, transformation tool, orchestration tool, BI tool, data quality tool, and catalog. That is 6 tools, each requiring configuration, monitoring, upgrades, and troubleshooting. With 40 hours per week divided across 6 tools, stakeholder support, and actual pipeline development, each tool got approximately 3 hours per week of attention. That is not enough to maintain anything reliably. The architecture had to be simpler.
How is the system structured?
The system minimizes operational surface area by choosing tools that overlap in capability, that run with minimal infrastructure, and that a single person can debug at 2am without consulting documentation.
Step 1: Database (DuckDB + PostgreSQL)
DuckDB serves as the analytical engine for transformation and querying. It runs as an embedded database (no server to maintain), handles columnar analytics on datasets up to 100GB on a single machine, and integrates natively with Python and dbt. PostgreSQL serves as the operational store for metadata, schedules, and application state. Two databases, one serverless and one managed, covering all storage needs. According to DuckDB’s design philosophy, an in-process analytical database eliminates an entire category of infrastructure management.
Step 2: Orchestration and transformation (Dagster + dbt)
Dagster orchestrates all pipelines with an asset-centric model that makes pipeline state visible in a single UI. dbt handles SQL transformations with testing and documentation built in. Dagster runs on a single VM ($40/month) with automatic restarts. The Airflow alternative was justified here because Dagster’s operational footprint is smaller for a solo operator: no separate scheduler, webserver, and worker processes.
Step 3: BI and serving (Metabase)
Metabase provides self-service dashboards and ad-hoc querying for stakeholders. It connects directly to PostgreSQL (for operational dashboards) and to DuckDB-exported Parquet files via DuckDB (for analytical dashboards). The managed Metabase Cloud plan ($85/month) eliminates server management entirely. The dashboard design discipline is critical here: with limited maintenance capacity, every dashboard must justify its existence.
How do you validate it works?
The one-person system succeeds when the operator spends less than 30% of their time on infrastructure maintenance, stakeholder satisfaction scores remain above 80%, and the platform can survive a 1-week absence without manual intervention.
I track three metrics. Maintenance ratio: hours spent on infrastructure versus hours spent on data development. Target is below 30%. Currently at 22%. Stakeholder NPS: quarterly survey of data consumers. Currently at 82 (up from 61 before the redesign). Hands-off survival: the longest period the platform ran without my intervention. Currently 12 days (during a vacation). The total monthly cost: $40 (VM) + $85 (Metabase) + $120 (managed PostgreSQL) + $50 (monitoring and storage) = $295. Even at peak, with additional compute for heavy processing months, the cost has not exceeded $600.