01 — Problem
Every Process Was a Snowflake
Across the workforce education operation I managed, nothing ran the same way twice. Enrollment processing involved 6 manual steps across 3 platforms. Credential verification required a human to cross-reference state databases, update the CRM, and notify the student — a process that took 25 minutes per record and was performed 40–60 times per month. Each process had been documented in a Word file somewhere, but the documentation was always stale because no one updated it after the process changed.
I didn’t need another documentation system. I needed an execution engine where the documentation was the process — a framework where defining a workflow in code meant it could be run, monitored, rolled back, and audited from the same interface.
02 — Architecture
Tasks as Nodes, Dependencies as Edges
The framework models every workflow as a directed acyclic graph (DAG) of tasks. Each task is a Python function decorated with metadata: inputs, outputs, timeout, retry policy, and rollback handler. The execution engine resolves the dependency graph, runs tasks in topological order, and captures structured logs at every transition.
Task Definition Layer
Tasks are defined as decorated Python functions with explicit type annotations. A @task decorator registers the function with the scheduler and enforces that its return type matches the next task’s input type. This catches integration errors at definition time, not at runtime.
Execution Engine (Celery + Docker)
Celery handles distributed task execution with Redis as the message broker. Each task runs in an isolated Docker container, which means a failing task can’t corrupt the state of other running tasks. Container isolation also makes the framework portable — the same workflow definition runs identically on my local machine and on a cloud instance.
Rollback and Audit Layer
Every task can optionally define a rollback() method. If task 4 in a 6-task pipeline fails, the engine calls rollback() on tasks 3, 2, and 1 in reverse order. Every state transition — start, success, failure, rollback — is written to a structured JSON log with timestamps, input hashes, and output snapshots.
Key Design Decisions
Why DAGs instead of linear sequences? Many real workflows have parallel branches. Credential verification and enrollment confirmation can run simultaneously — they don’t depend on each other. A DAG scheduler exploits this parallelism automatically, cutting wall-clock time without the developer thinking about concurrency.
Why container isolation per task? Early versions ran all tasks in the same Python process. A memory leak in one task would degrade all subsequent tasks in the pipeline. Container isolation added ~2 seconds of overhead per task but eliminated an entire class of cascading failure.
03 — Outcomes
Measured Results
Workflows Automated
from enrollment processing to credential verification
Time Reduction
on credential verification — from 25 min to 7 min per record
Audit Coverage
every task transition logged with inputs, outputs, and timestamps
Silent Failures
all errors surface immediately with structured context
04 — Reflection
Documentation That Executes
The framework solved the stale documentation problem by eliminating documentation as a separate artifact. The workflow definition is the documentation. When someone asks “how does credential verification work?”, the answer is the code itself — not a Word file that may or may not reflect the current process.
What I’d change: the Celery dependency makes local development heavier than it needs to be. For workflows with fewer than 10 tasks and no parallelism, a simple sequential executor would be sufficient. I’d add a mode="local" flag that bypasses Celery entirely for development and testing.
“A process that can’t be audited doesn’t exist. It’s just a habit that hasn’t failed visibly yet.”
Outcomes
8 workflows automated; 73% time reduction on credential verification; 100% audit log coverage; 0 silent failures in production