SEC Filing Intelligence Pipeline

01

What problem did this system solve?

The SEC filing analysis bottleneck: analysts manually reviewing thousands of regulatory documents spent 80% of their time on structural parsing (identifying sections, extracting tables, normalizing formats) and only 20% on the analytical work that required human judgment.

SEC filings on the EDGAR database follow a semi-structured format. Each filing contains structured header data (CIK number, filing date, filing type) and unstructured body content (management discussions, risk factors, financial statements embedded in HTML or XML). The challenge is that “semi-structured” means “inconsistently structured.” Filing formats vary by filer type, by year, by the particular software used to generate the submission.

I needed a system that could ingest the full corpus of filings for a given set of companies, normalize the inconsistencies, extract the analytically relevant sections, and present them in a format that analysts could interrogate directly. The goal was to shift the human effort from parsing to analysis.

Timeline: 6 weeks from initial design to production deployment.

Role: Sole architect and developer.

02

How was the architecture designed?

The pipeline follows a 5-stage ETL architecture: ingestion from EDGAR API, schema validation, format normalization, section extraction, and structured storage with anomaly flagging.

Stage 1: Ingestion. I built a Python client for the EDGAR XBRL API that downloads filings in batches of 100. Rate limiting was critical: EDGAR enforces 10 requests per second per user agent. I implemented exponential backoff with jitter to stay compliant while maximizing throughput.

Stage 2: Validation. Every downloaded filing is validated against an expected schema before entering the pipeline. Records that fail validation are quarantined with the specific validation error tagged. Of 36,791 filings, 8,471 (23%) required quarantine-level examination for structural anomalies.

Stage 3: Normalization. Date fields appeared in 14 formats. Company names contained encoding artifacts. Filing type values numbered 847 unique entries where the specification defined 73. The normalization stage handles each inconsistency with deterministic rules, logging every transformation for auditability.

Stage 4: Extraction. Section extraction uses a combination of regex patterns for well-formatted filings and NLP-based section detection for irregular formats. I trained a simple classifier on 500 manually labeled sections to identify Management Discussion, Risk Factors, and Financial Statements regardless of formatting.

Stage 5: Storage. Extracted sections are stored in PostgreSQL with full-text search indexes. A parallel vector index (using sentence-transformer embeddings) enables semantic search across the corpus for the RAG layer that analysts use for natural-language questions.

Tech stack: Python, PostgreSQL, SEC EDGAR API, sentence-transformers, FastAPI, Docker.

03

What were the measurable outcomes?

The pipeline reduced per-filing analysis time from an average of 45 minutes to 12 minutes (73% reduction) while surfacing anomalies in 23% of filings that manual review had not detected.

• Processing throughput: Full corpus ingestion and processing completes in 4.2 hours (previously required manual review over several weeks).
• Analyst time reallocation: Analysts shifted from 80% parsing / 20% analysis to 15% parsing / 85% analysis. The remaining parsing effort is reviewing quarantined records.
• Anomaly discovery: The 23% anomaly rate was significantly higher than expected. Manual processes had been overlooking formatting irregularities because human reviewers normalized inconsistencies unconsciously. The pipeline made these visible.
• RAG layer performance: The semantic search layer achieves 89% retrieval accuracy on targeted queries about specific filings, with structured metadata pre-filtering reducing incorrect retrievals from 11% to 4%.

04

What would I change in hindsight?

I would invest earlier in streaming architecture and build the anomaly classification as a supervised model rather than rule-based detection.

The batch processing approach works for the current corpus size but will not scale linearly. If the filing volume doubled, the 4.2-hour processing window would become a constraint. A streaming architecture that processes filings as they appear on EDGAR would eliminate the batch window entirely.

The anomaly detection is currently rule-based: specific format deviations, value ranges, and consistency checks. This catches known anomaly patterns but misses novel ones. A supervised anomaly classifier trained on the 8,471 quarantined records would generalize better to new anomaly types. I deprioritized this in favor of shipping a working system. The rule-based approach was sufficient for launch but will need replacement as the corpus grows.

The vector embedding model (sentence-transformers all-MiniLM-L6-v2) was chosen for speed over accuracy. A domain-specific embedding model fine-tuned on financial text would likely improve the 89% retrieval accuracy. The cost is fine-tuning effort and increased inference latency. I would make this tradeoff now that the base system has proven its value.