Building an LLM Evaluation Pipeline That Outlasts Models
An evaluation pipeline designed as durable infrastructure caught 23 regressions and reduced model migration from 6 weeks to 9 days across 4 model transitions.
Adam Mosley
I build systems that turn complexity into clarity — then document the thinking behind them.
Product-oriented leader with 8+ years developing, launching, and scaling digital platforms across the full product lifecycle. I've modernized enterprise systems, automated data workflows, managed portfolios of 1,000+ annual programs, and supported $1M+ in revenue through data-informed strategy. Currently building AI-powered tools and writing about systems, philosophy, and the human experience of building things.
8+
Years in Operations & Data
$1M+
Revenue Supported
30+
Products Launched
15K
Records Recovered & Rebuilt
9,000
Manual Steps Eliminated / Cycle
1,000+
Annual Programs Managed
Core Competencies
Product Development & Lifecycle
Directed development, enhancement, and launch of 30+ digital offerings. Defined product scope, delivery models, pricing, and go-to-market strategy. Built modular learning content and micro-credential pathways from concept through market delivery.
Product lifecycle, Requirements definition, Pricing strategy, Market analysis, Go-to-market planning
Platform Integration & Digital Transformation
Led enterprise platform implementations coordinating functional and technical requirements across IT, vendors, and engineering stakeholders. Integrated enrollment, communication, and credentialing workflows across 5+ systems. Designed automated badge and credential logic.
Canvas LMS, Modern Campus, Salesforce, Cirrus SSO, Apporto, Mailchimp, Power Automate
Data Operations & Analytics
Recovered and rebuilt a 15,000-record SQL database, restoring historical integrity and improving lead conversion. Designed automated data workflows and centralized datasets supporting enrollment, revenue, marketing, and executive reporting. Built dashboards for forecasting and performance tracking.
SQL, Advanced Excel, Power BI, ETL workflows, Data validation, Revenue modeling, Reporting automation
AI Engineering & Automation
Designed multi-agent orchestration systems and local LLM infrastructure. Built automation pipelines replacing hours of manual work with single CLI commands. Developed job application assistants, content publishing workflows, and SEC filing data pipelines processing 36,791 records.
Python, CrewAI, LangChain, FastAPI, Docker, LM Studio, Playwright, MCP
Program Management & Operations
Oversaw delivery of 1,000+ annual programs ensuring scalable operations and consistent quality. Built multi-system scheduling and payroll automation eliminating 9,000 manual interactions per cycle. Coordinated 15+ instructors and managed cross-functional execution across marketing, enrollment, and platform teams.
Jira, Confluence, SharePoint, Process automation, Vendor management, Stakeholder coordination
Financial Analysis & Revenue Strategy
Performed financial analysis, revenue modeling, and multi-year forecasting supporting $1M+ in gross revenue. Designed comprehensive pricing proposals and employer-sponsored training frameworks. Analyzed enrollment funnels to identify conversion gaps and optimize marketing spend.
Revenue modeling, Pricing strategy, Forecasting, Funnel analysis, IRS §127 frameworks
Experience
2025 – Present
Senior Assistant Director
Directing product development and launch of 30+ workforce offerings. Leading enterprise platform implementations across 5+ systems. Performing financial analysis and revenue modeling supporting $1M+ in gross revenue. Managing facility relocation and hybrid learning environment buildout.
2024 – Present
Co-Founder & Consultant
Launched consulting practice focused on small business operations and marketing systems. Established Jira-based project workflows, developed social media strategy, and built vendor outreach and communication systems for client engagements.
2022 – 2025
Senior Program Manager
Oversaw delivery of 1,000+ annual non-credit programs. Recovered and rebuilt a 15,000-record learner database. Built Canvas-based product components and micro-credential pathways. Streamlined instructor onboarding, payroll, and scheduling workflows across 15+ instructors. Scaled operations from 4 staff to 20+.
2018 – 2022
Program Coordinator / Business Support Specialist
Supported registration, logistics, and customer service for 1,000+ annual training sessions. Built a multi-system scheduling and payroll automation tool eliminating 9,000 manual data interactions per cycle. Improved communication workflows supporting 20,000+ annual student interactions.
2017 – 2018
Financial Advisor
Financial analysis, client reporting, regulatory documentation, and compliance.
Prior
Research Assistant / QA Tester
Data validation, QA testing protocols, technical documentation, and test script development in engineering environments.
Selected Case Studies
Local LLMs as Strategic Infrastructure
A hybrid inference architecture routing between local and cloud models cut costs 67% while eliminating data sovereignty concerns for 45,000 daily healthcare queries.
RAG as Data Infrastructure, Not Feature
Redesigning RAG as foundational data infrastructure reduced per-query costs 75% and improved answer accuracy from 67% to 91% across 2.3 million monthly queries.
Recent Essays
-
2 min Architecture
The architecture of meaning: How the tools we build shape what we think is possible
-
4 min Philosophy
Tillich’s Ultimate Concern and the Idol of Productivity
Paul Tillich defined idolatry as elevating a finite thing to ultimate significance. When 61% of technology workers measure self-worth by productivity, productivity has become the industry's idol.
-
4 min AI Systems
Ethics of AI-Assisted Decision Making in Government
Six government AI systems reviewed, none meeting transparency standards required of equivalent human processes. Public systems demand higher ethical standards, yet the opposite is often true.
-
4 min Philosophy
The Absurdity of Optimizing Deprecated Systems
The average enterprise system lives 6.2 years before deprecation. Camus's absurdism reveals that meaning in engineering comes not from permanence but from the quality of the work itself.
-
4 min Data
Data Privacy Engineering Is a Data Engineering Discipline
Implementing tokenization and differential privacy at the pipeline level reduced PII exposure incidents by 89% while adding less than 3% to processing time.
Education
B.A. Psychology
University of Missouri-St. Louis · Expected 2026
Coursework in psychology, philosophy of science, and statistics. Pursuing Clinical Psychology PhD specializing in existential psychology.
Certificates
Excel Power User · SharePoint Collaboration Specialist · Power Automate Efficiency · Access Database Management
Research Interests
- Existential psychology and meaning-making through systems work
- Philosophy of technology, AI ethics, and automation
- Cognitive offloading and human-AI collaboration
- Religious anti-realism and Christian universalism
- Knowledge architecture and personal productivity systems
- Stoic practice applied to knowledge work and leadership
Technical Skills
PythonSQLFastAPIDocker
CrewAILangChainPlaywrightLM Studio
Advanced ExcelPower BIPower AutomateSharePoint
Salesforce CRMCanvas LMSModern Campus
JiraConfluenceGitWordPress
ReportLabPandasPostgreSQL
From the Notebook
🌱 Seedling
Static Site Generation as Architectural Statement
Static site generation eliminates entire categories of runtime failure and security vulnerability. It is a philosophical commitment to simplicity, not a limitation.
🌱 Seedling
The Strangler Fig Pattern and Incremental Change
The Strangler Fig pattern replaces legacy systems incrementally, avoiding the big-bang rewrite that fails 62% of the time. Steady progress over dramatic overhaul.
🌱 Seedling
Vector Database Selection as Data Architecture
Vector database selection is a standard data architecture decision. Evaluate query patterns, scale, operational overhead, integration, and 24-month total cost.
🌱 Seedling
Designing for Context Window Limits
Context window limits force editorial decisions about what a model gets to know. The strategies for managing this constraint are epistemological choices, not just optimizations.