Evaluating LLMs Is the Hardest Problem in AI Engineering

Why is evaluation harder than building the AI system itself?

Evaluation is harder because it requires you to define “correct” for tasks where correctness is subjective, multidimensional, and context-dependent, a problem that is fundamentally philosophical before it is technical.

Consider a simple question: “Is this summary good?” To answer that, you need criteria. Factual accuracy? Completeness? Conciseness? Readability? Relevance to the user’s intent? Each criterion requires its own measurement methodology. Factual accuracy can be verified against source documents, but completeness is relative to the user’s unstated information needs. Conciseness and readability are preferences that vary by audience. Relevance depends on context that may not be fully captured in the evaluation dataset.

Now multiply that complexity across every task your application performs. A customer support agent needs evaluation for answer accuracy, tone appropriateness, policy compliance, and escalation judgment. A document analysis tool needs evaluation for extraction precision, classification accuracy, and handling of ambiguous inputs. Each dimension requires test cases, scoring rubrics, and ideally both automated metrics and human judgment.

What is the difference between model benchmarks and product evals?

Model benchmarks measure a model’s general capabilities in isolation, while product evals measure how well the model performs within your specific application’s context, data, and constraints, and the correlation between the two is often weak.

I have seen teams select a model based on its MMLU score and then discover it performs poorly in their application. MMLU measures multi-task language understanding across academic domains. It tells you nothing about whether the model can correctly parse your company’s invoice format or follow your specific system prompt’s instructions about tone. A model that scores 89% on MMLU might score 62% on your product eval because your task has specific requirements (domain vocabulary, output format constraints, edge cases) that general benchmarks do not capture.

Product evals must be built from your data, your users’ queries, and your definition of success. I start every evaluation project by collecting 200-500 real user interactions, having domain experts label them (correct/incorrect, with annotations for why), and using these as the foundation for an automated eval suite. This investment, typically 40-80 hours of labeling work, is the single highest-ROI activity in AI application development.

How do you build an evaluation pipeline that actually works?

A production evaluation pipeline requires 4 layers: unit evals (testing individual components), integration evals (testing the full pipeline), regression evals (ensuring changes do not break existing capabilities), and continuous monitoring (detecting quality degradation in production).

• Unit evals: Test each component of your AI pipeline independently. For a RAG system, this means evaluating retrieval precision separately from generation quality. I use NDCG@10 for retrieval and a rubric-based LLM judge for generation. Separating these reveals whether a quality problem originates in retrieval or generation.
• Integration evals: Test the full pipeline end-to-end on representative inputs. These catch interaction effects that unit evals miss. A retrieval system that returns relevant documents and a generation model that produces good summaries can still fail when combined if the context assembly step introduces formatting that confuses the model.
• Regression evals: Run a fixed test suite before every deployment. I maintain a “golden set” of 150-300 examples that covers core capabilities, known edge cases, and previously discovered bugs. If a deployment degrades performance on any golden set category by more than 2 percentage points, it is blocked.
• Continuous monitoring: Production quality monitoring using LLM-as-judge on a sample of live traffic. I sample 5% of production queries, run them through an evaluation model (typically a different model than the production model to avoid self-evaluation bias), and track quality scores over time. This detects gradual degradation that regression evals on static data cannot catch.

Why is LLM-as-judge both necessary and problematic?

LLM-as-judge (using one language model to evaluate another’s output) is necessary because human evaluation does not scale, but problematic because models have systematic biases (verbosity preference, sycophancy, position bias) that can corrupt evaluation results.

I use LLM judges extensively. The alternative, human evaluation of every output, costs $2-8 per evaluation at production volumes. For a system handling 10,000 queries per day, even sampling 5% would require 500 human evaluations daily, roughly $2,000/day. An LLM judge costs $0.002 per evaluation.

But the biases are real. I have measured a consistent 12-15% inflation in quality scores when using the same model family as judge and subject (GPT-4 judging GPT-4 output, for example). Models prefer verbose answers over concise ones, agreeable answers over critical ones, and the first option in multiple-choice evaluations over later options. Controlling for these biases requires careful rubric design, model diversity (using a different model family as judge), and periodic calibration against human judgments.

The calibration process I follow: every 2 weeks, I send 50 production outputs to human evaluators and compare their scores to the LLM judge’s scores. If the correlation coefficient drops below 0.75, I retune the judge’s rubric. This calibration loop is the mechanism that keeps automated evaluation honest.

What is the philosophy of measurement in AI systems?

Every evaluation metric embeds a value judgment about what matters, and the decision to measure one dimension over another shapes the system’s development trajectory more powerfully than any architectural choice.

There is a concept in philosophy of science called the “theory-ladenness of observation,” the idea that what you observe depends on the theoretical framework you bring to the observation. Evaluation in AI engineering is theory-laden in exactly this way. If you measure accuracy, you optimize for correctness. If you measure user satisfaction, you optimize for helpfulness (which may diverge from correctness). If you measure latency alongside accuracy, you optimize for the tradeoff between speed and quality.

I have watched teams optimize for the wrong metric and produce systems that score well on their eval suite but fail in production. A customer support agent evaluated purely on answer accuracy produced responses that were technically correct but emotionally tone-deaf. A document extraction system evaluated on precision but not recall missed 34% of relevant information, which users experienced as the system “not working.” The metrics you choose are not neutral instruments. They are architectural decisions that encode your values into the system.

The hardest part of evaluation is not building the pipeline. It is deciding what to measure. That decision requires understanding your users, your domain, and your own biases about what “good” means. It is, at its core, an exercise in applied epistemology: how do you know that your system knows what it claims to know? The teams that take this question seriously build better systems. The teams that skip it are guessing.