Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics

Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics

How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment

The Problem: Why "Vibe Checks" Fail in Production

Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing - we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation. Our test process was literally "ask 5 questions, read answers, thumbs up."

What Production Evaluation Actually Needs

Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs:

• Domain-specific judges - Your criteria, not generic "helpfulness"
• Speed - Evaluation must run in CI/CD, not overnight
• Regression detection - Know immediately when a prompt change breaks things
• CI/CD integration - Block merges that degrade quality
• Golden dataset management - Versioned, stratified, growing test cases

Architecture: The Evaluation Pipeline

┌─────────────┐   ┌──────────────┐   ┌────────────────────┐   ┌──────────────┐
│ Test Cases  │───▶│ LLM Under    │───▶│ Judge Ensemble     │───▶│ Metrics &    │
│ (Golden Set)│   │ Test         │   │ - Faithfulness     │   │ Regression   │
└─────────────┘   └──────────────┘   │ - Instruction F.    │   │ Detection    │
                                     │ - JSON Schema       │   └──────┬───────┘
                                     │ - Custom LLM        │          ▼
                                     └────────────────────┘   ┌──────────────┐
                                                              │ Dashboard/   │
                                                              │ PR Comments  │
                                                              └──────────────┘

Core Abstractions

# eval/base.py
@dataclass(frozen=True)
class TestCase:
    id: str
    input: dict[str, Any]
    expected: dict[str, Any] | None = None
    tags: list[str] = field(default_factory=list)  # ["edge-case", "long-context"]

@dataclass(frozen=True)
class EvaluationResult:
    test_case_id: str
    judge_name: str
    score: float
    passed: bool
    reasoning: str

class Judge(ABC):
    @abstractmethod
    async def evaluate(self, test_case: TestCase, response: Any) -> EvaluationResult: ...

class EvaluationHarness:
    def __init__(self, judges: list[Judge]):
        self.judges = judges

    async def evaluate_all(self, test_cases, generate_fn, concurrency=10):
        # Runs all cases through all judges with controlled concurrency
        ...

The Judge Ensemble: Beyond RAGAS

RAGAS gives you faithfulness and answer relevance. But production needs more:

Custom LLM Judge with Few-Shot

# eval/judges.py
class LLMJudge(Judge):
    def __init__(self, name, criteria, model="gpt-4o-mini", few_shot_examples=None):
        self.name = name
        self.criteria = criteria
        self.model = model
        self.few_shot = few_shot_examples or []

    async def evaluate(self, test_case, response):
        client = instructor.from_openai(AsyncOpenAI())
        class Output(BaseModel):
            score: float = Field(ge=0, le=1)
            reasoning: str
            passed: bool

        result = await client.chat.completions.create(
            model=self.model,
            response_model=Output,
            messages=[
                {"role": "system", "content": self._system_prompt()},
                *self._few_shot_messages(),
                {"role": "user", "content": self._build_prompt(test_case, response)},
            ],
            temperature=0.0,
        )
        return EvaluationResult(...)

Faithfulness Judge (Production-Ready)

def create_faithfulness_judge() -> LLMJudge:
    return LLMJudge(
        name="faithfulness",
        criteria="""
Evaluate whether the ANSWER is faithful to the CONTEXT.
- Score 1.0: All claims in answer are directly supported by context
- Score 0.5: Some claims unsupported but not contradictory
- Score 0.0: Answer contains claims directly contradicted by context
""",
        threshold=0.8,
        few_shot_examples=[
            {
                "input": {"context": "Company founded in 2019. Revenue $10M in 2023.", "answer": "The company was founded in 2019 and reached $10M revenue in 2023."},
                "output": {"score": 1.0, "reasoning": "All claims supported", "passed": True}
            },
            {
                "input": {"context": "Product launched in January 2024.", "answer": "The product launched in March 2024 after extensive beta testing."},
                "output": {"score": 0.0, "reasoning": "Contradicts launch date", "passed": False}
            },
        ]
    )

Golden Dataset Strategy

Don't start with 1000 cases. Start with 50 real production cases.

# eval/golden_set.jsonl
{"id": "support-001", "input": {"question": "How do I reset my password?", "context": "..."}, "expected": {"answer": "Use the 'Forgot Password' link..."}, "tags": ["basic", "auth"]}
{"id": "support-042", "input": {"question": "Why was I charged twice?", "context": "..."}, "expected": null, "tags": ["billing", "edge-case"]}

Stratification matters:

• 40% basic/happy-path
• 30% edge cases (ambiguous, multi-step)
• 20% adversarial (injection, off-topic)
• 10% multilingual/long-context

Version your dataset: Git-track it. Every production failure becomes a new test case.

Regression Detection That Works

def regression_report(self, baseline: dict[str, float]) -> dict[str, Any]:
    current = self.summary()
    report = {}
    for judge_name, metrics in current.items():
        if judge_name not in baseline:
            continue
        baseline_mean = baseline[judge_name]
        current_mean = metrics["mean_score"]
        diff = current_mean - baseline_mean
        # Statistical test (simplified - use proper stats in prod)
        report[judge_name] = {
            "baseline": baseline_mean,
            "current": current_mean,
            "delta": diff,
            "regressed": diff < -0.05,  # 5% drop = regression
            "improved": diff > 0.02,
        }
    return report

CI/CD Integration: GitHub Actions

# .github/workflows/llm-eval.yml
name: LLM Evaluation
on:
  pull_request:
    paths: ['prompts/**', 'eval/**']
  schedule: ['0 2 * * *']  # Nightly

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.11' }
      - name: Install deps
        run: pip install -e .[dev]
      - name: Run evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python -m eval.run_suite --config config.yaml --output results.json
      - name: Check regressions
        run: |
          python -m eval.check_regression --baseline baseline.json --current results.json
      - name: Comment PR
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const results = JSON.parse(fs.readFileSync('results.json'));
            const body = `## LLM Evaluation Results
            | Judge | Pass Rate | Mean Score |
            |-------|-----------|------------|
            ${Object.entries(results.summary).map(([k,v]) => `| ${k} | ${(v.pass_rate*100).toFixed(1)}% | ${v.mean_score.toFixed(3)} |`).join('\n')}
            `;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body
            });

Results: 6 Months of Production Evaluation

| Metric | Before | After | Change |
|----------------------------|------------------|----------------|---------------|
| Hallucination catch rate | ~67% (human) | 92% (auto) | +25% |
| Prompt iteration cycle | 2 hours | 15 minutes | 8x faster |
| Production incidents |