CI/CD for AI: Automated Evals

Lesson updated November 3, 2026 based on OpenAI documentation: temperature and top_p are only accepted with a reasoning effort of none, whereas GPT-5.6 models reason by default (medium). Calls that set these parameters must therefore specify reasoning={"effort": "none"} (Responses API) or reasoning_effort="none" (Chat Completions). Sources: Using GPT-6 · GPT-5.6 Terra.

Integrating Evals into Your CI/CD Pipeline

Deploying a prompt change to production without evaluation is like deploying code without tests. The comparison is more accurate than it might seem: in both cases, the change seems trivial, it was manually verified on one or two examples, and the side effect shows up three days later for a user who won’t tell you about it. The difference is that a prompt regression doesn’t raise an exception—the application keeps responding, just worse.

The goal is to make automated evaluation a mandatory gate in your deployment process, ensuring every change maintains or improves quality. That’s exactly what we’ll cover next.

The Principle: A Gate, Not a Report

The eval pipeline integrates into your deployment flow using a familiar mechanism. A developer modifies a prompt or parameter, the commit triggers the CI pipeline, and evals run automatically against the reference dataset. If scores meet the thresholds, deployment continues; otherwise, it’s blocked, with a detailed report indicating which evaluation failed and by how much.

The critical point is that blocking behavior. A CI that produces a pretty report without stopping the pipeline becomes an artifact nobody checks after two weeks. Only a non-zero exit code turns measurement into a guardrail.

The script below reads a configuration, loads each dataset, runs the cases, and compares the average score to the declared threshold. The scoring method—simple keyword inclusion—is intentionally rudimentary: a CI check must be fast, deterministic, and independent of a judge whose score might vary from one run to the next.

import json
import sys
import openai
from pathlib import Path

client = openai.OpenAI()

def load_eval_config(path: str = "evals/config.json") -> dict:
    """Loads the evaluation configuration."""
    with open(path) as f:
        return json.load(f)

def load_dataset(path: str) -> list[dict]:
    """Loads a JSONL dataset."""
    cases = []
    with open(path) as f:
        for line in f:
            cases.append(json.loads(line))
    return cases

def run_ci_evals(config: dict) -> dict:
    """Runs evaluations for CI."""
    results = {}

    for eval_config in config["evaluations"]:
        name = eval_config["name"]
        dataset = load_dataset(eval_config["dataset"])
        threshold = eval_config["min_threshold"]
        model = eval_config["model"]
        prompt = Path(eval_config["prompt_file"]).read_text()

        scores = []
        for case in dataset:
            response = client.responses.create(
                model=model,
                reasoning={"effort": "none"},  # temperature requires effort none
                instructions=prompt,
                input=case["input"],
                temperature=0.0,
            )

            # Simple evaluation by keyword inclusion
            if "keywords" in case:
                response_lower = response.output_text.lower()
                found = sum(
                    1 for word in case["keywords"]
                    if word.lower() in response_lower
                )
                score = found / len(case["keywords"])
            else:
                score = 1.0  # No specific criterion

            scores.append(score)

        avg_score = sum(scores) / len(scores)
        passed = avg_score >= threshold

        results[name] = {
            "score": avg_score,
            "threshold": threshold,
            "passed": passed,
            "num_cases": len(dataset),
        }

    return results

def ci_report(results: dict) -> tuple[str, bool]:
    """Generates a report for CI and indicates if it's OK."""
    lines = ["# AI Evaluation Report\n"]
    all_pass = True

    for name, r in results.items():
        status = "PASS" if r["passed"] else "FAIL"
        if not r["passed"]:
            all_pass = False
        lines.append(
            f"- [{status}] {name}: {r['score']:.2%} "
            f"(threshold: {r['threshold']:.0%}, {r['num_cases']} cases)"
        )

    report = "\n".join(lines)
    return report, all_pass

Notice that the prompt is read from a file rather than written directly in the script. This is what makes triggering possible: a prompt versioned in the repository produces a diff, and a diff can trigger an evaluation. As long as your prompts live as string literals scattered throughout your application code, proper automation isn’t possible.

The configuration declares one evaluation per task, each with its own threshold.

{
  "evaluations": [
    {
      "name": "ticket-classification",
      "dataset": "evals/datasets/classification.jsonl",
      "prompt_file": "prompts/classification.txt",
      "model": "gpt-5.6-terra",
      "min_threshold": 0.90
    },
    {
      "name": "resume-generation",
      "dataset": "evals/datasets/resumes.jsonl",
      "prompt_file": "prompts/resume.txt",
      "model": "gpt-5.6-terra",
      "min_threshold": 0.75
    }
  ]
}

The gap between these two thresholds is the heart of the method. Ticket classification is a closed task: below 0.90, the feature doesn’t deliver the expected service. Summarization is an open task, judged on keywords that a good text might legitimately phrase differently; requiring 0.90 would block every deployment and quickly lead to disabling the check. Calibrate each threshold based on scores observed in production rather than on a round number that feels reassuring.

Wiring It All Up in GitHub Actions

The workflow triggers only on relevant paths—prompts, evals, AI integration code—so a stylesheet change doesn’t consume API calls.

# .github/workflows/eval-ai.yml
name: AI Evaluation

on:
  pull_request:
    paths:
      - "prompts/**"
      - "evals/**"
      - "src/ai/**"

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - run: pip install openai

      - name: Run evals
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python evals/run_ci.py

      - name: Publish report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-report
          path: evals/report.md

Two details are worth noting. The API key passes through repository secrets, never through the workflow file, which is readable by anyone with access to the code. And the report publication uses if: always(), otherwise the artifact wouldn’t be produced when evals fail—which is precisely the only case where you need to read it.

The entry script ties it together: it writes the report to disk for the artifact, prints it to the logs, and sets the exit code that allows or blocks the rest of the pipeline.

#!/usr/bin/env python3
"""Evaluation script for CI — evals/run_ci.py"""

import sys
from pathlib import Path

def main():
    config = load_eval_config("evals/config.json")
    results = run_ci_evals(config)
    report, all_pass = ci_report(results)

    # Write the report
    Path("evals/report.md").write_text(report)
    print(report)

    # Exit code for CI
    if not all_pass:
        print("\nSome evaluations failed. Deployment blocked.")
        sys.exit(1)
    else:
        print("\nAll evaluations pass.")
        sys.exit(0)

if __name__ == "__main__":
    main()

Compare Rather Than Judge in Absolute Terms

A fixed threshold answers the question “Is this good enough?” It doesn’t answer “Is this better than before?”, which is often the real question. A rewritten prompt can stay above the threshold while losing three points compared to the previous version: CI turns green and the degradation enters production without anyone noticing.

The regression test therefore runs both versions on the same dataset, in the same campaign, and compares the averages. A 5% tolerance absorbs the noise inherent in generative outputs; beyond that, the verdict is explicit.

def prompt_regression_test(
    current_prompt: str,
    previous_prompt: str,
    dataset: list[dict],
    model: str,
    degradation_threshold: float = 0.05,
) -> dict:
    """Verifies that a new prompt does not degrade performance."""
    scores_current = []
    scores_previous = []

    for case in dataset:
        # Evaluate with current prompt
        r1 = client.responses.create(
            model=model, instructions=current_prompt,
            reasoning={"effort": "none"},  # temperature requires effort none
            input=case["input"], temperature=0.0,
        )
        # Evaluate with previous prompt
        r2 = client.responses.create(
            model=model, instructions=previous_prompt,
            reasoning={"effort": "none"},  # temperature requires effort none
            input=case["input"], temperature=0.0,
        )

        s1 = evaluate_response(r1.output_text, case)
        s2 = evaluate_response(r2.output_text, case)
        scores_current.append(s1)
        scores_previous.append(s2)

    avg_current = sum(scores_current) / len(scores_current)
    avg_previous = sum(scores_previous) / len(scores_previous)
    delta = avg_current - avg_previous

    return {
        "current_score": avg_current,
        "previous_score": avg_previous,
        "delta": delta,
        "regression": delta < -degradation_threshold,
        "verdict": "OK" if delta >= -degradation_threshold else "REGRESSION",
    }

This test doubles the number of calls, and thus the cost and duration. Reserve it for prompt changes, keeping it off the path of ordinary commits, and run it on a representative subset rather than your entire reference dataset.

Key Takeaways

• Integrate evals into your CI/CD pipeline like unit tests
• Version prompts in files: without a diff, no triggering is possible
• Define minimum thresholds per evaluation type, calibrated on production data
• Block deployments via exit code when scores drop below thresholds
• Test for regressions by comparing new and old prompts on the same dataset