CI/CD Evaluation Gates is a deployment pattern that automatically blocks unsafe prompt, tool, or model changes from reaching production by requiring them to pass strict behavioral, policy, and trajectory tests during the continuous integration pipeline. Treat every change to a prompt, tool, policy, or model version as a deployment event that must pass defined gates before it reaches production.
The Risk of Unsafe Agent Deployments
Your agent system passes evaluation today. You change the system prompt, update a tool definition, or swap the underlying model version. The agent ships. The assumption is that a change small enough to seem safe is safe. In a deterministic system, that assumption is often reasonable. In an agent system, where prompt wording shifts behavior and tool composition creates new paths, it is not.
The weaker assumption underneath this — that ordinary unit tests cover agent behavior — is also not quite right. Unit tests are necessary, but they do not cover model behavior, policy compliance, trajectory quality, or semantic regressions. Agent systems need both software tests and behavior gates.
Implementing Evaluation Gates in CI/CD Pipelines
Run deterministic tests first. Treat slow or provider-backed evals as optional stages until the project needs them. Required gates block the release when they fail.
flowchart TD
A[Pull request or release] --> B[Collect CI signals]
B --> C[Unit tests]
B --> D[Pattern tests]
B --> E[Golden trajectory evals]
B --> F[Injection evals]
B --> G[Redaction checks]
B --> H[Policy compliance]
B --> I[Cost regression]
C --> J[GateRunner]
D --> J
E --> J
F --> J
G --> J
H --> J
I --> J
J --> K{Blocking failures?}
K -->|no| L[Release may proceed]
K -->|yes| M[Block deployment]
A release pipeline is itself a graph. Unlike the runtime patterns, the agent under test is the artifact flowing through this graph, not a node in it: run_evals collects behavior scores for a candidate, gate compares them against the baseline with EvaluationGate, and the decision routes to a terminal deploy or block:
gate = EvaluationGate(baseline_scores=[0.90, 0.95, 0.92])
def gate_node(state: PipelineState) -> dict:
passed, reason = gate.evaluate_deployment(state["candidate_scores"])
return {"passed": passed, "reason": reason}
def route_gate(state) -> Literal["deploy", "block"]:
return "deploy" if state["passed"] else "block"
builder.add_edge(START, "run_evals")
builder.add_edge("run_evals", "gate")
builder.add_conditional_edges("gate", route_gate, {"deploy": "deploy", "block": "block"})Designing Effective Continuous Evaluation Gates
Consider the following factors when you implement this pattern:
- Gate design. Make the first gate cheap and deterministic. Unit tests and policy tests should run on every pull request. Add slower eval suites — golden trajectory evals, injection evals, redaction checks, cost regression — for prompt, tool, model, or behavior changes.
- Continuous Evaluation. AI systems are dynamic; they are frequently retrained and fine-tuned. Security controls must evolve at the same pace. Your gates should create Fast Feedback Loops to adapt mitigations based on rapid deployment cycles.
- The “more helpful” regression. The regression these gates are built for rarely looks dangerous in the diff. A developer tweaks a system prompt to make the agent more helpful, and the same change makes it more willing to share an internal document when a user pushes back.