Agent Release Evaluation Gate

In AI and automation work, a small change to an agent can break things nobody meant to touch: a smaller model routes a billing question to the wrong team, a new instruction accepts a dispute the policy declines, or a tool gets called when a person should have taken over. This Botlit concept shows a release evaluation gate that runs before a new agent version publishes, whether it carries new instructions, a different model, a changed knowledge base or a new tool, so the change is tested against what matters before customers see it.

The illustration compares a v18 candidate of a sample triage agent with the live v17, beside a red Held badge counting three blocking failures, with Rerun evaluation and Override with note buttons. A strip records the change: a smaller model and new instructions, with the knowledge unchanged. A tag shows the change was submitted from a VibeControls plan. Four tiles set the candidate against live for cases passed, correct routing, answers with citations and average cost per case, with arrows on the tiles where the candidate moved: it falls behind on passes and routing while costing less per case.

A Failed cases table lists each case with the expected outcome, what the candidate did, what the live version did and a severity. A duplicate card charge goes to technical support instead of the billing agent, a late dispute is accepted when the expected answer declines it and cites the policy, a receipts export answer loses its citation and is marked minor, and a request to close an account calls the close-account tool when the expected outcome is a handoff with no tool call. With that last case selected, a side panel shows the expected and candidate exchanges one above the other, with a badge showing the case was added from an incident.

The design envisions each evaluation set as a golden set of questions with expected answers, routing, citations and actions that must not be taken, with incident cases added from the execution record in one step. The release is designed to stay held, with stated reasons, until a person accepts, fixes or overrides it with a note. A footer names the golden set, its case count and how many cases came from incidents, beside a note that results are sent to BigConsole via FluidGrids. In the design, engineers can submit changes from VibeControls, Botlit runs the gate, and a FluidGrids workflow is designed to pass the results to a BigConsole cost and quality console. The concept is meant for AI engineers, AI operations leads and the support owners who sign off agent releases.