CourseAgentic AI Engineer: The Complete Course · Module 13: Evaluation · part 75 of 97
Part 75 · Module 13: Evaluation

Lab: An eval suite that gates every change

38 min read·13 Oct 2026
Module 13 · Lab · About 75 minutes · Needs: lessons 1 and 2 of this module (golden states, grade, verdict, pass^k, the gate) and the Module 3 lab's tools.py

What you are building

RETURNS-DESK AFTER MODULE 13 suite.py 16 tasks runner.py fresh world tools.py call, unchanged BrightCart simulated agent scripted or real checks.py reads the world gate.py SHIP: exit 0 BLOCK: exit 1 on every PR
Your new evals/ code (blue) runs each task in a fresh world through Module 3's unchanged door (grey), grades the world the agent (amber) left, and ships (green) or blocks (red).

It is Wednesday 24 June 2026, week 11. Yesterday a prompt change shipped without the eval gate, hand-offs rose, and Dev rolled it back the same day. Priya wants it never to happen again. This lab moves the lessons' measures into evals/, beside the returns_desk package: the task suite, a runner that gives every trial a fresh world, deterministic checks, seeded trials with pass^k, and a gate that exits 1 when a change makes anything worse. A GitHub Actions workflow runs it on every pull request. Everything at BrightCart is simulated.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside