Sulba
ReviewSign inAccount
Theme
Editor theme
Light
Dark

Intermediate · Agents

Scoring an agent

Score the help desk's agents over repeated trials of each task: pass@k and pass^k, with intervals that count each task once; the median and 90th percentile of the tasks' seconds and tokens, with intervals from the order statistics; traces of an agent's runs in OpenTelemetry's conventions for generative AI, and each run's row from them; a review of an assistant's scorecard; and a release gate that compares a candidate with the current agent task by task.

You build
scores.py, then costs.py, then traces.py, then scorecard.py, then gate.py
Stages
3 Build, 1 Catch, 1 Ship
Time
About 3 hours
Runs on
Your machine

Loading the workspace…