Score the help desk's agents over repeated trials of each task: pass@k and pass^k, with intervals that count each task once; the median and 90th percentile of the tasks' seconds and tokens, with intervals from the order statistics; traces of an agent's runs in OpenTelemetry's conventions for generative AI, and each run's row from them; a review of an assistant's scorecard; and a release gate that compares a candidate with the current agent task by task.
You build
scores.py, then costs.py, then traces.py, then scorecard.py, then gate.py