Intermediate · Evals
Test sets Measure the help desk's replies on questions it never tuned on: a test set split by page and service, checks of each reply written in code, the margin of error the set gives with each page's questions counted together, a review of an assistant's test set, and a release report that says how many questions a margin needs.
You build testset.py, then checks.py, then margin.py, then evalset.py, then release.py
Stages 3 Build, 1 Catch, 1 Ship
Time About 2.5 hours
Runs on Your machine Loading the workspace…
Loading the workspace…
Data: MultiDoc2Dial , by Song Feng, Siva Sankalp Patel, Hui Wan, Sachindra Joshi, CC BY 3.0 . The opening question of each of the dataset's test dialogues whose next turn answers it from the same page: 340 questions, each with its page and the page's domain.
Data: MultiDoc2Dial , by Song Feng, Siva Sankalp Patel, Hui Wan, Sachindra Joshi, CC BY 3.0 . For each question, the passages that hold at least half of a part of the page its answer was given from, and every passage with the same text as one of them.
Data: MultiDoc2Dial , by Song Feng, Siva Sankalp Patel, Hui Wan, Sachindra Joshi, CC BY 3.0 . For each question, the first 5 passages of the help desk's reranked search: the first 20 texts of a reciprocal rank fusion, with k = 60, of the passages' BM25 ranking and their vectors' ranking, in the order of the scores Qwen3-Reranker-0.6B at 8 bits gave them, through llama.cpp's server, release b11734, on 20 October 2026.
Data: MultiDoc2Dial , by Song Feng, Siva Sankalp Patel, Hui Wan, Sachindra Joshi, CC BY 3.0 . For each question, the reply Qwen3.5-2B at 8 bits wrote from its 5 passages, through llama.cpp's server, release b11734, on 21 October 2026, at the sampling settings its model card gives for text, with a seed of its own, the reason it stopped and the tokens it wrote.