Course
AI Engineering
Beginner
- Modules
- 8
- Labs
- 41
- Time
- About 133 hours
- Needs
- Any laptop: every lab runs in your browser
- Progress
1 Python for programmers Show
- Values and references
- Iterators and generators
- Closures and decorators
- Classes and dataclasses
- Exceptions and context managers
- Concurrency with asyncio
- Types, tests and packages
- 1.1
Values and references
Start porting superstruct, a JavaScript library that checks data, to Python: tell values apart as JavaScript does, compare them as its strict equality does, and fill in defaults without changing the data you were given.
- You build
- values.py, then intake.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- 1.2
Iterators and generators
Port how superstruct checks a whole record: build schemas with comprehensions, walk nested data with a generator that yields each failure with its path, and check a day's records as they stream in, reading no more than you need.
- You build
- walk.py, then walker.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- 1.3
Closures and decorators
Port superstruct's factories and refinements: functions that make checks and remember what they were made with, refinements wrapped around a check, and decorators that turn a plain test into a named check and count the failures each check finds.
- You build
- refine.py, then form.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- 1.4
Classes and dataclasses
Port superstruct's structs to classes: a dataclass for each failure, a class for each kind of struct that shares one walk through its base class, and the special methods that join structs with a bar, test whether a value passes and show each struct as it's made.
- You build
- structs.py, then service.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- 1.5
Exceptions and context managers
Port how superstruct reports a value that fails: an error class of its own, raised, returned or tested for, notes and chaining for a day's records, groups of errors raised together, and context managers that clean up whatever happens.
- You build
- errors.py, then daily.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- 1.6
Concurrency with asyncio
Check each sample's code against the clinic's registry with asyncio: coroutines that wait without holding the service up, lookups made at once in tasks and task groups, a limit and a timeout on a slow registry, and a day's codes checked within them.
- You build
- lookups.py, then day_lookups.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- 1.7
Types, tests and packages
Port superstruct's types: type hints that mypy checks in strict mode, generic structs that narrow what a value is, tests with pytest that catch a change that breaks the port, and a check of the packages uv locks for the service against the clinic's approved list.
- You build
- typed.py, then test_typed.py, then intake.py, then approvals.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
2 The maths behind ML, as code Show
- 2.1
Arrays and broadcasting
Find the past samples most like each new breast mass sample, in NumPy: standardize thirty measurements at once, measure every distance by broadcasting, and keep the search within its memory as the records grow.
- You build
- similar.py, then search.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- 2.2
Vectors and matrices
Treat each breast mass sample as a vector in NumPy: score it along the line from the average benign sample to the average malignant one, and work out every distance between samples at once with the matrix product.
- You build
- vectors.py, then screen.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- 2.3
Derivatives and gradients
Estimate derivatives from finite differences in NumPy, find the step where rounding takes over, work out the gradient of the clinic's first neural network by the chain rule, and check a large model's gradient along random directions.
- You build
- gradients.py, then network.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- 2.4
Sampling
Draw the random batches a model trains on, in NumPy: shuffle every epoch, work out how far a batch's average strays from the whole, balance the classes by weighted sampling, and share each epoch's batches out across workers.
- You build
- sampling.py, then batches.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- 2.5
The bootstrap
Put an interval around a number measured on a sample, in NumPy: resample the clinic's results, read confidence intervals off the bootstrap distribution, correct them for bias and skew, and compare two models on the same samples.
- You build
- bootstrap.py, then evaluation.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- 2.1
3 Working with data Show
- 3.1
Cleaning data
Turn a hospital network's raw extract into a table you can trust, in pandas: read its missing values right, mark the codes that mean unknown, keep each patient's first stay, and clean an extract too big to read at once.
- You build
- cleaning.py, then clean.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- 3.2
Joins and group-by
Count a hospital network's readmissions by what its codes mean, in pandas: split a file of lookup tables, join them to the encounters without losing or doubling one, count by group with the unknowns kept, and give each encounter its patient's history for a model.
- You build
- report.py, then monthly.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- 3.3
Querying with SQL
Ask a hospital network's questions of a database, in SQL: load the extract into SQLite, join its codes without losing an encounter, count by group with NULL kept, and build each encounter's history with window functions, with answers that agree with pandas.
- You build
- queries.py, then dashboard.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- 3.4
Checking data
Check a hospital network's monthly extract before anything uses it, in pandas: hold it to a contract from the dataset's documentation, check its keys and codes, find what a patient's encounters can't all say at once, and compare each month's extract with the last.
- You build
- checks.py, then validate.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- 3.1
4 Learning from data Show
- Linear regression
- Logistic regression
- Overfitting and regularization
- Cross-validation
- Measuring a classifier
- 4.1
Linear regression
Build a model in NumPy that forecasts a power plant's electrical output for each hour from the weather around it.
- You build
- model.py, then forecaster.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- 4.2
Logistic regression
Build a classifier in NumPy that takes measurements of the cell nuclei in a sample from a breast mass and gives the probability that the mass is malignant.
- You build
- model.py, then screener.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- 4.3
Overfitting and regularization
Build a model in NumPy that predicts a concrete mix's compressive strength from its recipe and age, and find how flexible a model the plant's records can support.
- You build
- model.py, then strength.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- 4.4
Cross-validation
Score the plant's concrete models on every one of its mixes in turn with k-fold cross-validation, choose by it, and measure the error of the choosing itself.
- You build
- cv.py, then tuning.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- 4.5
Measuring a classifier
Measure a pulsar survey's classifier the way its astronomers will use it: how well it ranks the candidates, what share of those it flags are pulsars, and whether its probabilities can be counted on.
- You build
- metrics.py, then report.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
5 Models that win on tables Show
- Nearest neighbours
- Decision trees
- Random forests
- Gradient boosting
- k-means and PCA
- Leakage and model comparison
- 5.1
Nearest neighbours
Name a bean's variety from the beans most like it, in NumPy: put its measurements on one scale, find its nearest neighbours and take their vote, choose how many to ask, and sort a whole line of beans, saying which it can't place.
- You build
- sorter.py, then line.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- 5.2
Decision trees
Rank a bank's clients by their chance of buying a term deposit with a decision tree grown in NumPy: find the split that leaves the least impurity, grow the tree split by split, choose its depth, and turn it into rules a call centre can read.
- You build
- tree.py, then call_list.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- 5.3
Random forests
Rank the bank's clients with a forest of decision trees: grow each tree on its own sample of the calls, give each split columns drawn at random, estimate the forest from the calls each tree left out, and measure which details of a client its ranking leans on.
- You build
- forest.py, then report.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- 5.4
Gradient boosting
Rank the bank's clients with boosted trees: grow each tree on what the trees before it got wrong, choose the learning rate and the rounds on validation calls, work out the splits XGBoost makes, and ship a LightGBM model that never raises a client for being called again.
- You build
- boost.py, then ranking.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- 5.5
k-means and PCA
Find the groups in the beans' shapes without their varieties: put the measurements' variance into a few principal components, group the beans with k-means, choose how many groups by how well they separate, and estimate a new harvest's make-up from named groups.
- You build
- groups.py, then report.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- 5.6
Leakage and model comparison
Compare the module's four models with a logistic regression on the bank's calls: encode a category by its share of yes without leaking the answer, score every model on the same folds, put error bars on the differences that allow for folds sharing their calls, and ship the review that decides which model the bank calls from.
- You build
- encoding.py, then compare.py, then report.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
6 Text as numbers Show
- 6.1
Tokens and byte-pair encoding
Build an online shop's tokenizer from its product titles: learn byte-pair merges, encode and decode titles as tiktoken does, measure what titles cost in tokens in three languages, and ship a shortener that fits any title into a token limit.
- You build
- bpe.py, then prompts.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- 6.2
Logits and sampling
Ask a small language model whether products match shoppers' searches, and read its answers from the logits: turn them into probabilities, reshape them with temperature, top-k and top-p, draw answers as generation does, and ship probabilities that mean what they say.
- You build
- judge.py, then sampler.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- 6.3
Embeddings and search
Search the shop's products with a small embedding model: compare queries and titles by cosine similarity, find each query's nearest products, score the results against people's judgements with recall, MRR and nDCG, and ship an index that searches a fraction of the catalogue.
- You build
- search.py, then evaluate.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- 6.1
7 Coding interviews Show
- 7.1
Arrays
Five interview problems on lists of numbers and a grid: sums over ranges, the best trade, the largest run, the products of the others and a spiral.
- You build
- ranges.py, then trade.py, then runs.py, then products.py, then spiral.py
- Stages
- 5 Problems
- Time
- About 1.5 hours
- 7.2
Hash maps
Five interview problems where a dict or a set turns a search into one lookup: pair sums, the first unique character, groups of anagrams, the longest streak and runs that add up.
- You build
- pairs.py, then unique.py, then anagrams.py, then streaks.py, then subarrays.py
- Stages
- 5 Problems
- Time
- About 1.5 hours
- 7.3
Strings
Five interview problems on text: a palindrome check, the common prefix, run lengths, the same pattern and the longest palindrome inside a text.
- You build
- palindromes.py, then prefixes.py, then lengths.py, then patterns.py, then longest.py
- Stages
- 5 Problems
- Time
- About 1.5 hours
- 7.4
Two pointers
Five interview problems where two positions move through a list: a pair in sorted order, duplicates removed in place, the most water, triples that sum to zero and trapped water.
- You build
- sorted_pair.py, then dedupe.py, then water.py, then triples.py, then rain.py
- Stages
- 5 Problems
- Time
- About 2 hours
- 7.5
Sliding windows
Five interview problems solved by a window that slides along a list or a text: the best window, no repeats, reaching a total, at most k kinds and the smallest cover.
- You build
- windows.py, then no_repeats.py, then reach.py, then kinds.py, then cover.py
- Stages
- 5 Problems
- Time
- About 2 hours
- 7.6
Stacks
Five interview problems that a stack solves as it reads its input: balanced brackets, reverse Polish notation, days until warmer, a stack that knows its minimum and decoding repeats.
- You build
- brackets.py, then rpn.py, then warmer.py, then min_stack.py, then repeats.py
- Stages
- 5 Problems
- Time
- About 1.5 hours
- 7.7
Binary search
Five interview problems that halve what's left to search at each step: the insertion point, the first and last places of a value, a rotated list, a whole-number square root and the least capacity to ship in time.
- You build
- insertion.py, then first_last.py, then rotated.py, then square_root.py, then shipping.py
- Stages
- 5 Problems
- Time
- About 1.5 hours
- 7.8
Sorting
Five interview problems that come apart once their input is in order: merging intervals, meeting rooms, the kth largest, the largest number and the most frequent items.
- You build
- intervals.py, then rooms.py, then kth.py, then largest_number.py, then frequent.py
- Stages
- 5 Problems
- Time
- About 1.5 hours
- 7.9
Mock interview
Two interview problems in 45 minutes, as in a coding screen: merging sorted lists, then the anagrams of a word inside a text.
- You build
- merge.py, then anagram_starts.py
- Stages
- 2 Problems
- Time
- About 45 minutes
- 7.1
8 Capstone Show
- 8.1
The take-home
A first version of the hospital network's readmission service in three hours: read its extract, train a model on each patient's first stay, and serve the chance of a readmission within 30 days from a FastAPI endpoint.
- You build
- service.py
- Stages
- 1 Problem
- Time
- About 3 hours
- 8.2
The readmission service
The full build of the hospital network's readmission service: inputs made one way for training and serving, a model chosen with error bars and checked for calibration, a threshold for the calls planners can make, an API held to a latency budget, your own tests, and a one-page write-up.
- You build
- features.py, then model.py, then train.py, then app.py, then call_list.py, then test_app.py, then report.py
- Stages
- 10 Build, 2 Catch, 2 Ship
- Time
- About 16.5 hours
- 8.1
Intermediate
- Modules
- 17, 2 of them optional
- Labs
- 66
- Time
- About 194.5 hours, and about 21.5 hours more in the optional modules
- Needs
- A laptop with uv, 8 GB of memory and about 10 GB free: Windows or Linux on x86-64, Linux on Arm, or macOS 14 or later on Apple silicon. Module 1, the coding interviews, most of Working with customers, the design round, the capstone's take-home and the mock loop run in your browser, and the rest runs on your machine
- Progress
1 Neural networks from scratch Show
- Layers and backpropagation
- An autograd engine
- Optimizers
- Initialization and normalization
- Debugging a training run
- 1.1
Layers and backpropagation
Build a network from layers in NumPy: linear layers, ReLU and a cross-entropy loss, each with a backward pass whose gradients match PyTorch's, and train it to read handwritten digits from people whose writing it has never seen.
- You build
- layers.py, then network.py, then train.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- 1.2
An autograd engine
Build an engine that works out gradients by itself, as PyTorch's autograd does: values and arrays that remember how they were made, a backward pass in topological order, broadcasting summed back, and gradients that match PyTorch's on random expressions.
- You build
- engine.py, then autograd.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4.5 hours
- 1.3
Optimizers
Build PyTorch's optimizers in NumPy: SGD with momentum as PyTorch writes it, Adam with its bias correction, and AdamW with its weight decay kept apart, each step matched to torch.optim's, then train the office's network on a budget of steps.
- You build
- optimizers.py, then optim.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- 1.4
Initialization and normalization
Train a deep network: start its weights as torch.nn.init starts them, so that the signal keeps its size through every layer, then build batch and layer normalization, each matched to PyTorch's outputs and gradients, until the network trains where it stalled.
- You build
- init.py, then norms.py, then normalization.py, then deep.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- 1.5
Debugging a training run
Find what's wrong with a training run before it costs a day: the loss a network should start at, a batch it should learn by heart, gradient norms and clipping as torch.nn.utils has them and a learning-rate range test, then review an assistant's training script and ship a run whose report holds up.
- You build
- network.py, then clipping.py, then range_test.py, then train.py, then run.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
2 PyTorch Show
- 2.1
Tensors and autograd
Port the records office's digit reader from NumPy to PyTorch: its digits as tensors, its gradients from autograd, a loss with a backward pass of your own, and a reader that trains step for step as the NumPy one did.
- You build
- digits.py, then network.py, then trainer.py, then reader.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- Runs on
- Your machine
- 2.2
Modules and training loops
Train the records office's deep reader the way PyTorch code does: its layers as modules, its digits from a data loader, its validation in evaluation mode, and checkpoints that pick up a stopped run exactly where it was.
- You build
- reader.py, then data.py, then checkpoints.py, then trainer.py, then run.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- Runs on
- Your machine
- 2.3
Convolutional networks
Sort an online shop's photos of clothes with a convolutional network: convolution and pooling by hand, the photos' files read and checked, a network that trains on any device and in mixed precision, a review of an assistant's classifier, and a classifier that beats the shop's perceptron within a budget of compute.
- You build
- conv.py, then photos.py, then net.py, then classifier.py, then shop.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4.5 hours
- Runs on
- Your machine
- 2.4
Transfer learning
Sort a land-use survey's satellite tiles with a backbone trained by others: the tiles read from their zip, a vision transformer loaded from its checkpoint, a linear probe on its features, a review of an assistant's fine-tuning, and a classifier that learns from a few tiles of each kind.
- You build
- tiles.py, then backbone.py, then probe.py, then finetune.py, then survey.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- Runs on
- Your machine
- 2.1
3 Transformers from scratch Show
- 3.1
Attention
Build the attention a reading app's story model runs in every layer: scaled dot-product attention with its causal mask, heads split and merged, masks for batches of stories, and attention in tiles that never holds a whole row of scores.
- You build
- attention.py, then self_attention.py, then tiles.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- Runs on
- Your machine
- 3.2
The transformer block
Build the app's GPT as GPT-2 builds it: layer normalization by hand, the MLP and the pre-norm residual block, a GPT with learned positions and a tied output layer, and what a GPT costs, worked out from its configuration.
- You build
- block.py, then gpt.py, then costs.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 3.3
Training a small GPT
Train the reading app's story model on your own machine: a tokenizer learnt from the stories, batches of tokens packed as GPT-3 packs its documents, a training loop with AdamW, warm-up, cosine decay and clipping, a review of an assistant's trainer, and a model that beats a target loss within a budget of compute.
- You build
- tokenizer.py, then data.py, then train.py, then trainer.py, then story.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- Runs on
- Your machine
- 3.4
GPT-2's weights
Run OpenAI's GPT-2 in the reading app's own GPT: its weights read by hand from their file, its tokenizer built from its published vocabulary and merges, its tensors mapped into your model until its predictions match Hugging Face's, a review of an assistant's loader, and a loader for any GPT-2 that refuses a damaged file.
- You build
- weights.py, then tokenizer.py, then gpt2.py, then loader.py, then load.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 3.5
Rotary positions and the KV cache
Run Hugging Face's SmolLM2-135M in your own code: rotary positions, RMSNorm, SwiGLU and grouped-query attention matched to transformers' Llama, a KV cache that writes each token without reading the story again, a review of an assistant's cached generator, and a generator for batches of prompts.
- You build
- rope.py, then smollm.py, then cache.py, then generator.py, then batch.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 3.1
4 Building on model APIs Show
- 4.1
Calling a model
Connect a bicycle repair shop's assistant to a model through the chat completions API: requests and replies by hand over HTTP, the tokens a conversation takes and the model's window, a month of the shop's questions priced from providers' published prices, a review of an assistant's client, and a client that retries what it can within a deadline.
- You build
- chat.py, then window.py, then costs.py, then client.py, then retry.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
- 4.2
Structured output
Hold the bicycle repair shop's assistant to JSON its calendar can take: a JSON Schema validator for the keywords the shop uses, replies checked and asked for again with their errors, constrained decoding that masks every token that can't continue a valid reply, a review of an assistant's extractor, and an extractor that turns the shop's emails into bookings.
- You build
- schema.py, then extract.py, then constrain.py, then bookings.py, then inbox.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 4.3
Tools, streaming and MCP
Give the bicycle repair shop's assistant tools: the shop's functions described for the model, the loop that runs the calls it makes, its replies read as they stream in, a review of an assistant's tool loop, and the shop's tools served over the Model Context Protocol.
- You build
- tools.py, then loop.py, then stream.py, then assistant.py, then server.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 4.4
Securing the assistant
Keep the bicycle repair shop's assistant safe when the model does what an attacker writes: tools that answer only for the customer signed in, actions the customer confirms, replies the chat window can show, a review of an assistant's guardrails, and the shop's chat shipped against prompt injections.
- You build
- shop.py, then actions.py, then render.py, then assistant.py, then chat.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 4.1
5 Retrieval Show
- Embeddings for search
- Vector search in Postgres
- Chunking
- Hybrid search and reranking
- Evaluating retrieval
- 5.1
Embeddings for search
Search a help desk's pages with an embedding model on your machine: embedding requests in batches with the model's prefixes, a cache that embeds each text once, vectors cut short and kept in a byte a number, a review of an assistant's search, and an index that fits in a few hundred bytes a passage.
- You build
- embed.py, then cache.py, then shorten.py, then search.py, then index.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
- 5.2
Vector search in Postgres
Search the help desk's passages in Postgres with pgvector: a table of passages and their vectors, an approximate index, searches within one service, a review of an assistant's queries, and a search whose indexes fit a small budget.
- You build
- store.py, then index.py, then service.py, then queries.py, then desk.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
- 5.3
Chunking
Cut the help desk's pages into chunks and compare the ways of cutting them by the answers they hold: windows of words and joined blocks, recall and precision by an answer's characters, chunkings compared at one budget of words, a review of an assistant's chunker, and a chunker whose chunks last when a page changes.
- You build
- chunk.py, then score.py, then compare.py, then splitter.py, then chunker.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
- 5.4
Hybrid search and reranking
Search the help desk's passages by their words as well as their vectors, and have a second model put what the two find in order: full-text search in Postgres, BM25 by hand, two rankings fused by their ranks, a review of an assistant's hybrid search, and a search reranked by Qwen3-Reranker-0.6B on your machine.
- You build
- fts.py, then bm25.py, then fuse.py, then hybrid.py, then desk.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
- 5.5
Evaluating retrieval
Measure the help desk's searches as its readers meet them: hit rate, recall, MRR and nDCG over every question, each text judged once, each part's share found by taking it away and testing the difference question by question, a review of an assistant's report, and a release check that decides whether a new search replaces the old.
- You build
- measures.py, then texts.py, then compare.py, then report.py, then release.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
6 Evals Show
- 6.1
Test sets
Measure the help desk's replies on questions it never tuned on: a test set split by page and service, checks of each reply written in code, the margin of error the set gives with each page's questions counted together, a review of an assistant's test set, and a release report that says how many questions a margin needs.
- You build
- testset.py, then checks.py, then margin.py, then evalset.py, then release.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 2.5 hours
- Runs on
- Your machine
- 6.2
Grading with a model
Grade the help desk's replies with Granite Guardian 3.1 2B on your machine: its prompt rendered from its own template and its verdict read from its likeliest first tokens, its agreement with people's labels, a check on the help desk's own pages, a review of an assistant's grading run, and a report of how many of a release's replies keep to their passages.
- You build
- grader.py, then agreement.py, then known.py, then judge.py, then grounding.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
- 6.3
Regression gates
Gate the help desk's releases in CI: several replies to each question and what they take out of the standard error, two releases compared question by question, the smallest drop the test set can be counted on to find, a review of an assistant's gate, and a gate that stops a planted regression and ships a rerun.
- You build
- samples.py, then compare.py, then power.py, then gate.py, then release_gate.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 2.5 hours
- Runs on
- Your machine
- 6.1
7 Agents Show
- 7.1
Workflows and agents
Have the help desk do its own work as well as answer: the desk's tools over a caller's tickets and its advisers' callbacks, a workflow that sorts each message and follows a fixed path, an agent that calls the tools itself, a review of an assistant's agent, and a router that sends the workflow what it can follow and the agent the rest.
- You build
- desk.py, then workflow.py, then agent.py, then desk_agent.py, then router.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
- 7.2
Tools and planning
Write the help desk's tools for the model that calls them: search results it can cite, lists that say what they leave out, and errors that say how to call again; tickets and callbacks the model proposes and the desk makes only on the caller's yes, after looking up what they need; the agent on those tools; a review of an assistant's proposals; and callbacks that move.
- You build
- desk.py, then actions.py, then agent.py, then desk_actions.py, then moves.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 7.3
Memory and sub-agents
Keep the help desk's long conversations within a budget of tokens: old tool results cleared first, notes the model saves that outlast a summary of the turns before, and researchers that search each service's pages in a conversation of their own; a review of an assistant's memory; and notes that last from one conversation to the next.
- You build
- context.py, then memory.py, then researchers.py, then desk_memory.py, then notebook.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
- 7.4
Scoring an agent
Score the help desk's agents over repeated trials of each task: pass@k and pass^k, with intervals that count each task once; the median and 90th percentile of the tasks' seconds and tokens, with intervals from the order statistics; traces of an agent's runs in OpenTelemetry's conventions for generative AI, and each run's row from them; a review of an assistant's scorecard; and a release gate that compares a candidate with the current agent task by task.
- You build
- scores.py, then costs.py, then traces.py, then scorecard.py, then gate.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
- 7.1
8 Shipping full stack Show
- 8.1
The desk's API
Put the help desk's agent behind a web API: conversations kept by id, with every error as a problem details object; a gate in front of the model, with a time limit on each reply; the reply streamed as server-sent events, kept only once it's whole; a review of an assistant's replies; and the app, ready to serve, with its settings from the environment.
- You build
- conversations.py, then gate.py, then replies.py, then desk_replies.py, then app.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 8.2
Conversations in Postgres
Keep the desk's conversations in Postgres: migrations applied once each, in order, by one app at a time; a store whose turns are kept whole, in one transaction, on pooled connections; each turn kept with what it used, however it ended; a review of an assistant's store; and each day's usage for the dashboard, found through an index.
- You build
- migrations.py, then store.py, then replies.py, then desk_store.py, then app.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 8.3
Signing in
Sign the desk's callers in with OpenID Connect: the authorization request with its state, nonce and PKCE challenge; the code exchanged and the ID token checked against the provider's keys; sessions kept in Postgres, as hashes of their tokens; each person's conversations theirs alone; a review of an assistant's sign-in routes; and the app signed in, with its cookies sent only over HTTPS and no change taken from another site.
- You build
- sign_in.py, then tokens.py, then sessions.py, then store.py, then desk_auth.py, then app.py
- Stages
- 4 Build, 1 Catch, 1 Ship
- Time
- About 4.5 hours
- Runs on
- Your machine
- 8.4
The chat page
The desk's page, in TypeScript: a typed client for its API with its problem details as errors; the reply's event stream read from fetch as browsers read one; the chat, with each message put on the page as text and a log a screen reader can follow; a review of an assistant's chat; and a dashboard of usage and cost by UTC day, in a table a screen reader can read.
- You build
- api.ts, then stream.ts, then chat.ts, then desk_chat.ts, then dashboard.ts
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 7 hours
- Runs on
- Your machine
- 8.5
Shipping the desk
The desk ready for a host: logs as JSON lines, each with its request's id and no secrets; readiness that says whether the database and the model answer; the server's settings from the host's environment, with a clean stop; a review of an assistant's setup; and the desk's image, built in stages and run as a user without root.
- You build
- logs.py, then health.py, then main.py, then desk_ops.py, then Dockerfile
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- Runs on
- Your machine
- 8.1
9 Data pipelines Show
- 9.1
Analytical SQL
Answer a cab dispatcher's questions about a month of New York's yellow cab trips in Postgres: each borough's busiest zones with ties kept, an hour-by-hour series with running totals, subtotals by borough and payment, a review of an analyst's report, and a daily summary that a run replaces whole.
- You build
- zones.py, then hours.py, then payments.py, then report.py, then summary.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
- 9.2
Spark DataFrames
Work through a month of New York's yellow and green cab trips with Spark's DataFrame API: one table from two files laid out differently, a flag for each record that can't be taken at its word, each zone's days with a seven-day mean over the calendar, a review of an assistant's report, and daily Parquet files that a run replaces whole.
- You build
- trips.py, then flags.py, then days.py, then report.py, then store.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
- 9.3
Testing a pipeline
Join a month of New York's cab trips to the taxi zones and to Central Park's weather with Spark, and test the pipeline that does it: NOAA's daily weather file read into units, each borough's trips on every day of August, wet days against dry days of the same weekday, a review of an assistant's daily table, and tests of your own that catch the mistakes planted in the pipeline.
- You build
- weather.py, then daily.py, then rain.py, then daily_table.py, then test_pipeline.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3 hours
- Runs on
- Your machine
- 9.1
10 Working with customers Show
- 10.1
Estimating the work
Turn a team's three estimates of each task into a date and a price for a customer: the triangular distribution's mean and quantiles, the project's finish over the tasks it waits for, many runs of it drawn at random, a promise from their percentiles, a review of an assistant's estimate, and a quote in working days and whole cents.
- You build
- estimates.py, then schedule.py, then promise.py, then estimate.py, then quote.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 2.5 hours
- 10.2
Measuring value
Work out what the help desk is worth to the benefits office from its agent's trials: each kind of contact's share done, counting each task once; the share of the office's own contacts, weighted by its mix; the net saving a month and the months the pilot takes to pay for itself, each with its interval; a review of an assistant's value report; and the summary the office reads, with every interval rounded outward.
- You build
- kinds.py, then mix.py, then worth.py, then report.py, then summary.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 2.5 hours
- 10.3
Inside the customer's walls
Bring the desk inside the benefits office: a search of its staff guidance that finds only what each person may read, groups from a sign-in whose token checks out, an audit log that names no one, a review of an assistant's search for staff, and a transport that lets no request leave the office's hosts.
- You build
- access.py, then staff.py, then audit.py, then staff_search.py, then walls.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 2.5 hours
- Runs on
- Your machine
- 10.4
A demo against the clock
Two demos for the benefits office, each from its written brief and against the clock, with the brief's points planned as tests first: its web form's messages filed to the right queue, with Social Security numbers kept from the model, and answers from the office's own answers that cite the ones they use.
- You build
- triage.py, then faq.py
- Stages
- 2 Problems
- Time
- About 1.5 hours
- 10.1
11 Coding interviews: lists, trees and graphs Show
- 11.1
Linked lists
Seven interview problems on linked lists: reversing one, finding where a cycle starts, removing a node from the end, adding numbers held as digits, reordering a list from both ends, copying a list with random links and a least recently used cache.
- You build
- reverse.py, then cycle.py, then remove.py, then add.py, then reorder.py, then copy_list.py, then lru.py
- Stages
- 7 Problems
- Time
- About 2.5 hours
- 11.2
Binary trees
Six interview problems on binary trees: reading one level by level, its diameter, the lowest common ancestor of two nodes, writing a tree down and reading it back, the paths that add up to a target and the largest path sum.
- You build
- levels.py, then diameter.py, then ancestor.py, then codec.py, then paths.py, then path_sum.py
- Stages
- 6 Problems
- Time
- About 2.5 hours
- 11.3
Graph search
Six interview problems on searching graphs, breadth first and depth first: counting islands on a map, the shortest way out of a maze, a fire spreading from many places at once, copying a graph with cycles, a word ladder whose links are never written down, and splitting a team into two groups.
- You build
- islands.py, then maze.py, then fire.py, then copy_graph.py, then ladder.py, then groups.py
- Stages
- 6 Problems
- Time
- About 2.5 hours
- 11.4
Depth-first search
Six interview problems a depth-first search answers, kept on a stack of its own: rooms opened by the keys inside, regions of a board cut off from its edge, rain that reaches two oceans, a cycle among steps, the longest climb through a grid, and ratios from equations.
- You build
- keys.py, then regions.py, then oceans.py, then cycle.py, then climb.py, then ratios.py
- Stages
- 6 Problems
- Time
- About 2.5 hours
- 11.5
Graph order and paths
Six interview problems on ordering and connecting a graph's nodes and finding its cheapest paths: an order for courses, the order of an alphabet from sorted words, the cable that closes a loop, merging accounts that share addresses, the time a message takes to reach a network, and the cheapest route within a number of stops.
- You build
- order.py, then letters.py, then cables.py, then accounts.py, then delay.py, then route.py
- Stages
- 6 Problems
- Time
- About 3 hours
- 11.1
12 Coding interviews: heaps, recursion and intervals Show
- 12.1
Heaps
Six interview problems a heap answers: the cheapest way to join ropes, the median of a stream as it grows, many sorted streams merged into one, the kth largest of a stream so far, a string rearranged so no two letters alike touch, and the pairs with the smallest sums.
- You build
- ropes.py, then medians.py, then merging.py, then kth.py, then reorganize.py, then pairs.py
- Stages
- 6 Problems
- Time
- About 2.5 hours
- 12.2
Recursion
Six interview problems solved from smaller cases of themselves: a symbol deep in a table of rows, Fibonacci numbers by halving, the last one left in a circle, prefix expressions, a list's inversions counted by merge sort, and the closest pair of points.
- You build
- symbols.py, then fibonacci.py, then circle.py, then expressions.py, then inversions.py, then closest.py
- Stages
- 6 Problems
- Time
- About 2.5 hours
- 12.3
Backtracking
Six interview problems solved by trying a choice, going on, and taking it back: every balanced bracketing, the combinations that add up to a total, queens on a board, a word traced through a grid of letters, the splits of a text into palindromes, and a sudoku.
- You build
- brackets.py, then totals.py, then queens.py, then grid.py, then palindromes.py, then sudoku.py
- Stages
- 6 Problems
- Time
- About 2 hours
- 12.4
Intervals
Six interview problems on spans of time and number: an interval inserted into a sorted list, where two schedules meet, the fewest meetings to cancel, a calendar that refuses double bookings, everyone's free time, and the smallest interval that holds each query.
- You build
- insert.py, then meet.py, then remove.py, then bookings.py, then free.py, then queries.py
- Stages
- 6 Problems
- Time
- About 2 hours
- 12.5
Mock interview
Three interview problems in 65 minutes, as in an onsite round: tasks on a machine with a cooldown, the addresses a string of digits can be, and a text cut into parts that share no letter.
- You build
- cooldown.py, then addresses.py, then parts.py
- Stages
- 3 Problems
- Time
- About 1 hour
- 12.1
13 The design round Show
- 13.1
Designing assistants
Design four assistants for a business in writing, then defend each design against an interviewer's follow-up questions.
- Stages
- 4 Designs
- Time
- About 3 hours
- 13.2
Designing agents
Design four agents that act for a business in writing, then defend each design against an interviewer's follow-up questions.
- Stages
- 4 Designs
- Time
- About 3 hours
- 13.3
Running LLM features
Design how LLM features are run, released, moderated and kept on a customer's own servers, then defend each design against follow-up questions.
- Stages
- 4 Designs
- Time
- About 3 hours
- 13.1
14 Capstone Show
- 14.1
The take-home
A first version of Harbor College's aid office assistant in three hours: the office's help pages in passages, a search that finds the page that answers a question, and answers from the model that cite their pages, or pass the question to staff.
- You build
- assistant.py
- Stages
- 1 Problem
- Time
- About 3 hours
- 14.2
The office's search
Harbor College's aid office's search, for its own server: the office's passages in Postgres, kept in step as its pages change, a search by meaning and by words that gives pages, a limit that turns away questions the pages don't answer, a review of an assistant's search, and the search the office's assistant calls.
- You build
- index.py, then search.py, then scope.py, then finder.py, then office.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 5 hours
- Runs on
- Your machine
- 14.3
The office's answers
Harbor College's aid office's assistant, answering students: answers from the served model that cite the office's pages, each student's own aid from records behind row-level security, two releases compared on the office's questions, a review of an assistant's answers, and the desk the office's chat page calls.
- You build
- answers.py, then records.py, then evals.py, then desk_answers.py, then desk.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 5 hours
- Runs on
- Your machine
- 14.4
Shipped to the office
Harbor College's aid desk on the office's own server: the answer streamed as the model writes it, the desk's API behind the college's sign-in, its latency and capacity from timed replies, a review of an assistant's API, and the write-up the office decides by.
- You build
- streamed.py, then app.py, then capacity.py, then desk_api.py, then report.py
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 5 hours
- Runs on
- Your machine
- 14.1
15 The mock loop Show
- 15.1
An onsite loop
Four rounds against the clock, as in an onsite loop: two coding problems, a design round and a scoping exercise from a customer's brief.
- You build
- search_tree.py, then climb.py
- Stages
- 2 Problems, 2 Designs
- Time
- About 2 hours
- 15.1
16 Go for programmers Show
- 16.1
Types, errors and interfaces
The start of a model gateway, in Go: a chat request read and checked, with its errors as values; routes from each model to the upstreams that serve it; a fallback from one upstream to the next, behind an interface; a review of an assistant's usage counter; and the gateway's answers when a request fails.
- You build
- request.go, then routes.go, then fallback.go, then usage.go, then failures.go
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 16.2
Goroutines, channels and context
The gateway's limits, in Go: a token bucket for each key; retries with backoff, inside the caller's deadline; a limit on the calls in flight, and a batch sent at once; a review of an assistant's fan-out to every upstream; and a retry budget that holds an outage's retries in check.
- You build
- bucket.go, then retry.go, then flight.go, then fanout.go, then budget.go
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- Runs on
- Your machine
- 16.3
Serving HTTP
The gateway on HTTP, in Go: a server with keys, rate limits and a cap on bodies, in front of its routes; upstreams asked over HTTP; replies streamed as server-sent events; a review of an assistant's cache; and a server ready for traffic, with timeouts and a graceful shutdown.
- You build
- server.go, then upstream.go, then stream.go, then cache.go, then serve.go
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 4 hours
- Runs on
- Your machine
- 16.1
17 Java for programmers Show
- 17.1
Records, interfaces and exceptions
The help desk's model, in Java: a ticket as a record that checks itself; the desk's tickets, callbacks and advisers' free times, with its refusals as checked exceptions; the tools a model calls, each reading its arguments from JSON into a record; a review of an assistant's schedule; and a problem details object for every error.
- You build
- Ticket.java, then Desk.java, then Tools.java, then Schedule.java, then Problems.java
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 17.2
Collections, streams and the JVM
The desk's reports and memory, in Java: a day's questions counted with streams; an index of the help pages' words; a cache that forgets the conversations used least lately; a review of an assistant's report; and counts kept right by thousands of threads at once.
- You build
- Reports.java, then Index.java, then Recent.java, then Stats.java, then Tally.java
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 17.3
A REST service
The desk's API on the JDK's own HTTP server: the tickets and callbacks as JSON, with every error as problem details; a gate in front of the model; the reply streamed as server-sent events; a review of an assistant's server; and the service, ready to serve, with its settings from the environment and a stop that lets replies finish.
- You build
- Api.java, then Gate.java, then Replies.java, then Server.java, then App.java
- Stages
- 3 Build, 1 Catch, 1 Ship
- Time
- About 3.5 hours
- Runs on
- Your machine
- 17.1