Sulba
ReviewSign inAccount
Theme

Course

AI Engineering

Beginner

Modules
8
Labs
41
Time
About 133 hours
Needs
Any laptop: every lab runs in your browser
  1. 1 Python for programmers

    7 labs · About 22.5 hours

    1. 1.1

      Values and references

      Start porting superstruct, a JavaScript library that checks data, to Python: tell values apart as JavaScript does, compare them as its strict equality does, and fill in defaults without changing the data you were given.

      You build
      values.py, then intake.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours

    2. 1.2

      Iterators and generators

      Port how superstruct checks a whole record: build schemas with comprehensions, walk nested data with a generator that yields each failure with its path, and check a day's records as they stream in, reading no more than you need.

      You build
      walk.py, then walker.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours

    3. 1.3

      Closures and decorators

      Port superstruct's factories and refinements: functions that make checks and remember what they were made with, refinements wrapped around a check, and decorators that turn a plain test into a named check and count the failures each check finds.

      You build
      refine.py, then form.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

    4. 1.4

      Classes and dataclasses

      Port superstruct's structs to classes: a dataclass for each failure, a class for each kind of struct that shares one walk through its base class, and the special methods that join structs with a bar, test whether a value passes and show each struct as it's made.

      You build
      structs.py, then service.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

    5. 1.5

      Exceptions and context managers

      Port how superstruct reports a value that fails: an error class of its own, raised, returned or tested for, notes and chaining for a day's records, groups of errors raised together, and context managers that clean up whatever happens.

      You build
      errors.py, then daily.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

    6. 1.6

      Concurrency with asyncio

      Check each sample's code against the clinic's registry with asyncio: coroutines that wait without holding the service up, lookups made at once in tasks and task groups, a limit and a timeout on a slow registry, and a day's codes checked within them.

      You build
      lookups.py, then day_lookups.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

    7. 1.7

      Types, tests and packages

      Port superstruct's types: type hints that mypy checks in strict mode, generic structs that narrow what a value is, tests with pytest that catch a change that breaks the port, and a check of the packages uv locks for the service against the clinic's approved list.

      You build
      typed.py, then test_typed.py, then intake.py, then approvals.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

  2. 2 The maths behind ML, as code

    5 labs · About 17.5 hours

    1. 2.1

      Arrays and broadcasting

      Find the past samples most like each new breast mass sample, in NumPy: standardize thirty measurements at once, measure every distance by broadcasting, and keep the search within its memory as the records grow.

      You build
      similar.py, then search.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

    2. 2.2

      Vectors and matrices

      Treat each breast mass sample as a vector in NumPy: score it along the line from the average benign sample to the average malignant one, and work out every distance between samples at once with the matrix product.

      You build
      vectors.py, then screen.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

    3. 2.3

      Derivatives and gradients

      Estimate derivatives from finite differences in NumPy, find the step where rounding takes over, work out the gradient of the clinic's first neural network by the chain rule, and check a large model's gradient along random directions.

      You build
      gradients.py, then network.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

    4. 2.4

      Sampling

      Draw the random batches a model trains on, in NumPy: shuffle every epoch, work out how far a batch's average strays from the whole, balance the classes by weighted sampling, and share each epoch's batches out across workers.

      You build
      sampling.py, then batches.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

    5. 2.5

      The bootstrap

      Put an interval around a number measured on a sample, in NumPy: resample the clinic's results, read confidence intervals off the bootstrap distribution, correct them for bias and skew, and compare two models on the same samples.

      You build
      bootstrap.py, then evaluation.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

  3. 3 Working with data

    4 labs · About 11.5 hours

    1. 3.1

      Cleaning data

      Turn a hospital network's raw extract into a table you can trust, in pandas: read its missing values right, mark the codes that mean unknown, keep each patient's first stay, and clean an extract too big to read at once.

      You build
      cleaning.py, then clean.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours

    2. 3.2

      Joins and group-by

      Count a hospital network's readmissions by what its codes mean, in pandas: split a file of lookup tables, join them to the encounters without losing or doubling one, count by group with the unknowns kept, and give each encounter its patient's history for a model.

      You build
      report.py, then monthly.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours

    3. 3.3

      Querying with SQL

      Ask a hospital network's questions of a database, in SQL: load the extract into SQLite, join its codes without losing an encounter, count by group with NULL kept, and build each encounter's history with window functions, with answers that agree with pandas.

      You build
      queries.py, then dashboard.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours

    4. 3.4

      Checking data

      Check a hospital network's monthly extract before anything uses it, in pandas: hold it to a contract from the dataset's documentation, check its keys and codes, find what a patient's encounters can't all say at once, and compare each month's extract with the last.

      You build
      checks.py, then validate.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours

  4. 4 Learning from data

    5 labs · About 20 hours

    1. 4.1

      Linear regression

      Build a model in NumPy that forecasts a power plant's electrical output for each hour from the weather around it.

      You build
      model.py, then forecaster.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours

    2. 4.2

      Logistic regression

      Build a classifier in NumPy that takes measurements of the cell nuclei in a sample from a breast mass and gives the probability that the mass is malignant.

      You build
      model.py, then screener.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours

    3. 4.3

      Overfitting and regularization

      Build a model in NumPy that predicts a concrete mix's compressive strength from its recipe and age, and find how flexible a model the plant's records can support.

      You build
      model.py, then strength.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours

    4. 4.4

      Cross-validation

      Score the plant's concrete models on every one of its mixes in turn with k-fold cross-validation, choose by it, and measure the error of the choosing itself.

      You build
      cv.py, then tuning.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours

    5. 4.5

      Measuring a classifier

      Measure a pulsar survey's classifier the way its astronomers will use it: how well it ranks the candidates, what share of those it flags are pulsars, and whether its probabilities can be counted on.

      You build
      metrics.py, then report.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours

  5. 5 Models that win on tables

    6 labs · About 18 hours

    1. 5.1

      Nearest neighbours

      Name a bean's variety from the beans most like it, in NumPy: put its measurements on one scale, find its nearest neighbours and take their vote, choose how many to ask, and sort a whole line of beans, saying which it can't place.

      You build
      sorter.py, then line.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours

    2. 5.2

      Decision trees

      Rank a bank's clients by their chance of buying a term deposit with a decision tree grown in NumPy: find the split that leaves the least impurity, grow the tree split by split, choose its depth, and turn it into rules a call centre can read.

      You build
      tree.py, then call_list.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours

    3. 5.3

      Random forests

      Rank the bank's clients with a forest of decision trees: grow each tree on its own sample of the calls, give each split columns drawn at random, estimate the forest from the calls each tree left out, and measure which details of a client its ranking leans on.

      You build
      forest.py, then report.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours

    4. 5.4

      Gradient boosting

      Rank the bank's clients with boosted trees: grow each tree on what the trees before it got wrong, choose the learning rate and the rounds on validation calls, work out the splits XGBoost makes, and ship a LightGBM model that never raises a client for being called again.

      You build
      boost.py, then ranking.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours

    5. 5.5

      k-means and PCA

      Find the groups in the beans' shapes without their varieties: put the measurements' variance into a few principal components, group the beans with k-means, choose how many groups by how well they separate, and estimate a new harvest's make-up from named groups.

      You build
      groups.py, then report.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours

    6. 5.6

      Leakage and model comparison

      Compare the module's four models with a logistic regression on the bank's calls: encode a category by its share of yes without leaking the answer, score every model on the same folds, put error bars on the differences that allow for folds sharing their calls, and ship the review that decides which model the bank calls from.

      You build
      encoding.py, then compare.py, then report.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

  6. 6 Text as numbers

    3 labs · About 10 hours

    1. 6.1

      Tokens and byte-pair encoding

      Build an online shop's tokenizer from its product titles: learn byte-pair merges, encode and decode titles as tiktoken does, measure what titles cost in tokens in three languages, and ship a shortener that fits any title into a token limit.

      You build
      bpe.py, then prompts.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

    2. 6.2

      Logits and sampling

      Ask a small language model whether products match shoppers' searches, and read its answers from the logits: turn them into probabilities, reshape them with temperature, top-k and top-p, draw answers as generation does, and ship probabilities that mean what they say.

      You build
      judge.py, then sampler.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

    3. 6.3

      Embeddings and search

      Search the shop's products with a small embedding model: compare queries and titles by cosine similarity, find each query's nearest products, score the results against people's judgements with recall, MRR and nDCG, and ship an index that searches a fraction of the catalogue.

      You build
      search.py, then evaluate.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

  7. 7 Coding interviews

    9 labs · About 14 hours

    1. 7.1

      Arrays

      Five interview problems on lists of numbers and a grid: sums over ranges, the best trade, the largest run, the products of the others and a spiral.

      You build
      ranges.py, then trade.py, then runs.py, then products.py, then spiral.py
      Stages
      5 Problems
      Time
      About 1.5 hours

    2. 7.2

      Hash maps

      Five interview problems where a dict or a set turns a search into one lookup: pair sums, the first unique character, groups of anagrams, the longest streak and runs that add up.

      You build
      pairs.py, then unique.py, then anagrams.py, then streaks.py, then subarrays.py
      Stages
      5 Problems
      Time
      About 1.5 hours

    3. 7.3

      Strings

      Five interview problems on text: a palindrome check, the common prefix, run lengths, the same pattern and the longest palindrome inside a text.

      You build
      palindromes.py, then prefixes.py, then lengths.py, then patterns.py, then longest.py
      Stages
      5 Problems
      Time
      About 1.5 hours

    4. 7.4

      Two pointers

      Five interview problems where two positions move through a list: a pair in sorted order, duplicates removed in place, the most water, triples that sum to zero and trapped water.

      You build
      sorted_pair.py, then dedupe.py, then water.py, then triples.py, then rain.py
      Stages
      5 Problems
      Time
      About 2 hours

    5. 7.5

      Sliding windows

      Five interview problems solved by a window that slides along a list or a text: the best window, no repeats, reaching a total, at most k kinds and the smallest cover.

      You build
      windows.py, then no_repeats.py, then reach.py, then kinds.py, then cover.py
      Stages
      5 Problems
      Time
      About 2 hours

    6. 7.6

      Stacks

      Five interview problems that a stack solves as it reads its input: balanced brackets, reverse Polish notation, days until warmer, a stack that knows its minimum and decoding repeats.

      You build
      brackets.py, then rpn.py, then warmer.py, then min_stack.py, then repeats.py
      Stages
      5 Problems
      Time
      About 1.5 hours

    7. 7.7

      Binary search

      Five interview problems that halve what's left to search at each step: the insertion point, the first and last places of a value, a rotated list, a whole-number square root and the least capacity to ship in time.

      You build
      insertion.py, then first_last.py, then rotated.py, then square_root.py, then shipping.py
      Stages
      5 Problems
      Time
      About 1.5 hours

    8. 7.8

      Sorting

      Five interview problems that come apart once their input is in order: merging intervals, meeting rooms, the kth largest, the largest number and the most frequent items.

      You build
      intervals.py, then rooms.py, then kth.py, then largest_number.py, then frequent.py
      Stages
      5 Problems
      Time
      About 1.5 hours

    9. 7.9

      Mock interview

      Two interview problems in 45 minutes, as in a coding screen: merging sorted lists, then the anagrams of a word inside a text.

      You build
      merge.py, then anagram_starts.py
      Stages
      2 Problems
      Time
      About 45 minutes

  8. 8 Capstone

    2 labs · About 19.5 hours

    1. 8.1

      The take-home

      A first version of the hospital network's readmission service in three hours: read its extract, train a model on each patient's first stay, and serve the chance of a readmission within 30 days from a FastAPI endpoint.

      You build
      service.py
      Stages
      1 Problem
      Time
      About 3 hours

    2. 8.2

      The readmission service

      The full build of the hospital network's readmission service: inputs made one way for training and serving, a model chosen with error bars and checked for calibration, a threshold for the calls planners can make, an API held to a latency budget, your own tests, and a one-page write-up.

      You build
      features.py, then model.py, then train.py, then app.py, then call_list.py, then test_app.py, then report.py
      Stages
      10 Build, 2 Catch, 2 Ship
      Time
      About 16.5 hours

Intermediate

Modules
17, 2 of them optional
Labs
66
Time
About 194.5 hours, and about 21.5 hours more in the optional modules
Needs
A laptop with uv, 8 GB of memory and about 10 GB free: Windows or Linux on x86-64, Linux on Arm, or macOS 14 or later on Apple silicon. Module 1, the coding interviews, most of Working with customers, the design round, the capstone's take-home and the mock loop run in your browser, and the rest runs on your machine
  1. 1 Neural networks from scratch

    5 labs · About 19.5 hours

    1. 1.1

      Layers and backpropagation

      Build a network from layers in NumPy: linear layers, ReLU and a cross-entropy loss, each with a backward pass whose gradients match PyTorch's, and train it to read handwritten digits from people whose writing it has never seen.

      You build
      layers.py, then network.py, then train.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours

    2. 1.2

      An autograd engine

      Build an engine that works out gradients by itself, as PyTorch's autograd does: values and arrays that remember how they were made, a backward pass in topological order, broadcasting summed back, and gradients that match PyTorch's on random expressions.

      You build
      engine.py, then autograd.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4.5 hours

    3. 1.3

      Optimizers

      Build PyTorch's optimizers in NumPy: SGD with momentum as PyTorch writes it, Adam with its bias correction, and AdamW with its weight decay kept apart, each step matched to torch.optim's, then train the office's network on a budget of steps.

      You build
      optimizers.py, then optim.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours

    4. 1.4

      Initialization and normalization

      Train a deep network: start its weights as torch.nn.init starts them, so that the signal keeps its size through every layer, then build batch and layer normalization, each matched to PyTorch's outputs and gradients, until the network trains where it stalled.

      You build
      init.py, then norms.py, then normalization.py, then deep.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours

    5. 1.5

      Debugging a training run

      Find what's wrong with a training run before it costs a day: the loss a network should start at, a batch it should learn by heart, gradient norms and clipping as torch.nn.utils has them and a learning-rate range test, then review an assistant's training script and ship a run whose report holds up.

      You build
      network.py, then clipping.py, then range_test.py, then train.py, then run.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours

  2. 2 PyTorch

    4 labs · About 16.5 hours

    1. 2.1

      Tensors and autograd

      Port the records office's digit reader from NumPy to PyTorch: its digits as tensors, its gradients from autograd, a loss with a backward pass of your own, and a reader that trains step for step as the NumPy one did.

      You build
      digits.py, then network.py, then trainer.py, then reader.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours
      Runs on
      Your machine

    2. 2.2

      Modules and training loops

      Train the records office's deep reader the way PyTorch code does: its layers as modules, its digits from a data loader, its validation in evaluation mode, and checkpoints that pick up a stopped run exactly where it was.

      You build
      reader.py, then data.py, then checkpoints.py, then trainer.py, then run.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours
      Runs on
      Your machine

    3. 2.3

      Convolutional networks

      Sort an online shop's photos of clothes with a convolutional network: convolution and pooling by hand, the photos' files read and checked, a network that trains on any device and in mixed precision, a review of an assistant's classifier, and a classifier that beats the shop's perceptron within a budget of compute.

      You build
      conv.py, then photos.py, then net.py, then classifier.py, then shop.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4.5 hours
      Runs on
      Your machine

    4. 2.4

      Transfer learning

      Sort a land-use survey's satellite tiles with a backbone trained by others: the tiles read from their zip, a vision transformer loaded from its checkpoint, a linear probe on its features, a review of an assistant's fine-tuning, and a classifier that learns from a few tiles of each kind.

      You build
      tiles.py, then backbone.py, then probe.py, then finetune.py, then survey.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours
      Runs on
      Your machine

  3. 3 Transformers from scratch

    5 labs · About 18.5 hours

    1. 3.1

      Attention

      Build the attention a reading app's story model runs in every layer: scaled dot-product attention with its causal mask, heads split and merged, masks for batches of stories, and attention in tiles that never holds a whole row of scores.

      You build
      attention.py, then self_attention.py, then tiles.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours
      Runs on
      Your machine

    2. 3.2

      The transformer block

      Build the app's GPT as GPT-2 builds it: layer normalization by hand, the MLP and the pre-norm residual block, a GPT with learned positions and a tied output layer, and what a GPT costs, worked out from its configuration.

      You build
      block.py, then gpt.py, then costs.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine

    3. 3.3

      Training a small GPT

      Train the reading app's story model on your own machine: a tokenizer learnt from the stories, batches of tokens packed as GPT-3 packs its documents, a training loop with AdamW, warm-up, cosine decay and clipping, a review of an assistant's trainer, and a model that beats a target loss within a budget of compute.

      You build
      tokenizer.py, then data.py, then train.py, then trainer.py, then story.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours
      Runs on
      Your machine

    4. 3.4

      GPT-2's weights

      Run OpenAI's GPT-2 in the reading app's own GPT: its weights read by hand from their file, its tokenizer built from its published vocabulary and merges, its tensors mapped into your model until its predictions match Hugging Face's, a review of an assistant's loader, and a loader for any GPT-2 that refuses a damaged file.

      You build
      weights.py, then tokenizer.py, then gpt2.py, then loader.py, then load.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine

    5. 3.5

      Rotary positions and the KV cache

      Run Hugging Face's SmolLM2-135M in your own code: rotary positions, RMSNorm, SwiGLU and grouped-query attention matched to transformers' Llama, a KV cache that writes each token without reading the story again, a review of an assistant's cached generator, and a generator for batches of prompts.

      You build
      rope.py, then smollm.py, then cache.py, then generator.py, then batch.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine

  4. 4 Building on model APIs

    4 labs · About 13 hours

    1. 4.1

      Calling a model

      Connect a bicycle repair shop's assistant to a model through the chat completions API: requests and replies by hand over HTTP, the tokens a conversation takes and the model's window, a month of the shop's questions priced from providers' published prices, a review of an assistant's client, and a client that retries what it can within a deadline.

      You build
      chat.py, then window.py, then costs.py, then client.py, then retry.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

    2. 4.2

      Structured output

      Hold the bicycle repair shop's assistant to JSON its calendar can take: a JSON Schema validator for the keywords the shop uses, replies checked and asked for again with their errors, constrained decoding that masks every token that can't continue a valid reply, a review of an assistant's extractor, and an extractor that turns the shop's emails into bookings.

      You build
      schema.py, then extract.py, then constrain.py, then bookings.py, then inbox.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine

    3. 4.3

      Tools, streaming and MCP

      Give the bicycle repair shop's assistant tools: the shop's functions described for the model, the loop that runs the calls it makes, its replies read as they stream in, a review of an assistant's tool loop, and the shop's tools served over the Model Context Protocol.

      You build
      tools.py, then loop.py, then stream.py, then assistant.py, then server.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine

    4. 4.4

      Securing the assistant

      Keep the bicycle repair shop's assistant safe when the model does what an attacker writes: tools that answer only for the customer signed in, actions the customer confirms, replies the chat window can show, a review of an assistant's guardrails, and the shop's chat shipped against prompt injections.

      You build
      shop.py, then actions.py, then render.py, then assistant.py, then chat.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine

  5. 5 Retrieval

    5 labs · About 15.5 hours

    1. 5.1

      Embeddings for search

      Search a help desk's pages with an embedding model on your machine: embedding requests in batches with the model's prefixes, a cache that embeds each text once, vectors cut short and kept in a byte a number, a review of an assistant's search, and an index that fits in a few hundred bytes a passage.

      You build
      embed.py, then cache.py, then shorten.py, then search.py, then index.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

    2. 5.2

      Vector search in Postgres

      Search the help desk's passages in Postgres with pgvector: a table of passages and their vectors, an approximate index, searches within one service, a review of an assistant's queries, and a search whose indexes fit a small budget.

      You build
      store.py, then index.py, then service.py, then queries.py, then desk.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

    3. 5.3

      Chunking

      Cut the help desk's pages into chunks and compare the ways of cutting them by the answers they hold: windows of words and joined blocks, recall and precision by an answer's characters, chunkings compared at one budget of words, a review of an assistant's chunker, and a chunker whose chunks last when a page changes.

      You build
      chunk.py, then score.py, then compare.py, then splitter.py, then chunker.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

    4. 5.4

      Hybrid search and reranking

      Search the help desk's passages by their words as well as their vectors, and have a second model put what the two find in order: full-text search in Postgres, BM25 by hand, two rankings fused by their ranks, a review of an assistant's hybrid search, and a search reranked by Qwen3-Reranker-0.6B on your machine.

      You build
      fts.py, then bm25.py, then fuse.py, then hybrid.py, then desk.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

    5. 5.5

      Evaluating retrieval

      Measure the help desk's searches as its readers meet them: hit rate, recall, MRR and nDCG over every question, each text judged once, each part's share found by taking it away and testing the difference question by question, a review of an assistant's report, and a release check that decides whether a new search replaces the old.

      You build
      measures.py, then texts.py, then compare.py, then report.py, then release.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

  6. 6 Evals

    3 labs · About 8 hours

    1. 6.1

      Test sets

      Measure the help desk's replies on questions it never tuned on: a test set split by page and service, checks of each reply written in code, the margin of error the set gives with each page's questions counted together, a review of an assistant's test set, and a release report that says how many questions a margin needs.

      You build
      testset.py, then checks.py, then margin.py, then evalset.py, then release.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 2.5 hours
      Runs on
      Your machine

    2. 6.2

      Grading with a model

      Grade the help desk's replies with Granite Guardian 3.1 2B on your machine: its prompt rendered from its own template and its verdict read from its likeliest first tokens, its agreement with people's labels, a check on the help desk's own pages, a review of an assistant's grading run, and a report of how many of a release's replies keep to their passages.

      You build
      grader.py, then agreement.py, then known.py, then judge.py, then grounding.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

    3. 6.3

      Regression gates

      Gate the help desk's releases in CI: several replies to each question and what they take out of the standard error, two releases compared question by question, the smallest drop the test set can be counted on to find, a review of an assistant's gate, and a gate that stops a planted regression and ships a rerun.

      You build
      samples.py, then compare.py, then power.py, then gate.py, then release_gate.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 2.5 hours
      Runs on
      Your machine

  7. 7 Agents

    4 labs · About 12.5 hours

    1. 7.1

      Workflows and agents

      Have the help desk do its own work as well as answer: the desk's tools over a caller's tickets and its advisers' callbacks, a workflow that sorts each message and follows a fixed path, an agent that calls the tools itself, a review of an assistant's agent, and a router that sends the workflow what it can follow and the agent the rest.

      You build
      desk.py, then workflow.py, then agent.py, then desk_agent.py, then router.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

    2. 7.2

      Tools and planning

      Write the help desk's tools for the model that calls them: search results it can cite, lists that say what they leave out, and errors that say how to call again; tickets and callbacks the model proposes and the desk makes only on the caller's yes, after looking up what they need; the agent on those tools; a review of an assistant's proposals; and callbacks that move.

      You build
      desk.py, then actions.py, then agent.py, then desk_actions.py, then moves.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine

    3. 7.3

      Memory and sub-agents

      Keep the help desk's long conversations within a budget of tokens: old tool results cleared first, notes the model saves that outlast a summary of the turns before, and researchers that search each service's pages in a conversation of their own; a review of an assistant's memory; and notes that last from one conversation to the next.

      You build
      context.py, then memory.py, then researchers.py, then desk_memory.py, then notebook.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

    4. 7.4

      Scoring an agent

      Score the help desk's agents over repeated trials of each task: pass@k and pass^k, with intervals that count each task once; the median and 90th percentile of the tasks' seconds and tokens, with intervals from the order statistics; traces of an agent's runs in OpenTelemetry's conventions for generative AI, and each run's row from them; a review of an assistant's scorecard; and a release gate that compares a candidate with the current agent task by task.

      You build
      scores.py, then costs.py, then traces.py, then scorecard.py, then gate.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

  8. 8 Shipping full stack

    5 labs · About 22.5 hours

    1. 8.1

      The desk's API

      Put the help desk's agent behind a web API: conversations kept by id, with every error as a problem details object; a gate in front of the model, with a time limit on each reply; the reply streamed as server-sent events, kept only once it's whole; a review of an assistant's replies; and the app, ready to serve, with its settings from the environment.

      You build
      conversations.py, then gate.py, then replies.py, then desk_replies.py, then app.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine

    2. 8.2

      Conversations in Postgres

      Keep the desk's conversations in Postgres: migrations applied once each, in order, by one app at a time; a store whose turns are kept whole, in one transaction, on pooled connections; each turn kept with what it used, however it ended; a review of an assistant's store; and each day's usage for the dashboard, found through an index.

      You build
      migrations.py, then store.py, then replies.py, then desk_store.py, then app.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine

    3. 8.3

      Signing in

      Sign the desk's callers in with OpenID Connect: the authorization request with its state, nonce and PKCE challenge; the code exchanged and the ID token checked against the provider's keys; sessions kept in Postgres, as hashes of their tokens; each person's conversations theirs alone; a review of an assistant's sign-in routes; and the app signed in, with its cookies sent only over HTTPS and no change taken from another site.

      You build
      sign_in.py, then tokens.py, then sessions.py, then store.py, then desk_auth.py, then app.py
      Stages
      4 Build, 1 Catch, 1 Ship
      Time
      About 4.5 hours
      Runs on
      Your machine

    4. 8.4

      The chat page

      The desk's page, in TypeScript: a typed client for its API with its problem details as errors; the reply's event stream read from fetch as browsers read one; the chat, with each message put on the page as text and a log a screen reader can follow; a review of an assistant's chat; and a dashboard of usage and cost by UTC day, in a table a screen reader can read.

      You build
      api.ts, then stream.ts, then chat.ts, then desk_chat.ts, then dashboard.ts
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 7 hours
      Runs on
      Your machine

    5. 8.5

      Shipping the desk

      The desk ready for a host: logs as JSON lines, each with its request's id and no secrets; readiness that says whether the database and the model answer; the server's settings from the host's environment, with a clean stop; a review of an assistant's setup; and the desk's image, built in stages and run as a user without root.

      You build
      logs.py, then health.py, then main.py, then desk_ops.py, then Dockerfile
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours
      Runs on
      Your machine

  9. 9 Data pipelines

    3 labs · About 9 hours

    1. 9.1

      Analytical SQL

      Answer a cab dispatcher's questions about a month of New York's yellow cab trips in Postgres: each borough's busiest zones with ties kept, an hour-by-hour series with running totals, subtotals by borough and payment, a review of an analyst's report, and a daily summary that a run replaces whole.

      You build
      zones.py, then hours.py, then payments.py, then report.py, then summary.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

    2. 9.2

      Spark DataFrames

      Work through a month of New York's yellow and green cab trips with Spark's DataFrame API: one table from two files laid out differently, a flag for each record that can't be taken at its word, each zone's days with a seven-day mean over the calendar, a review of an assistant's report, and daily Parquet files that a run replaces whole.

      You build
      trips.py, then flags.py, then days.py, then report.py, then store.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

    3. 9.3

      Testing a pipeline

      Join a month of New York's cab trips to the taxi zones and to Central Park's weather with Spark, and test the pipeline that does it: NOAA's daily weather file read into units, each borough's trips on every day of August, wet days against dry days of the same weekday, a review of an assistant's daily table, and tests of your own that catch the mistakes planted in the pipeline.

      You build
      weather.py, then daily.py, then rain.py, then daily_table.py, then test_pipeline.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3 hours
      Runs on
      Your machine

  10. 10 Working with customers

    4 labs · About 9 hours

    1. 10.1

      Estimating the work

      Turn a team's three estimates of each task into a date and a price for a customer: the triangular distribution's mean and quantiles, the project's finish over the tasks it waits for, many runs of it drawn at random, a promise from their percentiles, a review of an assistant's estimate, and a quote in working days and whole cents.

      You build
      estimates.py, then schedule.py, then promise.py, then estimate.py, then quote.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 2.5 hours

    2. 10.2

      Measuring value

      Work out what the help desk is worth to the benefits office from its agent's trials: each kind of contact's share done, counting each task once; the share of the office's own contacts, weighted by its mix; the net saving a month and the months the pilot takes to pay for itself, each with its interval; a review of an assistant's value report; and the summary the office reads, with every interval rounded outward.

      You build
      kinds.py, then mix.py, then worth.py, then report.py, then summary.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 2.5 hours

    3. 10.3

      Inside the customer's walls

      Bring the desk inside the benefits office: a search of its staff guidance that finds only what each person may read, groups from a sign-in whose token checks out, an audit log that names no one, a review of an assistant's search for staff, and a transport that lets no request leave the office's hosts.

      You build
      access.py, then staff.py, then audit.py, then staff_search.py, then walls.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 2.5 hours
      Runs on
      Your machine

    4. 10.4

      A demo against the clock

      Two demos for the benefits office, each from its written brief and against the clock, with the brief's points planned as tests first: its web form's messages filed to the right queue, with Social Security numbers kept from the model, and answers from the office's own answers that cite the ones they use.

      You build
      triage.py, then faq.py
      Stages
      2 Problems
      Time
      About 1.5 hours

  11. 11 Coding interviews: lists, trees and graphs

    5 labs · About 12.5 hours

    1. 11.1

      Linked lists

      Seven interview problems on linked lists: reversing one, finding where a cycle starts, removing a node from the end, adding numbers held as digits, reordering a list from both ends, copying a list with random links and a least recently used cache.

      You build
      reverse.py, then cycle.py, then remove.py, then add.py, then reorder.py, then copy_list.py, then lru.py
      Stages
      7 Problems
      Time
      About 2.5 hours

    2. 11.2

      Binary trees

      Six interview problems on binary trees: reading one level by level, its diameter, the lowest common ancestor of two nodes, writing a tree down and reading it back, the paths that add up to a target and the largest path sum.

      You build
      levels.py, then diameter.py, then ancestor.py, then codec.py, then paths.py, then path_sum.py
      Stages
      6 Problems
      Time
      About 2.5 hours

    3. 11.3

      Graph search

      Six interview problems on searching graphs, breadth first and depth first: counting islands on a map, the shortest way out of a maze, a fire spreading from many places at once, copying a graph with cycles, a word ladder whose links are never written down, and splitting a team into two groups.

      You build
      islands.py, then maze.py, then fire.py, then copy_graph.py, then ladder.py, then groups.py
      Stages
      6 Problems
      Time
      About 2.5 hours

    4. 11.4

      Depth-first search

      Six interview problems a depth-first search answers, kept on a stack of its own: rooms opened by the keys inside, regions of a board cut off from its edge, rain that reaches two oceans, a cycle among steps, the longest climb through a grid, and ratios from equations.

      You build
      keys.py, then regions.py, then oceans.py, then cycle.py, then climb.py, then ratios.py
      Stages
      6 Problems
      Time
      About 2.5 hours

    5. 11.5

      Graph order and paths

      Six interview problems on ordering and connecting a graph's nodes and finding its cheapest paths: an order for courses, the order of an alphabet from sorted words, the cable that closes a loop, merging accounts that share addresses, the time a message takes to reach a network, and the cheapest route within a number of stops.

      You build
      order.py, then letters.py, then cables.py, then accounts.py, then delay.py, then route.py
      Stages
      6 Problems
      Time
      About 3 hours

  12. 12 Coding interviews: heaps, recursion and intervals

    5 labs · About 10 hours

    1. 12.1

      Heaps

      Six interview problems a heap answers: the cheapest way to join ropes, the median of a stream as it grows, many sorted streams merged into one, the kth largest of a stream so far, a string rearranged so no two letters alike touch, and the pairs with the smallest sums.

      You build
      ropes.py, then medians.py, then merging.py, then kth.py, then reorganize.py, then pairs.py
      Stages
      6 Problems
      Time
      About 2.5 hours

    2. 12.2

      Recursion

      Six interview problems solved from smaller cases of themselves: a symbol deep in a table of rows, Fibonacci numbers by halving, the last one left in a circle, prefix expressions, a list's inversions counted by merge sort, and the closest pair of points.

      You build
      symbols.py, then fibonacci.py, then circle.py, then expressions.py, then inversions.py, then closest.py
      Stages
      6 Problems
      Time
      About 2.5 hours

    3. 12.3

      Backtracking

      Six interview problems solved by trying a choice, going on, and taking it back: every balanced bracketing, the combinations that add up to a total, queens on a board, a word traced through a grid of letters, the splits of a text into palindromes, and a sudoku.

      You build
      brackets.py, then totals.py, then queens.py, then grid.py, then palindromes.py, then sudoku.py
      Stages
      6 Problems
      Time
      About 2 hours

    4. 12.4

      Intervals

      Six interview problems on spans of time and number: an interval inserted into a sorted list, where two schedules meet, the fewest meetings to cancel, a calendar that refuses double bookings, everyone's free time, and the smallest interval that holds each query.

      You build
      insert.py, then meet.py, then remove.py, then bookings.py, then free.py, then queries.py
      Stages
      6 Problems
      Time
      About 2 hours

    5. 12.5

      Mock interview

      Three interview problems in 65 minutes, as in an onsite round: tasks on a machine with a cooldown, the addresses a string of digits can be, and a text cut into parts that share no letter.

      You build
      cooldown.py, then addresses.py, then parts.py
      Stages
      3 Problems
      Time
      About 1 hour

  13. 13 The design round

    3 labs · About 9 hours

    1. 13.1

      Designing assistants

      Design four assistants for a business in writing, then defend each design against an interviewer's follow-up questions.

      Stages
      4 Designs
      Time
      About 3 hours

    2. 13.2

      Designing agents

      Design four agents that act for a business in writing, then defend each design against an interviewer's follow-up questions.

      Stages
      4 Designs
      Time
      About 3 hours

    3. 13.3

      Running LLM features

      Design how LLM features are run, released, moderated and kept on a customer's own servers, then defend each design against follow-up questions.

      Stages
      4 Designs
      Time
      About 3 hours

  14. 14 Capstone

    4 labs · About 17.5 hours

    1. 14.1

      The take-home

      A first version of Harbor College's aid office assistant in three hours: the office's help pages in passages, a search that finds the page that answers a question, and answers from the model that cite their pages, or pass the question to staff.

      You build
      assistant.py
      Stages
      1 Problem
      Time
      About 3 hours

    2. 14.2

      The office's search

      Harbor College's aid office's search, for its own server: the office's passages in Postgres, kept in step as its pages change, a search by meaning and by words that gives pages, a limit that turns away questions the pages don't answer, a review of an assistant's search, and the search the office's assistant calls.

      You build
      index.py, then search.py, then scope.py, then finder.py, then office.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 5 hours
      Runs on
      Your machine

    3. 14.3

      The office's answers

      Harbor College's aid office's assistant, answering students: answers from the served model that cite the office's pages, each student's own aid from records behind row-level security, two releases compared on the office's questions, a review of an assistant's answers, and the desk the office's chat page calls.

      You build
      answers.py, then records.py, then evals.py, then desk_answers.py, then desk.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 5 hours
      Runs on
      Your machine

    4. 14.4

      Shipped to the office

      Harbor College's aid desk on the office's own server: the answer streamed as the model writes it, the desk's API behind the college's sign-in, its latency and capacity from timed replies, a review of an assistant's API, and the write-up the office decides by.

      You build
      streamed.py, then app.py, then capacity.py, then desk_api.py, then report.py
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 5 hours
      Runs on
      Your machine

  15. 15 The mock loop

    1 lab · About 2 hours

    1. 15.1

      An onsite loop

      Four rounds against the clock, as in an onsite loop: two coding problems, a design round and a scoping exercise from a customer's brief.

      You build
      search_tree.py, then climb.py
      Stages
      2 Problems, 2 Designs
      Time
      About 2 hours

  16. 16 Go for programmers

    Optional · 3 labs · About 11 hours

    1. 16.1

      Types, errors and interfaces

      The start of a model gateway, in Go: a chat request read and checked, with its errors as values; routes from each model to the upstreams that serve it; a fallback from one upstream to the next, behind an interface; a review of an assistant's usage counter; and the gateway's answers when a request fails.

      You build
      request.go, then routes.go, then fallback.go, then usage.go, then failures.go
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine

    2. 16.2

      Goroutines, channels and context

      The gateway's limits, in Go: a token bucket for each key; retries with backoff, inside the caller's deadline; a limit on the calls in flight, and a batch sent at once; a review of an assistant's fan-out to every upstream; and a retry budget that holds an outage's retries in check.

      You build
      bucket.go, then retry.go, then flight.go, then fanout.go, then budget.go
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours
      Runs on
      Your machine

    3. 16.3

      Serving HTTP

      The gateway on HTTP, in Go: a server with keys, rate limits and a cap on bodies, in front of its routes; upstreams asked over HTTP; replies streamed as server-sent events; a review of an assistant's cache; and a server ready for traffic, with timeouts and a graceful shutdown.

      You build
      server.go, then upstream.go, then stream.go, then cache.go, then serve.go
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 4 hours
      Runs on
      Your machine

  17. 17 Java for programmers

    Optional · 3 labs · About 10.5 hours

    1. 17.1

      Records, interfaces and exceptions

      The help desk's model, in Java: a ticket as a record that checks itself; the desk's tickets, callbacks and advisers' free times, with its refusals as checked exceptions; the tools a model calls, each reading its arguments from JSON into a record; a review of an assistant's schedule; and a problem details object for every error.

      You build
      Ticket.java, then Desk.java, then Tools.java, then Schedule.java, then Problems.java
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine

    2. 17.2

      Collections, streams and the JVM

      The desk's reports and memory, in Java: a day's questions counted with streams; an index of the help pages' words; a cache that forgets the conversations used least lately; a review of an assistant's report; and counts kept right by thousands of threads at once.

      You build
      Reports.java, then Index.java, then Recent.java, then Stats.java, then Tally.java
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine

    3. 17.3

      A REST service

      The desk's API on the JDK's own HTTP server: the tickets and callbacks as JSON, with every error as problem details; a gate in front of the model; the reply streamed as server-sent events; a review of an assistant's server; and the service, ready to serve, with its settings from the environment and a stop that lets replies finish.

      You build
      Api.java, then Gate.java, then Replies.java, then Server.java, then App.java
      Stages
      3 Build, 1 Catch, 1 Ship
      Time
      About 3.5 hours
      Runs on
      Your machine