Derivatives
A derivative is the slope of a function at a point: how fast it is changing there. Learn to measure it, work it out with a few rules, and use it to walk down to the lowest point of a loss, which is how every model is trained.
Warm-up
One question before the lesson. Choose an answer and check it.
f(x) = x² gets steeper as x grows. Near x = 3, about how much does f(x) go up for each 1 that x moves to the right?
Show the answer
C: About 6: for a tiny step, f(x) rises about 6 times as much as x does. From x = 3 to 3.001, x² goes from 9 to 9.006001: a rise of 0.006001 over a run of 0.001, a slope of 6.001. The slope at a point is called the derivative, and this lesson’s power rule gives it exactly: 2x = 6.
Step 1 The slope of a line
A function is a rule that turns one number into another. f(x) = 2x + 1 (say “f of x equals 2x plus 1”) takes any x, doubles it and adds 1: f(3) = 2 × 3 + 1 = 7.
| x | f(x) |
|---|---|
| 0 | 2 × 0 + 1 = 1 |
| 1 | 2 × 1 + 1 = 3 |
| 2 | 2 × 2 + 1 = 5 |
| 3 | 2 × 3 + 1 = 7 |
| 4 | 2 × 4 + 1 = 9 |
Drawn as a graph, with x across and f(x) up, this function is a straight line. Its slope is how steep it is: rise over run, how much f(x) goes up for each 1 that x goes across.
rise = f(3) − f(1) = 7 − 3 = 4
run = 3 − 1 = 2
slope = 4 ÷ 2 = 2
A straight line has the same slope everywhere, and for a line written m·x + b the slope is m: here 2. It is the m of Linear regression.
Try it yourself
Pick where to start and how big the steps are, then step down the bowl. Push the learning rate past 0.5 and the steps overshoot the bottom; past 1, they run away.
f(x) the steps the slope where you stand
step 0: x = 0
f(x) = (0 − 3)² + 1 = 10
f′(x) = 2 × 0 − 6 = −6
new x = 0 − 0.25 × (−6) = 1.5
At 0.25, each step moves closer to 3 without passing it.
Each step multiplies the distance to 3 by 1 − 2 × 0.25 = 0.5.
Practice problems
Work each problem out on paper, then type your answer and press Check. Every problem has hints and a full solution.
Score: 0 of 10 points
Problem 1
1 pointWhat is the slope of f(x) = x⁵ at x = 2?
Hint 1
The power rule: the derivative of x⁵ is 5x⁴.
Solution
f′(x) = 5x⁴
f′(2) = 5 × 2⁴ = 5 × 16 = 80
Problem 2
2 pointsf(x) = 3x² − 4x + 2. What is f′(2)?
Hint 1
Take each piece in turn: the power rule for x², a number in front stays in front, and a number on its own has slope 0.
Solution
f′(x) = 6x − 4
f′(2) = 6 × 2 − 4 = 8
Problem 3
2 pointsMeasure the slope of f(x) = x² at x = 5, with h = 0.1: (f(5.1) − f(5)) ÷ 0.1.
Hint 1
f(5.1) = 5.1² = 26.01.
Hint 2
The rise is that minus f(5) = 25. Then divide by h.
Solution
rise = 26.01 − 25 = 1.01
slope = 1.01 ÷ 0.1 = 10.1
Problem 4
2 pointsf(x) = (x − 4)², so f′(x) = 2x − 8. Start at x = 1 and take one step of gradient descent with a learning rate of 0.1. What is the new x?
Hint 1
Work out the slope where you stand first: f′(1).
Hint 2
new x = x − learning rate × f′(x).
Solution
f′(1) = 2 × 1 − 8 = −6
new x = 1 − 0.1 × (−6) = 1.6
Problem 5
3 pointsf(x) = 2x² − 12x + 5. At what x is f(x) lowest?
Hint 1
At the lowest point the curve is flat: its slope is 0.
Hint 2
Work out f′(x), then solve f′(x) = 0.
Solution
f′(x) = 4x − 12
4x − 12 = 0
4x = 12
x = 12 ÷ 4 = 3
Programming exercise
Measure slopes, apply the power rule and run gradient descent in plain Python. Save derivatives.py and test_derivatives.py in the same folder, fill in each function in derivatives.py, and run the tests:
python test_derivatives.py
"""Derivatives: programming exercise. Measure slopes, use the power rule and run gradient descent in plain Python,then run the tests from this folder: python test_derivatives.py A function is passed in like any other value: f = lambda x: x * x is x squared,and f(3) is 9.""" def slope_between(f, a, b): """Return the slope of f between x = a and x = b: rise over run.""" raise NotImplementedError def measured_slope(f, x, h=1e-5): """Return the slope of f at x, measured: (f(x + h) - f(x - h)) / (2h).""" raise NotImplementedError def power_rule(n, x): """Return the derivative of x to the power n, at x: n times x to the power n - 1.""" raise NotImplementedError def poly_derivative(coeffs): """Return the derivative of a polynomial, as a list of coefficients. coeffs[k] is the number in front of x to the power k, so [5, 2, 3] is 5 + 2x + 3x². Its derivative, 2 + 6x, is [2, 6]. """ raise NotImplementedError def descend(slope, x, rate, steps): """Run gradient descent: take `steps` steps of x = x - rate * slope(x), and return the final x.""" raise NotImplementedError Stuck? Every function here is a line or two: a rise over a run, a rule from step 3, or the step from step 6 in a loop. The Solution tab has one way to write it.
In practice: derivatives in AI
You will rarely work out a derivative yourself in practice. PyTorch, the library most models are trained with, works out the derivative of any loss you write: you call loss.backward() and every number in the model gets its slope. Module 7 builds a small version of that machinery, and the chain rule in the next lesson is what makes it work.
Checking a derivative. When you do work one out yourself, compare it with a measured slope, (f(x + h) − f(x − h)) ÷ 2h for a small h such as 0.00001. If the two disagree past the first few decimal places, the formula is wrong. This is called a gradient check.
The learning rate is the setting tuned most often. On this lesson’s bowl, any rate above 1 makes the steps grow instead of shrink, because the slope changes by 2 for every 1 that x moves. Real losses curve by different amounts in different places, which is why the rate is found by trying.
Test your knowledge
01What is the derivative of x⁴?Show answer
By the power rule, bring the 4 down in front and lower the power by 1: 4x³.
02f(x) = 5x² − 3x + 7. What is f′(x), and what is f′(2)?Show answer
Piece by piece: 5x² → 10x, −3x → −3, 7 → 0. So f′(x) = 10x − 3, and f′(2) = 10 × 2 − 3 = 17.
03f′(x) is negative at x = 4. Should gradient descent move x up or down?Show answer
Up. A negative slope means f goes down as x goes up. The step x − learning rate × f′(x) subtracts a negative number, so x gets bigger.
04Why does a learning rate that is too big make the loss blow up?Show answer
On f(x) = (x − 3)² + 1, one step multiplies the distance from 3 by 1 − 2 × the learning rate. With a rate of 1.5 that is −2: each step lands twice as far away, on the other side. The loss grows every step instead of shrinking.
Exit ticket
One last question on the main idea of the lesson.
What does the derivative f′(x) tell you?
Show the answer
A: How steep f is at x: how fast f(x) changes as x changes. f′(x) is the slope of f at x: the rise over run between x and a point a tiny step away. Where it is 0 the curve is flat, which is how the bottom of a loss is found, and gradient descent steps against it to walk downhill.