Linear regression

Warm-up

One question before the lesson. Choose an answer and check it.

Try it yourself

Drag the points, click the loss surface to choose a starting line, and press Run. Then raise the learning rate until each step overshoots, and keep going until the loss grows instead of shrinking.

Practice problems

Work each problem out on paper, then type your answer and press Check. Every problem has hints and a full solution.

Programming exercise

Implement linear regression in plain Python. Save line.py and test_line.py in the same folder, fill in each function in line.py, and run the tests:

python test_line.py

"""Linear regression: programming exercise. Implement each function below, then run the tests from this folder:     python test_line.py A line is two numbers: its slope m and its intercept b, so its y at x ism * x + b. The data is a list of x values and a list of y values, the samelength. Use plain Python (no NumPy)."""  def predict(m, b, xs):    """Return the line's y at each x, as a list."""    raise NotImplementedError  def loss(m, b, xs, ys):    """Return the mean squared error: the average of (predicted y - actual y) squared."""    raise NotImplementedError  def gradient(m, b, xs, ys):    """Return the gradient of the loss as a pair (dL_dm, dL_db).     dL_dm = 2 * mean(error * x) and dL_db = 2 * mean(error),    where error = predicted y - actual y at each point.    """    raise NotImplementedError  def step(m, b, xs, ys, rate):    """Take one step of gradient descent with learning rate `rate`.     Move m and b against their gradients: m - rate * dL_dm, b - rate * dL_db.    Return the new pair (m, b).    """    raise NotImplementedError  def fit(xs, ys, rate, steps, m=0.0, b=0.0):    """Run `steps` steps of gradient descent from (m, b) and return the final (m, b)."""    raise NotImplementedError 

Stuck? “Show the calculation” in Try it yourself gives the loss and the gradient for any line, so you can compare your numbers. The Solution tab has one way to write it, with every line explained.

In practice: feature scaling

Add 10 to every x. It is the same data measured from a different starting point, and the best line has the same slope. But the bowl from step 4 stretches into a long, narrow valley: in its steepest direction it becomes about 40 times steeper. The largest learning rate that still converges (settles at the bottom) drops from 0.349 to 0.0088, and at a safe rate training takes 36,373 steps instead of 20.

In a real training run, this looks like a loss that drops fast for a few steps and then barely moves for hours. Or a learning rate that used to work (0.1 here) suddenly makes the loss blow up to NaN ("not a number", what you get when a value overflows) within a few steps.

The fix is feature scaling. Before training, standardise each input: subtract its mean and divide by its standard deviation (how spread out its values are). The bowl becomes round again, and one learning rate suits every direction. Module 7 comes back to this inside neural networks, as normalisation.

Interview questions

1. 01Why square the errors instead of just adding them up?Show answer

   Errors above and below the line have opposite signs, so added up they cancel out, and a terrible line can score zero. Squaring makes every error positive and penalises big misses more than small ones. It also gives a smooth loss with a gradient everywhere. Absolute error avoids the cancelling too, but it has a sharp corner at zero and is less sensitive to outliers. Bonus: if the noise is Gaussian (bell-shaped), minimising squared error gives the most likely line.

2. 02What does the learning rate do, and how do you pick one?Show answer

   It sets how big each step is. Too small and training is slow. Too large and each step overshoots the minimum: the loss zigzags, and past a certain point it grows without limit. For a bowl-shaped loss like this one, that limit is exactly 2 divided by the bowl’s curvature (how quickly its slope changes) in its steepest direction. In practice you try values a factor of 10 apart (0.001, 0.01, 0.1), watch the loss curve, and often use a schedule: warm up at the start, then lower the rate as training goes on.

3. 03Can a single step of gradient descent make the loss go up?Show answer

   Yes. The gradient only tells you the slope where you are standing. If the step is long enough to cross the bottom of the valley and land higher on the other side, the loss goes up. On a perfect bowl that can only happen when the learning rate times the curvature (how quickly the slope changes) is more than 2. In a real network the curvature changes as you move, which is one reason loss curves spike.

Exit ticket

One last question on the main idea of the lesson.