Precision, recall and other metrics

Warm-up

One question before the lesson. Choose an answer and check it.

Try it yourself

Move the threshold and watch the confusion matrix, precision, recall and F1 change, and the point move along the ROC curve. Find the threshold with the best F1, then the lowest threshold with no false alarms.

Practice problems

Work each problem out on paper, then type your answer and press Check. Every problem has hints and a full solution.

Programming exercise

Write the confusion matrix, accuracy, precision, recall, F1, the ROC curve and the area under it, in plain Python. Save metrics.py and test_metrics.py in the same folder, fill in each function in metrics.py, and run the tests:

python test_metrics.py

"""Precision, recall and other metrics: programming exercise. Write the measures of a classifier in plain Python: the four counts of theconfusion matrix, accuracy, precision, recall, F1, and the ROC curve with thearea under it. Run the tests from this folder:     python test_metrics.py actual is a list of the true classes: 1 for positive (such as ill) and 0 fornegative (healthy). scores is a list of the model's chances that each exampleis positive, in the same order, and predicted is a list of 1s and 0s."""  def predict(scores, threshold):    """Return a list with 1 for each score that is at least threshold, and 0 for every other score."""    raise NotImplementedError  def confusion_counts(actual, predicted):    """Return (tp, fp, fn, tn).     tp: positive and predicted positive. fp: negative but predicted positive.    fn: positive but predicted negative. tn: negative and predicted negative.    """    raise NotImplementedError  def accuracy(tp, fp, fn, tn):    """Return the share of all predictions that are right: (tp + tn) / (tp + fp + fn + tn)."""    raise NotImplementedError  def precision(tp, fp):    """Return tp / (tp + fp), the share of positive predictions that are right.     If nothing is predicted positive, tp + fp is 0: return 0.0, as scikit-learn does.    """    raise NotImplementedError  def recall(tp, fn):    """Return tp / (tp + fn), the share of the positives that are found. Return 0.0 if there are no positives."""    raise NotImplementedError  def f1(tp, fp, fn):    """Return the F1 score, 2 * tp / (2 * tp + fp + fn). Return 0.0 if the bottom is 0."""    raise NotImplementedError  def roc_points(actual, scores):    """Return the ROC curve as a list of (false positive rate, true positive rate) pairs.     Start at (0.0, 0.0), where nothing is predicted positive. Then use each    different score as the threshold, from the highest to the lowest, and add    the point it gives. Equal scores are predicted positive together, so they    make one point. The last point is (1.0, 1.0).     The false positive rate is fp / (fp + tn), and the true positive rate is tp / (tp + fn).    """    raise NotImplementedError  def auc(points):    """Return the area under a curve given as (x, y) points in order of x.     Add up one trapezoid for each pair of neighbouring points:    (x2 - x1) * (y1 + y2) / 2.    """    raise NotImplementedError 

Stuck? Count TP, FP, FN and TN first: every other measure is worked out from those four. For the ROC curve, go through the scores from highest to lowest and flag one more group of equal scores at each step. The Solution tab has one way to write each function.

In practice: measuring a classifier

Choose the threshold on the validation set. If you pick the threshold with the best F1 on the test set and then report that F1, the number is too good, because the threshold was fitted to those examples. Choose it on the validation set of Lesson 2, then measure the model once on the test set.

The share of positives changes after launch. A fraud model tested on a sample where 1 payment in 10 is fraud shows a far higher precision than it gets in use, where fraud may be 1 in 1,000. Recall and the false positive rate stay the same, so work out the precision for the real share, as in step 8, and keep measuring it once the model is live.

More than two classes. scikit-learn’s classification_report gives precision, recall and F1 for each class, then two averages. The macro average weights every class equally. The weighted average weights each class by how many examples it has, so a large class doing well can hide a small one doing badly. Read the rows for each class before the averages.

Test your knowledge

1. 01A model is 99% accurate on data where 1% of the examples are positive. What would you check first?Show answer

   The confusion matrix, and the recall on the positive class. A model that always answers no is also 99% accurate on that data, and it finds nothing.

2. 02When would you choose high recall over high precision?Show answer

   When a miss costs more than a false alarm. A screening test for a serious illness should catch nearly every case, and a second test can clear the false alarms. A spam filter is the other way round: hiding a real email costs more than letting a spam through.

3. 03Why is F1 a harmonic mean rather than an ordinary average?Show answer

   The harmonic mean is pulled towards the smaller number, so a model cannot score well by being excellent at one and poor at the other. With precision 1 and recall 0.1, the ordinary average is 0.55 but F1 is 2 × 1 × 0.1 ÷ (1 + 0.1) = 0.18.

4. 04What does an AUC of 0.5 mean, and an AUC of 0.93?Show answer

   AUC is the chance that the model gives a random positive a higher score than a random negative. 0.5 is what guessing gets. 0.93 means 93 of every 100 such pairs are in the right order. It does not depend on the threshold, so it compares models before a threshold is chosen.

5. 05Why can a model with an AUC of 0.95 still be poor at finding a rare disease?Show answer

   Neither of the ROC curve’s rates depends on how rare the disease is. If 1 person in 1,000 has it, a test that catches 90% of cases and flags just 1% of healthy people still flags about 10 healthy people for every person who is ill, so fewer than 1 flag in 10 is right. Judge it by precision and recall on data as rare as the real thing.

Exit ticket

One last question on the main idea of the lesson.