Eigenvectors, the SVD and PCA

Warm-up

One question before the lesson. Choose an answer and check it.

Try it yourself

Turn a direction through the students’ marks and watch the spread along it change, with every student’s position worked out. Find the direction with the most spread. Then mark physics out of 10, or standardise both tests, and see the first component move.

Practice problems

Work each problem out on paper, then type your answer and press Check. Every problem has hints and a full solution.

Programming exercise

Write PCA for two inputs in plain Python: the covariance matrix, its eigenvalues and eigenvectors, the scores and the rebuilt points, and power iteration. Save pca.py and test_pca.py in the same folder, fill in each function in pca.py, and run the tests:

python test_pca.py

"""Eigenvectors, the SVD and PCA: programming exercise. Write principal component analysis for data with two inputs in plainPython: centre the data, build its covariance matrix, find the matrix'seigenvalues and eigenvectors, score each point along a direction and rebuildit from its score, and find the first component a second way, by poweriteration. Run the tests from this folder:     python test_pca.py A point is a list of two numbers, such as [maths mark, physics mark]. A2 x 2 matrix is a list of two rows, such as [[134, 90], [90, 81.5]]. Adirection is a list of two numbers whose squares add up to 1, such as[0.8, 0.6].""" import math  # noqa: F401  (you will need math.sqrt)  def mean_centre(points):    """Return (means, centred): the mean of each input, and every point with the means taken away."""    raise NotImplementedError  def covariance(centred):    """Return the covariance matrix of centred points, dividing by n - 1.     It is [[s00, s01], [s01, s11]], where s00 adds up p[0] * p[0] over the    points, s01 adds up p[0] * p[1] and s11 adds up p[1] * p[1], each divided    by n - 1.    """    raise NotImplementedError  def variance_along(cov, u):    """Return the spread along the direction u: u[0]**2 * cov[0][0] + 2 * u[0] * u[1] * cov[0][1] + u[1]**2 * cov[1][1]."""    raise NotImplementedError  def eigen2(cov):    """Return ((l1, l2), (v1, v2)): a 2 x 2 covariance matrix's eigenvalues, largest first, and their eigenvectors.     With a = cov[0][0], b = cov[0][1] and d = cov[1][1], the eigenvalues are    (a + d) / 2 plus or minus sqrt(((a - d) / 2)**2 + b**2). The eigenvector    for an eigenvalue l is [b, l - a] divided by its length. If its first part    is negative, multiply both parts by -1. You may assume b is not 0.    """    raise NotImplementedError  def project(centred, v):    """Return each centred point's score along the direction v: v[0] * p[0] + v[1] * p[1]."""    raise NotImplementedError  def rebuild(means, scores, v):    """Return each point rebuilt from its score alone: [means[0] + score * v[0], means[1] + score * v[1]]."""    raise NotImplementedError  def explained(values):    """Return each eigenvalue's share of their total."""    raise NotImplementedError  def power_iteration(cov, start, steps):    """Return the direction power iteration reaches.     Start from start divided by its length. Then, steps times, multiply the    direction by cov and divide the result by its length.    """    raise NotImplementedError 

Stuck? Write mean_centre and covariance first. eigen2 uses the formula from step 4, with the eigenvector (Σ₁₂, λ − Σ₁₁) divided by its length. project, rebuild and explained take a line or two each. The Solution tab has one way to write each function.

In practice: PCA in real projects

Scale the inputs first. PCA follows the inputs with the biggest numbers, so inputs in different units, such as kilograms and centimetres, should be standardised first, as in Lesson 6. scikit-learn’s StandardScaler does this.

scikit-learn. PCA(n_components=k) centres the data itself, but does not scale it. It finds the components with the SVD or, when there are many more rows than inputs, from the covariance matrix. components_ holds the components, one per row, and explained_variance_ratio_ each one’s share of the spread. PCA(n_components=0.95, svd_solver="full") keeps as many components as it takes to hold 95% of the spread. Fit it on the training data only, then apply the same components to the validation and test data.

Signs. If v is an eigenvector, so is −v: the spread along both is the same. Two libraries, or two versions of one, may give the same component with opposite signs, so compare components up to their sign.

Uses. PCA squeezes many inputs into a few before training another model, which can make it faster and less likely to overfit. It plots data with many inputs in two dimensions, and it removes noise by rebuilding from the first few components. It does not look at the labels, so the direction with the most spread need not be the one that best separates the classes.

Test your knowledge

1. 01What does PCA do?Show answer

   It finds new directions for the data, the principal components, in order of how much the data spreads along each. They are the eigenvectors of the covariance matrix, and each eigenvalue is the spread along its component. Keeping the first few components describes each point with fewer numbers and keeps most of the spread.

2. 02What are an eigenvector and an eigenvalue?Show answer

   An eigenvector of a matrix is a direction the matrix does not turn: multiplying by the matrix only stretches it. The number it is stretched by is its eigenvalue, so Av = λv. A symmetric matrix, such as a covariance matrix, has eigenvectors at right angles to each other.

3. 03Why standardise the inputs before PCA?Show answer

   PCA finds the directions with the most spread, and spread depends on units. An input measured in big numbers spreads more and takes over the first component, whether or not it says more. Standardising gives every input a standard deviation of 1, so each counts the same.

4. 04How do you choose how many components to keep?Show answer

   Add up the components’ shares of the spread, largest first, until they reach a target such as 90% or 95%. Or look for an elbow, where the next component adds little. If the components feed another model, keep the number that gives the best validation score.

5. 05How is PCA related to the SVD?Show answer

   Write the centred data as X = USVᵀ. The columns of V are the principal components, each eigenvalue is a singular value squared divided by n − 1, and US holds the scores. Libraries mostly compute PCA this way, because building the covariance matrix multiplies the data by itself, which loses precision when some directions have far less spread than others.

Exit ticket

One last question on the main idea of the lesson.