Matrices and the matrix product
A matrix is a grid of numbers. Learn to multiply a matrix by a vector and by another matrix, why the shapes must match, and why the order matters. Every layer of a neural network is a matrix product.
Step 1 What a matrix is
A matrix is a grid of numbers arranged in rows (across) and columns (down). This one, called A, has 2 rows and 3 columns, so its shape is 2 × 3 (say “2 by 3”). Rows always come first.
| Symbol | Say | Means | In this lesson |
|---|---|---|---|
| A | A matrix: a grid of numbers. Matrices get capital letters, vectors small ones. | 2 × 3 | |
| a i j | The entry of A in row i and column j. | a₂₃ = 2 | |
| m by n | A shape: m rows and n columns. Rows always come first. | 2 × 3 |
Each number is an entry, named by its row and its column: aᵢⱼ is the entry in row i and column j. Here is every entry of A:
| Row i | Column j | Entry aᵢⱼ |
|---|---|---|
| 1 | 1 | a₁₁ = 1 |
| 1 | 2 | a₁₂ = 2 |
| 1 | 3 | a₁₃ = 0 |
| 2 | 1 | a₂₁ = 3 |
| 2 | 2 | a₂₂ = −1 |
| 2 | 3 | a₂₃ = 2 |
A vector is a matrix with a single column, so everything in this lesson works on vectors too.
Try it yourself
Change the four numbers of the matrix, or pick one of the examples, and drag the tip of x. The matrix moves the arrow and the square, and Mx is worked out row by row.
x Mx the square, moved by M
x = (3, 4)
row 1 · x = 0 × 3 + (−1) × 4 = −4
row 2 · x = 1 × 3 + 0 × 4 = 3
Mx = (−4, 3)
The shaded square has corners (0, 0), (1, 0), (1, 1) and (0, 1). Its outline after M shows what M does to every arrow at once: the first column of M is where (1, 0) lands, and the second is where (0, 1) lands.
Programming exercise
Write matrix operations in plain Python, with a matrix as a list of rows. Save matrices.py and test_matrices.py in the same folder, fill in each function in matrices.py, and run the tests:
python test_matrices.py
"""Matrices and the matrix product: programming exercise. Write matrix operations in plain Python, with a matrix as a list of rows:[[1, 2, 0], [3, -1, 2]] is the 2 x 3 matrix A from the lesson. Then run thetests from this folder: python test_matrices.py""" def shape(A): """Return (rows, columns) of A, as a tuple.""" raise NotImplementedError def transpose(A): """Return A with its rows turned into columns: row i of the answer is column i of A.""" raise NotImplementedError def mat_vec(A, x): """Return A times the vector x: the dot product of each row of A with x, as a list.""" raise NotImplementedError def mat_mul(A, B): """Return the matrix product AB. The entry in row i, column j is the dot product of row i of A with column j of B. If A's columns don't match B's rows, raise ValueError. """ raise NotImplementedError def identity(n): """Return the n x n identity matrix: 1 on the diagonal from top left to bottom right, 0 elsewhere. It leaves any vector as it is: identity(n) times x is x. """ raise NotImplementedError Stuck? Every entry of a product is a dot product, so the Vectors lesson’s dot function does most of the work. The Solution tab has one way to write it.
In practice: matrices in AI
A neural network layer is a matrix times a vector, plus a bias: y = Wx + b. W has one row per output and one column per input, so a layer from 4,096 numbers to 4,096 numbers is a 4,096 × 4,096 matrix. Module 2 builds layers like this.
Many inputs at once. Put many input vectors side by side as the columns of one matrix X, and the single matrix product WX computes the layer for all of them. This is what training in batches means, and it keeps a GPU busy.
Check the shapes first. Most bugs in this kind of code are shapes that do not match. NumPy multiplies matrices with the @ sign, and when the shapes are wrong it stops with an error that ends “(size 2 is different from 3)”. Print the shape of each matrix before anything else.
Test your knowledge
01A has rows (1, 2), (3, 4) and (5, 6). What is its shape, and what is a₃₂?Show answer
It has 3 rows and 2 columns, so its shape is 3 × 2. a₃₂ is the entry in row 3, column 2: 6.
02What is the matrix with rows (1, 2) and (3, 4), times the vector (1, 1)?Show answer
Row 1: 1 × 1 + 2 × 1 = 3. Row 2: 3 × 1 + 4 × 1 = 7. The answer is (3, 7).
03Can a 2 × 3 matrix be multiplied by another 2 × 3 matrix?Show answer
No. In AB, the number of columns of A must equal the number of rows of B, and here those are 3 and 2. A 2 × 3 matrix can be multiplied by any matrix with 3 rows.
04How many multiplications does a 100 × 200 matrix times a 200 × 50 matrix take?Show answer
The answer is 100 × 50, so it has 5,000 entries. Each is a dot product of 200 pairs. That is 5,000 × 200 = 1,000,000 multiplications.