Matrices and the matrix product

A matrix is a grid of numbers. Learn to multiply a matrix by a vector and by another matrix, why the shapes must match, and why the order matters. Every layer of a neural network is a matrix product.

Lesson 6 stepsPractice 5 problemsExercise Python, 5 functionsQuiz 4 questions

Warm-up

One question before the lesson. Choose an answer and check it.

With ordinary numbers, 2 × 3 = 3 × 2. For two matrices A and B, is AB always the same as BA?

No. AB and BA can be different, and can even have different shapes.Yes. Multiplying never depends on the order.Only when both matrices are square.

Show the answer

A: No. AB and BA can be different, and can even have different shapes. In this lesson A is 2 × 3 and B is 3 × 2, so AB is 2 × 2 and BA is 3 × 3: not even the same shape. Even square matrices usually give a different AB and BA. With matrices, the order matters.

Step 1 What a matrix is

01/06

A matrix is a grid of numbers arranged in rows (across) and columns (down). This one, called A, has 2 rows and 3 columns, so its shape is 2 × 3 (say “2 by 3”). Rows always come first.

A = [\begin{matrix} 1 & 2 & 0 \\ 3 & −1 & 2 \end{matrix}]

Symbol	Say	Means	In this lesson
$A$	A	A matrix: a grid of numbers. Matrices get capital letters, vectors small ones.	2 × 3
$a_{i j}$	a i j	The entry of A in row i and column j.	a₂₃ = 2
$m \times n$	m by n	A shape: m rows and n columns. Rows always come first.	2 × 3

Each number is an entry, named by its row and its column: aᵢⱼ is the entry in row i and column j. Here is every entry of A:

Row i	Column j	Entry aᵢⱼ
1	1	a₁₁ = 1
1	2	a₁₂ = 2
1	3	a₁₃ = 0
2	1	a₂₁ = 3
2	2	a₂₂ = −1
2	3	a₂₃ = 2

A vector is a matrix with a single column, so everything in this lesson works on vectors too.

Try it yourself

Change the four numbers of the matrix, or pick one of the examples, and drag the tip of x. The matrix moves the arrow and the square, and Mx is worked out row by row.

x Mx the square, moved by M

M =

x = (3, 4)

row 1 · x = 0 × 3 + (−1) × 4 = −4

row 2 · x = 1 × 3 + 0 × 4 = 3

Mx = (−4, 3)

The shaded square has corners (0, 0), (1, 0), (1, 1) and (0, 1). Its outline after M shows what M does to every arrow at once: the first column of M is where (1, 0) lands, and the second is where (0, 1) lands.

Practice problems

Work each problem out on paper, then type your answer and press Check. Every problem has hints and a full solution.

Score: 0 of 10 points

Problem 1
1 point
A matrix has rows (2, 0, 1), (4, 3, 5) and (7, 6, 8). What is a₂₃?
Hint 1
aᵢⱼ is the entry in row i and column j. The row comes first.
Solution
row 2 is (4, 3, 5), and its 3rd entry is 5
Problem 2
2 points
M has rows (1, 2) and (3, 4), and x = (3, −1). What is the second number of Mx?
Hint 1
Each number of Mx is one row of M dotted with x.
Hint 2
The second number comes from row 2, (3, 4).
Solution
row 2 · x = 3 × 3 + 4 × (−1)
= 9 − 4 = 5
Problem 3
2 points
A is 3 × 4 and B is 4 × 2. How many entries does AB have?
Hint 1
AB has as many rows as A and as many columns as B.
Solution
AB is 3 × 2
entries = 3 × 2 = 6
Problem 4
2 points
How many multiplications does it take to work out that AB?
Hint 1
Each entry of AB is a dot product of 4 pairs.
Hint 2
multiplications = rows of A × columns of B × columns of A.
Solution
entries × pairs per entry = 6 × 4
= 24
Problem 5
3 points
P has rows (1, 2) and (0, 3), and Q has rows (4, 1) and (2, 6). What is the entry in row 1, column 2 of PQ?
Hint 1
Entry (1, 2) of PQ is row 1 of P dotted with column 2 of Q.
Hint 2
Column 2 of Q is its second number in each row: (1, 6).
Solution
row 1 of P = (1, 2), column 2 of Q = (1, 6)
(PQ)₁₂ = 1 × 1 + 2 × 6 = 13

Programming exercise

Write matrix operations in plain Python, with a matrix as a list of rows. Save matrices.py and test_matrices.py in the same folder, fill in each function in matrices.py, and run the tests:

python test_matrices.py

"""Matrices and the matrix product: programming exercise. Write matrix operations in plain Python, with a matrix as a list of rows:[[1, 2, 0], [3, -1, 2]] is the 2 x 3 matrix A from the lesson. Then run thetests from this folder:     python test_matrices.py"""  def shape(A):    """Return (rows, columns) of A, as a tuple."""    raise NotImplementedError  def transpose(A):    """Return A with its rows turned into columns: row i of the answer is column i of A."""    raise NotImplementedError  def mat_vec(A, x):    """Return A times the vector x: the dot product of each row of A with x, as a list."""    raise NotImplementedError  def mat_mul(A, B):    """Return the matrix product AB.     The entry in row i, column j is the dot product of row i of A with column    j of B. If A's columns don't match B's rows, raise ValueError.    """    raise NotImplementedError  def identity(n):    """Return the n x n identity matrix: 1 on the diagonal from top left to bottom right, 0 elsewhere.     It leaves any vector as it is: identity(n) times x is x.    """    raise NotImplementedError

Stuck? Every entry of a product is a dot product, so the Vectors lesson’s dot function does most of the work. The Solution tab has one way to write it.

In practice: matrices in AI

A neural network layer is a matrix times a vector, plus a bias: y = Wx + b. W has one row per output and one column per input, so a layer from 4,096 numbers to 4,096 numbers is a 4,096 × 4,096 matrix. Module 7 builds layers like this.

Many inputs at once. Put many input vectors side by side as the columns of one matrix X, and the single matrix product WX computes the layer for all of them. This is what training in batches means, and it keeps a GPU busy.

Check the shapes first. Most bugs in this kind of code are shapes that do not match. NumPy multiplies matrices with the @ sign, and when the shapes are wrong it stops with an error that ends “(size 2 is different from 3)”. Print the shape of each matrix before anything else.

Test your knowledge

01A has rows (1, 2), (3, 4) and (5, 6). What is its shape, and what is a₃₂?Show answer
It has 3 rows and 2 columns, so its shape is 3 × 2. a₃₂ is the entry in row 3, column 2: 6.
02What is the matrix with rows (1, 2) and (3, 4), times the vector (1, 1)?Show answer
Row 1: 1 × 1 + 2 × 1 = 3. Row 2: 3 × 1 + 4 × 1 = 7. The answer is (3, 7).
03Can a 2 × 3 matrix be multiplied by another 2 × 3 matrix?Show answer
No. In AB, the number of columns of A must equal the number of rows of B, and here those are 3 and 2. A 2 × 3 matrix can be multiplied by any matrix with 3 rows.
04How many multiplications does a 100 × 200 matrix times a 200 × 50 matrix take?Show answer
The answer is 100 × 50, so it has 5,000 entries. Each is a dot product of 200 pairs. That is 5,000 × 200 = 1,000,000 multiplications.

Exit ticket

One last question on the main idea of the lesson.

What is each entry of a matrix product AB?

A row of A times the same row of B, entry by entry.An entry of A times the entry of B in the same place.The dot product of a row of A with a column of B.

Show the answer

C: The dot product of a row of A with a column of B. Entry (i, j) of AB is row i of A dotted with column j of B. That is why the rows of A and the columns of B must be the same length.