The shell, git and a debugger
The tools engineers use every day: the shell to run programs and move between folders, git to save every version of your work, a debugger to stop a program and look inside it, and a profiler to find what is slow. Every lesson from here on uses them.
Warm-up
One question before the lesson. Choose an answer and check it.
Your program prints the wrong answer. Which is usually the quickest way to find the bug?
Show the answer
B: Stop the program where the value goes wrong, and look at every value there. A debugger stops your program on any line you choose and shows every value at that moment, so you can see which number is wrong instead of guessing. This lesson covers the debugger, and the shell, git and a profiler around it.
Step 1 The shell
A shell is a program that runs other programs. You type a command, press Enter, and it prints the result. Engineers use it to run code, install tools, look at files and work on servers. In this course every command shown after a $ is typed into a shell. The $ is the prompt, which the shell prints, so you don’t type it.
On macOS and Linux, open the Terminal app. On Windows, install WSL (Windows Subsystem for Linux), which gives you a real Linux shell: open PowerShell as administrator, run wsl --install, and restart. The course’s commands are written for that shell.
Here is a first session. Each line after a $ is a command, and the lines under it are what it printed:
$ pwd
/home/you
$ ls
course
$ cd course
$ ls
data line.py
$ mkdir notes
$ ls
data line.py notes
| Command | What it does | What it printed |
|---|---|---|
| pwd | Print the folder you are in. | /home/you |
| ls | List what is in a folder. | course |
| cd course | Change to another folder. | nothing |
| mkdir notes | Make a new folder. | nothing |
Many commands print nothing when they work, as cd and mkdir did. The shell speaks up when there is something to show, or when something went wrong.
Try it yourself
A practice shell with a small set of folders. Type the commands from the lesson and work through the tasks. Nothing you type here touches your computer.
- Print the folder you are in:
pwd - Go into the course folder:
cd course - List what is in it:
ls - Make a folder called notes inside course:
mkdir notes - Make an empty file called todo.txt inside notes:
touch notes/todo.txt - Go back to your home folder:
cd ~
0 of 6 done. Type help to see every command this shell knows.
Practice problems
Work each problem out on paper, then type your answer and press Check. Every problem has hints and a full solution.
Score: 0 of 10 points
Problem 1
1 pointYou are in /home/you/course/data. How many times must you type cd .. to reach /home?
Hint 1
Each cd .. moves up one folder: to the folder that holds the one you are in.
Solution
cd .. (1) → /home/you/course
cd .. (2) → /home/you
cd .. (3) → /home
Problem 2
2 pointsYou make 3 commits on main. Then you make a branch called experiment, switch to it, and make 2 more commits. How many commits does git log list on experiment?
Hint 1
A new branch starts from the commit you were on, with all the history before it.
Solution
experiment starts with main’s 3 commits
then adds 2: 3 + 2 = 5
Problem 3
2 pointsA function should return the mean of a list, but its last line is return total / (len(xs) − 1). At a breakpoint, p total prints 15 and p len(xs) prints 5. What does the function return?
Hint 1
Work out what the code divides by: len(xs) − 1 = 5 − 1.
Solution
len(xs) − 1 = 5 − 1 = 4
15 ÷ 4 = 3.75
Problem 4
2 pointsA profile of a 4-second training run shows that the loss function takes 3.2 seconds of it. What share of the run is spent in loss? Answer as a share such as 0.5, or a percentage such as 50%.
Hint 1
share = time in loss ÷ time for the whole run.
Solution
3.2 ÷ 4 = 0.8, or 80%
Problem 5
3 pointsYou rewrite loss with NumPy and it becomes 20 times faster. Nothing else changes. How many seconds does the whole run take now?
Hint 1
Only loss gets faster: 3.2 seconds becomes 3.2 ÷ 20.
Hint 2
The other 0.8 seconds stay as they were.
Solution
rest of the run = 4 − 3.2 = 0.8 seconds
new loss = 3.2 ÷ 20 = 0.16 seconds
new run = 0.8 + 0.16 = 0.96 seconds
Programming exercise
metrics.py has three functions, and each has one bug. Run the tests to see which fail, then stop inside each failing function with breakpoint(), look at the values, and fix it. Save metrics.py and test_metrics.py in the same folder and run:
python test_metrics.py
"""The shell, git and a debugger: programming exercise. Each function below has one bug. Run the tests from this folder: python test_metrics.py For each test that fails, put breakpoint() at the top of that function, runthe tests again, and use the debugger to find the line that goes wrong:n runs the next line, p followed by a name prints it, c carries on.Fix the bug, delete the breakpoint() line, and run the tests again.""" def mean(xs): """Return the mean of xs: their sum divided by how many there are.""" total = sum(xs) return total / (len(xs) - 1) def mse(ys, preds): """Return the mean squared error: the average of (pred - y) squared, over every pair.""" total = 0.0 for i in range(1, len(ys)): total += (preds[i] - ys[i]) ** 2 return total / len(ys) def accuracy(labels, preds): """Return the share of predictions that match their labels.""" right = sum(1 for y, p in zip(labels, preds) if y != p) return right / len(labels) Stuck? Put breakpoint() on the first line of a function, run the tests, then type n to go one line at a time and p followed by a name to print it. The Solution tab shows each fix.
In practice: tools at work
Commit small and often, with messages that say why. A commit that does one thing is easy to review and easy to undo with git revert, which adds a new commit that reverses it. Before every commit, read git diff to check exactly what is going in.
Never commit secrets. An API key or password committed to a repository stays in its history even after you delete the file. Keep secrets in environment variables or in a .env file listed in .gitignore, the file that tells git what to leave out.
Use the debugger before adding print statements. Editors such as VS Code have one built in: click in the margin to set a breakpoint, run, then step through and hover over any variable to see its value. Before a long training run, stop inside the first step this way and check the shape and first few values of every array.
Test your knowledge
01You are in /home/you/course. You type cd data, then cd .. . Where are you now?Show answer
Back in /home/you/course. cd data moves into the data folder inside it, and .. always means one folder up.
02What is the difference between git add and git commit?Show answer
git add chooses which changes go into the next commit, which is called staging them. git commit saves the staged changes as a new version in the repository’s history, with a message and an id.
03Your program has stopped at breakpoint(). How do you see the value of a variable called loss, run one more line, and then let the program carry on?Show answer
Type p loss to print it, n to run the next line, and c to continue until the program ends or reaches another breakpoint.
04A training script is slow. How do you find which function takes the time?Show answer
Run it under the profiler: python -m cProfile -s cumulative train.py. It lists every function with how many times it was called and how long it took, the slowest first.
Exit ticket
One last question on the main idea of the lesson.
You have changed three files, but only want to save one of them in your next commit. What do you do?
Show the answer
A: git add that one file, then git commit. git add stages just the changes you name, and git commit saves only what is staged. The other two files stay changed on your disk, ready for a later commit. git restore would throw their changes away.