Chapter 30

Points That Read Digits

We know how the reader makes a decision. Each digit supplies a direction and an offset, the image supplies a point, and ten dot products decide which region contains it. But we have been given all the numbers that make this work. Where did they come from?

Suppose we erase them. There are ten lists to fill in, each containing a bias and 64 weights. We could try guessing. Write down 650 numbers, run some images through the resulting reader, and see how it does. If it does badly, change the numbers and try again.

This sounds hopelessly unstructured. But it also sounds like something we have seen before: choosing a point, evaluating a function, and looking for a better point. The first task is to work out what the point and the function should be.

30.1This time the reader is the point

In the first chapter an image became a point of $ℝ 64$ . A list of 64 numbers was all we needed; the fact that the numbers happened to record ink did not change the mathematics.

Now apply the same idea to the reader. Put its ten parameter lists end to end:

Θ = (𝑏 0, 𝑤 0, 1, \dots, 𝑤 0, 64, 𝑏 1, 𝑤 1, 1, \dots, 𝑤 1, 64, \dots, 𝑏 9, 𝑤 9, 1, \dots, 𝑤 9, 64) .

Another point in a high-dimensional space! This time it is a point of $ℝ 650$ , and it represents the machine rather than the picture. Change one coordinate and we change one stored number. Move to another point and we have another reader. Even the reader whose entire list is zero is a point in this space. It gives every image ten zero scores and ten probabilities of $1 / 10$ .

There are now two spaces to keep track of. When we use a reader, we fix $Θ$ and vary the image $⃗ 𝑥$ . When we teach a reader, we fix a collection of images and vary $Θ$ . The dot product is the same calculation in either case. What changes is which list we regard as its input.

Figure 30.1 Hold one training image fixed and change the reader. Then hold the reader fixed and change a pixel. Watch the probabilities in both experiments, and identify which space contains the moving point.

Of course, watching probabilities change does not tell us whether they have improved. We need some answers to compare them with.

30.2What counts as a better reader?

The data include images whose written digits are already known. We call these digits the labels. Our training set consists of 3,823 pairs

(⃗ 𝑥 (1), 𝑦 1), \dots, (⃗ 𝑥 (𝑁), 𝑦 𝑁), 𝑁 = 3823,

where each $⃗ 𝑥 (𝑖)$ is an image vector and each $𝑦 𝑖$ is one of $0, 1, \dots, 9$ . The superscript numbers the images; it is not an exponent.

There is a wonderfully direct way to compare readers: count their mistakes. Feed every training image to the reader and count how often the largest score belongs to the wrong digit. This gives a function on $ℝ 650$ with integer values. Smaller is better.

But imagine a reader that assigns a known three probability $0.01$ . After we change its parameters, that probability becomes $0.40$ . Perhaps a five still has the largest probability, so the answer is still wrong. The mistake count cannot see the improvement. Conversely, a correct answer can become much less certain without changing the count.

We can make this problem particularly visible with the reader we already have. Call its parameter list $Θ ⋆$ , and consider the line

Θ (𝑡) = 𝑡 Θ ⋆ .

Every score is multiplied by $𝑡$ . If $𝑡 > 0$ , this preserves their order: the largest score remains the largest. Every prediction on every image stays the same! But the probabilities involve exponentials, so they do change. Near $𝑡 = 0$ , they are nearly equal. As $𝑡$ grows, the winning digit takes more and more of the probability.

Figure 30.2 These are actual evaluations on the training set along

Θ = 𝑡 Θ ⋆

. All positive values of

𝑡

give the same 75 mistakes. The second graph measures something the first graph cannot see; we will construct its function next.

More generally, a small change of parameters usually leaves all the winning digits alone. Away from ties, the mistake count is locally constant. Its partial derivatives are zero there, even when the reader has much to learn. At a change of winner it can jump. Neither behavior gives us the smooth function we would like to study with calculus.

30.3Ask for probability on the known answer

For a labeled image $(⃗ 𝑥, 𝑦)$ , let $𝑝 𝑦 (⃗ 𝑥; Θ)$ be the probability the reader assigns to its known digit. The semicolon is a reminder of the two roles: the image is fixed training data, while the parameters are what we will change.

A good reader should put a large probability on $𝑦$ . Equivalently, it should make

ℓ (Θ; ⃗ 𝑥, 𝑦) = - l o g 𝑝 𝑦 (⃗ 𝑥; Θ)

small. We call this quantity the loss for the image. Here and throughout the laboratory, $l o g$ means the natural logarithm.

Why this particular function? First, it has the right direction: $- l o g 𝑝$ decreases as $𝑝$ increases, and it approaches zero as $𝑝$ approaches one. Second, it distinguishes degrees of error. Assigning the correct digit probability $1 / 2$ costs about $0.693$ , probability $1 / 10$ costs about $2.303$ , and probability $1 / 100$ costs about $4.605$ . A confident wrong answer is expensive.

There is also a useful way to understand adding these losses. If we multiply the probabilities assigned to all the known answers, then

- l o g (𝑁 \prod 𝑖 = 1 𝑝 𝑦 𝑖 (⃗ 𝑥 (𝑖); Θ)) = 𝑁 \sum 𝑖 = 1 - l o g 𝑝 𝑦 𝑖 (⃗ 𝑥 (𝑖); Θ) .

Thus minimizing the sum of losses is equivalent to maximizing that product. We are asking the reader to account for the whole collection of known answers. One probability close to zero matters, even if many others are good. This identity is algebraic: it does not require us to regard the images as independent random events.

Figure 30.3 Choose a labeled training image and find its contribution on the graph of

- l o g 𝑝

. The table also shows the first four training contributions. The full training loss is the same addition with all 3,823 terms.

We can express the loss directly in terms of the scores. Recall that

𝑝 𝑦 = 𝑒 𝑠 𝑦 \sum 9 𝑘 = 0 𝑒 𝑠 𝑘 .

Taking the negative logarithm gives

ℓ = l o g (9 \sum 𝑘 = 0 𝑒 𝑠 𝑘) - 𝑠 𝑦 .

This is a smooth function of ten scores, and each score is a linear function of the stored parameters. We have turned the reader's answer into a function we can differentiate.

30.4Keeping the numbers in check

There is one more choice to make. Good predictions do not, by themselves, single out sensible parameter sizes.

For example, add the same vector $⃗ 𝑎$ to all ten weight vectors, and the same number $𝑐$ to all ten biases. On a given image every score increases by the same amount $⃗ 𝑎 \cdot ⃗ 𝑥 + 𝑐$ . This common shift cancels out of softmax. The probabilities do not change at all, although the parameter list can become arbitrarily large.

Also, increasing all scores by a common positive scale makes a winning answer more confident. That helps on images the reader gets right and hurts on images it gets wrong. We should not confuse increasing confidence with improving the decisions. Our existing reader makes 75 training mistakes, so this example does not say that scaling it forever improves its loss.

We will add a cost for large parameters:

𝜆 2 ‖ Θ ‖ 2 .

This is the familiar squared distance from the origin in parameter space. It penalizes all 650 entries, including the biases. The positive constant $𝜆$ lets us decide how much that size matters relative to fitting the training data. In this laboratory we use $𝜆 = 1$ .

Our complete objective is therefore

𝐽 (Θ) = 𝑁 \sum 𝑖 = 1 [l o g (9 \sum 𝑘 = 0 𝑒 𝑠 𝑘 (⃗ 𝑥 (𝑖); Θ)) - 𝑠 𝑦 𝑖 (⃗ 𝑥 (𝑖); Θ)] + 𝜆 2 ‖ Θ ‖ 2 .

It is a long expression, but it asks for two understandable things: put probability on the known digits, and pay for large stored numbers. We use a sum, rather than an average, of image losses. Changing to an average while leaving $𝜆$ alone would change the balance between the two terms and produce a different optimization problem.

Figure 30.4 Restrict the full objective to the line

Θ = 𝑡 Θ ⋆

. The blue curve is the training loss, the gold curve is the size penalty, and the rust curve is their sum. Every plotted value comes from the ten-digit reader and the complete training set. A curve along one line is something we can draw, even when the surrounding space has 650 dimensions.

The lowest point on this particular line is near $𝑡 = 1$ , because $Θ ⋆$ is the reader obtained by approximately minimizing this objective. That is a useful check on the picture, but it would not be a method for finding $Θ ⋆$ : we used its coordinates to choose the line! Starting with a zero reader, we must discover which directions to move in.

30.5What we will teach, and what we will test

The training images define $𝐽$ . They will determine every step we take through parameter space. Separately, the data provide 1,797 test images. We keep these out of the objective and use them to ask how the completed reader handles examples it was not taught on.

This distinction is easy to lose when we have a drawing pad that responds immediately. A reader can improve on the examples used to choose its numbers without improving by the same amount on new handwriting. Training performance tells us how well we solved the fitting problem. Test performance asks a different question about the resulting reader.

We have now supplied a precise meaning for “find a good reader”: find a point where $𝐽$ is small. This is a multivariable optimization problem. Its dimension is unusual; its purpose is unusual; the calculus question is familiar. How do we find a minimum?

30.6Things to try

Compute the loss when the reader assigns the known digit probabilities $0.9$ , $0.5$ , and $0.01$ . Which change in probability produces the larger change in loss?
Show directly that adding the same constant to all ten scores leaves each softmax probability unchanged. Explain why this differs from multiplying all scores by a positive constant.
Suppose we replace the sum of losses by their average. What coefficient on $‖ Θ ‖ 2 / 2$ gives an objective with exactly the same minimizer as the one above?