Taking Apart a Digit Reader

28.1What Goes into the Function?

From the outside, the reader takes a picture and produces probabilities. This sounds like a function! For example, we could ask just for the probability it assigns to 3:

The output is a real number. But what is the input? We know how to give a function numbers; how do we give it handwriting?

Look closely at one of the supplied images. It is a square array of tiny pixels, with 32 rows and 32 columns. Each pixel either contains ink or it doesn't. We can record these possibilities with 1 and 0. Already, the picture has become a list of 32 ⋅32 =1024 numbers.

Our reader uses a smaller list. Divide the picture into blocks, each containing 4 rows and 4 columns of pixels. In each block, count how many of its 16 pixels contain ink. An empty block has count 0, a completely filled block has count 16, and a partly filled block has a count somewhere in between.

There are 32/4 =8 blocks in each row and 8 rows of blocks. Thus we obtain an 8 ×8 array of 64 counts. Finally, divide each count by 16, so every measurement lies between 0 and 1:

Here 0 means no ink and 1 means a full block of ink. If a block contains 12 ink pixels, its coordinate is 12/16 =0.75. We can display that number as a shade of gray, which is why the smaller image still looks like a picture. But the numbers are what the reader receives.

This measurement forgets something. Suppose two ink pixels move to different positions within the same block. The original picture changes, but the count is still 2, and the corresponding coordinate is still 0.125. If every block keeps the same count, the reader receives exactly the same input. It has no way to recover the arrangements we discarded.

For a freehand drawing there is one earlier step: the pad centers and resizes the strokes and adjusts their thickness to make a bitmap like the supplied examples. Once that bitmap is made, the block counts are calculated in exactly the same way. From here on, our mathematical input is the resulting list of 64 measurements.

28.2One Image Is One Point

How many dimensions does it take to describe this input?

It is tempting to say two: the image has a horizontal direction and a vertical direction. But those two directions locate a pixel within an image. We want to describe the entire image. This is the same distinction we encountered when a whole can became a point in the space of possible cans: the coordinates describe the object we are choosing.

Let's make a very small picture with just two pixels. We can specify it by giving the amount of ink in its first pixel and the amount in its second:

The picture with ink amounts 0.25 and 0.75 is the point (0.25,0.75). Increasing the ink in the first pixel moves this point in the 𝑥1 direction. Increasing the ink in the second moves it in the 𝑥2 direction. Each point of the square [0,1] ×[0,1] describes a different two-pixel picture.

Now add a third pixel. We need a third coordinate, and the square becomes a cube. Add a fourth pixel, and we need a fourth coordinate. Nothing about the description stops working when the number of pixels exceeds three; we simply run out of room to draw its coordinate axes on the page!

For our actual images, read the top row from left to right to obtain 𝑥1,…,𝑥8. The next row gives 𝑥9,…,𝑥16, and so on. One complete image is represented by the point

The possible brightness lists fill the 64-dimensional cube [0,1]64. Our block-count images occupy particular points in this cube: each coordinate is a multiple of 1/16. Most points in the cube do not resemble handwritten digits at all, but the scoring formulas will still be able to act on them.

Notice that writing the array as a list discards no additional information. If I give you the list and tell you the row order, you can put every entry back where it belongs. The loss of information happened when we replaced each 4 ×4 block by its count, not when we wrote the counts in a different arrangement.

We can now state the input to our function using familiar mathematical language. The reader's probability of three is a function of 64 variables:

It is a function of many variables because changing any one of these pixel measurements can affect the answer. This is a rather concrete reason to care about functions on spaces we cannot draw.

28.3One Input, Ten Scores

Let's follow our point a little farther into the machine.

The reader first calculates ten numbers, one for each possible digit. These are called scores, and we will write them as

For now, leave the calculation of each score inside its box. What matters is that the same image goes into all ten calculations. The score for 3 measures how strongly its calculation favors that answer; the score for 5 comes from another calculation, and so on. The reader chooses the digit with the largest score.

Scores are ordinary real numbers. They can be positive or negative, and they need not add to anything in particular. So they cannot yet be the probabilities we saw in the opening display. We need one more operation.

First take the exponential of each score. This makes all ten numbers positive, and preserves their order: a larger score gives a larger exponential. Then divide each exponential by their total:

Each fraction is positive, and the ten fractions add to 1, because adding their numerators gives exactly the denominator. The largest score still gives the largest fraction. This operation is called softmax.

To see the arithmetic without a long list of decimals, imagine an image with score 2 for three and score 0 for every other digit. The probability of three would be

Each of the other digits receives

Three is the clear winner among the ten answers, but the other nine possibilities together receive more probability than three does. That explains how our hesitant three can win with only 37.36%: it beats each competitor individually. It does not have to beat their combined total.

There is another useful experiment in the figure. Add the same number 𝑐 to every score. This changes their values but leaves every probability unchanged:

The common factor cancels. Thus a score's absolute size cannot by itself tell us a probability. The scores must be compared with one another. In particular, the probability assigned to three can change even if its own score stays fixed: changing the other scores changes the shared denominator.

We have now taken apart the reader into two stages:

The complete probability output is the list 𝑃(𝑥) =(𝑃0(𝑥),…,𝑃9(𝑥)). Each entry is a scalar-valued function on image space. The displayed digit is obtained by choosing the largest entry.

28.4What Is Stored Inside?

Let's open the calculation for three. Inside, we find a long list of numbers. Here is how the beginning of that list might look in Python, with the actual stored numbers rounded to six decimal places:

weights = [0.000000, 0.331608, 0.286236, 0.073579, ...]
bias = 0.013382

The dots stand for the rest of the list: there are 64 entries altogether, one for each coordinate of the image. These numbers are called weights. The bias is one additional number, stored separately here. The image is also a list of 64 numbers, so we can line up their entries:

What happens next is remarkably simple. Multiply each image entry by the weight underneath it, add all 64 products, and then add the bias:

In Python, the whole calculation can be written as

score = sum(xi * wi for xi, wi in zip(image, weights)) + bias

Here zip pairs the first image entry with the first weight, the second with the second, and so on. The expression xi * wi multiplies one pair, and sum adds the resulting products. There is a loop hiding in that line, but the arithmetic is just multiplication and addition.

Do you recognize the sum? It is a dot product! If we collect the weights for three into the vector ⃗𝑤3, the score is

where ⃗𝑥 is the position vector of our image point. So this part of the reader consists of a stored vector, a dot product, and one final addition.

It is worth watching these operations happen to the actual image. Put its 64 entries back on their grid, and let each entry be the height of a bar. Then arrange the 64 weights on a matching grid. A positive weight rises above the grid; a negative weight extends below it.

Multiplication rescales each weight's bar by the corresponding image value. If the image value is 0, that bar becomes flat. If it is 1, the bar keeps its height. If it is 1/4, the bar shrinks to a quarter of its height. The result is another grid of bars, now with heights 𝑥𝑖𝑤3,𝑖.

Adding the bars means adding their signed heights. Bars above zero increase the total; bars below zero decrease it. For example, an image entry of 0.75 paired with weight −2 contributes −1.5 to the sum. The other 63 entries make their own contributions, and the bias is added at the end. The number we obtain is exactly the score shown earlier in the reader.

The height plots are a way to inspect the entries of our two lists. Their horizontal grid records which pixel an entry belongs to; their vertical height records its value. We have rearranged the numbers to make the arithmetic visible, just as we earlier rearranged the image's coordinates into a list.

Now do the same thing for each digit. Zero has a stored vector ⃗𝑤0 and bias 𝑏0, one has ⃗𝑤1 and 𝑏1, and so on through nine. Feed the same image to every one of them:

Ten dot products and ten biases give our ten scores. Those scores go through the exponential normalization we already investigated, and out come the ten probabilities. The stored part of the whole reader consists of

numbers.

There are two different kinds of numbers in play. When you draw a new image, its coordinates change, so the products, sums, and probabilities change. The stored vectors and biases stay fixed. Changing those stored numbers would change the reader itself: it would change how the machine responds to images. Later, we will choose those numbers ourselves using labeled examples.

But first there is a more immediate question. We now know what the reader computes. Why should a dot product with a particular vector tell us anything about whether an image looks like a three? How can a sum of pixel values, scaled by some stored numbers, distinguish one kind of handwriting from another?

That is the question for the next chapter. We can carry out the arithmetic; now we need to understand its geometry.

The supplied handwriting comes from E. Alpaydin and C. Kaynak's Optical Recognition of Handwritten Digits dataset, distributed through the UCI Machine Learning Repository under CC BY 4.0. The examples used here come from its held-out test set; they were not used to fit the reader.