Chapter 11

Partial Derivatives

How do we differentiate a function with multiple inputs? We will learn several ways to do this throughout the course. But all methods rely on one fundamental idea: the partial derivative.

11.1Partial Derivatives

If we take a function $𝑓 (𝑥, 𝑦)$ and hold the $𝑦$ value constant, we get a function of just $𝑥$ . For example, if $𝑓 (𝑥, 𝑦) = s i n (𝑥 𝑦)$ and $𝑦 = 2$ we get the function $𝑓 (𝑥, 2) = s i n (2 𝑥)$ . This sort of function we already know how to take the derivative of!

𝑓 (𝑥, 2)' = 𝑑 𝑑 𝑥 𝑓 (𝑥, 2) = 2 c o s (2 𝑥) .

What does this derivative mean? Well, we are measuring the slope in the $𝑥$ -direction along the line where $𝑦 = 2$ .

Figure 11.1 Read it from the bottom up. The plane under the graph is the domain, carrying the point and the line through it along which one variable is held fixed. The sheet of glass standing on that line cuts the surface, and the curve it cuts is drawn flat beside it. The partial derivative is the slope of that curve—an ordinary derivative of an ordinary graph, and the whole content of

𝜕 𝑓 / 𝜕 𝑥

. The point sweeps along the line by itself so the slope can be watched changing; take hold of it to put it where you like, along the line or on a different one.

What happens if we instead look at the slice where $𝑦 = 7$ ? Then we'd have $𝑓 (𝑥, 7) = s i n (7 𝑥)$ and the derivative would be

𝑓 (𝑥, 7)' = 𝑑 𝑑 𝑥 𝑓 (𝑥, 7) = 7 c o s (7 𝑥) .

Thus, whatever $𝑦$ is, we see that it appears as a coefficient via the chain rule. If we call the constant $𝑦$ without specifying its numerical value, we get

𝑓 (𝑥, 𝑦)' = 𝑑 𝑑 𝑥 𝑓 (𝑥, 𝑦) = 𝑦 c o s (𝑥 𝑦) .

The only thing confusing here is that unless we know what we are doing, it's hard to tell what the prime means. So we should probably not use this notation when there's more than one variable.

In fact, to signify that we are taking the derivative of a multivariable function, it is customary to write the $𝑑$ a little fancy as well, using the italic $𝜕$ .

Definition 11.1 ($x$-Partial Derivative). If $𝑓 (𝑥, 𝑦, 𝑧, \dots)$ is a function of multiple variables, the partial derivative with respect to $𝑥$ is the result of treating all other variables as constants and differentiating with respect to $𝑥$ . It is denoted

𝜕 𝑓 𝜕 𝑥 = l i m ℎ \to 0 𝑓 (𝑥 + ℎ, 𝑦, 𝑧, \dots) - 𝑓 (𝑥, 𝑦, 𝑧, \dots) ℎ

A partial derivative is therefore an ordinary one-variable derivative of a slice. Computationally, we hold every other input fixed and use the familiar rules of single-variable calculus.

Partial derivatives are ubiquitous in the sciences, and because they are used in so many fields, there are several common notations for them. I will use these notations interchangeably in class; the most common ones appear below for reference.

Definition 11.2 (Notations for Partial Differentiation). The partial derivative of $𝑓$ with respect to $𝑥$ may be written as

𝜕 𝑓 𝜕 𝑥 = 𝜕 𝜕 𝑥 𝑓 = 𝜕 𝑥 𝑓 = 𝑓 𝑥

The last notation takes some getting used to at first: the subscript represents differentiation! But because of its conciseness, it is very commonly used when performing calculations.

For a quick example with three variables, let

𝑓 (𝑥, 𝑦, 𝑧) = 𝑥 2 𝑦 𝑒 𝑧 .

Holding the other two variables constant in turn gives

𝑓 𝑥 = 2 𝑥 𝑦 𝑒 𝑧, 𝑓 𝑦 = 𝑥 2 𝑒 𝑧, 𝑓 𝑧 = 𝑥 2 𝑦 𝑒 𝑧 .

In single variable calculus there was just a single first derivative: $𝑓'$ . Now, the number of first derivatives depends on the number of variables - $𝑓 (𝑥, 𝑦)$ has two first derivatives $𝑓 𝑥$ and $𝑓 𝑦$ , whereas $𝑔 (𝑢, 𝑣, 𝑤)$ has three! It turns out that it is very convenient to package all of this information together into a single vector, called the gradient.

Definition 11.3 (The Gradient Vector). Given a multivariable function $𝑓 (𝑥 1, \dots, 𝑥 𝑛)$ , its gradient is the vector of all first partial derivatives

\nabla 𝑓 = ⟨ 𝜕 𝑥 1 𝑓, 𝜕 𝑥 2 𝑓, \dots, 𝜕 𝑥 𝑛 𝑓 ⟩

The symbol $\nabla$ used in the notation of the gradient will become quite commonplace throughout vector calculus, though this is our first encounter with it. Alone, it is pronounced nabla, or del. Right now, it is alright to just think of the gradient as a convenient bookkeeping device storing all of the partial derivatives in one handy place. But soon we will see that its direction and magnitude are actually quite meaningful to the geometry of $𝑓$ , and because of this, the gradient vector lies at the heart of many modern optimization techniques in Machine Learning.

11.2Second Derivatives

Higher order partial derivatives are no more difficult: each time you take the derivative, you just treat all other variables as constants.

For instance, the second partial $𝑥$ derivative is just what you get by taking the $𝑥$ derivative twice:

𝜕 𝑥 𝜕 𝑥 (c o s (𝑥 𝑦)) = 𝜕 𝑥 (- 𝑦 s i n (𝑥 𝑦)) = - 𝑦 2 c o s (𝑥 𝑦)

But you can also take partials with respect to different variables.

𝜕 𝑥 𝜕 𝑦 (𝑥 3 𝑦 2) = 𝜕 𝑥 (2 𝑥 3 𝑦) = 6 𝑥 2 𝑦

Figure 11.2 Once a variable is held fixed there is only one variable left, so the slice can be differentiated exactly as in a first course—and then differentiated again. The three graphs on the left are that chain: the slice, its derivative, and its derivative's derivative. Where

𝑓

has a horizontal tangent,

𝑓 𝑥

crosses zero; where

𝑓

bends upward,

𝑓 𝑥 𝑥

is positive. Nothing is new except the bookkeeping, and the second variable's only job is to choose which slice.

Definition 11.4. A higher partial derivative is just the result of taking the partial derivative more than once (perhaps with respect to different variables). When doing this, one needs to be careful with notation: the "derivative notations" are all read like function composition

𝜕 𝑥 𝜕 𝑦 𝜕 𝑧 𝑓 = 𝜕 𝜕 𝑥 𝜕 𝜕 𝑦 𝜕 𝜕 𝑧 𝑓

both mean do the z partial, then the y partial, then the x partial.

The subscript notation is read from inside out:

𝑓 𝑧 𝑦 𝑥 = ((𝑓 𝑧) 𝑦) 𝑥

is equivalent to the expression above: differentiate first with respect to $𝑧$ , then $𝑦$ , and finally $𝑥$ .

Theorem 11.5 (Equality of Mixed Partials). If the mixed partial derivatives are continuous in a neighborhood of a point, their order may be exchanged at that point:

𝜕 𝑥 𝜕 𝑦 𝑓 = 𝜕 𝑦 𝜕 𝑥 𝑓 .

This works with higher order derivatives as well

𝑓 𝑥 𝑦 𝑥 𝑥 = 𝑓 𝑥 𝑥 𝑥 𝑦 = 𝑓 𝑦 𝑥 𝑥 𝑥 = \dots

Just like we packaged all of the first partial derivatives together into one nice object, the Gradient, we do the same with the second partials:

Definition 11.6 (The Hessian (Matrix of 2nd Derivatives)). Given a twice differentiable function $𝑓 (𝑥, 𝑦)$ its Hessian Matrix is the $2 \times 2$ array of all second derivatives

𝐻 𝑓 = (𝑓 𝑥 𝑥 𝑓 𝑥 𝑦 𝑓 𝑦 𝑥 𝑓 𝑦 𝑦)

The Laplacian is another way of collecting second-order information. It keeps only the non-mixed second partial derivatives and adds them.

Definition 11.7 (The Laplacian). The Laplacian operator is the sum of the non-mixed second order partial derivatives. It is sometimes written as $Δ$ and sometimes as $\nabla 2$ : in the plane,

Δ 𝑓 = \nabla 2 𝑓 = 𝑓 𝑥 𝑥 + 𝑓 𝑦 𝑦 = t r (𝐻 𝑓) .

In higher dimensions, we add one second partial derivative for each input variable.

One way to imagine what the Laplacian is measuring is a net sum of coordinate-direction concavities: it adds up the concavity in both the $𝑥$ and $𝑦$ directions. Thus, a function like $𝑥 2 + 𝑦 2$ has Laplacian

Δ (𝑥 2 + 𝑦 2) = 2 + 2 = 4,

while $- (𝑥 2 + 𝑦 2)$ has Laplacian $- 4$ , and $𝑥 2 - 𝑦 2$ has Laplacian equal to zero: it is concave up in one coordinate direction and concave down in the other, so added together they cancel. Functions whose Laplacian is zero are called harmonic functions and play a huge role in understanding differential equations, physics, and engineering.

A positive Laplacian means that these coordinate-direction concavities have a positive sum. It does not by itself mean that the function is convex: one direction may curve downward while another curves upward more strongly.

For now, $\nabla 𝑓$ , $𝐻 𝑓$ , and $Δ 𝑓$ are bookkeeping devices for first and second derivatives. The following chapters will show how the gradient and Hessian control approximation and geometry.

11.3Partial Differential Equations

Partial derivatives are the language in which much of modern science is written. We saw in the last portion of the course that vector valued differential equations are the right language to describe the motion of single particles: but what about quantities that depend on more than one variable?

A first example is simply waves on a string: when a guitar string is pulled taut, if you try to pluck it away from rest it pulls back on you - the farther you pull it away, the harder it pulls back.

The amount a string curves away from its straight line equilibrium is captured (roughly) by its concavity. And so one simple model of string motion would say the bigger the concavity the faster it wants to "snap back". Said more precisely:

The acceleration of the string is proportional to its concavity.

Writing this in math - if the string's displacement at position $𝑥$ and time $𝑡$ is given by the function $𝑊 (𝑥, 𝑡)$ , we have the partial differential equation

𝑊 𝑡 𝑡 = 𝑐 2 𝑊 𝑥 𝑥,

where the positive constant $𝑐$ is the wave speed.

In two dimensions, a wave equation measures the displacement of a circular membrane, like a drumhead or the interior speaker of an earbud. Here we have to account for displacements in both the $𝑥$ and $𝑦$ directions. Thus, the 2 dimensional wave equation is

𝑊 𝑡 𝑡 = 𝑐 2 (𝑊 𝑥 𝑥 + 𝑊 𝑦 𝑦) = 𝑐 2 Δ 𝑊 .

Figure 11.3 The rim is fixed, but the interior of the membrane can move. This radial mode is built from a function whose value is zero at the boundary, so every point on the highlighted circle stays still while the center oscillates. The equation

𝑊 𝑡 𝑡 = 𝑐 2 Δ 𝑊

says that the membrane's acceleration is controlled by its spatial concavity—the two-variable version of the vibrating string.

This same wave equation in three dimensions describes the propagation of electromagnetic waves - or light! This was a triumph of 19th century physics, where James Clerk Maxwell derived a wave equation from his equations for the electromagnetic field.

Similar partial differential equations occur throughout physics and engineering. Quantum mechanics, heat flow, fluid motion, elasticity, and acoustics all replace the evolution of a few coordinates with the evolution of fields depending on space and time.

Partial derivatives tell us how a scalar field changes while one input varies. In the next chapter, we combine those derivatives to construct the best linear and quadratic approximations near a point.