The Story — A Single Ticket and a Season Pass
A normal GradientTape is the single ticket. After one call to tape.gradient(), TensorFlow frees the recording. A persistent=True tape is the season pass. You can call gradient() many times, and you delete it when done.
And what about the "gradient of a gradient"? Think of a car. The gradient of position is speed. The gradient of speed is acceleration. That is a second-order gradient.
A Normal Tape Works Only Once
To save memory, a tape throws away its recording right after the first gradient() call.
import tensorflow as tf
x = tf.Variable(3.0)
with tf.GradientTape() as tape:
y = x ** 2
z = x ** 3
print("dy/dx =", tape.gradient(y, x).numpy())
try:
tape.gradient(z, x) # second call
except RuntimeError as e:
print("RuntimeError:", str(e).split(".")[0])persistent=True — Many Gradients From One Recording
x = tf.Variable(3.0)
with tf.GradientTape(persistent=True) as tape:
y = x ** 2
z = x ** 3
print("dy/dx =", tape.gradient(y, x).numpy()) # 2x = 6
print("dz/dx =", tape.gradient(z, x).numpy()) # 3x² = 27
del tape # free the memory yourselfA persistent tape keeps every intermediate tensor alive until the tape is deleted. With a big model this can be many gigabytes. Always del tape when you are finished, and only use persistent when you really need more than one call.
Gradients of one target for many variables come from a single call: tape.gradient(loss, [w, b]). You only need persistent for several targets (for example two different losses) or for jacobians.
Try It — Tape Lifecycle Simulator
Choose the tape type, press Record, then ask for gradients. Watch when the tape gets released. Here x = 3.0, y = x ** 2, z = x ** 3.
Higher-Order Gradients — Nested Tapes
To get the gradient of a gradient, put one tape inside another. The inner tape computes the first gradient. Because that happens inside the outer tape's block, the outer tape records it — and can differentiate it again.
x = tf.Variable(2.0)
with tf.GradientTape() as outer:
with tf.GradientTape() as inner:
y = x ** 3 # y = x³ = 8
dy_dx = inner.gradient(y, x) # y' = 3x² = 12 (recorded by outer)
d2y_dx2 = outer.gradient(dy_dx, x) # y'' = 6x = 12
print("y =", y.numpy())
print("y' =", dy_dx.numpy())
print("y'' =", d2y_dx2.numpy())Blue: the inner tape finds the first derivative. Purple: the outer tape differentiates that result again.
A Persistent Tape Can Do It Too
x = tf.Variable(2.0)
with tf.GradientTape(persistent=True) as tape:
y = x ** 3
dy = tape.gradient(y, x) # computed while still recording
d2y = tape.gradient(dy, x)
print(dy.numpy(), d2y.numpy())
del tapeTry It — The Derivative Ladder
Type a function of x or pick a preset. Move the slider. See the value, the slope (f′) and the curvature (f″). Where f′ = 0 and f″ > 0 you are at the bottom of a valley — a minimum.
Where Second Derivatives Are Used
One classic use is Newton's method. Plain gradient descent only knows the slope. Newton's method also knows the curvature, so it can jump much closer to the minimum in one step: x ← x − f′(x) / f″(x).
x = tf.Variable(2.0) # minimise f(x) = x**4 - 3x
for step in range(5):
with tf.GradientTape() as outer:
with tf.GradientTape() as inner:
f = x ** 4 - 3 * x
g = inner.gradient(f, x) # f'
h = outer.gradient(g, x) # f''
x.assign_sub(g / h) # Newton step
print(f"step {step}: x = {x.numpy():.6f} f'(x) = {g.numpy():+.4f}")
print("exact answer:", round((3 / 4) ** (1 / 3), 6))| Use | What it needs | Tool |
|---|---|---|
| Two losses from one forward pass | Several gradient calls | persistent=True |
| Newton's method, curvature checks | 2nd derivative | Nested tapes |
| WGAN-GP gradient penalty | Gradient of a gradient norm | Nested tapes |
| Physics-informed neural networks | du/dx, d²u/dx² of the network | Nested tapes |
| Per-output gradients | A full Jacobian | tape.jacobian, tape.batch_jacobian |
Golden Rules
gradient() call. A second call raises RuntimeError.persistent=True for several targets, then del tape to free memory.