tf.GradientTape Explained: Your First Gradient in TensorFlow

The Story — A Video Camera in the Kitchen

Training a neural network means asking one question again and again: "If I change this weight a little, does the loss go up or down, and by how much?" The answer is the gradient. TensorFlow computes it for you automatically. This is called automatic differentiation.

What Is a Gradient? (No Heavy Maths)

A gradient is a slope. It tells you how fast the output changes when you push the input a tiny bit. For one input, the gradient is the same as the derivative you may know from school.

Your First Gradient in Three Lines

import tensorflow as tf

x = tf.Variable(3.0)

with tf.GradientTape() as tape:     # 1. start recording
    y = x ** 2                      # 2. do the maths inside the block

dy_dx = tape.gradient(y, x)         # 3. play it backwards
print("y     =", y.numpy())
print("dy/dx =", dy_dx.numpy())

Try It — Slope Explorer

The Chain Rule Happens for You

Real models chain many ops together. The gradient of a chain is the product of each step's local slope. This is the chain rule. The tape applies it automatically, step by step, backwards.

x = tf.Variable(2.0)

with tf.GradientTape() as tape:
    a = 3 * x + 1        # a = 7
    y = a ** 2           # y = 49

print("tape says :", tape.gradient(y, x).numpy())
print("by hand   :", 2 * (3 * 2.0 + 1) * 3)     # dy/da * da/dx = 2a * 3

What Does the Tape Watch?

The tape automatically watches every trainable float tf.Variable. It does not watch plain tensors. If the tape did not watch the source, tape.gradient returns None — no error, just None.

c = tf.constant(3.0)

with tf.GradientTape() as tape:
    y = c ** 2
print("constant, not watched :", tape.gradient(y, c))

with tf.GradientTape() as tape:
    tape.watch(c)                     # ask the tape to watch it
    y = c ** 2
print("constant, watched     :", tape.gradient(y, c).numpy())

Common Reasons for a None Gradient

def grad_of(x, fn):
    with tf.GradientTape() as tape:
        y = fn(x)
    return tape.gradient(y, x)

print("int variable     :", grad_of(tf.Variable(3), lambda v: v * v))
print("trainable=False  :", grad_of(tf.Variable(3.0, trainable=False), lambda v: v * v))
print("left TF (.numpy) :", grad_of(tf.Variable(3.0), lambda v: tf.constant(v.numpy() ** 2)))
print("no-gradient op   :", grad_of(tf.Variable(3.0), lambda v: tf.round(v)))

Try It — Will I Get a Gradient?

Python Control Flow Is Fine

The tape records the ops that actually ran. So if, for and while all work.

x = tf.Variable(2.0)

with tf.GradientTape() as tape:
    y = x
    for _ in range(3):        # y = x ** 8 after three squarings
        y = y * y

print("y     :", y.numpy())
print("dy/dx :", tape.gradient(y, x).numpy(), "(= 8 * 2**7)")

Golden Rules