Activations: relu, sigmoid, softmax

Section 01

The Story — Why Straight Lines Are Not Enough

📖 Real World Analogy

Stacking Rulers Still Gives a Ruler

Put ten straight rulers end to end. What do you get? A longer straight ruler. You can never draw a curve with only straight rulers joined in a line.

A Dense layer without an activation is a straight line (a linear function). Stack ten of them and — just like the rulers — you still have one linear function. The network could never learn a curved border between cats and dogs.

An activation function is a bend put after each layer. With a bend after every ruler, you can trace any shape you like.

In this lesson you will prove that stacked linear layers collapse into one, then meet the activations in tf.nn you will use every day.

Section 02

Proof: Two Linear Layers = One Linear Layer

import tensorflow as tf
tf.random.set_seed(0)

W1, b1 = tf.random.normal([3, 5]), tf.random.normal([5])
W2, b2 = tf.random.normal([5, 2]), tf.random.normal([2])
x = tf.random.normal([4, 3])

two_layers = (x @ W1 + b1) @ W2 + b2
W, b = W1 @ W2, b1[tf.newaxis] @ W2 + b2      # merge them into ONE layer
one_layer = x @ W + b

print("max difference:", float(tf.reduce_max(tf.abs(two_layers - one_layer))))

with_relu = tf.nn.relu(x @ W1 + b1) @ W2 + b2
print("with relu, difference from one layer:", round(float(tf.reduce_max(tf.abs(with_relu - one_layer))), 3))

OUTPUT

max difference: 4.76837158203125e-07 with relu, difference from one layer: 2.727

Without activation the difference is only float rounding: the two layers are one layer. With a ReLU in between, the result can no longer be squashed into a single layer.

Section 03

The Activations You Need

⤴️

relu

max(0, z). Fast and simple. The default for hidden layers.

tf.nn.relu — range [0, ∞)

↗️

leaky_relu

Like relu, but a small slope for z < 0, so neurons cannot "die".

tf.nn.leaky_relu(z, alpha=0.2)

〜

gelu

A smooth relu. Used in Transformers such as BERT and GPT.

tf.nn.gelu

∫

sigmoid

Squashes to (0, 1). Use on the output for yes/no (binary) problems.

tf.nn.sigmoid — range (0, 1)

〜

tanh

Squashes to (−1, 1), centred at 0. Common inside RNNs.

tf.nn.tanh — range (−1, 1)

📊

softmax

Turns a vector of scores into probabilities that add up to 1. For multi-class outputs.

tf.nn.softmax — works on a whole vector

z = tf.constant([-3.0, -1.0, 0.0, 1.0, 3.0])

print("z          :", z.numpy())
print("relu       :", tf.nn.relu(z).numpy())
print("leaky_relu :", tf.nn.leaky_relu(z, alpha=0.1).numpy().round(3))
print("sigmoid    :", tf.nn.sigmoid(z).numpy().round(3))
print("tanh       :", tf.nn.tanh(z).numpy().round(3))
print("gelu       :", tf.nn.gelu(z).numpy().round(3))

OUTPUT

z : [-3. -1. 0. 1. 3.] relu : [0. 0. 0. 1. 3.] leaky_relu : [-0.3 -0.1 0. 1. 3. ] sigmoid : [0.047 0.269 0.5 0.731 0.953] tanh : [-0.995 -0.762 0. 0.762 0.995] gelu : [-0.004 -0.159 0. 0.841 2.996]

Section 04

Try It — Activation Explorer

〜 Shape and Slope of Each Activation Interactive

Tick the activations to compare. The solid line is the function; the dashed line is its gradient (slope). Drag on the chart to move z. Type your own vector below to see what each function does to it.

z = 1.00 show gradients

activation	f(z)	gradient f′(z)	note

your vector z

Section 05

Gradients Matter: Saturation and Dead Neurons

z = tf.Variable([-6.0, -2.0, 0.0, 2.0, 6.0])

for name, fn in [("sigmoid", tf.nn.sigmoid), ("tanh", tf.nn.tanh), ("relu", tf.nn.relu)]:
    with tf.GradientTape() as tape:
        y = tf.reduce_sum(fn(z))
    print(f"{name:8s} gradient:", tape.gradient(y, z).numpy().round(4))

OUTPUT

sigmoid gradient: [0.0025 0.105 0.25 0.105 0.0025] tanh gradient: [0. 0.0707 1. 0.0707 0. ] relu gradient: [0. 0. 0. 1. 1.]

Saturation (sigmoid, tanh)

At z = ±6 the slope is almost 0

Deep stacks multiply many tiny slopes

→ "vanishing gradients", slow learning

ReLU

Slope is exactly 1 for z > 0 — no shrinking

Slope is 0 for z < 0

A neuron stuck below 0 is "dead" → try leaky_relu

Section 06

Softmax — From Scores to Probabilities

Softmax

p_i = exp(z_i) / Σ exp(z_j)

Every p is between 0 and 1, and they add up to 1.

Stable Version

exp(z_i − max(z)) / Σ …

Subtracting the max gives the same answer but never overflows.

logits = tf.constant([2.0, 1.0, 0.1])
p = tf.nn.softmax(logits)
print("probabilities:", p.numpy().round(3), " sum =", float(tf.reduce_sum(p)))

big = tf.constant([1000.0, 999.0, 998.0])
naive = tf.exp(big) / tf.reduce_sum(tf.exp(big))
stable = tf.exp(big - tf.reduce_max(big)) / tf.reduce_sum(tf.exp(big - tf.reduce_max(big)))
print("naive softmax :", naive.numpy())
print("stable softmax:", stable.numpy().round(4))
print("tf.nn.softmax :", tf.nn.softmax(big).numpy().round(4))

OUTPUT

probabilities: [0.659 0.242 0.099] sum = 0.9999999403953552 naive softmax : [nan nan nan] stable softmax: [0.6652 0.2447 0.09 ] tf.nn.softmax : [0.6652 0.2447 0.09 ]

💡

softmax Works Along an Axis

For a batch of shape (batch, classes), tf.nn.softmax(logits) uses the last axis by default: one probability vector per sample. The raw scores before softmax are called logits. You will meet that word a lot in the next lesson.

Section 07

Try It — Softmax Playground

📊 Push the Logits, Watch the Probabilities Interactive

Move the sliders to change each class's logit. The bars show the softmax probabilities. Change the temperature: low temperature makes the model very sure; high temperature makes it unsure. Press +1 to all and notice nothing changes.

temperature: 1.00

Section 08

Which Activation Where?

Place	Task	Activation
Hidden layers	Any	`relu` (or leaky_relu, gelu)
Output layer	Regression (a number)	None (linear)
Output layer	Binary yes/no	`sigmoid` — or output logits and use a "from logits" loss
Output layer	One of many classes	`softmax` — or output logits and use a "from logits" loss
Output layer	Several labels at once	`sigmoid` on each output

Section 09

Golden Rules

〜 Activations — Rules to Remember

Without activations, any number of Dense layers equals one Dense layer.

Use tf.nn.relu in hidden layers by default. Pair it with He initialisation.

Sigmoid and tanh saturate: their gradients vanish for large |z|. Avoid them in deep hidden stacks.

Softmax turns logits into probabilities along the last axis. Adding a constant to all logits changes nothing.

Use tf.nn.softmax, never a hand-made exp / sum: the built-in version is numerically stable.