Loss Functions: MSE and Cross-Entropy

Section 01

The Story — The Strict Teacher

📖 Real World Analogy

One Number That Says "How Wrong?"

A teacher marks a test. For a maths answer, she checks how far off the number is: 98 instead of 100 is nearly right; 50 is very wrong.

For a multiple-choice question she cares about something else: how confident were you in the right answer? A student who said "I'm 90% sure it's B" (and B was right) did better than one who said "maybe B, 40%". A student who was 99% sure of the wrong answer gets the harshest mark of all.

A loss function is the teacher. For numbers we use MSE. For classes we use cross-entropy. Training is simply making this one number smaller.

Section 02

Regression Loss: Mean Squared Error

MSE

mean((y − ŷ)²)

Squares the error: big mistakes are punished a lot.

MAE

mean(|y − ŷ|)

Plain distance: every unit of error counts the same.

import tensorflow as tf

y_true = tf.constant([3.0, 5.0, 2.5, 7.0])
y_pred = tf.constant([2.5, 5.0, 4.0, 8.0])

mse = tf.reduce_mean(tf.square(y_true - y_pred))
mae = tf.reduce_mean(tf.abs(y_true - y_pred))
print("MSE by hand :", mse.numpy())
print("MSE (Keras) :", tf.keras.losses.MeanSquaredError()(y_true, y_pred).numpy())
print("MAE by hand :", mae.numpy())

y_pred_outlier = tf.constant([2.5, 5.0, 4.0, 17.0])   # one very bad prediction
print("with one outlier -> MSE", tf.reduce_mean(tf.square(y_true - y_pred_outlier)).numpy(),
      " MAE", tf.reduce_mean(tf.abs(y_true - y_pred_outlier)).numpy())

OUTPUT

MSE by hand : 0.875 MSE (Keras) : 0.875 MAE by hand : 0.75 with one outlier -> MSE 25.625 MAE 3.0

One bad prediction made MSE jump about 30 times, but MAE only about 4 times. MSE listens very hard to outliers.

Section 03

Try It — Regression Loss Explorer

🎯 How Much Does Each Loss Punish an Error? Interactive

Set the true value and drag the prediction (or click the chart). The curves show each loss for every possible prediction. Watch MSE grow much faster than MAE as the error gets big.

true y: 2.0 prediction ŷ: 0.5 Huber δ

MSE (squared)MAE (absolute)Huber (squared near, absolute far)

Section 04

Binary Cross-Entropy (Yes / No)

Binary Cross-Entropy

−[y·log(p) + (1−y)·log(1−p)]

y is 0 or 1; p is the predicted probability of "1".

In TensorFlow

tf.nn.sigmoid_cross_entropy_with_logits

Takes raw scores (logits), applies sigmoid inside, safely.

labels = tf.constant([1.0, 0.0, 1.0])
logits = tf.constant([2.0, -1.0, -3.0])       # raw model outputs

p = tf.sigmoid(logits)
by_hand = -(labels * tf.math.log(p) + (1 - labels) * tf.math.log(1 - p))
builtin = tf.nn.sigmoid_cross_entropy_with_logits(labels=labels, logits=logits)
print("p       :", p.numpy().round(3))
print("by hand :", by_hand.numpy().round(4))
print("built-in:", builtin.numpy().round(4))
print("mean    :", float(tf.reduce_mean(builtin)))

OUTPUT

p : [0.881 0.269 0.047] by hand : [0.1269 0.3133 3.0486] built-in: [0.1269 0.3133 3.0486] mean : 1.1629256010055542

The third sample is a "1" but the model gave it only 4.7% — so it gets by far the biggest loss.

Section 05

Categorical Cross-Entropy (Many Classes)

Cross-Entropy

−Σ y_i · log(p_i) = −log(p_true)

With one-hot labels, only the true class counts.

Two TF Versions

softmax_…_with_logits / sparse_…

One-hot labels vs integer labels. Both take logits.

logits = tf.constant([[2.0, 1.0, 0.1],       # sample 1: true class 0
                      [0.5, 2.5, 0.3]])      # sample 2: true class 2
labels_int = tf.constant([0, 2])
labels_1hot = tf.one_hot(labels_int, depth=3)

probs = tf.nn.softmax(logits)
by_hand = -tf.reduce_sum(labels_1hot * tf.math.log(probs), axis=1)
dense = tf.nn.softmax_cross_entropy_with_logits(labels=labels_1hot, logits=logits)
sparse = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=labels_int, logits=logits)

print("p(true) :", tf.reduce_sum(labels_1hot * probs, axis=1).numpy().round(3))
print("by hand :", by_hand.numpy().round(4))
print("one-hot :", dense.numpy().round(4))
print("sparse  :", sparse.numpy().round(4))

OUTPUT

p(true) : [0.659 0.089] by hand : [0.417 2.42 ] one-hot : [0.417 2.42 ] sparse : [0.417 2.42 ]

Why "with_logits"? Because log(0) = −∞

logits = tf.constant([[120.0, 0.0, -50.0]])     # very sure of class 0
label = tf.constant([[0.0, 1.0, 0.0]])          # ...but the truth is class 1

p = tf.nn.softmax(logits)
print("softmax probs :", p.numpy())
print("by hand loss  :", float(-tf.reduce_sum(label * tf.math.log(p))))
print("with_logits   :", float(tf.nn.softmax_cross_entropy_with_logits(labels=label, logits=logits)[0]))

OUTPUT

softmax probs : [[1. 0. 0.]] by hand loss : nan with_logits : 120.0

⚠️

Always Pass Logits, Not Probabilities

The softmax probabilities rounded to exactly 0, so log(p) became −inf and the hand-made loss turned into nan. One nan and training breaks. The "with_logits" function does the maths in a safe order (log-sum-exp) and returns the correct large number. So: the last layer outputs logits (no softmax), and the loss applies softmax itself.

A Beautiful Result: The Gradient Is p − y

logits = tf.Variable([[2.0, 1.0, 0.1]])
label = tf.constant([[0.0, 1.0, 0.0]])
with tf.GradientTape() as tape:
    loss = tf.nn.softmax_cross_entropy_with_logits(labels=label, logits=logits)
print("gradient   :", tape.gradient(loss, logits).numpy().round(4))
print("softmax - y:", (tf.nn.softmax(logits) - label).numpy().round(4))

OUTPUT

gradient : [[ 0.659 -0.7576 0.0986]] softmax - y: [[ 0.659 -0.7576 0.0986]]

The gradient pushes the true class logit up (negative gradient) and every wrong class down, in proportion to how much probability it wrongly took.

Section 06

Try It — Cross-Entropy Calculator

🏹 Confidence vs Punishment Interactive

Move the logit sliders and choose the true class. See the probabilities, the loss −log(p_true) on its curve, and the gradient p − y for each logit. Try making the model very sure of a wrong class.

true class

softmax probabilities & gradient p − y

loss = −log(p_true)

Section 07

Which Loss for Which Task?

Task	Last layer outputs	Loss (tf.nn / by hand)	Keras name
Predict a number	1 value, no activation	`reduce_mean(square(y - ŷ))`	MeanSquaredError
Number, with outliers	1 value, no activation	Huber / MAE	Huber, MeanAbsoluteError
Yes / no	1 logit	`sigmoid_cross_entropy_with_logits`	BinaryCrossentropy(from_logits=True)
One of K classes, one-hot labels	K logits	`softmax_cross_entropy_with_logits`	CategoricalCrossentropy(from_logits=True)
One of K classes, integer labels	K logits	`sparse_softmax_cross_entropy_with_logits`	SparseCategoricalCrossentropy(from_logits=True)

Section 08

Golden Rules

🎯 Losses — Rules to Remember

Regression → MSE (or Huber/MAE if there are outliers). Classification → cross-entropy.

Cross-entropy is −log(p_true): being confident and wrong costs the most.

Pass logits to …_with_logits functions. Never compute log(softmax(...)) by hand in training.

Integer labels → sparse_softmax_cross_entropy_with_logits. One-hot → softmax_cross_entropy_with_logits.

These functions return one loss per sample. Take tf.reduce_mean to get the scalar you differentiate.