The Story — A Spreadsheet That Thinks in Blocks
He adds a fixed tax of ₹5 to every price. That is broadcasting — one number stretched over the whole table.
He multiplies each price by its own quantity sold. That is an element-wise operation — cell by matching cell.
He works out each shop's total bill using a price list and quantity list at once. That is a matrix multiply.
Finally he adds up each shop's column into one total. That is a reduction.
These four kinds of maths are almost everything a neural network does. A dense layer is a matrix multiply, plus a broadcast bias, then an element-wise activation. The loss is a reduction. Learn these four well, and deep learning code becomes easy to read.
Element-wise Operations
An element-wise op works on each position on its own. Cell [0, 0] of the result uses only cell [0, 0] of the inputs.
import tensorflow as tf
a = tf.constant([[1., 2., 3.],
[4., 5., 6.]])
b = tf.constant([[10., 20., 30.],
[40., 50., 60.]])
print(a + b) # same as tf.add(a, b)
print((a * b).numpy()) # tf.multiply — NOT matrix multiply
print((b / a).numpy()) # tf.divide
print(tf.maximum(a, 3.0).numpy())Common Element-wise Functions
| Operator | Function | Example |
|---|---|---|
+ - * / | tf.add, tf.subtract, tf.multiply, tf.divide | a * b |
** | tf.pow | a ** 2 |
// % | tf.math.floordiv, tf.math.floormod | a // 2 |
tf.square, tf.sqrt, tf.abs | tf.sqrt(a) | |
tf.exp, tf.math.log | tf.math.log(a) | |
tf.maximum, tf.minimum | tf.maximum(a, 0.) = ReLU | |
> == < | tf.greater, tf.equal, tf.less | returns bool |
x = tf.constant([-2.0, -0.5, 0.0, 1.0, 3.0])
print("square :", tf.square(x).numpy())
print("abs :", tf.abs(x).numpy())
print("exp :", tf.exp(x).numpy().round(3))
print("relu :", tf.nn.relu(x).numpy())
print("sigmoid:", tf.sigmoid(x).numpy().round(3))
(1) Both inputs must have the same dtype. int32 + float32 is an error. Use tf.cast.
(2) Dividing two int32 tensors with / gives float64, not float32.
Cast to float32 first to keep your model in float32.
i = tf.constant([1, 2, 3])
print((i / 2).dtype) # float64!
print((tf.cast(i, tf.float32) / 2).dtype)
print((i // 2).numpy()) # floor division keeps int32Try It — Element-wise Calculator
Click any cell in A or B to change its number. Pick an operation and press Compute. Each result cell is built from the matching cells only.
Matrix Multiplication — tf.matmul
Matrix multiply is different from element-wise *. Each result cell is a dot product: take one row of A and one column of B, multiply the pairs, and add them up.
A = tf.constant([[1, 2, 3],
[4, 5, 6]]) # (2, 3)
B = tf.constant([[7, 8],
[9, 10],
[11, 12]]) # (3, 2)
C = tf.matmul(A, B) # or: A @ B
print(C)
try:
tf.matmul(A, A) # (2,3) @ (2,3): inner sizes 3 != 2
except tf.errors.InvalidArgumentError as e:
print("Error:", e.message.split("} ")[-1].split(" [Op")[0])| Shapes must match (or broadcast) |
| C[i, j] = A[i, j] × B[i, j] |
| Used for masks, scaling, gates |
| Inner sizes must match |
| C[i, j] = row i · column j |
| Used in every dense layer |
Useful matmul Options
X = tf.random.normal([4, 3])
W = tf.random.normal([5, 3])
print(tf.matmul(X, W, transpose_b=True).shape) # X @ W.T -> (4, 5)
batch_a = tf.ones([10, 2, 3]) # a batch of 10 matrices
batch_b = tf.ones([10, 3, 4])
print(tf.matmul(batch_a, batch_b).shape) # batched matmul -> (10, 2, 4)Try It — Matrix Multiply, Step by Step
Set the sizes of A and B, then press Step to compute one cell, or Play to watch them all. Click cells to edit values. Try making the inner sizes different to see the error.
Broadcasting — Stretching to Fit
What if the shapes do not match? TensorFlow tries to broadcast. It stretches the smaller tensor, without copying memory, so the shapes match. This is how one bias vector gets added to every row of a batch.
prices = tf.constant([[100., 200., 300.],
[150., 250., 350.]]) # (2, 3)
print(prices + 5) # scalar -> every cell
print(prices * tf.constant([1., 0.5, 2.])) # (3,) -> every row
print(prices - tf.constant([[10.], [20.]])) # (2, 1) -> every column
col = tf.constant([[1], [2], [3]]) # (3, 1)
row = tf.constant([10, 20, 30, 40]) # (4,)
print((col + row).shape) # (3, 4): both stretch!The column is copied across 4 times. The row is copied down 3 times. Then the matching cells are added.
Try It — Broadcasting Checker
Type two shapes. Use an empty box for a scalar. The table lines them up from the right and checks each axis. Small 2-D cases are also drawn.
Reductions — tf.reduce_*
A reduction collapses an axis into a single value. The axis argument says which axis disappears. With no axis, the whole tensor becomes one number.
sales = tf.constant([[3., 5., 2., 8.],
[1., 4., 6., 2.],
[7., 2., 3., 5.]]) # 3 shops x 4 days
print("total :", tf.reduce_sum(sales).numpy())
print("per day :", tf.reduce_sum(sales, axis=0).numpy()) # collapse shops
print("per shop :", tf.reduce_sum(sales, axis=1).numpy()) # collapse days
print("mean :", tf.reduce_mean(sales, axis=1).numpy())
print("best day :", tf.argmax(sales, axis=1).numpy()) # index of max
print("keepdims :", tf.reduce_sum(sales, axis=1, keepdims=True).shape)| Function | Returns | Deep learning use |
|---|---|---|
tf.reduce_sum | Sum | Total loss, counting |
tf.reduce_mean | Average | Mean loss over a batch |
tf.reduce_max / min | Largest / smallest | Max pooling, stable softmax |
tf.reduce_prod | Product | Counting elements in a shape |
tf.reduce_all / any | AND / OR of bools | Checks and assertions |
tf.argmax / argmin | Index of largest / smallest | Predicted class from scores |
tf.reduce_mean(tf.constant([1, 2])) gives 1, not 1.5. The result keeps the int dtype.
Cast to float32 before taking a mean.
Try It — Reduction Explorer
Pick a function and an axis. Cells with the same colour are combined into one result value. Click cells to change the numbers.
Putting It Together — A Dense Layer by Hand
Now combine all four ideas. This is exactly what a Keras Dense layer computes.
tf.random.set_seed(0)
x = tf.random.normal([4, 3]) # batch of 4 samples, 3 features
W = tf.random.normal([3, 2]) # 3 inputs -> 2 units
b = tf.constant([0.1, -0.2]) # one bias per unit
z = tf.matmul(x, W) + b # matmul + broadcast -> (4, 2)
y = tf.nn.relu(z) # element-wise -> (4, 2)
loss = tf.reduce_mean(tf.square(y)) # element-wise + reduce -> ()
print("z shape:", z.shape, " y shape:", y.shape)
print("loss :", round(float(loss), 4))Golden Rules
* is element-wise. @ or tf.matmul is matrix multiply. Do not mix them up.axis is the axis that disappears. Use keepdims=True to keep it as size 1 for later broadcasting.activation(x @ W + b). You now know every piece of it.