TFRecord Files: Write, Read, Parse

Section 01

The Story — Shipping Containers

📖 Real World Analogy

Thousands of Loose Boxes vs a Few Big Containers

Imagine shipping a million small parcels one by one. Each needs its own label, its own handling, its own trip. It is slow and messy. Ports solved this with shipping containers: pack many parcels into one standard box, and move the box.

Training data has the same problem. A million tiny image files are slow to open one by one, especially from cloud storage. A TFRecord file is a container: it packs many examples into one big file that is read from start to end, very fast. Each example inside is a standard "parcel" called tf.train.Example.

Animated Diagram — Inside a TFRecord File

Section 02

tf.train.Example — The Standard Parcel

An Example is a dictionary from feature names to lists of values. There are only three list types:

📦

BytesList

tf.train.BytesList

Strings and raw bytes: text, encoded JPEG/PNG images, serialized tensors.

🌡️

FloatList

tf.train.FloatList

float32 numbers (float64 is stored as float32).

🔢

Int64List

tf.train.Int64List

Integers and booleans: labels, IDs, counts.

import tensorflow as tf

def bytes_feature(v):  return tf.train.Feature(bytes_list=tf.train.BytesList(value=v))
def float_feature(v):  return tf.train.Feature(float_list=tf.train.FloatList(value=v))
def int64_feature(v):  return tf.train.Feature(int64_list=tf.train.Int64List(value=v))

example = tf.train.Example(features=tf.train.Features(feature={
    "city":   bytes_feature([b"Baddi"]),
    "income": float_feature([91000.0]),
    "age":    int64_feature([47]),
    "scores": float_feature([0.5, 0.9, 0.2]),
}))
data = example.SerializeToString(deterministic=True)
print(example)
print("serialized size:", len(data), "bytes")

OUTPUT

features { feature { key: "scores" value { float_list { value: 0.5 value: 0.9 value: 0.2 } } } feature { key: "income" value { float_list { value: 91000 } } } feature { key: "city" value { bytes_list { value: "Baddi" } } } feature { key: "age" value { int64_list { value: 47 } } } } serialized size: 83 bytes

Section 03

Try It — Example Builder

🔧 Build an Example, See the Bytes Interactive

Add features, pick their type and type values separated by commas. The builder shows the Python code, the parsing spec, the exact serialized size and the raw bytes — computed with the real protobuf encoding rules. Notice how small integers take fewer bytes than floats.

Python

Parsing spec

Section 04

Writing TFRecord Files

import numpy as np, os
rng = np.random.default_rng(0)

def make_example(i):
    return tf.train.Example(features=tf.train.Features(feature={
        "x":     float_feature(rng.normal(size=8).astype("float32").tolist()),
        "label": int64_feature([int(i % 3)]),
        "id":    bytes_feature([f"sample-{i}".encode()]),
    })).SerializeToString()

with tf.io.TFRecordWriter("/tmp/data.tfrecord") as w:
    for i in range(1000):
        w.write(make_example(i))

gzip = tf.io.TFRecordOptions(compression_type="GZIP")
with tf.io.TFRecordWriter("/tmp/data.tfrecord.gz", options=gzip) as w:
    for i in range(1000):
        w.write(make_example(i))

print("plain:", os.path.getsize("/tmp/data.tfrecord"), "bytes")
print("gzip :", os.path.getsize("/tmp/data.tfrecord.gz"), "bytes")

OUTPUT

plain: 98890 bytes gzip : 40934 bytes

📦

What Compresses Well?

Here GZIP cut the file by more than half, because the feature names and labels repeat in every record. Random float values themselves barely compress, and data that is already compressed (JPEG, PNG) hardly shrinks at all. Compression saves disk and network, but costs some CPU when reading.

Section 05

Reading and Parsing

A TFRecordDataset gives raw bytes. You describe the features you expect, then parse.

feature_spec = {
    "x":     tf.io.FixedLenFeature([8], tf.float32),
    "label": tf.io.FixedLenFeature([], tf.int64),
    "id":    tf.io.FixedLenFeature([], tf.string),
}

raw = tf.data.TFRecordDataset("/tmp/data.tfrecord")
print("raw element:", raw.element_spec)

def parse(record):
    ex = tf.io.parse_single_example(record, feature_spec)
    return ex["x"], ex["label"]

ds = raw.map(parse, num_parallel_calls=tf.data.AUTOTUNE)
for x, y in ds.take(2):
    print(x.numpy().round(2), y.numpy())

OUTPUT

raw element: TensorSpec(shape=(), dtype=tf.string, name=None) [ 0.13 -0.13 0.64 0.1 -0.54 0.36 1.3 0.95] 0 [-0.7 -1.27 -0.62 0.04 -2.33 -0.22 -1.25 -0.73] 1

Faster: Batch First, Then Parse the Whole Batch

def parse_batch(records):
    ex = tf.io.parse_example(records, feature_spec)       # parses many at once
    return ex["x"], ex["label"]

fast = (tf.data.TFRecordDataset("/tmp/data.tfrecord.gz", compression_type="GZIP")
          .batch(32)
          .map(parse_batch, num_parallel_calls=tf.data.AUTOTUNE)
          .prefetch(tf.data.AUTOTUNE))
x, y = next(iter(fast))
print(x.shape, y.shape, y[:10].numpy())

OUTPUT

(32, 8) (32,) [0 1 2 0 1 2 0 1 2 0]

Spec	Use for	Gives
`FixedLenFeature([n], dtype)`	Every example has exactly n values	A dense tensor
`FixedLenFeature([], dtype, default_value=…)`	One value; may be missing	A scalar (default if missing)
`VarLenFeature(dtype)`	Lists of different lengths (e.g. tags)	A `SparseTensor`
`RaggedFeature(dtype)`	Lists of different lengths	A `RaggedTensor`

Section 06

Images and Whole Tensors

Store images as their encoded PNG/JPEG bytes (small) and decode while reading. For any other tensor, tf.io.serialize_tensor turns it into bytes.

img = tf.cast(tf.random.uniform([16, 16, 3], 0, 255, seed=1), tf.uint8)
emb = tf.random.normal([4, 5], seed=2)

ex = tf.train.Example(features=tf.train.Features(feature={
    "image": bytes_feature([tf.io.encode_png(img).numpy()]),
    "embedding": bytes_feature([tf.io.serialize_tensor(emb).numpy()]),
}))

spec = {"image": tf.io.FixedLenFeature([], tf.string),
        "embedding": tf.io.FixedLenFeature([], tf.string)}
back = tf.io.parse_single_example(ex.SerializeToString(), spec)
img2 = tf.io.decode_png(back["image"])
emb2 = tf.io.parse_tensor(back["embedding"], out_type=tf.float32)
print("image  :", img2.shape, "same?", bool(tf.reduce_all(img == img2)))
print("tensor :", emb2.shape, "same?", bool(tf.reduce_all(emb == emb2)))

OUTPUT

image : (16, 16, 3) same? True tensor : (4, 5) same? True

Many Files: Sharding

# write:  train-00000-of-00016.tfrecord ... train-00015-of-00016.tfrecord
files = tf.data.Dataset.list_files("gs://my-bucket/train-*.tfrecord", shuffle=True)
ds = files.interleave(tf.data.TFRecordDataset,                  # read several files at once
                      cycle_length=8, num_parallel_calls=tf.data.AUTOTUNE)

Section 07

When Should I Use TFRecord?

✅

Use it when…

Data is large (GBs+), lives in cloud storage, has millions of small files, or is read by TPUs.

big, remote, many files

⚖️

Maybe…

Medium datasets on a local SSD. Try the simple way first and measure.

measure first

❌

Skip it when…

Data fits in memory as arrays. from_tensor_slices is simpler and just as fast.

small data

Section 08

Golden Rules

📦 TFRecord — Rules to Remember

A TFRecord file is a list of byte records. Usually each record is a serialized tf.train.Example.

An Example maps names to one of three lists: bytes, float or int64.

Write the same feature spec for reading that you used for writing.

Batch before parse_example for speed; read many shards with interleave.

Store images as encoded bytes and other tensors with serialize_tensor.