Python Iterators and Generators — Understanding yield, Lazy Evaluation, and Memory-Efficient Loops

The Story That Explains Iterators & Generators

In Python, iterables, iterators, and generators are three closely related concepts that beginners often confuse. All three let you loop through data — but the way they store, compute, and hand out values is fundamentally different. Master them and you will write code that handles gigabytes with the same ease as a handful of rows.

Containers & Iterables — The Foundation

A container is any Python object that holds a collection of values — list, tuple, dict, set, and str are the most common. An iterable is any object that implements the __iter__ method, which means Python knows how to produce an iterator from it.

# A list is iterable — but is NOT an iterator
list1 = [10, 20, 30]

print(hasattr(list1, '__iter__'))   # True  — it IS iterable
print(hasattr(list1, '__next__'))   # False — it is NOT an iterator

# This FAILS:
next(list1)
# TypeError: 'list' object is not an iterator

# This WORKS — first convert to iterator using iter()
it = iter(list1)
print(next(it))    # 10
print(next(it))    # 20
print(next(it))    # 30
print(next(it))    # StopIteration raised

Iterators — The Stateful Cursor

An iterator is an object that remembers where it is in the sequence. It implements two methods: __iter__ (which returns itself) and __next__ (which returns the next value, or raises StopIteration when done).

Building a Custom Iterator — MyRange

Let's build our own iterator that mimics Python 2's xrange() — producing numbers one at a time without ever holding the full list in memory.

class MyRange:
    """A minimal reimplementation of xrange as a class-based iterator."""

    def __init__(self, start, end):
        self.value = start
        self.end   = end

    def __iter__(self):
        return self            # iterators return themselves

    def __next__(self):
        if self.value >= self.end:
            raise StopIteration
        current = self.value
        self.value += 1
        return current


# Use it exactly like the built-in range
r = MyRange(1, 5)
for num in r:
    print(num, end=' ')

That is 15 lines of code to build a lazy sequence generator. Powerful — but verbose. Python has a much shorter route to the same behaviour: the yield keyword.

Enter Generators — Iterators with Superpowers

A generator is a special kind of iterator built using a function containing at least one yield statement. The moment Python sees yield anywhere in a function body, that function becomes a generator function — calling it does not run the code, it just returns a generator object.

def my_range(start, end):
    """A generator version of MyRange — 4 lines instead of 15."""
    while start < end:
        yield start
        start += 1


# Calling the function does NOT execute the body
gen = my_range(1, 5)
print(gen)
# <generator object my_range at 0x7f9b...>

# Values are produced only on demand
for num in gen:
    print(num, end=' ')

How yield Actually Works — Step by Step

Generator Expressions — Comprehensions with Parentheses

Just as list comprehensions give you compact list creation, generator expressions give you compact generator creation. The syntax is identical, except you use parentheses () instead of square brackets [].

# List comprehension — builds the whole list in memory
squares_list = [x * x for x in range(10)]
print(squares_list)
# [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

# Generator expression — builds a generator (nothing computed yet)
squares_gen = (x * x for x in range(10))
print(squares_gen)
# <generator object <genexpr> at 0x7f9b8c1a2d40>

# Consume values on demand
for sq in squares_gen:
    print(sq, end=' ')
# 0 1 4 9 16 25 36 49 64 81

Memory & Performance — The Real Difference

This is where generators earn their keep. Let's measure the memory footprint of a list comprehension against a generator expression, using the same computation for 10 million elements.

import sys

# List comprehension — allocates everything
lst = [x * 2 for x in range(10_000_000)]
print(f"List memory:      {sys.getsizeof(lst):,} bytes")

# Generator expression — allocates almost nothing
gen = (x * 2 for x in range(10_000_000))
print(f"Generator memory: {sys.getsizeof(gen):,} bytes")

Speed Comparison — Summing 100 Million Numbers

import time

# List comprehension — build then sum
t = time.perf_counter()
total = sum([x for x in range(100_000_000)])
print(f"List:      {time.perf_counter() - t:.2f}s")

# Generator expression — stream directly into sum
t = time.perf_counter()
total = sum(x for x in range(100_000_000))
print(f"Generator: {time.perf_counter() - t:.2f}s")

Practical Example 1 — Fibonacci Forever

A classic use of generators: producing an infinite sequence. A list of infinite Fibonacci numbers is impossible — a generator is not.

def fibonacci():
    """An infinite Fibonacci generator."""
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b


# Take the first 10 Fibonacci numbers
fib = fibonacci()
for _ in range(10):
    print(next(fib), end=' ')

Combined with itertools.islice

from itertools import islice

# Take Fibonacci numbers 20 through 30 without building a list
first_ten_from_20 = list(islice(fibonacci(), 20, 30))
print(first_ten_from_20)

Practical Example 2 — Reading a Huge File

Suppose you must scan a 20 GB server log for the word "ERROR". Loading the file with file.readlines() would blow up your RAM. A generator handles it in megabytes of memory.

def grep_errors(path):
    """Yield every line in a huge log file that contains 'ERROR'."""
    with open(path) as f:
        for line in f:          # file objects are iterators too!
            if 'ERROR' in line:
                yield line.rstrip()


# Streams line-by-line — only one line in RAM at a time
errors = grep_errors('/var/log/app-20gb.log')

# Print the first 5 error lines without loading the file
for line, _ in zip(errors, range(5)):
    print(line)

Advanced — send(), close(), and yield from

Generators can do more than produce values. They can also receive them.

def echo():
    """A generator that echoes whatever you send into it."""
    while True:
        received = yield
        print(f"Got: {received}")


e = echo()
next(e)              # prime the generator (advance to first yield)
e.send('hello')      # Got: hello
e.send(42)           # Got: 42
e.close()            # cleanly stops the generator


# yield from — delegate to sub-generators
def countdown(n):
    while n > 0:
        yield n
        n -= 1

def combined():
    yield from countdown(3)
    yield from ['go!', 'done']

print(list(combined()))
# [3, 2, 1, 'go!', 'done']

The Three Concepts Side-by-Side

Common Pitfalls

Real-World Case Study — Streaming CSV Aggregation

Suppose you have a 5 GB sales CSV file with 200 million rows, and you need the total revenue per country. Loading it into pandas would swap your machine to a halt. A generator pipeline handles it in a few hundred MB of RAM.

import csv
from collections import defaultdict

def read_rows(path):
    """Stage 1 — yield each row as a dict, one at a time."""
    with open(path, newline='') as f:
        yield from csv.DictReader(f)

def only_2026(rows):
    """Stage 2 — filter to 2026 sales only."""
    for row in rows:
        if row['date'].startswith('2026'):
            yield row

def parse_amount(rows):
    """Stage 3 — parse the amount field once."""
    for row in rows:
        row['amount'] = float(row['amount'])
        yield row

# Compose the pipeline — nothing runs yet
pipeline = parse_amount(only_2026(read_rows('sales_2026.csv')))

# Only NOW do we pull rows through — one at a time
totals = defaultdict(float)
for row in pipeline:
    totals[row['country']] += row['amount']

for country, revenue in sorted(totals.items(), key=lambda x: -x[1])[:5]:
    print(f"{country:20s} ${revenue:,.2f}")

Golden Rules