Python Generators: Complete Guide with yield, iterators, and Patterns

Python generators are one of the most powerful but underused features. They let you write lazy, memory-efficient iterators with plain function syntax. This guide covers everything — from the basics of yield to advanced patterns like send(), yield from, and coroutines.

Quick Reference

Iterator Protocol

Before generators, you need to understand iterators. Any object with __iter__ and __next__ is an iterator:

class Counter:
    def __init__(self, stop):
        self.current = 0
        self.stop = stop

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.stop:
            raise StopIteration
        value = self.current
        self.current += 1
        return value

for n in Counter(3):
    print(n)  # 0, 1, 2

This works but is verbose. Generators give you the same result in three lines.

Generator Functions

A function becomes a generator the moment it contains a yield statement:

def counter(stop):
    current = 0
    while current < stop:
        yield current
        current += 1

for n in counter(3):
    print(n)  # 0, 1, 2

# Or manually
gen = counter(3)
print(next(gen))  # 0
print(next(gen))  # 1
print(next(gen))  # 2
print(next(gen))  # StopIteration raised

How it works: calling counter(3) does NOT run the function body. It returns a generator object. The body runs only when you call next() — and it pauses at each yield, resuming from that point next time.

Multiple yields

A generator can yield as many times as you want:

def weekdays():
    yield "Monday"
    yield "Tuesday"
    yield "Wednesday"
    yield "Thursday"
    yield "Friday"

days = list(weekdays())
# ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"]

Yield inside loops

The common pattern — yield inside a loop:

def read_large_file(path):
    with open(path) as f:
        for line in f:
            yield line.rstrip()

# Reads one line at a time — no matter how large the file
for line in read_large_file("huge.log"):
    process(line)

Generator Expressions

Compact syntax, like list comprehensions but with parentheses:

# List comprehension — builds entire list in memory
squares_list = [x**2 for x in range(1_000_000)]

# Generator expression — lazy, one value at a time
squares_gen = (x**2 for x in range(1_000_000))

print(sum(squares_gen))      # memory-efficient sum
print(list(squares_gen))     # now exhausted, returns []

Key rule: a generator can only be iterated once. To iterate again, create a new one.

Infinite Sequences

Generators are ideal for sequences with no end:

def integers(start=0):
    n = start
    while True:
        yield n
        n += 1

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

# Take first 10 fibonacci numbers
from itertools import islice
fibs = list(islice(fibonacci(), 10))
# [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Memory Comparison

import sys

# List: 8 MB for 1 million integers
big_list = [x for x in range(1_000_000)]
print(sys.getsizeof(big_list))  # ~8 000 056 bytes

# Generator: 128 bytes regardless of size
big_gen = (x for x in range(1_000_000))
print(sys.getsizeof(big_gen))   # 104 bytes

Use generators whenever you process large sequences one item at a time.

yield from — Delegating to Sub-Generators

yield from flattens nested iterables and delegates to sub-generators:

def chain(*iterables):
    for it in iterables:
        yield from it

list(chain([1, 2], [3, 4], [5]))  # [1, 2, 3, 4, 5]

# Flatten nested lists
def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item

list(flatten([1, [2, [3, 4]], 5]))  # [1, 2, 3, 4, 5]

Two-Way Communication with send()

send() lets you pass a value back into the generator at each yield:

def accumulator():
    total = 0
    while True:
        value = yield total   # yield sends total OUT, receives value IN
        if value is None:
            break
        total += value

gen = accumulator()
next(gen)        # must prime with next() first
gen.send(10)     # 10
gen.send(20)     # 30
gen.send(5)      # 35

Rule: the first call must be next(gen) (or gen.send(None)) to advance to the first yield.

throw() and close()

def guarded_gen():
    try:
        yield 1
        yield 2
        yield 3
    except ValueError as e:
        print(f"Caught: {e}")
        yield -1
    finally:
        print("Generator closed")

gen = guarded_gen()
print(next(gen))             # 1
print(gen.throw(ValueError, "bad input"))  # Caught: bad input → -1
gen.close()                  # Generator closed

Real-World Patterns

Pagination — fetch one page at a time

import urllib.request, json

def paginate(url, page_size=100):
    page = 1
    while True:
        response = urllib.request.urlopen(f"{url}?page={page}&size={page_size}")
        data = json.loads(response.read())
        if not data["items"]:
            return
        yield from data["items"]
        page += 1

for user in paginate("https://api.example.com/users"):
    process(user)

Pipeline of generators

def read_numbers(path):
    with open(path) as f:
        for line in f:
            line = line.strip()
            if line:
                yield float(line)

def filter_positive(numbers):
    for n in numbers:
        if n > 0:
            yield n

def square(numbers):
    for n in numbers:
        yield n ** 2

# Compose: data flows lazily through pipeline
numbers = read_numbers("data.txt")
positive = filter_positive(numbers)
squared = square(positive)

result = sum(squared)  # only now does any data flow

Context-aware generator

def batch(iterable, size):
    """Yield items in batches of given size."""
    batch = []
    for item in iterable:
        batch.append(item)
        if len(batch) == size:
            yield batch
            batch = []
    if batch:
        yield batch

for chunk in batch(range(10), 3):
    print(chunk)
# [0, 1, 2]
# [3, 4, 5]
# [6, 7, 8]
# [9]

itertools — The Generator Toolkit

from itertools import (
    count, cycle, repeat,
    islice, takewhile, dropwhile,
    chain, product, combinations, permutations,
    groupby, accumulate, starmap
)

# count: 0, 1, 2, 3, ... (infinite)
for n in islice(count(start=5, step=2), 5):
    print(n)  # 5, 7, 9, 11, 13

# cycle: repeats iterable forever
status = cycle(["active", "idle", "maintenance"])

# chain: concatenate multiple iterables
all_items = list(chain([1, 2], [3, 4], [5, 6]))

# takewhile: yield while condition is True
small = list(takewhile(lambda x: x < 5, [1, 2, 3, 6, 2]))  # [1, 2, 3]

# accumulate: running total
from itertools import accumulate
print(list(accumulate([1, 2, 3, 4])))  # [1, 3, 6, 10]

# groupby: group consecutive items by key
from itertools import groupby
data = [("a", 1), ("a", 2), ("b", 3), ("b", 4)]
for key, group in groupby(data, key=lambda x: x[0]):
    print(key, list(group))
# a [('a', 1), ('a', 2)]
# b [('b', 3), ('b', 4)]

# combinations and permutations
list(combinations("ABC", 2))    # [('A','B'), ('A','C'), ('B','C')]
list(permutations("AB", 2))     # [('A','B'), ('B','A')]
list(product([0, 1], repeat=3)) # all 3-bit binary numbers

Typing Generator Functions

from typing import Generator, Iterator

# Generator[YieldType, SendType, ReturnType]
def counter(n: int) -> Generator[int, None, None]:
    for i in range(n):
        yield i

# Simpler: just Iterator[YieldType]
def evens(n: int) -> Iterator[int]:
    return (x for x in range(n) if x % 2 == 0)

Common Mistakes

FAQ

What's the difference between a generator and an iterator?
Every generator is an iterator, but not every iterator is a generator. Generators are created with yield; iterators can also be built with __iter__/__next__ methods. Generators are usually simpler to write.

When should I use a generator vs a list?
Use a generator when: (a) the sequence is large or infinite, (b) you only need each item once, (c) you want to compose pipelines. Use a list when you need to index, slice, iterate multiple times, or know the length.

Can a generator function also return?
Yes. return in a generator causes StopIteration to be raised with the return value as its .value attribute. Useful with yield from in coroutines.

What is yield from for?
It delegates to a sub-generator, forwarding all next()/send()/throw() calls. It's also the foundation of Python's asyncio (before async/await).

Are generators thread-safe?
No. Don't share a generator across threads without a lock. Each thread should have its own generator instance.

How do I reset a generator?
You can't reset it — call the generator function again to get a fresh one. If you need multiple passes, either store results in a list or wrap the creation in a callable.