4. Iterators & Generators

Iterators are one of Python's most powerful features. At a high level, you might view iteration simply as a way to process items in a sequence, but there is much more than this such as creating your own iterator object, applying useful iterator patterns in itertools module, make generator functions etc.

4.1: Manually consuming an iterator

problem

You wan tto consume items in an iterable but for whatever reasons you can't or don't want to use a for loop

Solution

To manually consume the iterable, use the next() function and write code to catch the StopIteration exception as shown below.

with open("etc/passwd") as f:
    try:
        while True:
            line = next(f)
            print(line, end="")
    except StopIteration:
        pass

Normally StopIteration exception is used to signal the end of the sequence.

But, if you are using next, you can instruct next() to return a terminating value like None as shown below.

with open("etc/passwd") as f:
    try:
        while True:
            line = next(f, None)
            if line is None:
                break
            print(line, end="")

Discussion

In most cases the for is used to consume the iterable. However every now and then a problem statement calls for precise control over the underlying mechanism. The below example explains what happens underneath the iteration protocol.

>>> items= [1, 2, 3]        ## the iterable
>>> 
>>> ## Get the iterator
>>> it = iter(items)        ## invokes "items.__iter__()"
>>> next(it)                ## invokes "items.__next__()"
1
>>> next(it)
2
>>> next(it)
3
>>> next(it)    ## this triggers "StopIteration" exception b'coz the iterator "it" is empty now
Traceback (most recent call last):
    file "<stdin>", line 1, in <module>
StopIteration

4.2: Delegating Iteration

Problem

You have built a custom container object that internallt holds a list or tuple or other iterables. You want the iteration to work with this custom container.

Solution

Typically all you need to do is to define an internal iter() method that delegates iteration to internally held container as shown below.

class Node:
    def __init__(self, value):
        self._value = value
        self._children = []

    def __repr__(self):
        return f"Node{self._value!r}"

    def add_child(self, node):
        self._children.append(node)

    def __iter__(self):
        return iter(self._children)     ## forwars the iteration request to the "self._children" 

## ---------------------------------------------- ##

if __naame__ == "__main__":
    root = Node(0)
    child1 = Node(1)
    child2 = Node(2)
    root.add_child(child1)
    root.add_child(child2)

    for ch in root:
        print(ch)

>>> Node(1)
    Node(2)

In the above example, the __iter__() method simply forwards the iteration request to the internally held self._children attribute.

Discussion

Python's iterator protocol requires the __iter__() method to return a special iterator object that implements a __next__() method which actually carries out the actual iteration.

In the above example, we are delegating the implemeation of __next__() method by returning a list (iter(self._children)) which internally implements __iter__().

Calling the iter(x) function internally calls x.__iter__(). Note that this behaviour is similar to len(seq) and seq.__len__()

4.3: Creating new iteration patterns with GENERATORS