This post is the first of a series that will discuss coroutines in python, and how to leverage them to solve data crunching problems elegantly. Later in the series, We'll discuss native asynchronous programming introduced in PEP 492 (Coroutines with async and await syntax).
The first post will introduce coroutines, and explain a bit about how they work and how to use them.
<!-- more -->
- You're using python 2.7.x
- You're familiar with generators. otherwise, read either:
- 'yield' and Generators Explained by Jeff Knupp.
- Generator Tricks for Systems Programmers by David Beazley.
As David mentioned, If Python books are any guide, this is the most poorly documented, obscure, and apparently useless feature of Python.
The following example shows a naive usage of a coroutine:
from re import compile
Although, syntactically, generators and coroutines are similar, they are fundamentally different. You can mix and match both concepts - i.e: cause a function to generate and consume values.
The following example illustrates a "generator" that produces and receives values:
Although it works, its hard to follow and flaky:
Counting down from: 5
Try to avoid mixing both concepts. remember:
- Generators produce
- Coroutines consume & are not related to iteration.
Bootstrapping a coroutine
All coroutines need to get "bootstrapped" by first calling
.next(). These calls advance the function execution to the location of the first yield expression.
Remembering to bootstrap the coroutine is easy to forget. This is easily solved by wrapping coroutines with a decorator:
from functools import wraps
Now, our example will be simpler:
Closing a coroutine
A coroutine might run indefinitely. Use
.close() to shut it down. otherwise, the garbage collector will do it for you.
co = filter("^hello")
As opposed to regular functions, when a coroutine is closed, a
GeneratorExit exception is thrown, and every subsequent
.next() call raises a
A few things to note:
- GeneratorExit inherits directly from BaseException since it's technically not an error.
- GeneratorExit is raised when
close()is called, and is ignored by default.
- StopIteration is derived from Exception rather than StandardError, since this is not considered an error in its normal application.
Errors are supported in coroutines, and behave like you expect:
You can also cause a coroutine to raise an exception:
throw() is used to raise an exception inside the generator; the exception is raised by the yield expression where the generator’s execution is paused.
A bit of bytecode
# a generator
Let's look at the byte code:
from dis import dis
You've probably noticed that the
YIELD_VALUE opcode is used for both consuming and producing data. Although generators and coroutines are different, they share a lot of code behind the scenes.
That's the reason why the
a_mix coroutine has two
YIELD_VALUE opcodes, rather then one for consuming, and one for producing.
In other words, the distinction between generators and coroutines is mostly conceptual.
Go ahead and read the next part of the series, coroutines: pipelines & data flow.