A Curious Course on Coroutines and Concurrency
Copyright (C) 2009, All Rights Reserved
David Beazley
http://www.dabeaz.com
Presented at PyCon 2009, March 25, 2009.
Introduction
This tutorial is a practical exploration of using Python coroutines
(extended generators) for solving problems in data processing, event
handling, and concurrent programming. The material starts off with
generators and builds to writing a complete multitasking environment
that can run thousands of concurrent tasks without using threads or
using code based on event-driven callbacks (i.e., the "reactor"
model).
Note: This tutorial might be viewed as a sequel to the tutorial
Generator Tricks
for System Programmers I presented at PyCon'08 in Chicago. If
you have never used generator functions before, you might
want to look at that presentation for more information. This
coroutine tutorial is meant to stand on its own, but you'll get
a more complete picture if you combine it with the generator presentation.
Requirements and Support Data Files
This tutorial requires the use of Python 2.5 or newer. No third party
modules are required. Examples have been tested on both Unix and Windows XP.
Examples will work on Python 3 as long as you fix all of the print statements.
The following file contains some supporting data files that are used
by the various code samples. Download this to your machine to work with
the examples that follow.
This download also includes a PDF of the lecture slides.
Code Samples
Here are various code samples from the course. You can either cut and
paste these from the browser or simply work with them directly in the
"coroutines" directory. The order in which files are listed follow the
course material. These examples are written to run inside the
"coroutines" directory that gets created when you unzip the above file
containing the support data.
Part 1 : Introduction to Generators and Coroutines
- countdown.py. A trivially simple generator function.
- follow.py. A generator that follows lines
written to a real-time log file (like Unix 'tail -f'). To run this
program, you need to have a log-file to work with. Run the
program logsim.py to create a simulated
web-server log (written in the file access-log). Leave this program running in the background for the
next few parts.
- pipeline.py. An example of using generators to set up a simple processing pipeline.
Print all server log entries containing the word 'python'.
- grep.py. A first example of a coroutine function. This function receives lines
and prints out those that contain a substring.
- coroutine.py. A decorator function that eliminates the need to call .next() when
starting a coroutine.
- grepclose.py. An example of a coroutine that catches the close() operation.
- bogus.py. An example of a bogus generator that generates and receives values (not a recommended
coding style).
Part 2 : Coroutines, Pipelines, and Dataflow
- cofollow.py. A simple example of feeding data from a data source into a coroutine. This
mirrors the 'tail -f' example from earlier.
- copipe.py. An example of setting up a processing pipeline with coroutines.
- cobroadcast.py. An example of a coroutine broadcaster. This fans a data stream out to
multiple targets.
- cobroadcast2.py. An example of broadcasting with a slightly different data handling pattern.
- benchmark.py. A small benchmark comparing the performance of sending data into a coroutine
vs. sending data into an instance of a class.
Part 3 : Coroutines and Event Dispatching
- basicsax.py. A very basic example of the SAX API for parsing XML documents (does
not involve coroutines).
- cosax.py. An example that pushes SAX events into a coroutine.
- buses.py. An example of parsing and filtering XML data with a
series of connected coroutines.
- coexpat.py. An XML parser that turns events generated by the expat XML library into
coroutines. Compare with cosax.py above.
- cxml/cxmlparse.c. A bare-bones C
extension module that uses the expat library and pushes events into
coroutines. To build this code on your own machine, you will need to
first build and install expat and then use python setup.py
build_ext --inplace. Note: This main focus of this class is not
C extension building so you might have to mess around with this to get
it to work. Key point: you can dispatch data into a coroutine
directly from C.
- iterbus.py. An example of incremental
XML parsing with the ElementTree module (for comparison with
coroutines).
Part 4 : From Data Processing to Concurrent Programming
- cothread.py. A thread object that runs
a coroutine.
- coprocess.py. An example of running a
coroutine in a subprocess. This example runs a separate
program busproc.py in a subprocess and sends
data to it.
- cocrash.py. An example of crashing a
thread by having it send data into an already executing coroutine.
Since this example depends on thread synchronization, it may
occasionally "work" by accident. Run it a few more times.
Part 7 : Writing an Operating System
- pyos1.py. A simple object representing a "task." It is a wrapper around
a coroutine.
- pyos2.py. A simple task scheduler that alternates between
tasks whenever they yield.
- taskcrash.py. An example showing how the schedule crashes
if one of the tasks terminates.
- pyos3.py. An improved scheduler that properly handles
task termination.
- pyos4.py. A scheduler with support for "system calls."
- pyos5.py. A scheduler with system calls for basic task
creation and termination.
- pyos6.py. A scheduler that adds support for task waiting.
- echobad.py. A broken example of trying to use coroutines
to implement a multitasking network server. It breaks due to blocking I/O operations.
- pyos7.py. A scheduler that adds support for I/O waiting.
- echogood.py. A slightly modified version of the echo server
above that properly handles blocking I/O.
- echogood2.py. An echo server without the "I'm alive" messages.
Part 8 : The Problem with Subroutines and the Stack
- trampoline.py. An example of "trampolining" between coroutines.
- pyos8.py. The OS with an enhanced Task
object that allows coroutines to transfer control to other coroutines
like subroutines.
- echoserver.py. A concurrent echo server using coroutines.
- sockwrap.py. An class wrapper that shows how you can emulate the
socket interface with a collection of coroutine methods.
- echoserver2.py. An echo server using the class-based interface.
- twistedecho.py. A basic echo server
written for Twisted. You will need to
install Twisted to run this.
One reason why I have included this here is that the low-level I/O
handling is very similar.
Specifically, in our "operating system", I/O events are tied into task
scheduling. In Twisted, I/O events are tied into event handlers
(Reactors).
Part 9 : Final words
- blaster.py. A very simple script that
opens up 300 socket connections with an echo server and randomly
blasts it with 1024 bytes messages. The
program runblaster.py will run multiple
copies of this program in different subprocesses to increase the load.
Note: these scripts are the same ones I used to do a basic benchmark
on Twisted vs. Coroutines described in the handout. The main goal of
this class has not been performance measurement so you might want to play
around with the scripts--changing different settings
for the number of connections, number of processes, message size, etc. To
allow a large number of socket connections, you might have to fiddle
with system settings. For example, on my Mac, I first have to use
'ulimit -n 1024' in the shell to change the number of file descriptors
on the server (otherwise, only 256 connections can be handled).
- echothread.py. A threaded implementation of
the echo server--built using the SocketServer library module. It is
interesting to play around with the above benchmark script on coroutines and
then try it on a threaded server.
Design Commentary
One of the most tricky parts of working with a language feature like
coroutines is figuring out how they should interact with other
program elements such as functions and classes. If you look at various
libraries and frameworks, you often find that they all vary slightly in
how they address this issue.
The design of the task scheduler I wrote for this class is strongly
biased towards my prior experience teaching courses on Operating
System design. One of the most critical parts of writing an OS is
both protecting the system and keeping user applications isolated. In
the sample code, the Scheduler class represents a kind of
"operating system." You will notice that the tasks defined in this
course never directly interact with this scheduler (other than using
yield to execute scheduler traps). That is, they do not hold
references to the scheduler object, they do not invoke methods on the
scheduler, they do not inspect internal scheduler data, and they don't
hold references to other tasks. For all practical purposes, the
scheduler and the tasks are two completely different execution
domains. There is a good reason for keeping this separation. Namely
it promotes a loose coupling between tasks and their execution
environment. One could imagine creating other kinds of task
schedulers that run tasks within threads or subprocesses. If you've
set things up right and taken great care to promote the separation of
tasks and scheduling, such schedulers would be able to run existing
tasks without modification. Of course, the devil is in the details
:-).
Contact me
Concurrency is a topic that generally interests me. I welcome all
feedback, comments, and suggestions for improvement on this course
material. Please feel free to contact me by sending an email to
"dave" at "dabeaz.com".
|