A Curious Course on Coroutines and Concurrency

Copyright (C) 2009, All Rights Reserved
David Beazley
http://www.dabeaz.com

Presented at PyCon 2009, March 25, 2009.

Introduction

This tutorial is a practical exploration of using Python coroutines (extended generators) for solving problems in data processing, event handling, and concurrent programming. The material starts off with generators and builds to writing a complete multitasking environment that can run thousands of concurrent tasks without using threads or using code based on event-driven callbacks (i.e., the "reactor" model).

• Presentation Slides (PDF)

Note: This tutorial might be viewed as a sequel to the tutorial Generator Tricks for System Programmers I presented at PyCon'08 in Chicago. If you have never used generator functions before, you might want to look at that presentation for more information. This coroutine tutorial is meant to stand on its own, but you'll get a more complete picture if you combine it with the generator presentation.

Requirements and Support Data Files

This tutorial requires the use of Python 2.5 or newer. No third party modules are required. Examples have been tested on both Unix and Windows XP. Examples will work on Python 3 as long as you fix all of the print statements.

The following file contains some supporting data files that are used by the various code samples. Download this to your machine to work with the examples that follow.

• coroutines.zip

This download also includes a PDF of the lecture slides.

Code Samples

Here are various code samples from the course. You can either cut and paste these from the browser or simply work with them directly in the "coroutines" directory. The order in which files are listed follow the course material. These examples are written to run inside the "coroutines" directory that gets created when you unzip the above file containing the support data.

Part 1 : Introduction to Generators and Coroutines

• countdown.py. A trivially simple generator function.

• follow.py. A generator that follows lines written to a real-time log file (like Unix 'tail -f'). To run this program, you need to have a log-file to work with. Run the program logsim.py to create a simulated web-server log (written in the file access-log). Leave this program running in the background for the next few parts.

• pipeline.py. An example of using generators to set up a simple processing pipeline. Print all server log entries containing the word 'python'.

• grep.py. A first example of a coroutine function. This function receives lines and prints out those that contain a substring.

• coroutine.py. A decorator function that eliminates the need to call .next() when starting a coroutine.

• grepclose.py. An example of a coroutine that catches the close() operation.

• bogus.py. An example of a bogus generator that generates and receives values (not a recommended coding style).

• cofollow.py. A simple example of feeding data from a data source into a coroutine. This mirrors the 'tail -f' example from earlier.

• copipe.py. An example of setting up a processing pipeline with coroutines.

• cobroadcast.py. An example of a coroutine broadcaster. This fans a data stream out to multiple targets.

• cobroadcast2.py. An example of broadcasting with a slightly different data handling pattern.

• benchmark.py. A small benchmark comparing the performance of sending data into a coroutine vs. sending data into an instance of a class.

• basicsax.py. A very basic example of the SAX API for parsing XML documents (does not involve coroutines).

• cosax.py. An example that pushes SAX events into a coroutine.

• buses.py. An example of parsing and filtering XML data with a series of connected coroutines.

• coexpat.py. An XML parser that turns events generated by the expat XML library into coroutines. Compare with cosax.py above.

• cxml/cxmlparse.c. A bare-bones C extension module that uses the expat library and pushes events into coroutines. To build this code on your own machine, you will need to first build and install expat and then use python setup.py build_ext --inplace. Note: This main focus of this class is not C extension building so you might have to mess around with this to get it to work. Key point: you can dispatch data into a coroutine directly from C.

• iterbus.py. An example of incremental XML parsing with the ElementTree module (for comparison with coroutines).

• cothread.py. A thread object that runs a coroutine.

• coprocess.py. An example of running a coroutine in a subprocess. This example runs a separate program busproc.py in a subprocess and sends data to it.

• cocrash.py. An example of crashing a thread by having it send data into an already executing coroutine. Since this example depends on thread synchronization, it may occasionally "work" by accident. Run it a few more times.

• pyos1.py. A simple object representing a "task." It is a wrapper around a coroutine.

• pyos2.py. A simple task scheduler that alternates between tasks whenever they yield.

• taskcrash.py. An example showing how the schedule crashes if one of the tasks terminates.

• pyos3.py. An improved scheduler that properly handles task termination.

• pyos4.py. A scheduler with support for "system calls."

• pyos5.py. A scheduler with system calls for basic task creation and termination.

• pyos6.py. A scheduler that adds support for task waiting.

• echobad.py. A broken example of trying to use coroutines to implement a multitasking network server. It breaks due to blocking I/O operations.

• pyos7.py. A scheduler that adds support for I/O waiting.

• echogood.py. A slightly modified version of the echo server above that properly handles blocking I/O.

• echogood2.py. An echo server without the "I'm alive" messages.

• trampoline.py. An example of "trampolining" between coroutines.

• pyos8.py. The OS with an enhanced Task object that allows coroutines to transfer control to other coroutines like subroutines.

• echoserver.py. A concurrent echo server using coroutines.

• sockwrap.py. An class wrapper that shows how you can emulate the socket interface with a collection of coroutine methods.

• echoserver2.py. An echo server using the class-based interface.

• twistedecho.py. A basic echo server written for Twisted. You will need to install Twisted to run this. One reason why I have included this here is that the low-level I/O handling is very similar. Specifically, in our "operating system", I/O events are tied into task scheduling. In Twisted, I/O events are tied into event handlers (Reactors).

• blaster.py. A very simple script that opens up 300 socket connections with an echo server and randomly blasts it with 1024 bytes messages. The program runblaster.py will run multiple copies of this program in different subprocesses to increase the load.

  Note: these scripts are the same ones I used to do a basic benchmark on Twisted vs. Coroutines described in the handout. The main goal of this class has not been performance measurement so you might want to play around with the scripts--changing different settings for the number of connections, number of processes, message size, etc. To allow a large number of socket connections, you might have to fiddle with system settings. For example, on my Mac, I first have to use 'ulimit -n 1024' in the shell to change the number of file descriptors on the server (otherwise, only 256 connections can be handled).

• echothread.py. A threaded implementation of the echo server--built using the SocketServer library module. It is interesting to play around with the above benchmark script on coroutines and then try it on a threaded server.

Design Commentary

One of the most tricky parts of working with a language feature like coroutines is figuring out how they should interact with other program elements such as functions and classes. If you look at various libraries and frameworks, you often find that they all vary slightly in how they address this issue.

The design of the task scheduler I wrote for this class is strongly biased towards my prior experience teaching courses on Operating System design. One of the most critical parts of writing an OS is both protecting the system and keeping user applications isolated. In the sample code, the Scheduler class represents a kind of "operating system." You will notice that the tasks defined in this course never directly interact with this scheduler (other than using yield to execute scheduler traps). That is, they do not hold references to the scheduler object, they do not invoke methods on the scheduler, they do not inspect internal scheduler data, and they don't hold references to other tasks. For all practical purposes, the scheduler and the tasks are two completely different execution domains. There is a good reason for keeping this separation. Namely it promotes a loose coupling between tasks and their execution environment. One could imagine creating other kinds of task schedulers that run tasks within threads or subprocesses. If you've set things up right and taken great care to promote the separation of tasks and scheduling, such schedulers would be able to run existing tasks without modification. Of course, the devil is in the details :-).

Contact me

Concurrency is a topic that generally interests me. I welcome all feedback, comments, and suggestions for improvement on this course material. Please feel free to contact me by sending an email to "dave" at "dabeaz.com".