inportb

I got a bit closer to integrating Twisted with gevent. So far, I’m liking gevent’s speed and blocking style, but it does lack Twisted’s extensive library. Well, this new Twisted reactor runs on gevent and makes it a bit painful to use both frameworks together.

Here is where the magic happens.

Note that this code is for using Twisted with gevent (i.e. libevent and greenlet). For using Twisted with greenlet alone, there is corotwine. Twisted can also be used with Eventlet.

Now, Twisted is a pretty awesome framework. It’s not only a networking library, but it’s a framework that provides all sorts of functionality. There’s also plenty of example code, and many things are already implemented. It is somewhat slow, however, and it likes to take control of the whole program. Keeping track of all the deferreds is not really a problem for me, but greenlet’s blocking syntax is quite nice.

If you haven’t seen corotwine yet, I’d urge you to have a look. It lets you use Twisted with a blocking syntax. It’s not quite what I wanted, but it does what it says on the tin. I wanted to use Twisted with Twisted syntax, greenlet with greenlet syntax, and maybe even let the Twisted bits talk with the greenlet bits. It is apparently quite easy with gevent.

The first thing you want to do is monkey-patch select.select. This is the method that Twisted uses to wait for file descriptors, and we can let gevent handle that with libevent and greenlet. Next, you run reactor.run either in the main greenlet or in a new greenlet.

As long as Twisted and greenlet code stay separate, everything is peachy. There are some problems when they start mixing. Because the Twisted reactor runs in a single greenlet, calling gevent.sleep (or doing anything else that involves a greenlet switch) actually blocks all of Twisted. Long-running jobs could be done in new greenlets, though. And if another greenlet wants to work with a Twisted transport (i.e. to send or receive data), it’s not going to happen until gevent sees some network activity destined for Twisted and switches to its reactor. These are the same problems you’d see with corotwine, and I suspect they might be alleviated with a greenlet-aware reactor.

Update: gTwist, the original code for gevent-Twisted integration, has been replaced with a more robust system called geventreactor. As its name implies, geventreactor is a gevent-powered Twisted reactor.