http://www.tomaz.me

Hosting APT repository on Rackspace CloudFiles

2012-07-22T00:00:00+02:00

Hosting APT repository on Rackspace CloudFiles

In this post I will describe how to host APT repository on Rackspace Cloud Files.

First you need to create a CDN enabled container. You can do this using Libcloud or the new Rackspace control panel. Depending on how frequently you are going update your APT repository it is also recommended to lower the container TTL from the default value of 3 days to 900 seconds.

After you have created a CDN enabled container you need to install Python script I wrote which allows you to sychronize files from a local directory to a container hosted in one of the Storage providers supported by Libcloud.

  sudo pip install file-syncer

Next we are going to create a dummy APT repository::

mkdir -p /tmp/apt-test/conf
cat > /tmp/apt-test/conf/distributions <<DELIM
codename: precise
Components: main
Architectures: i386 amd64
DELIM

Add a test package using reprepro:

wget http://us.archive.ubuntu.com/ubuntu/ubuntu/ubuntu/pool/main/a/acpi/acpi_0.09-3ubuntu1_amd64.deb
reprepro -b /tmp/apt-test/ includedeb precise acpi_0.09-3ubuntu1_amd64.deb

And test the script:

file-syncer --log-level=DEBUG --directory=/tmp/apt-test/ --username=your username --key=your api key --container-name=container name

If everything went well, you should see an output similar to this one:

22 Jul 2012 22:36:05 : INFO     : Using provider: CloudFiles (US)
22 Jul 2012 22:36:07 : DEBUG    : Found 15 local files
22 Jul 2012 22:36:08 : DEBUG    : Manifest doesn't exist, assuming that there are no remote files
22 Jul 2012 22:36:08 : DEBUG    : Found 0 remote files
22 Jul 2012 22:36:08 : INFO     : To remove: 0, to upload: 15
22 Jul 2012 22:36:08 : DEBUG    : Uploading object: db/contents.cache.db
22 Jul 2012 22:36:08 : DEBUG    : Uploading object: dists/precise/main/binary-amd64/Packages
22 Jul 2012 22:36:08 : DEBUG    : Uploading object: dists/precise/main/binary-i386/Release
22 Jul 2012 22:36:08 : DEBUG    : Uploading object: dists/precise/main/binary-i386/Packages
22 Jul 2012 22:36:08 : DEBUG    : Uploading object: db/packages.db
22 Jul 2012 22:36:08 : DEBUG    : Uploading object: pool/main/a/acpi/acpi_0.09-3ubuntu1_amd64.deb
22 Jul 2012 22:36:08 : DEBUG    : Uploading object: db/release.caches.db
22 Jul 2012 22:36:08 : DEBUG    : Uploading object: db/version
22 Jul 2012 22:36:08 : DEBUG    : Uploading object: conf/distributions
22 Jul 2012 22:36:08 : DEBUG    : Uploading object: db/references.db
22 Jul 2012 22:36:09 : DEBUG    : Object uploaded: dists/precise/main/binary-amd64/Packages
22 Jul 2012 22:36:09 : DEBUG    : Uploading object: dists/precise/main/binary-i386/Packages.gz
22 Jul 2012 22:36:09 : DEBUG    : Object uploaded: dists/precise/main/binary-i386/Packages
22 Jul 2012 22:36:09 : DEBUG    : Uploading object: dists/precise/main/binary-amd64/Packages.gz
22 Jul 2012 22:36:09 : DEBUG    : Object uploaded: db/release.caches.db
22 Jul 2012 22:36:09 : DEBUG    : Uploading object: db/checksums.db
22 Jul 2012 22:36:09 : DEBUG    : Object uploaded: dists/precise/main/binary-i386/Release
22 Jul 2012 22:36:09 : DEBUG    : Uploading object: dists/precise/main/binary-amd64/Release
22 Jul 2012 22:36:09 : DEBUG    : Object uploaded: conf/distributions
22 Jul 2012 22:36:09 : DEBUG    : Uploading object: dists/precise/Release
22 Jul 2012 22:36:09 : DEBUG    : Object uploaded: db/packages.db
22 Jul 2012 22:36:09 : DEBUG    : Object uploaded: db/contents.cache.db
22 Jul 2012 22:36:09 : DEBUG    : Object uploaded: db/references.db
22 Jul 2012 22:36:09 : DEBUG    : Object uploaded: db/version
22 Jul 2012 22:36:09 : DEBUG    : Object uploaded: pool/main/a/acpi/acpi_0.09-3ubuntu1_amd64.deb
22 Jul 2012 22:36:10 : DEBUG    : Object uploaded: dists/precise/main/binary-amd64/Packages.gz
22 Jul 2012 22:36:10 : DEBUG    : Object uploaded: dists/precise/main/binary-i386/Packages.gz
22 Jul 2012 22:36:10 : DEBUG    : Object uploaded: db/checksums.db
22 Jul 2012 22:36:10 : DEBUG    : Object uploaded: dists/precise/Release
22 Jul 2012 22:36:10 : DEBUG    : Object uploaded: dists/precise/main/binary-amd64/Release
22 Jul 2012 22:36:11 : INFO     : Synchronization complete, took: 4.62 seconds

After the upload has finished, you can test the repository by adding it to your APT sources list:

  echo "deb http://c15173579.r79.cf2.rackcdn.com precise main" | sudo tee -a /etc/apt/sources.list.d/test-api-repo.list

We are also going to disable APT HTTP request pipeling. This step is required, because a nginx proxy running in front of the CDN doesn’t seem to support HTTP/1.1 pipelining. If you skip this test apt-get update will still work, but it will get stuck on [Waiting for headers] for a longer period of time.

  echo "Acquire::http::Pipeline-Depth "0";" | sudo tee -a /etc/apt/apt.conf.d/pipeline-workaround.conf

You can test that everything is working by issuing the following command:

  sudo apt-get update ; sudo apt-cache policy acpi

You should get an output similar to this one:

acpi:
  Installed: (none)
  Candidate: 1.6-1
  Version table:
     1.6-1 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise/universe amd64 Packages
     0.09-3ubuntu1 0
        500 http://c15173579.r79.cf2.rackcdn.com/ precise/main amd64 Packages

That is it. To make sure your container is always up to date, you need synchronize it every time you add a new packages. This can be achieved in multiple ways:

adding a script which runs synchronize command after adding a package
adding a cron job which periodically runs the sync command

Libcloud Update (April 2012) - 0.8.0, 0.9.1, GSoC

2012-04-28T00:00:00+02:00

Libcloud Update (April 2012) - 0.8.0, 0.9.1, GSoC

Welcome to Libcloud April 2012 update post. Sorry for skipping a couple of months, but I have been pretty busy again. In any case, here is a short but sweet April update.

What has been accomplished in the past few months

Libcloud 0.8.1 with support for compressed responses (gzip, deflate) has been released.
Libcloud 0.9.1 with improvements in deploy functionality and OpenStack driver has been released.
I have attended PyCon US 2012 in Santa Clara in March. As always, it was a lot of fun talking with old and meeting new friends. This year we didn’t hold a Libcloud development sprint, but I still managed to talk with some Libcloud users and prompted Libcloud at the AWS open space session.
Libcloud has applied to Google Summer of Code 2012 under Apache organization. We have received 1 slot. Student Ilgiz Islamgulov will be working on the Libcloud REST interface this summer.

Dancing robots at PyCon

What is currently going on

We are currently moving towards 1.0 release which means mostly polishing the code, fixing bugs and avoiding big and API breaking changes. A lot of code, especially compute API has already been battle tested, but there are still some parts which I want to see improved (deployment functionality for example) before releasing 1.0.

Libcloud Monthly Update (December 2011) - 0.6.1, 0.6.2, 0.7.1, Python 3

2011-12-07T00:00:00+01:00

Libcloud Monthly Update (December 2011) - 0.6.1, 0.6.2, 0.7.1, Python 3

Past few months have been pretty busy here and you may have noticed that I didn’t post a monthly update post in the last two months. This doesn’t mean nothing has been going on though! In fact, quite the contrary. Last few months have been very busy and we have shipped two new releases and a voting thread for a new 0.7.1 release was just started a few days ago. More about that bellow.

What has been accomplished in the past two months

Libcloud 0.6.1 with a brand new DNS API (among many other improvements and additions) has been released.
A few days after releasing 0.6.1 we have also released 0.6.2 which was primary a bug-fix release, but it also includes some new features and improvements such as support for OpenStack Auth 2.0 API, support for new Amazon location (Oregon) and a CloudStack driver.
A few weeks ago a new committer, Hutson Betts (hbetts) has joined our team. He has previously mostly contributed to the OpenNebula driver and we believe giving him commit access will allow him to contribute more directly and easily and is a good thing for the whole project.

What is currently going on

A lot of things!
This weekend I have just finished adding support for Python 3 to Libcloud. Now we have a single code base which supports both, Python 2 and 3 and as far as I know we are the first Python cloud library which does that. I hope other projects will follow. If you are interested in more details you can read my post titled Lessons learned while porting Libcloud to Python 3 where I have described some of the issues which I have encountered while porting the library. Armin has recently also wrote a good blog post about Python 3 which you should read - Thoughts on Python 3.
I am moving to San Francisco where I will work full time for Rackspace. I will try to organize a Bay Area Libcloud meet-up in the upcoming weeks (hey, free pizza and beer!). I will post more info on the website and mailing list when all the details are fleshed out.

Lessons learned while porting Libcloud to Python 3

2011-12-03T00:00:00+01:00

Lessons learned while porting Libcloud to Python 3

Yesterday after seeing and being inspired by the Django Python 3 port news, I have decided it’s finally time to port Libcloud to Python 3. There have already been some talks about doing that in the past, but nobody actually managed to make a lot of progress.

In general, our goal is pretty similar to the Django one - have a single code base which works with Python 2.5, 2.6, 2.7 / PyPy and Python 3.

Alternative approach to having a single code base is using a tool like 2to3 to automatically convert 2.x version to the 3.x one or having multiple code bases / branches - one for 2.x and one for 3.x.

Early on when we talked about porting to Python 3, we have decided that we will go with a “single code base” approach. This approach allows us to keep a fast development pace and it’s also more friendlier for our users.

In this post I will describe some of the issues which I have encountered while porting the library and how I have solved them.

1. Handling renamed libraries and moved functionality

httplib

In Python 3 httplib has been renamed to http.client. To solve this problem, I have used an aliased import - import http.client as httplib.

urllib & urllib2

All of the functionality from urllib2 has been merged to urllib. This problem can also be easily solved using an aliased import - import urllib as urllib2.

urlparse

Functionality from urlparse has been moved to urllib.parse. We only use two functions from this module (quote and urlencode) so simple aliased import did the trick:

from urllib.parse import quote as urlquote
from urllib.parse import urlencode as urlencode

xmlrpclib

xmlrpclib has been moved to xmlrpc.client. Simple aliased import also solved this problem - import xmlrpc.client as xmlrpclib.

StringIO

StringIO has also been moved. from io import StringIO did the trick.

2. `file` type and file-like objects

file type has been removed in Python 3. To resolve this problem, I have used code similar to the one bellow in the places where we use file type.

if PY3:
    from io import FileIO as file

class MyFileLikeObject(file):
    ...

3. Generators and `.next()` method.

For consistency with other magic methods, next method in Python 3 has been renamed to __next__. To make it work with all the versions, I have used built-in next function in Python >= 3 and object .next() method in older versions.

if sys.version_info >= (3, 0):
    next = __builtins__['next']
else:
    def next(i):
        return i.next()

4. Exception handling

Sadly, there is no unified way to handle exceptions and extract the exception object in Python 2.5 and Python 3.x. This means I needed to use a hacky sys.exc_info()[1] approach to extract the raised exception

Old code:

try:
    foo
except Exception, e:
    print e

New code:

try:
    foo
except Exception:
    e = sys.exc_info()[1]
    print e

One of the PyPy developers has posted on reddit that this approach is very slow in PyPy. Luckily, besides the tests, there aren’t many places in our code where we need access to the exception object so this should be a good compromise for now.

5. filter, map, dict.keys()

In Python 2 those functions return a list, but in Python 3 they return a special object. Compatibility can be preserved by casting a result from this function to a list - e.g. list(filter(lamba x: x.name == 'test', nodes)).

6. iteritems, xrange

In Python 3, iteritems method has been removed and functionality from xrange has been merged into range. I have simplify replaced iteritems with items and xrange with range. We never used xrange with a lot of values so storing a whole list in memory in Python 2.x shouldn’t be a huge deal.

7. `xml.etree.ElementTree.tostring` and encoding

In Python 3 this method returns bytes by default. To preserve the old behavior and get a string back, I have used a code similar to one bellow:

if PY3:
    encoding = 'unicode'
else:
    encoding = None

data = tostring(root, encoding=encoding)

8. encode(‘hex’)

We had multiple places in the code where we did something like this:

value = os.urandom(8).encode('hex')

Hex encoding has been removed from Python 3. I have preserved backward compatibility by using binascii module:

value = binascii.hexlify(os.urandom(8))

9. Octal numbers

In Python 3 there is a special backward-incompatible (and strange) syntax for octal numbers - e.g. 0o755. We only use octal number in one place and this has been easily resolved by using int to convert a string to a number with base 8 - int('755', 8).

Those are just some of the issues I have encountered during porting. If you want to view all of the issues and how I have resolved them, you can see a full diff here.

Overall, I’m pretty satisfied with the outcome. I have managed to keep most of the Python 2 and Python 3 compatibility code in a single module (libcloud.py3) and it probably took me less then 5 hours to do the whole port including the research.

Bellow you can also find some links which I have found helpful while porting the code:

Whiskey Node.js test runner now with more goodness - introducing process runner

2011-11-27T00:00:00+01:00

First a short-introduction for people who aren’t familiar with Whiskey.

Whiskey is a powerful test runner for Node.js applications. It supports async testing, code coverage, scope leaks reporting, Makefile generation, test timing and lot more. Be sure to check out the github page which lists all the features.

New version (0.6.0) which has been released today includes a process runner and a support for managing external test dependencies. Test dependency is any kind of process on which the (integration) tests depend on.

Examples include, but are not limited to:

database,
some kind of api server,
web server,
other external services

Process runner is configured using a simple JSON configuration file. Most of the options have sane default values, which means if you don’t have any special requirements you can configure it very quickly.

Example configuration file which we use for our monitoring system integration test suite at Rackspace can be found here.

Each process can also specify its dependencies in the depends option which allows Whiskey to start unrelated processes concurrently.

Before Whiskey process runner was available we have been using scons for managing and running all the test dependencies. Test dependencies related section in our SConstruct file was long and hard to maintain which means switching to Whiskey process runner was a nice improvement.

Process runner can be used by passing --dependencies <configuration_file_path.json> option to whiskey binary. By default all the dependencies specified in the configuration file are started, but there is also --only-esential-dependencies option available which will make Whiskey first inspect the test files and only start the processes which are required by the tests which will be ran.

Each test file can specify on which processes it depends by exporting dependencies attribute. This attribute must be an array and contain the names of the processes as defined in the configuration file.

If you have any questions or suggestions you can find me on #Node.js IRC channel on freenode (nick Kami_). If you find a bug or a problem you can also open an ticket on the project issue tracker.

libcloud Monthly Update (September 2011) - FLOSS weekly, OpenStack driver improvements, DNS API

2011-09-24T00:00:00+02:00

libcloud Monthly Update (September 2011) - FLOSS weekly, OpenStack driver improvements, DNS API

Without further ado here is a Libcloud monthly update for September 2011.

What has been accomplished in the past month

I was a guest on FLOSS weekly (podcast about FOSS software) where I talked about Libcloud. You can find video and audio recording of the show on twit.tv.
OpenStack and Rackspace drivers have received a lot of needed attention and refactoring. Rackspace driver now properly inherits from the OpenStack one instead of vice versa (thanks Mike). This will make extending the Rackspace driver and developing other provider drivers which are based on OpenStack a lot easier. Rackspace drivers now also support authentication with Rackspace Auth 1.1.
Linode compute driver now supports new location in Japan.

Linode has recently added a new location (Tokyo, Japan) and this location is now also supported in Libcloud.
DNS API development has finally started.

Base API proposal can be found here. I have also just finished a reference implementation and a first driver for the Linode DNS as a service. The driver can be found in trunk. Feedback is welcome (and encouraged).

What is currently going on

Hacking on the DNS API continues. DNS API with at least two drivers is planned to be included in the next release (0.6.0) which should be out around November.

See you next month!

libcloud Monthly Update (August 2011) - 0.5.2 release, EuroPython 2011 recap, FLOSS weekly

2011-08-24T00:00:00+02:00

libcloud Monthly Update (August 2011) - 0.5.2 release, EuroPython 2011 recap, FLOSS weekly

As you might have noticed, I didn’t write a monthly update post in the last two months. The reason for that is that I have been busy with work and finishing my thesis for a bachelor degree.

In this post I will try to sum up what has happened during this time and what is currently going on.

What has happened in the past two months

Libcloud 0.5.2 has been released.

This is a primary a bug-fix release, but it also includes two new compute drivers for serverlove.com and skalicloud.com. Full changelog and release announcement can be found here.
EuroPython 2011 sprint was a success.

At a peek time we had a total of 6 sprinters :) A lot of stuff which has been accomplished at the sprint has been integrated into 0.5.2 release. You can also find slides and recording of my Libcloud talk on EuroPython 2011 website.
I have finally met another Libcloud developer in real-life.

This time I have met Roman a.k.a. mirrorbox in San Francisco. Most of our conversation was not Libcloud related, but we managed to talk a bit about the Libcloud future and topics such as DNS support, problems with our current blocking-model and asynchronous API’s and possible support for event-based libraries such as Twisted and gevent.

What is currently going on

Me and ~~Paul Querna~~ (Paul Querna can’t make it since he is away on 7th) will be guests on the FLOSS weekly podcast so don’t forget to tune-in on September 7, 2011 at 09:30 AM US Pacific Time. Show is streamed live, but as always recording will also be available a day or two later.

libcloud Monthly Update (May 2011) - 0.5.0 Release, Graduation, EuroPython

2011-05-26T00:00:00+02:00

libcloud Monthly Update (May 2011) - 0.5.0 Release, Graduation, EuroPython

May has been a very busy and important month for us. We have finally manged to finalize and release a long awaited version 0.5.0.

Part of the reason that we have finally managed to release 0.5.0 this month was that me and Paul Querna were present at the Apache Retreat in Ireland where we have spent some time hacking on Libcloud and polishing the last few features which were missing for the release.

Overall, Libcloud 0.5.0 is a big step forward and represents a big milestone for the project. It includes many new features, improvements and new compute drivers.

Major changes in Libcloud 0.5.0

New cloud Storage API

Version 0.5.0 includes a new storage API which allows you to manage services such as Amazon S3 and Rackspace CloudFiles. Our main priority for this release was defining a good base API and this is also the reason why this release only includes two provider drivers.

New Load-balancer API

Beside cloud storage we have also added a new API for managing load-balancers (LbaaS). Similar to the storage we were also focusing on defining a good base API so this release only includes Rackspace and GoGrid driver.

Changes in the existing API

To support the new APIs and services we had to refactor the existing API which means that all of the “compute” functionality has been moved to libcloud.compute.

Old module locations (libcloud.deployment, libcloud.providers, libcloud.types, etc.) have been deprecated and will be fully removed in version 0.6.0. At the moment you can still use them, but importing something from the old location will emit a deprecation warning so we encourage our users to update their code to use the new module locations.

New compute drivers

Among other changes and improvements, this release also includes 5 new compute drivers:

Bluebox (contributed by Christian Paredes)
Gandi.net (contributed by Aymeric Barantal)
Nimbus (contributed by David LaBissoniere)
OpenStack (contributed by Roman Bogorodskiy)
Opsource.net cloud (contributed by Joe Miller)

Full release announcement can be found on the mailing list.

Graduation to a Top Level Project

Second very important milestone for us this month was graduating from the Apache Incubator to a Top Level Project. This puts us on par with other Apache projects such as Apache Cassandra and Apache Subversion.

Graduation signifies that both the Apache Libcloud product and community have been well-governed under the Foundation’s meritocratic, consensus-driven process and principles.

Graduating to a Top Level Project means that now we have a Project Management Committee (PMC) which will overlook our operations and make sure everything is running smoothly.

To graduate we also had to select a project chair. Other members have proposed me for this role and I have accepted it. My primary role as a project chair will be to communicate with the board (quarterly status reports, etc.) and making sure there aren’t any conflicts and the project is running smoothly.

As part of graduation we also had to move our website and SVN repositories. You can find all the new address in this thread on the mailing list.

Official graduation announcement / press release can be found on the Apache blog - The Apache Software Foundation Announces Apache Libcloud as a Top-Level Project.

Libcloud at EuroPython 2011 in Florence, Italy

I will be at EuroPython in Italy next month where I will give an introductory talk about Libcloud.

Beside giving a talk we will also host a development sprint there. This is a great opportunity for anyone who wants to contribute to the project or learn something new to join us. I will post more details about the sprint in the upcoming weeks on the libcloud mailing list.

So that is it for May. See you next month when I will hopefully be able to report about multiple new storage and compute drivers and other improvements.

libcloud Monthly Update (April 2011) - Storage API, load-balancer API, new drivers

2011-04-26T00:00:00+02:00

libcloud Monthly Update (April 2011) - Storage API, load-balancer API, new drivers

Another month is around and it is time for another libcloud monthly update post. I did not write one previous month, because I have written ”PyCon US 2011 Recap” post which also includes information about libcloud development which has happened during PyCon and in March.

What has been accomplished in April 2011

libcloud website has been ported to the Apache CMS. Now adding new and editing existing content should be a lot easier.
Extension method for modifying the instance attributes and changing the instance size has been added to the Amazon compute driver - r1084180
Gandi.net compute driver has been contributed by Aymeric Barantal - LIBCLOUD-76
CloudFiles storage driver and the base storage class have undergone a lot of improvements
CloudFiles storage driver code coverage has been increased for ~20% (from ~70% to ~89)
A new Amazon S3 storage driver has been committed intro trunk - s3.py
OpSource compute driver has been contributed by Joe Miller - LIBCLOUD-77
Work has started on the load-balancer API and a reference drivers for GoGrid and Rackspace have already been committed into trunk. r1095180
Community resources section has been added to the website. This section contains links to different articles, tutorials and presentations produced by the libcloud developers and users.

What is currently going on

Jeremy Whitlock is working on the libvirt driver. If you have any feedback or suggestions for this driver, please share it with others on the mailing list.
Work continues on the storage and the load-balancer API. Only some minor changes are still needed for the storage API to be considered stable enough so libcloud 0.5.0 can be released.

Misc

Roman (the main author of the libcloud load-balancer API) has written a blog post which describes some differences and caveats which you can encounter while working with the Rackspace and GoGrid load-balancer API so be sure to check it out - Overview of GoGrid and Rackspace Load Balancing Services.

As you can see, April was pretty busy and a lot of new stuff has been committed into trunk.

Storage API will hopefully be finished soon and one of the highlights of the next month update post will be a long-awaited libcloud 0.5.0 release :)

Making Django and Twisted tests faster

2011-04-03T00:00:00+02:00

Making Django and Twisted tests faster

As we all know writing tests is not particularly fun and having a test suite which takes a long time to complete makes everything even less enjoyable.

Currently at Cloudkick our test suite is not particularly large or slow, but I still wanted the tests to finish faster (when dealing with tests, every minute counts).

We have two types of tests - Django and Twisted tests.

One obvious approach to speed the tests up is to run them in parallel.

A similar solution which runs the tests in parallel already exists - multiprocessing plugin for the nose test runner. The main problem with this plugin is, that it is pretty useless where a lot of tests depend on each other. Even when I have defined all the dependencies properly, the tests were still around 60% slower.

In the end, I have decided to write a custom Django and Twisted test runner which runs the tests in parallel.

Keep in mind that even before writing a custom test runner we have used a “trick” which makes the tests run faster - MySQL data directory on our continuous integration server is stored on a ram disk.

Basically our new Django test runner works like this:

Create a pool of worker processes
Partition the tests so we can run each application tests in a separate worker process
Put pending tests in the pending_tests queue
Each worker waits for new a new item to appear in this queue and when available, runs the tests
When a worker has finished running the tests, results are formatted, pickled and put in a separate tests_results queue
Main process periodically checks for new items in the tests_results queue and prints the results when they are available

The approach sounds pretty simple, but there are some caveats:

because multiple processes run in parallel this means that the test output will get interleaved. The solution is to buffer each worker output and finally print it out in the main process after the worker has finished running the tests. The problem with this approach is that the output is not real-time, but it should work fine for most of the cases. If we really wanted a real-time output, I could have used a lock, but this would just add additional complexity and slow things down.
partitioning the tests - Currently our partitioning / grouping approach is really simple, but it works well. Django tests are grouped by application and the Twisted tests are grouped by the test module. Before trying this really simple partitioning / grouping scheme I have experimented with more complicated approaches, but it turned out that in our case, simple approach is better.
creating a separate database for each worker. Most of our Django tests manipulate the state in the database so a reasonable solution is to create a separate database for each worker. To make this work I had to override the setup_databases() function defined in the Django test runner class.

Because this function has changed in Django 1.3 I also had to create two separate versions - one for Django 1.2 and one for Django 1.3.

...
def setup_databases(self, **kwargs):
  if VERSION[0] == 1:
    if VERSION[1] == 2 and VERSION[2] < 4:
      return self.setup_databases_12(**kwargs)
    elif VERSION[2] >= 4 or VERSION[1] == 3:
      return self.setup_databases_13(**kwargs)

  raise Exception('Unsupported Django Version: %s' % (str(VERSION)))

def setup_databases_12(self, **kwargs):
  # Taken from django.test.simple
  old_names = []
  mirrors = []

  worker_index = kwargs.get('worker_index', None)
  for alias in connections:
    connection = connections[alias]
    database_name = 'test_%d_%s' % (worker_index, connection.settings_dict['NAME'])
    connection.settings_dict['TEST_NAME'] = database_name
    if connection.settings_dict['TEST_MIRROR']:
      mirrors.append((alias, connection))
      mirror_alias = connection.settings_dict['TEST_MIRROR']
      connections._connections[alias] = connections[mirror_alias]
    else:
      old_names.append((connection, connection.settings_dict['NAME']))
      connection.creation.create_test_db(verbosity=0, autoclobber=not self.interactive)
  return old_names, mirrors

def setup_databases_13(self, **kwargs):
  # Taken from django.test.simple
  from django.test.simple import dependency_ordered

  mirrored_aliases = {}
  test_databases = {}
  dependencies = {}

  worker_index = kwargs.get('worker_index', None)
  for alias in connections:
    connection = connections[alias]
    database_name = 'test_%d_%s' % (worker_index, connection.settings_dict['NAME'])
    connection.settings_dict['TEST_NAME'] = database_name

    item = test_databases.setdefault(
        connection.creation.test_db_signature(),
        (connection.settings_dict['NAME'], [])
    )
    item[1].append(alias)
    if alias != DEFAULT_DB_ALIAS:
      dependencies[alias] = connection.settings_dict.get('TEST_DEPENDENCIES', [DEFAULT_DB_ALIAS])

  old_names = []
  mirrors = []
  for signature, (db_name, aliases) in dependency_ordered(test_databases.items(), dependencies):
      connection = connections[aliases[0]]
      old_names.append((connection, db_name, True))
      test_db_name = connection.creation.create_test_db(verbosity=0, autoclobber=not self.interactive)
      for alias in aliases[1:]:
          connection = connections[alias]
          if db_name:
            old_names.append((connection, db_name, False))
            connection.settings_dict['NAME'] = test_db_name
          else:
            old_names.append((connection, db_name, True))
            connection.creation.create_test_db(verbosity=0, autoclobber=not self.interactive)

  for alias, mirror_alias in mirrored_aliases.items():
    mirrors.append((alias, connections[alias].settings_dict['NAME']))
    connections[alias].settings_dict['NAME'] = connections[mirror_alias].settings_dict['NAME']

  return old_names, mirrors
...

I have also used a very similar approach for the Twisted parallel test runner.

As a first thing, I have created a special base class which works similar as the Django TestCase class - it disables database transactions in setUp() and does a database roll-back in the tearDown() method (database rollback is a lot faster than re-creating all the tables).

For the Twisted runner to buffer the test output, I had to modify the trial _makeRunner function and pass it in a custom stream object.

class BufferWritesDevice(object):

  def __init__(self):
    self._data = []

  def write(self, string):
    self._data.append(string)

  def read(self):
    return ''.join(self._data)

  def flush(self, *args, **kwargs):
    pass

  def isatty(self):
    return False

....

def _tests_func(self, tests, worker_index):
  if not isinstance(tests, (list, set)):
    tests = [ tests ]

  args = [ '-e' ]
  args.extend(tests)

  config = Options()
  config.parseOptions(args)

  stream = BufferWritesDevice()
  runner = self._make_runner(config=config, stream=stream)
  suite = _getSuite(config)
  result = setup_test_db(worker_index, None, runner.run, suite)
  result = TestResult().from_trial_result(result)
  return result

...

def _make_runner(self, config, stream):
  # Based on twisted.scripts.trial._makeRunner
  mode = None
  if config['debug']:
      mode = TrialRunner.DEBUG
  if config['dry-run']:
      mode = TrialRunner.DRY_RUN
  return TrialRunner(config['reporter'],
                            mode=mode,
                            stream=stream,
                            profile=config['profile'],
                            logfile=config['logfile'],
                            tracebackFormat=config['tbformat'],
                            realTimeErrors=config['rterrors'],
                            uncleanWarnings=config['unclean-warnings'],
                            workingDirectory=config['temp-directory'],
                            forceGarbageCollection=config['force-gc'])

...

To make each worker use a separate database I also had to manually manipulate the connection settings_dict dictionary and adjust the value for the TEST_NAME item (I prepend worker index to each test database name).

There are still a lot of possible improvements left and some of them are already on my road-map.

Currently our number of tests and applications is not that high so it does not add much overhead to spawn a separate worker process for each application. Later on when our application number grows, it might make sense to use a smarter grouping method.

Because not all of the Django tests require access to the database, one obvious improvement would also be to spawn a separate worker process for those tests.

As I mentioned previously, our MySQL data directory is located on a ram disk so creating a database does not take that long, but every change which makes tests faster and is not too complex is worth considering.

In the end this modifications did take some time, but it was well wort it - both Django and Twisted tests now finish around 50% - 60% faster.

Update: Multiple people have asked me about the full runner source code so I have put it in Github - https://github.com/Kami/parallel-django-and-twisted-test-runner.

http://www.tomaz.me

Hosting APT repository on Rackspace CloudFiles

Libcloud Update (April 2012) - 0.8.0, 0.9.1, GSoC

What has been accomplished in the past few months

What is currently going on

Libcloud Monthly Update (December 2011) - 0.6.1, 0.6.2, 0.7.1, Python 3

What has been accomplished in the past two months

What is currently going on

Lessons learned while porting Libcloud to Python 3

1. Handling renamed libraries and moved functionality

httplib

urllib & urllib2

urlparse

xmlrpclib

StringIO

2. file type and file-like objects

3. Generators and .next() method.

4. Exception handling

5. filter, map, dict.keys()

6. iteritems, xrange

7. xml.etree.ElementTree.tostring and encoding

8. encode(‘hex’)

9. Octal numbers

Whiskey Node.js test runner now with more goodness - introducing process runner

libcloud Monthly Update (September 2011) - FLOSS weekly, OpenStack driver improvements, DNS API

What has been accomplished in the past month

What is currently going on

libcloud Monthly Update (August 2011) - 0.5.2 release, EuroPython 2011 recap, FLOSS weekly

What has happened in the past two months

What is currently going on

libcloud Monthly Update (May 2011) - 0.5.0 Release, Graduation, EuroPython

Major changes in Libcloud 0.5.0

New compute drivers

Graduation to a Top Level Project

Libcloud at EuroPython 2011 in Florence, Italy

libcloud Monthly Update (April 2011) - Storage API, load-balancer API, new drivers

What has been accomplished in April 2011

What is currently going on

Misc

Making Django and Twisted tests faster

2. `file` type and file-like objects

3. Generators and `.next()` method.

7. `xml.etree.ElementTree.tostring` and encoding