Yesterday after seeing and being inspired by the Django Python 3 port news, I have decided it’s finally time to port Libcloud to Python 3. There have already been some talks about doing that in the past, but nobody actually managed to make a lot of progress.
In general, our goal is pretty similar to the Django one - have a single code base which works with Python 2.5, 2.6, 2.7 / PyPy and Python 3.
Alternative approach to having a single code base is using a tool like 2to3 to automatically convert 2.x version to the 3.x one or having multiple code bases / branches - one for 2.x and one for 3.x.
Early on when we talked about porting to Python 3, we have decided that we will go with a “single code base” approach. This approach allows us to keep a fast development pace and it’s also more friendlier for our users.
In this post I will describe some of the issues which I have encountered while porting the library and how I have solved them.
1. Handling renamed libraries and moved functionality
httplib
In Python 3 httplib has been renamed to http.client. To solve this problem, I have used an aliased import - import http.client as httplib.
urllib & urllib2
All of the functionality from urllib2 has been merged to urllib. This problem can also be easily solved using an aliased import - import urllib as urllib2.
urlparse
Functionality from urlparse has been moved to urllib.parse. We only use two functions from this module (quote and urlencode) so simple aliased import did the trick:
from urllib.parse import quote as urlquote
from urllib.parse import urlencode as urlencode
xmlrpclib
xmlrpclib has been moved to xmlrpc.client. Simple aliased import also solved this problem - import xmlrpc.client as xmlrpclib.
StringIO
StringIO has also been moved. from io import StringIO did the trick.
2. file type and file-like objects
file type has been removed in Python 3. To resolve this problem, I have used code similar to the one bellow in the places where we use file type.
if PY3:
from io import FileIO as file
class MyFileLikeObject(file):
...
3. Generators and .next() method.
For consistency with other magic methods, next method in Python 3 has been renamed to __next__. To make it work with all the versions, I have used built-in next function in Python >= 3 and object .next() method in older versions.
if sys.version_info >= (3, 0):
next = __builtins__['next']
else:
def next(i):
return i.next()
4. Exception handling
Sadly, there is no unified way to handle exceptions and extract the exception object in Python 2.5 and Python 3.x. This means I needed to use a hacky sys.exc_info()[1] approach to extract the raised exception
Old code:
try:
foo
except Exception, e:
print e
New code:
try:
foo
except Exception:
e = sys.exc_info()[1]
print e
One of the PyPy developers has posted on reddit that this approach is very slow in PyPy. Luckily, besides the tests, there aren’t many places in our code where we need access to the exception object so this should be a good compromise for now.
5. filter, map, dict.keys()
In Python 2 those functions return a list, but in Python 3 they return a special object. Compatibility can be preserved by casting a result from this function to a list - e.g. list(filter(lamba x: x.name == 'test', nodes)).
6. iteritems, xrange
In Python 3, iteritems method has been removed and functionality from xrange has been merged into range. I have simplify replaced iteritems with items and xrange with range. We never used xrange with a lot of values so storing a whole list in memory in Python 2.x shouldn’t be a huge deal.
7. xml.etree.ElementTree.tostring and encoding
In Python 3 this method returns bytes by default. To preserve the old behavior and get a string back, I have used a code similar to one bellow:
if PY3:
encoding = 'unicode'
else:
encoding = None
data = tostring(root, encoding=encoding)
8. encode(‘hex’)
We had multiple places in the code where we did something like this:
value = os.urandom(8).encode('hex')
Hex encoding has been removed from Python 3. I have preserved backward compatibility by using binascii module:
value = binascii.hexlify(os.urandom(8))
9. Octal numbers
In Python 3 there is a special backward-incompatible (and strange) syntax for octal numbers - e.g. 0o755. We only use octal number in one place and this has been easily resolved by using int to convert a string to a number with base 8 - int('755', 8).
Those are just some of the issues I have encountered during porting. If you want to view all of the issues and how I have resolved them, you can see a full diff here.
Overall, I’m pretty satisfied with the outcome. I have managed to keep most of the Python 2 and Python 3 compatibility code in a single module (libcloud.py3) and it probably took me less then 5 hours to do the whole port including the research.
Bellow you can also find some links which I have found helpful while porting the code:
» Dec 03, 2011
First a short-introduction for people who aren’t familiar with Whiskey.
Whiskey is a powerful test runner for Node.js applications. It supports async testing, code coverage, scope leaks reporting, Makefile generation, test timing and lot more. Be sure to check out the github page which lists all the features.
New version (0.6.0) which has been released today includes a process runner and a support for managing external test dependencies. Test dependency is any kind of process on which the (integration) tests depend on.
Examples include, but are not limited to:
- database,
- some kind of api server,
- web server,
- other external services
Process runner is configured using a simple JSON configuration file. Most of the options have sane default values, which means if you don’t have any special requirements you can configure it very quickly.
Example configuration file which we use for our monitoring system integration test suite at Rackspace can be found here.
Each process can also specify its dependencies in the depends option which allows Whiskey to start unrelated processes concurrently.
Before Whiskey process runner was available we have been using scons for managing and running all the test dependencies. Test dependencies related section in our SConstruct file was long and hard to maintain which means switching to Whiskey process runner was a nice improvement.
Process runner can be used by passing --dependencies <configuration_file_path.json> option to whiskey binary. By default all the dependencies specified in the configuration file are started, but there is also --only-esential-dependencies option available which will make Whiskey first inspect the test files and only start the processes which are required by the tests which will be ran.
Each test file can specify on which processes it depends by exporting dependencies attribute. This attribute must be an array and contain the names of the processes as defined in the configuration file.
If you have any questions or suggestions you can find me on #Node.js IRC channel on freenode (nick Kami_). If you find a bug or a problem you can also open an ticket on the project issue tracker.
» Nov 27, 2011
Without further ado here is a Libcloud monthly update for September 2011.
What has been accomplished in the past month
-
I was a guest on FLOSS weekly (podcast about FOSS software) where I talked about Libcloud. You can find video and audio recording of the show on twit.tv.
-
OpenStack and Rackspace drivers have received a lot of needed attention and refactoring. Rackspace driver now properly inherits from the OpenStack one instead of vice versa (thanks Mike). This will make extending the Rackspace driver and developing other provider drivers which are based on OpenStack a lot easier. Rackspace drivers now also support authentication with Rackspace Auth 1.1.
-
Linode compute driver now supports new location in Japan.
Linode has recently added a new location (Tokyo, Japan) and this location is now also supported in Libcloud.
-
DNS API development has finally started.
Base API proposal can be found here. I have also just finished a reference implementation and a first driver for the Linode DNS as a service. The driver can be found in trunk. Feedback is welcome (and encouraged).
What is currently going on
- Hacking on the DNS API continues. DNS API with at least two drivers is planned to be included in the next release (0.6.0) which should be out around November.
See you next month!
» Sep 24, 2011
As you might have noticed, I didn’t write a monthly update post in the last two months. The reason for that is that I have been busy with work and finishing my thesis for a bachelor degree.
In this post I will try to sum up what has happened during this time and what is currently going on.
What has happened in the past two months
-
Libcloud 0.5.2 has been released.
This is a primary a bug-fix release, but it also includes two new compute drivers for serverlove.com and skalicloud.com. Full changelog and release announcement can be found here.
-
EuroPython 2011 sprint was a success.
At a peek time we had a total of 6 sprinters :) A lot of stuff which has been accomplished at the sprint has been integrated into 0.5.2 release. You can also find slides and recording of my Libcloud talk on EuroPython 2011 website.
-
I have finally met another Libcloud developer in real-life.
This time I have met Roman a.k.a. mirrorbox in San Francisco. Most of our conversation was not Libcloud related, but we managed to talk a bit about the Libcloud future and topics such as DNS support, problems with our current blocking-model and asynchronous API’s and possible support for event-based libraries such as Twisted and gevent.
What is currently going on
- Me and
Paul Querna (Paul Querna can’t make it since he is away on 7th) will be guests on the FLOSS weekly podcast so don’t forget to tune-in on September 7, 2011 at 09:30 AM US Pacific Time. Show is streamed live, but as always recording will also be available a day or two later.
» Aug 24, 2011
May has been a very busy and important month for us. We have finally manged to finalize and release a long awaited version 0.5.0.
Part of the reason that we have finally managed to release 0.5.0 this month was that me and Paul Querna were present at the Apache Retreat in Ireland where we have spent some time hacking on Libcloud and polishing the last few features which were missing for the release.
Overall, Libcloud 0.5.0 is a big step forward and represents a big milestone for the project. It includes many new features, improvements and new compute drivers.
Major changes in Libcloud 0.5.0
New cloud Storage API
Version 0.5.0 includes a new storage API which allows you to manage services such as Amazon S3 and Rackspace CloudFiles. Our main priority for this release was defining a good base API and this is also the reason why this release only includes two provider drivers.
New Load-balancer API
Beside cloud storage we have also added a new API for managing load-balancers (LbaaS). Similar to the storage we were also focusing on defining a good base API so this release only includes Rackspace and GoGrid driver.
Changes in the existing API
To support the new APIs and services we had to refactor the existing API which means that all of the “compute” functionality has been moved to libcloud.compute.
Old module locations (libcloud.deployment, libcloud.providers, libcloud.types, etc.) have been deprecated and will be fully removed in version 0.6.0. At the moment you can still use them, but importing something from the old location will emit a deprecation warning so we encourage our users to update their code to use the new module locations.
New compute drivers
Among other changes and improvements, this release also includes 5 new compute drivers:
- Bluebox (contributed by Christian Paredes)
- Gandi.net (contributed by Aymeric Barantal)
- Nimbus (contributed by David LaBissoniere)
- OpenStack (contributed by Roman Bogorodskiy)
- Opsource.net cloud (contributed by Joe Miller)
Full release announcement can be found on the mailing list.
Graduation to a Top Level Project
Second very important milestone for us this month was graduating from the Apache Incubator to a Top Level Project. This puts us on par with other Apache projects such as Apache Cassandra and Apache Subversion.
Graduation signifies that both the Apache Libcloud product and community have been well-governed under the Foundation’s meritocratic, consensus-driven process and principles.
Graduating to a Top Level Project means that now we have a Project Management Committee (PMC) which will overlook our operations and make sure everything is running smoothly.
To graduate we also had to select a project chair. Other members have proposed me for this role and I have accepted it. My primary role as a project chair will be to communicate with the board (quarterly status reports, etc.) and making sure there aren’t any conflicts and the project is running smoothly.
As part of graduation we also had to move our website and SVN repositories. You can find all the new address in this thread on the mailing list.
Official graduation announcement / press release can be found on the Apache blog - The Apache Software Foundation Announces Apache Libcloud as a Top-Level Project.
Libcloud at EuroPython 2011 in Florence, Italy
I will be at EuroPython in Italy next month where I will give an introductory talk about Libcloud.
Beside giving a talk we will also host a development sprint there. This is a great opportunity for anyone who wants to contribute to the project or learn something new to join us. I will post more details about the sprint in the upcoming weeks on the libcloud mailing list.
So that is it for May. See you next month when I will hopefully be able to report about multiple new storage and compute drivers and other improvements.
» May 26, 2011
Another month is around and it is time for another libcloud monthly update post. I did not write one previous month, because I have written ”PyCon US 2011 Recap” post which also includes information about libcloud development which has happened during PyCon and in March.
What has been accomplished in April 2011
- libcloud website has been ported to the Apache CMS. Now adding new and editing existing content should be a lot easier.
- Extension method for modifying the instance attributes and changing the instance size has been added to the Amazon compute driver - r1084180
- Gandi.net compute driver has been contributed by Aymeric Barantal - LIBCLOUD-76
- CloudFiles storage driver and the base storage class have undergone a lot of improvements
- CloudFiles storage driver code coverage has been increased for ~20% (from ~70% to ~89)
- A new Amazon S3 storage driver has been committed intro trunk - s3.py
- OpSource compute driver has been contributed by Joe Miller - LIBCLOUD-77
- Work has started on the load-balancer API and a reference drivers for GoGrid and Rackspace have already been committed into trunk. r1095180
- Community resources section has been added to the website. This section contains links to different articles, tutorials and presentations produced by the libcloud developers and users.
What is currently going on
- Jeremy Whitlock is working on the libvirt driver. If you have any feedback or suggestions for this driver, please share it with others on the mailing list.
- Work continues on the storage and the load-balancer API. Only some minor changes are still needed for the storage API to be considered stable enough so libcloud 0.5.0 can be released.
Misc
- Roman (the main author of the libcloud load-balancer API) has written a blog post which describes some differences and caveats which you can encounter while working with the Rackspace and GoGrid load-balancer API so be sure to check it out - Overview of GoGrid and Rackspace Load Balancing Services.
As you can see, April was pretty busy and a lot of new stuff has been committed into trunk.
Storage API will hopefully be finished soon and one of the highlights of the next month update post will be a long-awaited libcloud 0.5.0 release :)
» Apr 26, 2011
As we all know writing tests is not particularly fun and having a test suite which takes a long time to complete makes everything even less enjoyable.
Currently at Cloudkick our test suite is not particularly large or slow, but I still wanted the tests to finish faster (when dealing with tests, every minute counts).
We have two types of tests - Django and Twisted tests.
One obvious approach to speed the tests up is to run them in parallel.
A similar solution which runs the tests in parallel already exists - multiprocessing plugin for the nose test runner. The main problem with this plugin is, that it is pretty useless where a lot of tests depend on each other. Even when I have defined all the dependencies properly, the tests were still around 60% slower.
In the end, I have decided to write a custom Django and Twisted test runner which runs the tests in parallel.
Keep in mind that even before writing a custom test runner we have used a “trick” which makes the tests run faster - MySQL data directory on our continuous integration server is stored on a ram disk.
Basically our new Django test runner works like this:
- Create a pool of worker processes
- Partition the tests so we can run each application tests in a separate worker process
- Put pending tests in the
pending_tests queue
- Each worker waits for new a new item to appear in this queue and when available, runs the tests
- When a worker has finished running the tests, results are formatted, pickled and put in a separate
tests_results queue
- Main process periodically checks for new items in the
tests_results queue and prints the results when they are available
The approach sounds pretty simple, but there are some caveats:
-
because multiple processes run in parallel this means that the test output will get interleaved. The solution is to buffer each worker output and finally print it out in the main process after the worker has finished running the tests. The problem with this approach is that the output is not real-time, but it should work fine for most of the cases. If we really wanted a real-time output, I could have used a lock, but this would just add additional complexity and slow things down.
-
partitioning the tests - Currently our partitioning / grouping approach is really simple, but it works well. Django tests are grouped by application and the Twisted tests are grouped by the test module. Before trying this really simple partitioning / grouping scheme I have experimented with more complicated approaches, but it turned out that in our case, simple approach is better.
-
creating a separate database for each worker. Most of our Django tests manipulate the state in the database so a reasonable solution is to create a separate database for each worker. To make this work I had to override the setup_databases() function defined in the Django test runner class.
Because this function has changed in Django 1.3 I also had to create two separate versions - one for Django 1.2 and one for Django 1.3.
...
def setup_databases(self, **kwargs):
if VERSION[0] == 1:
if VERSION[1] == 2 and VERSION[2] < 4:
return self.setup_databases_12(**kwargs)
elif VERSION[2] >= 4 or VERSION[1] == 3:
return self.setup_databases_13(**kwargs)
raise Exception('Unsupported Django Version: %s' % (str(VERSION)))
def setup_databases_12(self, **kwargs):
# Taken from django.test.simple
old_names = []
mirrors = []
worker_index = kwargs.get('worker_index', None)
for alias in connections:
connection = connections[alias]
database_name = 'test_%d_%s' % (worker_index, connection.settings_dict['NAME'])
connection.settings_dict['TEST_NAME'] = database_name
if connection.settings_dict['TEST_MIRROR']:
mirrors.append((alias, connection))
mirror_alias = connection.settings_dict['TEST_MIRROR']
connections._connections[alias] = connections[mirror_alias]
else:
old_names.append((connection, connection.settings_dict['NAME']))
connection.creation.create_test_db(verbosity=0, autoclobber=not self.interactive)
return old_names, mirrors
def setup_databases_13(self, **kwargs):
# Taken from django.test.simple
from django.test.simple import dependency_ordered
mirrored_aliases = {}
test_databases = {}
dependencies = {}
worker_index = kwargs.get('worker_index', None)
for alias in connections:
connection = connections[alias]
database_name = 'test_%d_%s' % (worker_index, connection.settings_dict['NAME'])
connection.settings_dict['TEST_NAME'] = database_name
item = test_databases.setdefault(
connection.creation.test_db_signature(),
(connection.settings_dict['NAME'], [])
)
item[1].append(alias)
if alias != DEFAULT_DB_ALIAS:
dependencies[alias] = connection.settings_dict.get('TEST_DEPENDENCIES', [DEFAULT_DB_ALIAS])
old_names = []
mirrors = []
for signature, (db_name, aliases) in dependency_ordered(test_databases.items(), dependencies):
connection = connections[aliases[0]]
old_names.append((connection, db_name, True))
test_db_name = connection.creation.create_test_db(verbosity=0, autoclobber=not self.interactive)
for alias in aliases[1:]:
connection = connections[alias]
if db_name:
old_names.append((connection, db_name, False))
connection.settings_dict['NAME'] = test_db_name
else:
old_names.append((connection, db_name, True))
connection.creation.create_test_db(verbosity=0, autoclobber=not self.interactive)
for alias, mirror_alias in mirrored_aliases.items():
mirrors.append((alias, connections[alias].settings_dict['NAME']))
connections[alias].settings_dict['NAME'] = connections[mirror_alias].settings_dict['NAME']
return old_names, mirrors
...
I have also used a very similar approach for the Twisted parallel test runner.
As a first thing, I have created a special base class which works similar as the Django TestCase class - it disables database transactions in setUp() and does a database roll-back in the tearDown() method (database rollback is a lot faster than re-creating all the tables).
For the Twisted runner to buffer the test output, I had to modify the trial _makeRunner function and pass it in a custom stream object.
class BufferWritesDevice(object):
def __init__(self):
self._data = []
def write(self, string):
self._data.append(string)
def read(self):
return ''.join(self._data)
def flush(self, *args, **kwargs):
pass
def isatty(self):
return False
....
def _tests_func(self, tests, worker_index):
if not isinstance(tests, (list, set)):
tests = [ tests ]
args = [ '-e' ]
args.extend(tests)
config = Options()
config.parseOptions(args)
stream = BufferWritesDevice()
runner = self._make_runner(config=config, stream=stream)
suite = _getSuite(config)
result = setup_test_db(worker_index, None, runner.run, suite)
result = TestResult().from_trial_result(result)
return result
...
def _make_runner(self, config, stream):
# Based on twisted.scripts.trial._makeRunner
mode = None
if config['debug']:
mode = TrialRunner.DEBUG
if config['dry-run']:
mode = TrialRunner.DRY_RUN
return TrialRunner(config['reporter'],
mode=mode,
stream=stream,
profile=config['profile'],
logfile=config['logfile'],
tracebackFormat=config['tbformat'],
realTimeErrors=config['rterrors'],
uncleanWarnings=config['unclean-warnings'],
workingDirectory=config['temp-directory'],
forceGarbageCollection=config['force-gc'])
...
To make each worker use a separate database I also had to manually manipulate the connection settings_dict dictionary and adjust the value for the TEST_NAME item (I prepend worker index to each test database name).
There are still a lot of possible improvements left and some of them are already on my road-map.
Currently our number of tests and applications is not that high so it does not add much overhead to spawn a separate worker process for each application. Later on when our application number grows, it might make sense to use a smarter grouping method.
Because not all of the Django tests require access to the database, one obvious improvement would also be to spawn a separate worker process for those tests.
As I mentioned previously, our MySQL data directory is located on a ram disk so creating a database does not take that long, but every change which makes tests faster and is not too complex is worth considering.
In the end this modifications did take some time, but it was well wort it - both Django and Twisted tests now finish around 50% - 60% faster.
» Apr 03, 2011
