Welcome to Libcloud April 2012 update post. Sorry for skipping a couple of months, but I have been pretty busy again. In any case, here is a short but sweet April update.
What has been accomplished in the past few months
- Libcloud 0.8.1 with support for compressed responses (gzip, deflate) has been released.
- Libcloud 0.9.1 with improvements in deploy functionality and OpenStack driver has been released.
- I have attended PyCon US 2012 in Santa Clara in March. As always, it was a lot of fun talking with old and meeting new friends. This year we didn’t hold a Libcloud development sprint, but I still managed to talk with some Libcloud users and prompted Libcloud at the AWS open space session.
- Libcloud has applied to Google Summer of Code 2012 under Apache organization. We have received 1 slot. Student Ilgiz Islamgulov will be working on the Libcloud REST interface this summer.
Dancing robots at PyCon
What is currently going on
- We are currently moving towards 1.0 release which means mostly polishing the code, fixing bugs and avoiding big and API breaking changes. A lot of code, especially compute API has already been battle tested, but there are still some parts which I want to see improved (deployment functionality for example) before releasing 1.0.
» Apr 28, 2012
Past few months have been pretty busy here and you may have noticed that I didn’t post a monthly update post in the last two months. This doesn’t mean nothing has been going on though! In fact, quite the contrary. Last few months have been very busy and we have shipped two new releases and a voting thread for a new 0.7.1 release was just started a few days ago. More about that bellow.
What has been accomplished in the past two months
- Libcloud 0.6.1 with a brand new DNS API (among many other improvements and additions) has been released.
- A few days after releasing 0.6.1 we have also released 0.6.2 which was primary a bug-fix release, but it also includes some new features and improvements such as support for OpenStack Auth 2.0 API, support for new Amazon location (Oregon) and a CloudStack driver.
- A few weeks ago a new committer, Hutson Betts (hbetts) has joined our team. He has previously mostly contributed to the OpenNebula driver and we believe giving him commit access will allow him to contribute more directly and easily and is a good thing for the whole project.
What is currently going on
- A lot of things!
- This weekend I have just finished adding support for Python 3 to Libcloud. Now we have a single code base which supports both, Python 2 and 3 and as far as I know we are the first Python cloud library which does that. I hope other projects will follow. If you are interested in more details you can read my post titled Lessons learned while porting Libcloud to Python 3 where I have described some of the issues which I have encountered while porting the library. Armin has recently also wrote a good blog post about Python 3 which you should read - Thoughts on Python 3.
- I am moving to San Francisco where I will work full time for Rackspace. I will try to organize a Bay Area Libcloud meet-up in the upcoming weeks (hey, free pizza and beer!). I will post more info on the website and mailing list when all the details are fleshed out.
» Dec 07, 2011
Yesterday after seeing and being inspired by the Django Python 3 port news, I have decided it’s finally time to port Libcloud to Python 3. There have already been some talks about doing that in the past, but nobody actually managed to make a lot of progress.
In general, our goal is pretty similar to the Django one - have a single code base which works with Python 2.5, 2.6, 2.7 / PyPy and Python 3.
Alternative approach to having a single code base is using a tool like 2to3 to automatically convert 2.x version to the 3.x one or having multiple code bases / branches - one for 2.x and one for 3.x.
Early on when we talked about porting to Python 3, we have decided that we will go with a “single code base” approach. This approach allows us to keep a fast development pace and it’s also more friendlier for our users.
In this post I will describe some of the issues which I have encountered while porting the library and how I have solved them.
1. Handling renamed libraries and moved functionality
httplib
In Python 3 httplib has been renamed to http.client. To solve this problem, I have used an aliased import - import http.client as httplib.
urllib & urllib2
All of the functionality from urllib2 has been merged to urllib. This problem can also be easily solved using an aliased import - import urllib as urllib2.
urlparse
Functionality from urlparse has been moved to urllib.parse. We only use two functions from this module (quote and urlencode) so simple aliased import did the trick:
from urllib.parse import quote as urlquote
from urllib.parse import urlencode as urlencode
xmlrpclib
xmlrpclib has been moved to xmlrpc.client. Simple aliased import also solved this problem - import xmlrpc.client as xmlrpclib.
StringIO
StringIO has also been moved. from io import StringIO did the trick.
2. file type and file-like objects
file type has been removed in Python 3. To resolve this problem, I have used code similar to the one bellow in the places where we use file type.
if PY3:
from io import FileIO as file
class MyFileLikeObject(file):
...
3. Generators and .next() method.
For consistency with other magic methods, next method in Python 3 has been renamed to __next__. To make it work with all the versions, I have used built-in next function in Python >= 3 and object .next() method in older versions.
if sys.version_info >= (3, 0):
next = __builtins__['next']
else:
def next(i):
return i.next()
4. Exception handling
Sadly, there is no unified way to handle exceptions and extract the exception object in Python 2.5 and Python 3.x. This means I needed to use a hacky sys.exc_info()[1] approach to extract the raised exception
Old code:
try:
foo
except Exception, e:
print e
New code:
try:
foo
except Exception:
e = sys.exc_info()[1]
print e
One of the PyPy developers has posted on reddit that this approach is very slow in PyPy. Luckily, besides the tests, there aren’t many places in our code where we need access to the exception object so this should be a good compromise for now.
5. filter, map, dict.keys()
In Python 2 those functions return a list, but in Python 3 they return a special object. Compatibility can be preserved by casting a result from this function to a list - e.g. list(filter(lamba x: x.name == 'test', nodes)).
6. iteritems, xrange
In Python 3, iteritems method has been removed and functionality from xrange has been merged into range. I have simplify replaced iteritems with items and xrange with range. We never used xrange with a lot of values so storing a whole list in memory in Python 2.x shouldn’t be a huge deal.
7. xml.etree.ElementTree.tostring and encoding
In Python 3 this method returns bytes by default. To preserve the old behavior and get a string back, I have used a code similar to one bellow:
if PY3:
encoding = 'unicode'
else:
encoding = None
data = tostring(root, encoding=encoding)
8. encode(‘hex’)
We had multiple places in the code where we did something like this:
value = os.urandom(8).encode('hex')
Hex encoding has been removed from Python 3. I have preserved backward compatibility by using binascii module:
value = binascii.hexlify(os.urandom(8))
9. Octal numbers
In Python 3 there is a special backward-incompatible (and strange) syntax for octal numbers - e.g. 0o755. We only use octal number in one place and this has been easily resolved by using int to convert a string to a number with base 8 - int('755', 8).
Those are just some of the issues I have encountered during porting. If you want to view all of the issues and how I have resolved them, you can see a full diff here.
Overall, I’m pretty satisfied with the outcome. I have managed to keep most of the Python 2 and Python 3 compatibility code in a single module (libcloud.py3) and it probably took me less then 5 hours to do the whole port including the research.
Bellow you can also find some links which I have found helpful while porting the code:
» Dec 03, 2011
First a short-introduction for people who aren’t familiar with Whiskey.
Whiskey is a powerful test runner for Node.js applications. It supports async testing, code coverage, scope leaks reporting, Makefile generation, test timing and lot more. Be sure to check out the github page which lists all the features.
New version (0.6.0) which has been released today includes a process runner and a support for managing external test dependencies. Test dependency is any kind of process on which the (integration) tests depend on.
Examples include, but are not limited to:
- database,
- some kind of api server,
- web server,
- other external services
Process runner is configured using a simple JSON configuration file. Most of the options have sane default values, which means if you don’t have any special requirements you can configure it very quickly.
Example configuration file which we use for our monitoring system integration test suite at Rackspace can be found here.
Each process can also specify its dependencies in the depends option which allows Whiskey to start unrelated processes concurrently.
Before Whiskey process runner was available we have been using scons for managing and running all the test dependencies. Test dependencies related section in our SConstruct file was long and hard to maintain which means switching to Whiskey process runner was a nice improvement.
Process runner can be used by passing --dependencies <configuration_file_path.json> option to whiskey binary. By default all the dependencies specified in the configuration file are started, but there is also --only-esential-dependencies option available which will make Whiskey first inspect the test files and only start the processes which are required by the tests which will be ran.
Each test file can specify on which processes it depends by exporting dependencies attribute. This attribute must be an array and contain the names of the processes as defined in the configuration file.
If you have any questions or suggestions you can find me on #Node.js IRC channel on freenode (nick Kami_). If you find a bug or a problem you can also open an ticket on the project issue tracker.
» Nov 27, 2011
Without further ado here is a Libcloud monthly update for September 2011.
What has been accomplished in the past month
-
I was a guest on FLOSS weekly (podcast about FOSS software) where I talked about Libcloud. You can find video and audio recording of the show on twit.tv.
-
OpenStack and Rackspace drivers have received a lot of needed attention and refactoring. Rackspace driver now properly inherits from the OpenStack one instead of vice versa (thanks Mike). This will make extending the Rackspace driver and developing other provider drivers which are based on OpenStack a lot easier. Rackspace drivers now also support authentication with Rackspace Auth 1.1.
-
Linode compute driver now supports new location in Japan.
Linode has recently added a new location (Tokyo, Japan) and this location is now also supported in Libcloud.
-
DNS API development has finally started.
Base API proposal can be found here. I have also just finished a reference implementation and a first driver for the Linode DNS as a service. The driver can be found in trunk. Feedback is welcome (and encouraged).
What is currently going on
- Hacking on the DNS API continues. DNS API with at least two drivers is planned to be included in the next release (0.6.0) which should be out around November.
See you next month!
» Sep 24, 2011
