In this post I will describe how to host APT repository on Rackspace Cloud Files.
First you need to create a CDN enabled container. You can do this using Libcloud or the new Rackspace control panel. Depending on how frequently you are going update your APT repository it is also recommended to lower the container TTL from the default value of 3 days to 900 seconds.
After you have created a CDN enabled container you need to install Python script I wrote which allows you to sychronize files from a local directory to a container hosted in one of the Storage providers supported by Libcloud.
sudo pip install file-syncer
Next we are going to create a dummy APT repository::
mkdir -p /tmp/apt-test/conf
cat > /tmp/apt-test/conf/distributions <<DELIM
codename: precise
Components: main
Architectures: i386 amd64
DELIM
Add a test package using reprepro:
wget http://us.archive.ubuntu.com/ubuntu/ubuntu/ubuntu/pool/main/a/acpi/acpi_0.09-3ubuntu1_amd64.deb
reprepro -b /tmp/apt-test/ includedeb precise acpi_0.09-3ubuntu1_amd64.deb
And test the script:
file-syncer --log-level=DEBUG --directory=/tmp/apt-test/ --username=your username --key=your api key --container-name=container name
If everything went well, you should see an output similar to this one:
22 Jul 2012 22:36:05 : INFO : Using provider: CloudFiles (US)
22 Jul 2012 22:36:07 : DEBUG : Found 15 local files
22 Jul 2012 22:36:08 : DEBUG : Manifest doesn't exist, assuming that there are no remote files
22 Jul 2012 22:36:08 : DEBUG : Found 0 remote files
22 Jul 2012 22:36:08 : INFO : To remove: 0, to upload: 15
22 Jul 2012 22:36:08 : DEBUG : Uploading object: db/contents.cache.db
22 Jul 2012 22:36:08 : DEBUG : Uploading object: dists/precise/main/binary-amd64/Packages
22 Jul 2012 22:36:08 : DEBUG : Uploading object: dists/precise/main/binary-i386/Release
22 Jul 2012 22:36:08 : DEBUG : Uploading object: dists/precise/main/binary-i386/Packages
22 Jul 2012 22:36:08 : DEBUG : Uploading object: db/packages.db
22 Jul 2012 22:36:08 : DEBUG : Uploading object: pool/main/a/acpi/acpi_0.09-3ubuntu1_amd64.deb
22 Jul 2012 22:36:08 : DEBUG : Uploading object: db/release.caches.db
22 Jul 2012 22:36:08 : DEBUG : Uploading object: db/version
22 Jul 2012 22:36:08 : DEBUG : Uploading object: conf/distributions
22 Jul 2012 22:36:08 : DEBUG : Uploading object: db/references.db
22 Jul 2012 22:36:09 : DEBUG : Object uploaded: dists/precise/main/binary-amd64/Packages
22 Jul 2012 22:36:09 : DEBUG : Uploading object: dists/precise/main/binary-i386/Packages.gz
22 Jul 2012 22:36:09 : DEBUG : Object uploaded: dists/precise/main/binary-i386/Packages
22 Jul 2012 22:36:09 : DEBUG : Uploading object: dists/precise/main/binary-amd64/Packages.gz
22 Jul 2012 22:36:09 : DEBUG : Object uploaded: db/release.caches.db
22 Jul 2012 22:36:09 : DEBUG : Uploading object: db/checksums.db
22 Jul 2012 22:36:09 : DEBUG : Object uploaded: dists/precise/main/binary-i386/Release
22 Jul 2012 22:36:09 : DEBUG : Uploading object: dists/precise/main/binary-amd64/Release
22 Jul 2012 22:36:09 : DEBUG : Object uploaded: conf/distributions
22 Jul 2012 22:36:09 : DEBUG : Uploading object: dists/precise/Release
22 Jul 2012 22:36:09 : DEBUG : Object uploaded: db/packages.db
22 Jul 2012 22:36:09 : DEBUG : Object uploaded: db/contents.cache.db
22 Jul 2012 22:36:09 : DEBUG : Object uploaded: db/references.db
22 Jul 2012 22:36:09 : DEBUG : Object uploaded: db/version
22 Jul 2012 22:36:09 : DEBUG : Object uploaded: pool/main/a/acpi/acpi_0.09-3ubuntu1_amd64.deb
22 Jul 2012 22:36:10 : DEBUG : Object uploaded: dists/precise/main/binary-amd64/Packages.gz
22 Jul 2012 22:36:10 : DEBUG : Object uploaded: dists/precise/main/binary-i386/Packages.gz
22 Jul 2012 22:36:10 : DEBUG : Object uploaded: db/checksums.db
22 Jul 2012 22:36:10 : DEBUG : Object uploaded: dists/precise/Release
22 Jul 2012 22:36:10 : DEBUG : Object uploaded: dists/precise/main/binary-amd64/Release
22 Jul 2012 22:36:11 : INFO : Synchronization complete, took: 4.62 seconds
After the upload has finished, you can test the repository by adding it to your APT sources list:
echo "deb http://c15173579.r79.cf2.rackcdn.com precise main" | sudo tee -a /etc/apt/sources.list.d/test-api-repo.list
We are also going to disable APT HTTP request pipeling. This step is required, because a nginx proxy running in front of the CDN doesn’t seem to support HTTP/1.1 pipelining. If you skip this test apt-get update will still work, but it will get stuck on [Waiting for headers] for a longer period of time.
echo "Acquire::http::Pipeline-Depth "0";" | sudo tee -a /etc/apt/apt.conf.d/pipeline-workaround.conf
You can test that everything is working by issuing the following command:
sudo apt-get update ; sudo apt-cache policy acpi
You should get an output similar to this one:
acpi:
Installed: (none)
Candidate: 1.6-1
Version table:
1.6-1 0
500 http://us.archive.ubuntu.com/ubuntu/ precise/universe amd64 Packages
0.09-3ubuntu1 0
500 http://c15173579.r79.cf2.rackcdn.com/ precise/main amd64 Packages
That is it. To make sure your container is always up to date, you need synchronize it every time you add a new packages. This can be achieved in multiple ways:
- adding a script which runs synchronize command after adding a package
- adding a cron job which periodically runs the sync command
Welcome to Libcloud April 2012 update post. Sorry for skipping a couple of months, but I have been pretty busy again. In any case, here is a short but sweet April update.
What has been accomplished in the past few months
- Libcloud 0.8.1 with support for compressed responses (gzip, deflate) has been released.
- Libcloud 0.9.1 with improvements in deploy functionality and OpenStack driver has been released.
- I have attended PyCon US 2012 in Santa Clara in March. As always, it was a lot of fun talking with old and meeting new friends. This year we didn’t hold a Libcloud development sprint, but I still managed to talk with some Libcloud users and prompted Libcloud at the AWS open space session.
- Libcloud has applied to Google Summer of Code 2012 under Apache organization. We have received 1 slot. Student Ilgiz Islamgulov will be working on the Libcloud REST interface this summer.
Dancing robots at PyCon
What is currently going on
- We are currently moving towards 1.0 release which means mostly polishing the code, fixing bugs and avoiding big and API breaking changes. A lot of code, especially compute API has already been battle tested, but there are still some parts which I want to see improved (deployment functionality for example) before releasing 1.0.
Past few months have been pretty busy here and you may have noticed that I didn’t post a monthly update post in the last two months. This doesn’t mean nothing has been going on though! In fact, quite the contrary. Last few months have been very busy and we have shipped two new releases and a voting thread for a new 0.7.1 release was just started a few days ago. More about that bellow.
What has been accomplished in the past two months
- Libcloud 0.6.1 with a brand new DNS API (among many other improvements and additions) has been released.
- A few days after releasing 0.6.1 we have also released 0.6.2 which was primary a bug-fix release, but it also includes some new features and improvements such as support for OpenStack Auth 2.0 API, support for new Amazon location (Oregon) and a CloudStack driver.
- A few weeks ago a new committer, Hutson Betts (hbetts) has joined our team. He has previously mostly contributed to the OpenNebula driver and we believe giving him commit access will allow him to contribute more directly and easily and is a good thing for the whole project.
What is currently going on
- A lot of things!
- This weekend I have just finished adding support for Python 3 to Libcloud. Now we have a single code base which supports both, Python 2 and 3 and as far as I know we are the first Python cloud library which does that. I hope other projects will follow. If you are interested in more details you can read my post titled Lessons learned while porting Libcloud to Python 3 where I have described some of the issues which I have encountered while porting the library. Armin has recently also wrote a good blog post about Python 3 which you should read - Thoughts on Python 3.
- I am moving to San Francisco where I will work full time for Rackspace. I will try to organize a Bay Area Libcloud meet-up in the upcoming weeks (hey, free pizza and beer!). I will post more info on the website and mailing list when all the details are fleshed out.
Yesterday after seeing and being inspired by the Django Python 3 port news, I have decided it’s finally time to port Libcloud to Python 3. There have already been some talks about doing that in the past, but nobody actually managed to make a lot of progress.
In general, our goal is pretty similar to the Django one - have a single code base which works with Python 2.5, 2.6, 2.7 / PyPy and Python 3.
Alternative approach to having a single code base is using a tool like 2to3 to automatically convert 2.x version to the 3.x one or having multiple code bases / branches - one for 2.x and one for 3.x.
Early on when we talked about porting to Python 3, we have decided that we will go with a “single code base” approach. This approach allows us to keep a fast development pace and it’s also more friendlier for our users.
In this post I will describe some of the issues which I have encountered while porting the library and how I have solved them.
1. Handling renamed libraries and moved functionality
httplib
In Python 3 httplib has been renamed to http.client. To solve this problem, I have used an aliased import - import http.client as httplib.
urllib & urllib2
All of the functionality from urllib2 has been merged to urllib. This problem can also be easily solved using an aliased import - import urllib as urllib2.
urlparse
Functionality from urlparse has been moved to urllib.parse. We only use two functions from this module (quote and urlencode) so simple aliased import did the trick:
from urllib.parse import quote as urlquote
from urllib.parse import urlencode as urlencode
xmlrpclib
xmlrpclib has been moved to xmlrpc.client. Simple aliased import also solved this problem - import xmlrpc.client as xmlrpclib.
StringIO
StringIO has also been moved. from io import StringIO did the trick.
2. file type and file-like objects
file type has been removed in Python 3. To resolve this problem, I have used code similar to the one bellow in the places where we use file type.
if PY3:
from io import FileIO as file
class MyFileLikeObject(file):
...
3. Generators and .next() method.
For consistency with other magic methods, next method in Python 3 has been renamed to __next__. To make it work with all the versions, I have used built-in next function in Python >= 3 and object .next() method in older versions.
if sys.version_info >= (3, 0):
next = __builtins__['next']
else:
def next(i):
return i.next()
4. Exception handling
Sadly, there is no unified way to handle exceptions and extract the exception object in Python 2.5 and Python 3.x. This means I needed to use a hacky sys.exc_info()[1] approach to extract the raised exception
Old code:
try:
foo
except Exception, e:
print e
New code:
try:
foo
except Exception:
e = sys.exc_info()[1]
print e
One of the PyPy developers has posted on reddit that this approach is very slow in PyPy. Luckily, besides the tests, there aren’t many places in our code where we need access to the exception object so this should be a good compromise for now.
5. filter, map, dict.keys()
In Python 2 those functions return a list, but in Python 3 they return a special object. Compatibility can be preserved by casting a result from this function to a list - e.g. list(filter(lamba x: x.name == 'test', nodes)).
6. iteritems, xrange
In Python 3, iteritems method has been removed and functionality from xrange has been merged into range. I have simplify replaced iteritems with items and xrange with range. We never used xrange with a lot of values so storing a whole list in memory in Python 2.x shouldn’t be a huge deal.
7. xml.etree.ElementTree.tostring and encoding
In Python 3 this method returns bytes by default. To preserve the old behavior and get a string back, I have used a code similar to one bellow:
if PY3:
encoding = 'unicode'
else:
encoding = None
data = tostring(root, encoding=encoding)
8. encode(‘hex’)
We had multiple places in the code where we did something like this:
value = os.urandom(8).encode('hex')
Hex encoding has been removed from Python 3. I have preserved backward compatibility by using binascii module:
value = binascii.hexlify(os.urandom(8))
9. Octal numbers
In Python 3 there is a special backward-incompatible (and strange) syntax for octal numbers - e.g. 0o755. We only use octal number in one place and this has been easily resolved by using int to convert a string to a number with base 8 - int('755', 8).
Those are just some of the issues I have encountered during porting. If you want to view all of the issues and how I have resolved them, you can see a full diff here.
Overall, I’m pretty satisfied with the outcome. I have managed to keep most of the Python 2 and Python 3 compatibility code in a single module (libcloud.py3) and it probably took me less then 5 hours to do the whole port including the research.
Bellow you can also find some links which I have found helpful while porting the code:
First a short-introduction for people who aren’t familiar with Whiskey.
Whiskey is a powerful test runner for Node.js applications. It supports async testing, code coverage, scope leaks reporting, Makefile generation, test timing and lot more. Be sure to check out the github page which lists all the features.
New version (0.6.0) which has been released today includes a process runner and a support for managing external test dependencies. Test dependency is any kind of process on which the (integration) tests depend on.
Examples include, but are not limited to:
- database,
- some kind of api server,
- web server,
- other external services
Process runner is configured using a simple JSON configuration file. Most of the options have sane default values, which means if you don’t have any special requirements you can configure it very quickly.
Example configuration file which we use for our monitoring system integration test suite at Rackspace can be found here.
Each process can also specify its dependencies in the depends option which allows Whiskey to start unrelated processes concurrently.
Before Whiskey process runner was available we have been using scons for managing and running all the test dependencies. Test dependencies related section in our SConstruct file was long and hard to maintain which means switching to Whiskey process runner was a nice improvement.
Process runner can be used by passing --dependencies <configuration_file_path.json> option to whiskey binary. By default all the dependencies specified in the configuration file are started, but there is also --only-esential-dependencies option available which will make Whiskey first inspect the test files and only start the processes which are required by the tests which will be ran.
Each test file can specify on which processes it depends by exporting dependencies attribute. This attribute must be an array and contain the names of the processes as defined in the configuration file.
If you have any questions or suggestions you can find me on #Node.js IRC channel on freenode (nick Kami_). If you find a bug or a problem you can also open an ticket on the project issue tracker.
