Libcloud at ApacheCon NA, 2014 in Denver, Colorado

This is just a quick heads up that I will be attending ApacheCon North America, 2014 in Denver, Colorado next month.

I will be giving two talks. First one is titled 5 years of Libcloud. This is retrospective talk where I will tell a story of how Libcloud grew from a small project originally developed for the needs of Cloudkick product into a fully fledged and relatively popular Apache project.

I will go into details on some of the challenges we faced, what we learned from them and how we grew the community and the project.

Second one is titled Reducing barriers to contribution to an Apache project.

To give you some context, I first need to say that I’m a person who loves to move fast and loves lean and efficient approaches and teams. On top of that, I also have zero tolerance for unnecessary / useless processes and deep and mostly useless hierarchies.

All of that means I don’t see myself a big company person, where having useless processes, which among many other things, slow innovation down is usually the norm.

Apache is a relatively big organization which means it has it’s own fair share of (useless) proceses. A lot of “new era” developers who grew up with Github also consider Apache as slow, inflexible and place where projects go to die1.

In this talk I will go into details why this is not totally true and how Apache is (slowly) becoming more flexible and changing to adopt those new work-flows.

On top of that, I will also give some examples on how you can adopt those new work-flows, iterate fast and still receive all the benefits from being an Apache project. Those examples will be taken directly from the things we have learned at the Apache Libcloud project.

Depending on how many people will attend the talk, I think it would also be very interesting to turn this into a panel where other people can contribute their ideas and we can discuss how to reduce barriers even further and make Apache more attractive for “new-era projects”.

Besides my talks, Sebastien Goasguen, a long time contributor to the project who has recently joined the PMC is also giving a talk titled Apache Libcloud.

If you are around, you should stop by to listen to those talks and contribute your ideas to my second talk.


  1. Citation needed. I’m kinda lazy and tired atm, but you can Google it up.

Libcloud Google Summer of Code 2014 Call for Participation

This is call for participation / proposals for all the students who are interested in working on Apache Libcloud project during the Google Summer of Code 2014 program.

Before diving further, I just want to give a short disclaimer. We (Apache Software Foundation and Libcloud as a project) haven’t been accepted into Google Summer of Code 2014 yet, but we will apply, cross our fingers and hope we get a spot.

We will know if we have been accepted on February 24th when Google publishes a list of the accepted mentoring organizations.

What is Google Summer of Code?

Google Summer of Code is a program where Google sponsors students from around the world to spend their summer working on open-source projects. Student is paid 5500$ if they successfully complete all of their evaluations. More information about the program can be found on the project website.

This year is also special, because it’s a tenth anniversary of the program. To celebrate the anniversary, Google is, among other things, giving out 5500$ for successfully completed projects instead of the usual 5000$.

Google Summer of Code and Libcloud

Apache Software Foundation is not a stranger to Google Summer of Code since it has already participated in this program multiple times over the past years.

We (as in Libcloud project) have also participated in GSoC 2014 with one project. I have mentored a student Ilgiz Islamgulov from Russia who has worked and successfully completed (yeah, software is never really done, but in this case completed refers to him passing all the GSoC evaluations) a project called Libcloud REST interface.

If you want to know more about his project and our GSoC 2012 participation, you should check the following links:

Why should I participate / What do I get out of participating in GSoC?

Before looking at our call for proposals, let’s have a look at why you might want to participate in Google Summer of Code program.

For a moment, lets forget about the money and have a look at a couple of other reasons why participating in GSoC is great for you:

  • Instead of spending your summer flipping burgers (or similar) you can spend summer flipping bits (fun!)
  • You will learn how open source projects and communities work
  • You will get experience with working in distributed / remote teams
  • You will get experience with working across different timezones
  • You will make new friends in the open source community
  • It’s a great thing for your C.V. You will be able to show potential employers some concrete things you have worked on.

Libcloud Google Summer of Code 2014 Call For Proposals

This is the part where you, dear students come in. We would like to invite all the students who are interested in participating to start thinking about the things they could work on and start reaching out to the community.

It doesn’t matter if you are already using or contributing to the project or not, everyone is welcome (people who have or are already contributing to the project have a slight advantage though since we already know what they are capable of)!

Only pre-requisite is a good knowledge of Python, HTTP, REST APIs and a basic familiarity with different cloud services such as Amazon EC2 and others.

As noted in the opening paragraph, we haven’t been accepted yet and the student applications will open in about a month. The reason I’m already posting this is because we want to give potential candidates plenty of time to get familiar with the project and the community.

If you would like to participate, now is the right time to start exploring the existing ideas (you are also more than welcome to propose your own ideas), start thinking about the things you could work, getting familiar with the code base and start reaching out to the community.

Other links you might find useful:

New Libcloud website is now live

I’m happy to announce that a new Libcloud website is now live at libcloud.apache.org.

Design and layout wise, previous website hasn’t really changed since 2009 so a makeover was long overdue.

New website.

The new website includes many new features and improvements. One of the more important ones is a new and fully responsive design which means that the content can now also more easily be consumed on devices with smaller resolutions such as mobile phones and tablets.

On top of that the new website is now powered by Jekyll (same as my blog) which makes adding content and many other things easier.

Without further ado, I encourage you to go check out the new website and read the announcement blog post.

Say hello to Živa

This week I made a big decision. I finally decided to get a dog. In this blog post I’m going to introduce you to my new 24kg / 52 lbs heavy fluffy meatball friend called Živa.

Živa on the way to her new home.

Why get a dog in the first place?

First I need to say that I do like cats, but I was always more of a dog person. I always wanted my own dog, but haven’t really had a chance before.

When I still lived at my parents place, I couldn’t do it because we lived in an apartment and my parents didn’t allow it.

Later on when I moved to the US I was thinking of getting a dog again, but after some more thought I decided it would be irresponsible for me to do it. I did have a nice apartment with a patio, but the problem is that dogs need a lot of attention and I wasn’t really planning in staying in the US for the long term. I was working a lot and leaving the dog at home by itself for a half day or more just seems irresponsible to me.

On top of that, finding an apartment and landlord in San Francisco which allows larger dogs is quite hard and makes moving very painful.

Another opportunity showed itself again recently when I moved back to Slovenia. I decided that I want to stay in Europe for the foreseeable future and work from home. On top of that, my landlord here in Ljubljana has no problem with me having a dog inside the apartment.

Working from home means I can dedicate enough attention and love to the dog and that is also one of the main reasons why I decided to get it.

Adopting a dog from a shelter

A lot of people today buy puppies from local dog breeders. The problem is that all puppies are cute and a lot of people don’t realize that dogs are a big responsibility, especially when they grow up. Usually this leads to a lot of dogs being dumped once they grow up.

I’m not like that so I decided to adopt a slightly older dog. In top of that, I believe that dogs living in the shelters need more help so I decided to adopt one from a local shelter.

Before visiting the shelter in person, I have visited the website and immediately fell in love with a lovely young female called Živa. I know that dogs are very similar to people and outside look is not everything so I didn’t keep my hopes too high.

Luckily, when I visited the shelter in person it turned out that she’s one of the friendliest dogs there. Unlike other dogs, she was very friendly, didn’t bark and simply stuck her nose through the fence to greet me.

After getting to know her a little better, I came back on Monday and decided to adopt her.

Živa

Without further ado, lets get to know this lovely (not so) little creature.

Živa is a mixed breed (looks very much like a German shepherd) ~10 months old female which weighs around 24 kg / 52 lbs. She was found on the street and arrived in the shelter about two months ago. Her name is in Slovenian and stands for lively / playful / cheery. She got this name at the shelter and I decided to keep it since it describes her temperament really well.

Resting.

She is a very friendly, quiet and a kind dog, but she also has almost unlimited energy and needs a tons of play time.

Tummy rub time.
Tummy rub time.

Even though she is very friendly she is currently still afraid of a lot of things including moving vehicles, bridges, stairs and so on. Interesting enough, she is not afraid of other people and dogs. She loves to jump on and kiss people :-)

Dat look.

I don’t know her history, but being afraid of so many thing probably means that her previous owner didn’t socialize her much / enough.

First day with a dog

First day was very happy, but also very stressful for both of us. We both didn’t get a lot of sleep (she was constantly checking on me and I was checking on her) and there were some potty accidents. Potty accidents were mostly my fault because I didn’t recognize that she needs to go to the toilet (I thought she just wants to play).

Deer antler is nom nom.
Mmm, bone marrow...

Plans for the future

First couple of days were great, but there is still a tons of things to do in the future. She needs more socializing, I need to train her not to pull and she needs to be trained to not be so afraid of many things.

As far as the pulling goes, I tried some manual approaches without much success. I’ve ordered an anti pull harness which goes around her front legs and doesn’t hurt her. Hopefully that will help.

On top of that, I also plan to take her to the dog school. The dog school is actually more for me than her (this is my first dog).

Migrating from Zerigo to Rackspace Cloud DNS using Libcloud

In this blog post I’m going to describe how to migrate from Zerigo DNS to Rackspace Cloud DNS using a ~80 lines long Python script which utilizes Libcloud.

Background and Motivation

In September of the last year, I wrote how to export a Libcloud zone to the BIND zone format and use the BIND zone file to migrate between DNS providers.

At that time, my motivation for migrating away from Zerigo was mostly fueled by a very unreliable service which was a consequence of DDoS attacks and less than ideal service architecture.

I have a paid Zerigo plan, so back then, I only migrated the most important domains to a different provider. Not long after I have done this, Zerigo announced that they have partnered with Akamai and that going forward, they will outsource running of the DNS infrastructure to Akami and as such, the service should be way more stable and reliable.

I thought great, I won’t need to migrate rest of the domains away, but an unplesant surprise came earlier this month, when Zerigo announced pricing changes (see 1, 2, 3 & 4).

Previously, I have paid 19$ years per year, but with a new plan which matches my current one, I would need to pay 25$ per month. That’s with an existing customer loyalty discount. New customers will need to pay 38$ per month (what a great deal, instead of paying 24 times more, now I need to pay just 15 times more!). Yes, you have read this correctly, that’s more than one order of magniture per year more than I used to pay before.

I honestly don’t mind paying for a great software and services and I wouldn’t mind paying a little more if the service improved, but that kind or price increase is simply too much. That is especially true, because all of the ~15 domains that I still have at Zerigo are used to host non-profit and community websites and paying 25$ per month is simply too much.

Why Rackspace Cloud DNS?

Disclamer: I used to work at Rackspace, but I don’t work there anymore and I’m not affiliated with them in any way.

Before I dive further, lets have a look at why you might want to use Rackspace Cloud DNS.

The main reason for me to migrate to Rackspace is that they have a decent API, they are supported in Libcloud and best of all, the service is totally free for the existing cloud servers customers. On top of that, the service is supposed to use Anycast.

All of that made it a good fit for hosting my non-profit domains there.

I also need to add that I haven’t used the service a lot before, so I can’t really talk much about the service relablitity at this point. Only time and monitoring will tell how reliable the service really is.

Migrating from Zerigo DNS to Rackspace Cloud DNS using Libcloud

Instead of using Libcloud’s export to BIND zone file functionality, this script works by talking directly to both of the provider APIs.

The reason for that is that this approach is more robust and makes performing partial migrations and synchronizations easier. On top of that it also works with other providers which don’t support importing a BIND zone file.

It’s also important to note that the script relies on some Libcloud fixes which are currently only available in trunk. As such, you should use pip to install latest version from Git inside a virtual environment:

pip install -e git+https://github.com/apache/libcloud.git@trunk#egg=libcloud

After you have done this, you can use the script bellow to migrate all of your zones from Zerigo to Rackspace:

import hashlib

from libcloud.dns.types import Provider, RecordType
from libcloud.dns.providers import get_driver

ZERIGO_USERNAME = ''
ZERIGO_API_KEY = ''

RACKSPACE_USERNAME = ''
RACKSPACE_API_KEY = ''

CONTACT_EMAIL = ''  # Rackspace requires a valid email for every domain

ZONE_TTL = 30 * 60  # Default zone TTL (in seconds) which should be used
MIN_TTL = 300  # Minim TTL supported by the target provider
IGNORED_RECORD_TYPES = [RecordType.NS, RecordType.PTR]

source_cls = get_driver(Provider.ZERIGO)(ZERIGO_USERNAME, ZERIGO_API_KEY)
destination_cls = get_driver(Provider.RACKSPACE)(RACKSPACE_USERNAME,
                                                 RACKSPACE_API_KEY)


def get_record_hash(record):
    """
    Return a hash for the provided record. This is used to determine if the
    record already exists.
    """
    record_hash = hashlib.md5('%s-%s-%s' % (record.name, record.type,
                                            record.data)).hexdigest()
    return record_hash

source_zones = source_cls.list_zones()
destination_zones = destination_cls.list_zones()

destination_domains = [zone.domain for zone in destination_zones]

# 1. Create zones
for zone in source_zones:
    if zone.domain in destination_domains:
        print('Zone "%s" already exists, skipping...' % (zone.domain))
        continue

    extra = {'email': CONTACT_EMAIL}

    print('Creating zone: %s' % (zone.domain))
    destination_cls.create_zone(domain=zone.domain, ttl=ZONE_TTL,
                                extra=extra)

destination_zones = destination_cls.list_zones()

supported_record_type = destination_cls.list_record_types()

# 2. Create records
for source_zone in source_zones:
    destination_zone = [zone for zone in destination_zones
                        if zone.domain == source_zone.domain][0]

    source_records = source_zone.list_records()
    destination_records = destination_zone.list_records()

    for source_record in source_records:
        # Rackspace doesn't have a special SPF record type
        if source_record.type == RecordType.SPF:
            source_record.type = RecordType.TXT

        record_hash = get_record_hash(source_record)
        destination_record_hashes = [get_record_hash(record) for record
                                     in destination_records]

        if source_record.name:
            fqdn = '%s.%s' % (source_record.name, source_zone.domain)
        else:
            fqdn = source_zone.domain

        if record_hash in destination_record_hashes:
            print('Record "%s" already exists, skipping...' % (fqdn))
            continue

        if source_record.type in IGNORED_RECORD_TYPES:
            print(('Encountered ignored record type (type=%s,name=%s) '
                  'skipping...') % (source_record.type, fqdn))
            continue

        if type not in supported_record_type:
            print(('Encountered unsupported record type (type=%s,name=%s)'
                  ', skipping...') % (source_record.type, fqdn))
            continue

        extra = {}

        ttl = source_record.extra.get('ttl', None)
        priority = source_record.extra.get('priority', None)

        if ttl:
            if ttl < MIN_TTL:
                ttl = MIN_TTL
            extra['ttl'] = ttl

        if priority:
            extra['priority'] = priority

        name = source_record.name
        type = source_record.type
        data = source_record.data

        print('Creating a record: %s' % (fqdn))
        destination_zone.create_record(name=name, type=type, data=data,
                                       extra=extra)

Before proceeeding it’s worth knowing that there are some differences between the providers and some limitations you should be aware of:

  • Zerigo supports more record types. If you use more advanced record types which are not supported by Rackspace, then Rackspace might not be a good fit for you.
  • Rackspace only allows you to create PTR records for resources (cloud servers & load balancers) which are hosted in their data centers.
  • Rackspace doesn’t support SPF record type. This is not a big deal since this record type has been deprecated anyway and TXT can be used instead. This script transparently handled remapping of SPF to TXT for you.
  • Minimum supported TTL by Zerigo is 180 seconds and the minimum supported TTL by Rackspace is 300 seconds. If during the migration the script encounteres a TTL smaller than 300 seconds, it simply uses the smallest possible TTL which is 300 seconds.

To use it, simply plug in your API credentials and run it:

python migrate_dns_providers.py
Zerigo control panel.

If the script for some reason fails half-way through (bad connectivity, API issues, etc.), it’s safe to run it again since all the operations are idempotent.

Rackspace Cloud DNS control panel after the migration.

After you have run the script, you should check if everything looks OK and if it does, you can go ahead and change the DNS records for your domains to point to the Rackspace Cloud DNS servers (dns1.stabletransit.com & dns2.stabletransit.com).


For older posts, visit the Archive page.