Apache Libcloud

Cloudstack Collaboration Conference, Amsterdam, November 2013

Tomaz Muraus / @kamislo

Who am I?

  • Tomaz Muraus
  • Apache Libcloud Committer & Project chair
  • Previously: Cloudkick, Rackspace
  • Currently: Hacking away on my startup (email analytics)
  • Likes: Distributed systems, big data, open standards and open source systems, startups, ponies
  • tomaz.me
  • github.com/Kami
  • @kamislo
  • +TomazMuraus

Agenda

  • What is Libcloud?
  • Project History
  • Libcloud APIs
    • Compute and Block Storage
    • Object Storage and CDN
    • Load Balancer
    • DNS
  • CloudStack + Libcloud = ♥
    • Supported Functionality
    • Code Examples
    • Plans for the future

CloudStack + I

  • Not a whole lot of experience
  • Mostly got my hands dirty while reviewing patches and cross-checking the documentation
  • First used it with a real provider while working on the code examples for this presentation
  • ^^ Standard API exposed by Libcloud allows you to do that

Note About Code Examples

  • Examples assume you are using Libcloud trunk
  • Examples were executed against a CloudStack provider (exoscale), but should also work with other supported providers

What is Libcloud?

Libcloud is a Python library which hides differences between different cloud provider APIs and allows you to manage different cloud resources through a unified and easy to use API.

Turns this...


from pprint import pprint

from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver

from config import API_KEY, API_SECRET_KEY, API_HOST, API_PATH

cls = get_driver(Provider.CLOUDSTACK)
driver = cls(API_KEY, API_SECRET_KEY, host=API_HOST, path=API_PATH)

size = [size for size in driver.list_sizes() if size.name == 'Micro'][0]
image = [image for image in driver.list_images() if 'Ubuntu 12.04 LTS 64-bit'
         in image.name][0]

name = 'cloudstack-node'
node = driver.create_node(name=name, image=image, size=size)

pprint(node)

Into this

What is Libcloud?

  • Similar projects for other languages
    • fog (Ruby)
    • Apache jclouds (Java)
    • Apache Deltacloud (Ruby / HTTP)

How it came about?

  • We formed a committee and locked 10 people with a PhD into a room and tasked them with designing a standard interface
  • Not really :-)
  • Like many other "new era" projects and standards (think SPDY, etc.), Libcloud was developed as a pragmatic solution for our problem

How it came about?

How it came about?

How it came about?

  • Developed as a solution for our problem with talking to many different providers at Cloudkick
  • Released as an open-source Apache 2.0 licensed project
  • Joined Apache Incubator in late 2009, graduated to a top-level project in 2011
  • Mostly organic growth where interfaces and APIs have evolved over time

Supported Providers

Libcloud APIs

  • Compute - libcloud.compute.*
  • Storage - libcloud.storage.*
  • Load Balancer - libcloud.loadbalancer.*
  • DNS - libcloud.dns.*

Compute

  • Allows you to manage cloud and virtual servers and block storage
  • Services such as Amazon EC2, Rackspace Cloud, OpenStack, CloudStack...

Object Storage

  • Allows you to manage object storage and CDN
  • Services such as Amazon S3, Rackspace CloudFiles, ...

Load Balancer

  • Allows you to manage cloud load balancers
  • Services such as Rackspace Cloud Loadbalancers, GoGrid Loadbalancers, ...

DNS

  • Allows you to manage DNS as a Service
  • Services such as Amazon Route53, Zerigo DNS, ...

But wait, there is more!

  • libcloud.queues probably joining other APIs in the near future
  • Think Amazon SQS, OpenStack Marconi, Azure Queues...

Compute - Terminology

  • Node - represents a cloud / virtual server
  • NodeState - represents a state node is currently in (running, stopped, ...)
  • NodeSize - represents size of the server (cpu, memory, disk size)
  • NodeImage - represents an operating system

Compute - Base API

  • list_images()
  • list_sizes()
  • list_nodes()
  • create_node()
  • deploy_node()
  • destroy_node()
  • + volume management methods (block storage)

CloudStack + Libcloud

  • Standard Libcloud functionality
  • IP address allocation and management
  • Security group management
  • Firewall rules / port forwarding management
  • Keypair management
  • but wait, there is more!

Example 1 - Create a node


import random
from pprint import pprint

from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver

from config import API_KEY, API_SECRET_KEY

cls = get_driver(Provider.EXOSCALE)
driver = cls(API_KEY, API_SECRET_KEY)

size = [size for size in driver.list_sizes() if size.name == 'Micro'][0]
image = [image for image in driver.list_images() if 'Ubuntu 12.04 LTS 64-bit'
         in image.name][0]

name = 'cloudstack-node-' + str(random.randint(0,100))
node = driver.create_node(name=name, image=image, size=size)

pprint(node)

Example 2 - Deploy a node


import random
from pprint import pprint

from libcloud.compute.types import Provider
from libcloud.compute.providers import get_driver
from libcloud.compute.deployment import ScriptDeployment

from config import API_KEY, API_SECRET_KEY

cls = get_driver(Provider.EXOSCALE)
driver = cls(API_KEY, API_SECRET_KEY)

size = [size for size in driver.list_sizes() if size.name == 'Micro'][0]
image = [image for image in driver.list_images() if 'Ubuntu 12.04 LTS 64-bit'
         in image.name][0]

with open('./script.sh', 'r') as fp:
    content = fp.read()

name = 'cloudstack-node-' + str(random.randint(0, 100))
script = ScriptDeployment(script=content)
node = driver.deploy_node(name=name, image=image, size=size, deploy=script,
                          ex_keyname='default',
                          ssh_key_file='/home/kami/.ssh/id_rsa_cloudstack')

pprint(node)
pprint(script.stdout)

CloudStack Driver - Plans for the future

  • Better support for locations (zones)
  • Better support for advanced zones
  • Support for multiple API versions

Questions?

Thanks

https://libcloud.apache.org

https://libcloud.readthedocs.org

@kamislo / tomaz.me