Tomaz Muraus' personal blogcomputers, programming, startups and life.https://www.tomaz.me2022-11-26T21:25:27+01:00Tomaz Muraustomaz@tomaz.meMaking StackStorm Fasthttps://www.tomaz.me/2021/07/04/making-stackstorm-fast.html2021-07-04T00:00:00+02:00Tomaz Muraus<h2 id="making-stackstorm-fast"><a href="/2021/07/04/making-stackstorm-fast.html">Making StackStorm Fast</a></h2>
<p>In this post I will describe changes to the StackStorm database abstraction
layer which landed in <a href="https://stackstorm.com/2021/06/29/stackstorm-v3-5-0-released/">StackStorm v3.5.0</a>. Those changes will substantially
speed up action executions and workflow runs for most users.</p>
<div class="imginline">
<a href="http://www.stackstorm.com" target="_blank"><img src="/images/stackstorm-fast/st2_fast_2.png" class="inline" /></a>
</div>
<p>Based on the benchmarks and load testing we have performed, most actions
which return large results and workflows which pass large datasets around
should see speed ups in the range of up to 5-15x.</p>
<p>If you want to learn more about the details you can do that below. Alternatively
if you only care about the numbers, you can go directly to the
<a href="#numbers">Numbers, numbers, numbers</a> section.</p>
<h3 id="background-and-history">Background and History</h3>
<p>Today StackStorm is used for solving a very diverse set of problems – from IT
and infrastructure provisioning to complex CI/CD pipeline, automated remediation,
various data processing pipelines and more.</p>
<p>Solving a lot of those problems requires passing large datasets around – this
usually involves passing around large dictionary objects to the actions (which
can be in the range of many MBs) and then inside the workflow, filtering down
the result object and passing it to other tasks in the workflow.</p>
<p>This works fine when working with small objects, but it starts to break when
larger datasets are passed around (dictionaries over 500 KB).</p>
<p>In fact, passing large results around has been StackStorm’s achilles heel for
many years now (see some of the existing issues -
<a href="https://github.com/StackStorm/st2/issues/3712">#3718</a>,
<a href="https://github.com/StackStorm/st2/issues/4798">#4798</a>,
<a href="https://github.com/StackStorm/st2web/issues/625">#625</a>). Things will still work, but
executions and workflows which handle large datasets will get progressively
slower and waste progressively more CPU cycles and no one likes slow software
and wasting CPU cycles (looking at you bitcoin).</p>
<p>One of the more popular workarounds usually involves storage those larger
results / datasets in a 3d party system (such as a database) and then querying
this system and retrieving data inside the action.</p>
<p>There have been many attempts to improve that in the past (see
<a href="https://github.com/StackStorm/st2/pull/4837">#4837</a>,
<a href="https://github.com/StackStorm/st2/pull/4838">#4838</a>,
<a href="https://github.com/StackStorm/st2/pull/4846">#4846</a>) and we did make some smaller
incremental improvements over the years, but most of them were in the range of a
couple of 10% of an improvement maximum.</p>
<p>After an almost year long break from StackStorm due to the busy work and life
situation, I used StackStorm again to scratch my own itch. I noticed the age
old “large results” problem hasn’t been solved yet so I decided to take a
look at the issue again and try to make more progress on the PR I originally
started more than a year ago (https://github.com/StackStorm/st2/pull/4846).</p>
<p>It took many late nights, but I was finally able to make good progress on it.
This should bring substantial speed ups and improvements to all StackStorm
users.</p>
<h3 id="why-the-problem-exists-today">Why the problem exists today</h3>
<p>Before we look into the implemented solution, I want to briefly explain why
StackStorm today is slow and inefficient when working with large datasets.</p>
<p>Primary reason why StackStorm is slow when working with large datasets is
because we utilize <code class="language-plaintext highlighter-rouge">EscapedDictField()</code> and <code class="language-plaintext highlighter-rouge">EscapedDynamicField()</code>
mongoengine field types for storing execution results and workflow state.</p>
<p>Those field types seemed like good candidates when we started almost 7 years
ago (and they do work relatively OK for smaller results and other metadata
like fields), but over the years after people started to push more data
through it, it turned out they are very slow and inefficient for storing and
retrieving large datasets.</p>
<p>The slowness boils down to two main reasons:</p>
<ul>
<li>Field keys need to be escaped. Since <code class="language-plaintext highlighter-rouge">.</code> and <code class="language-plaintext highlighter-rouge">$</code> are special characters
in MongoDB used for querying, they need to be escaped recursively in all the
keys of a dictionary which is to be stored in the database. This can get slow
with large and deeply nested dictionaries.</li>
<li>mongoengine ORM library we use to interact with MongoDB is known to be be
very slow compared to using pymongo directly when working with large documents
(see <a href="https://github.com/MongoEngine/mongoengine/issues/1230">#1230</a> and
<a href="https://stackoverflow.com/questions/35257305/mongoengine-is-very-slow-on-large-documents-compared-to-native-pymongo-usage">https://stackoverflow.com/questions/35257305/mongoengine-is-very-slow-on-large-documents-compared-to-native-pymongo-usage)</a>.
This is mostly due to the complex and slow conversion of types mongoengine
performs when storing and retrieving documents.</li>
</ul>
<p>Those fields are also bad candidates for what we are using them for. Data we
are storing (results) is a more or less opaque binary blob to the database,
but we are storing it in a very rich field type which supports querying on
field keys and values. We don’t rely on any of that functionality and as
you know, nothing comes for free – querying on dictionary field values
requires more complex data structures internally in MongoDB and in some
cases also indexes. That’s wasteful and unnecessary in our case.</p>
<h3 id="solving-the-problem">Solving the Problem</h3>
<p>Over the years there have been many discussions on how to improve that. A
lot of users said we should switch away from MongoDB.</p>
<p>To begin with, I need to start and say I’m not a big fan of MongoDB, but
the actual database layer itself is not the problem here.</p>
<p>If switching to a different database technology was justified (aka the
bottleneck was the database itself and nor our code or libraries we depend
on), then I may say go for it, but the reality is that even then, such a
rewrite is not even close to being realistic.</p>
<p>We do have abstractions / ORM in place for working with the database
layer, but as anyone who was worked in a software project which has grown
organically over time knows, those abstractions get broken, misused or worked
around over time (for good or bad reasons, that’s it’s not even important for
this discussion).</p>
<p>Reality is that moving to a different database technology would likely
require many man months hours of work and we simply don’t have that. The
change would also be much more risky, very disruptive and likely result
in many regressions and bugs – I have participated in multiple major
rewrites in the past and no matter how many tests you have, how good are
the coding practices, the team, etc. there will always be bugs and
regressions. Nothing beats miles on the code and with a rewrite you are
removing all those miles and battle tested / hardened code with new code
which doesn’t have any of that.</p>
<p>Luckily after a bunch of research and prototyping I was able to come up with a
relatively simple solution which is much less invasive, fully backward
compatible and brings some serious improvements all across the board.</p>
<h3 id="implemented-approach">Implemented Approach</h3>
<p>Now that we know that using <code class="language-plaintext highlighter-rouge">DictField</code> and <code class="language-plaintext highlighter-rouge">DynamicField</code> is slow and
expensive, the challenge is to find a different field type which offers
much better performance.</p>
<p>After prototyping and benchmarking various approaches, I was able to find that
using binary data field type is the most efficient solution for our problem –
when using that field type, we can avoid all the escaping and most importantly,
very slow type conversions inside mongoengine.</p>
<p>This also works very well for us, since execution results, workflow results,
etc. are just an opaque blob to the database layer (we don’t perform any direct
queries on the result values or similar).</p>
<p>That’s all good, but in reality in StackStorm results are JSON dictionaries
which can contain all the simple types (dicts, lists, numbers, strings,
booleans - and as I recently learned, apparently even sets even though that’s
not a official JSON type, but mongoengine and some JSON libraries just
“silently” serialize it to a list). This means we still need to serialize data
in some fashion which can be deserialized fast and efficiently on retrieval
from the database.</p>
<p>Based on micro benchmark results, I decided to settle down on JSON,
specifically orjson library which offers very good performance on large
datasets. So with the new field type changes, execution result and various
other fields are now serialized as JSON string and stored in a database as a
binary blob (well, we did add some sugar coat on top of JSON, just to make it
a bit more future proof and allow us to change the format in future, if needed
and also implement things such as per field compression, etc.).</p>
<p>Technically using some kind of binary format (think Protobuf, msgpack,
flatbuffers, etc.) may be even faster, but those formats are primarily meant
for structured data (think all the fields and types are known up front) and
that’s not the case with our result and other fields – they can contain
arbitrary JSON dictionaries. While you can design a Protobuf structure which
would support our schemaless format, that would add a lot of overhead and very
likely in the end be slower than using JSON + orjson.</p>
<p>So even though the change sounds and looks really simple (remember – simple
code and designs are always better!) in reality it took a lot of time to get
everything to work and tests to pass (there were a lot of edge cases, code
breaking abstractions, etc.), but luckily all of that is behind us now.</p>
<p>This new field type is now used for various models (execution, live action,
workflow, task execution, trigger instance, etc.).</p>
<p>Most improvements should be seen in the action runner and workflow engine
service layer, but secondary improvements should also be seen in st2api (when
retrieving and listing execution results, etc.) and rules engine (when
evaluating rules against trigger instances with large payloads).</p>
<h3 id="-numbers-numbers-numbers"><a name="numbers"></a> Numbers, numbers, numbers</h3>
<p>Now that we know how the new changes and field type works, let’s look at the
most important thing – actual numbers.</p>
<h4 id="micro-benchmarks">Micro-benchmarks</h4>
<p>I believe all decisions like that should be made and backed up with data so I
started with some micro benchmarks for my proposed changes.</p>
<p>Those micro benchmarks measure how long it takes to insert and read a document
with a single large field from MongoDB comparing old and the new field type.</p>
<p>We also have micro benchmarks which cover more scenarios (think small values,
document with a lot of fields, document with single large field, etc.), but
those are not referenced here.</p>
<p><strong>1. Database writes</strong></p>
<div class="imginline">
<a href="/images/stackstorm-fast/image1.png" target="_blank"><img src="/images/stackstorm-fast/image1.png" class="inline" /></a>
<span class="image-caption">This screenshot shows that the new field type (json dict field) is
~10x faster over EscapedDynamicField and ~15x over EscapedDictField when saving 4 MB field
value in the database.</span>
</div>
<p><strong>2. Database reads</strong></p>
<div class="imginline">
<a href="/images/stackstorm-fast/image6.png" target="_blank"><img src="/images/stackstorm-fast/image6.png" class="inline" /></a>
<span class="image-caption">This screenshot shows that the new field is about ~7x faster
over EscapedDynamicField and ~40x over EscapedDictField..</span>
</div>
<p>P.S. You should only look at the relative change and not absolute numbers.
Those benchmarks ran on a relatively powerful server. On a smaller VMs
you may see different absolute numbers, but the relative change should be about
the same.</p>
<p>Those micro benchmarks also run daily as part of our CI to prevent regressions
and similar and you can view the complete results <a href="https://github.com/StackStorm/st2/actions/workflows/microbenchmarks.yaml">here</a>.</p>
<h4 id="end-to-end-load-tests">End to end load tests</h4>
<p>Micro benchmarks always serve as a good starting point, but in the end we care
about the complete picture.</p>
<p>Things never run in isolation, so we need to put all the pieces together and
measure how it performs in real-life scenarios.</p>
<p>To measure this, I utilized some synthetic and some more real-life like actions
and workflows.</p>
<p><strong>1. Python runner action</strong></p>
<p>Here we have a simple Python runner action which reads a 4 MB JSON file from
disk and returns it as an execution result.</p>
<p>Old field type</p>
<div class="imginline">
<a href="/images/stackstorm-fast/image2.png" target="_blank"><img src="/images/stackstorm-fast/image2.png" class="inline" /></a>
</div>
<p>New field type</p>
<div class="imginline">
<a href="/images/stackstorm-fast/image3.png" target="_blank"><img src="/images/stackstorm-fast/image3.png" class="inline" /></a>
</div>
<p>With the old field type it takes 12 seconds and with the new one it takes 1.</p>
<p>For the actual duration, please refer to the “log” field. Previous versions of
StackStorm contained a bug and didn’t accurately measure / reprt action run time –
the end_timestamp – start_timestamp only measures how long it took for action
execution to complete, but it didn’t include actual time it took to persist
execution result in the database (and with large results actual persistence
could easily take many 10s of seconds) – and execution is not actually
completed until data is persisted in the database.</p>
<p><strong>2. Orquesta Workflow</strong></p>
<p>In this test I utilized an orquesta workflow which runs Python runner action
which returns ~650 KB of data and this data is then passed to other tasks in the workflow.</p>
<p>Old field type</p>
<div class="imginline">
<a href="/images/stackstorm-fast/image4.png" target="_blank"><img src="/images/stackstorm-fast/image4.png" class="inline" /></a>
</div>
<p>New field type</p>
<div class="imginline">
<a href="/images/stackstorm-fast/image5.png" target="_blank"><img src="/images/stackstorm-fast/image5.png" class="inline" /></a>
</div>
<p>Here we see that with the old field type it takes 95 seconds and with the new
one it takes 10 seconds.</p>
<p>With workflows we see even larger improvements. The reason for that is that
actual workflow related models utilize multiple fields of this type and also
perform many more database operations (read and writes) compared to simple
non-workflow actions.</p>
<hr />
<p>You don’t need to take my word for it. You can download StackStorm v3.5.0 and
test the changes with your workloads.</p>
<p>Some of the early adopters have already tested those changes before StackStorm
v3.5.0 was released with their workloads and so far the feedback has been very
positive - speed up in the range of 5-15x.</p>
<h3 id="other-improvements">Other Improvements</h3>
<p>In addition to the database layer improvements which are the start of the v3.5.0
release, I also made various performance improvements in other parts of the
system:</p>
<ul>
<li>Various API and CLI operations have been sped up by switching to orjson for
serializarion and deserialization and various other optimizations.</li>
<li>Pack registration has been improved by reducing the number of redundant
queries and similar.</li>
<li>Various code which utilizes <code class="language-plaintext highlighter-rouge">yaml.safe_load</code> has been speed up by switching
to C versions of those functions.</li>
<li>ISO8601 / RFC3339 date time strings parsing has been speed up by switching to
<code class="language-plaintext highlighter-rouge">udatetime</code> library</li>
<li>Service start up time has been sped by utilizing <code class="language-plaintext highlighter-rouge">stevedore</code> library more
efficiently.</li>
<li>WebUI has been substantially sped up - we won’t retrieve and display very
large results by default anymore. In the past, WebUI would simply freeze the
browser window / tab when viewing the history tab. Do keep in mind that righ
now only the execution part has been optimized and in some other scenarios
WebUI will still try to load syntax highlighting very large datasets which
will result in browser freezing.</li>
</ul>
<h3 id="conclusion">Conclusion</h3>
<p>I’m personally very excited about those changes and hope you are as well.</p>
<p>They help address one of StackStorm’s long known pain points. And we are not
just talking about 10% here and there, but up to 10-15x improvements for
executions and workflows which work with larger datasets (> 500 KB).</p>
<p>That 10-15x speed up doesn’t just mean executions and workflows will complete
faster, but also much lower CPU utilization and less wasted CPU cycles (as
described above, due to the various conversions, storing large fields in the
database and to a lesser extent also reading them, was previously a very CPU
intensive task).</p>
<p>So in a sense, you can view of those changes as getting additional resources /
servers for free – previously you might have needed to add new pods / servers
running StackStorm services, but with those changes you should able to get
much better throughput (executions / seconds) with the existing resources
(you may even be able to scale down!). Hey, who doesn’t like free servers :)</p>
<p>This means many large StackStorm users will be able to save many hundreds
and thousands of $ per month in infrastructure costs. If this change will
benefit you and your can afford it, check <a href="https://stackstorm.com/donate/">Donate</a> page on how you can
help the project.</p>
<h3 id="thanks">Thanks</h3>
<p>I would like to thank everyone who has contributed to the performance
improvements in any way.</p>
<p>Thank to everyone who has helped to review that massive PR with over 100
commits (Winson, Drew, Jacob, Amanda), @guzzijones and others who have tested
the changes while they were still in development and more.</p>
<p>This also includes many of our long term uses such as Nick Maludy,
@jdmeyer3 and others who have reported this issue a long time ago and worked
around the limitations when working with larger datasets in various different
ways.</p>
<p>Special thanks also to v3.5.0 release managers <a href="https://github.com/amanda11">Amanda</a> and <a href="https://github.com/winem">Marcel</a>.</p>
Consuming AWS EventBridge Events inside StackStormhttps://www.tomaz.me/2019/07/13/consuming-aws-eventbridge-events-in-stackstorm.html2019-07-13T00:00:00+02:00Tomaz Muraus<h2 id="consuming-aws-eventbridge-events-inside-stackstorm"><a href="/2019/07/13/consuming-aws-eventbridge-events-in-stackstorm.html">Consuming AWS EventBridge Events inside StackStorm</a></h2>
<p>Amazon Web Services (AWS) recently launched a new product called <a href="https://aws.amazon.com/eventbridge/">Amazon
EventBridge</a>.</p>
<p>EventBridge has a lot of similarities to <a href="https://stackstorm.com">StackStorm</a>, a popular open-source
cross-domain event-driven infrastructure automation platform. In some ways, you
could think of it as a very light weight and limited version of StackStorm
as a service (SaaS).</p>
<p>In this blog post I will should you how you can extend StackStorm functionality
by consuming thousands of different events which are available through Amazon
EventsBridge.</p>
<h3 id="why">Why?</h3>
<p>First of all you might ask why you would want to do that.</p>
<p><a href="https://exchange.stackstorm.org/">StackStorm Exchange</a> already offers many different packs which allows users
to integrate with various popular projects and services (including AWS). In fact,
StackStorm Exchange integration integration packs expose over 1500 different
actions.</p>
<div class="imginline">
<a href="" target="_blank"><img src="/images/2019-07-13-consuming-aws-eventbridge-events-in-stackstorm/exchange.png" class="inline" /></a>
<span class="image-caption">StackStorm Exchange aka Pack Marketplace.</span>
</div>
<p>Even though StackStorm Exchange offers integration with many different products
and services, those integrations are still limited, especially on the incoming
events / triggers side.</p>
<p>Since event-driven automation is all about the events which can trigger various
actions and business logic, the more events you have access to, the better.</p>
<p>Run a workflow which runs Ansible provision, creates a CloudFlare DNS record,
adds new server to Nagios, adds server to the loadbalancer when a new EC2
instance is started? Check.</p>
<p>Honk your Tesla Model S horn when your satellite passes and establishes a
contact with <a href="https://aws.amazon.com/ground-station/">AWS Ground Station</a>? Check.</p>
<p>Having access to many thousands of different events exposed through EventBridge
opens up almost unlimited automation possibilities.</p>
<p>For a list of some of the events supported by EventsBridge, please refer to
<a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/event-types.html">their documentation</a>.</p>
<h3 id="consuming-eventbridge-events-inside-stackstorm">Consuming EventBridge Events Inside StackStorm</h3>
<p>There are many possible ways to integrate StackStorm and EventBridge and
consume EventBridge events inside StackStorm. Some more complex than others.</p>
<p>In this post, I will describe an approach which utilizes AWS Lambda function.</p>
<p>I decided to go with AWS Lambda approach because it’s simple and straightforward.
It looks like this:</p>
<div class="imginline">
<a href="https://exchange.stackstorm.org/" target="_blank"><img src="/images/2019-07-13-consuming-aws-eventbridge-events-in-stackstorm/eventbridge_stackstorm.png" class="inline" /></a>
<span class="image-caption">AWS / partner event -> AWS EventBridge -> AWS Lambda Function -> StackStorm Webhooks API</span>
</div>
<ol>
<li>Event is generated by AWS service or a partner SaaS product</li>
<li>EventBridge rule matches an event and triggers AWS Lambda Function (rule target)</li>
<li>AWS Lambda Function sends an event to StackStorm using StackStorm Webhooks
API endpoint</li>
</ol>
<h4 id="1-create-stackstorm-rule-which-exposes-a-new-webhook">1. Create StackStorm Rule Which Exposes a New Webhook</h4>
<p>First we need to create a StackStorm rule which exposes a new <code class="language-plaintext highlighter-rouge">eventbridge</code>
webhook. This webhook will be available through
<code class="language-plaintext highlighter-rouge">https://<example.com>/api/v1/webhooks/eventbridge</code> URL.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">wget https://gist.githubusercontent.com/Kami/204a8f676c0d1de39dc841b699054a68/raw/b3d63fd7749137da76fa35ca1c34b47fd574458d/write_eventbridge_data_to_file.yaml
st2 rule create write_eventbridge_data_to_file.yaml</code></pre></figure>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">name</span><span class="pi">:</span> <span class="s2">"</span><span class="s">write_eventbridge_data_to_file"</span>
<span class="na">pack</span><span class="pi">:</span> <span class="s2">"</span><span class="s">default"</span>
<span class="na">description</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Test</span><span class="nv"> </span><span class="s">rule</span><span class="nv"> </span><span class="s">which</span><span class="nv"> </span><span class="s">writes</span><span class="nv"> </span><span class="s">AWS</span><span class="nv"> </span><span class="s">EventBridge</span><span class="nv"> </span><span class="s">event</span><span class="nv"> </span><span class="s">data</span><span class="nv"> </span><span class="s">to</span><span class="nv"> </span><span class="s">file."</span>
<span class="na">enabled</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">trigger</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">core.st2.webhook"</span>
<span class="na">parameters</span><span class="pi">:</span>
<span class="na">url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">eventbridge"</span>
<span class="na">criteria</span><span class="pi">:</span>
<span class="na">trigger.body.detail.eventSource</span><span class="pi">:</span>
<span class="na">pattern</span><span class="pi">:</span> <span class="s2">"</span><span class="s">ec2.amazonaws.com"</span>
<span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">equals"</span>
<span class="na">trigger.body.detail.eventName</span><span class="pi">:</span>
<span class="na">pattern</span><span class="pi">:</span> <span class="s2">"</span><span class="s">RunInstances"</span>
<span class="na">type</span><span class="pi">:</span> <span class="s2">"</span><span class="s">equals"</span>
<span class="na">action</span><span class="pi">:</span>
<span class="na">ref</span><span class="pi">:</span> <span class="s2">"</span><span class="s">core.local"</span>
<span class="na">parameters</span><span class="pi">:</span>
<span class="na">cmd</span><span class="pi">:</span> <span class="s2">"</span><span class="s">echo</span><span class="nv"> </span><span class="se">\"</span><span class="s">{{trigger.body}}</span><span class="se">\"</span><span class="nv"> </span><span class="s">>></span><span class="nv"> </span><span class="s">~/st2.webhook.out"</span></code></pre></figure>
<p>You can have as many rules as you want with the same webhook URL parameter.
This means you can utilize the same webhook endpoint to match as many
different events and trigger as many different actions / workflows as you want.</p>
<p>In the <code class="language-plaintext highlighter-rouge">criteria</code> field we filter on events which correspond to new EC2
instance launches (<code class="language-plaintext highlighter-rouge">eventName</code> matches <code class="language-plaintext highlighter-rouge">RunInstances</code> and <code class="language-plaintext highlighter-rouge">eventSource</code>
matches <code class="language-plaintext highlighter-rouge">ec2.amazonaws.com</code>). StackStorm <a href="https://docs.stackstorm.com/rules.html#critera-comparison[">rule criteria comparison
operators</a> are quite expressive so you can also get more creative than that.</p>
<p>As this is just an example, we simply write a body of the matched event to
a file on disk (<code class="language-plaintext highlighter-rouge">/home/stanley/st2.webhook.out</code>). In a real life scenario,
you would likely utilize <a href="https://github.com/StackStorm/orquesta">Orquesta workflow</a> which runs your complex or less
complex business logic.</p>
<p>This could involve steps and actions such as:</p>
<ul>
<li>Add new instance to the load-balancer</li>
<li>Add new instance to your monitoring system</li>
<li>Notify Slack channel new instance has been started</li>
<li>Configure your firewall for the new instance</li>
<li>Run Ansible provision on it</li>
<li>etc.</li>
</ul>
<h4 id="2-configure-and-deploy-aws-lambda-function">2. Configure and Deploy AWS Lambda Function</h4>
<p>Once your rule is configured, you need to configure and deploy AWS Lambda
function.</p>
<p>You can find code for the Lambda Python function I wrote here -
<a href="https://github.com/Kami/aws-lambda-event-to-stackstorm">https://github.com/Kami/aws-lambda-event-to-stackstorm</a>.</p>
<p>I decided to use Lambda Python environment, but the actual handler is very
simple so I could easily use JavaScript and Node.js environment instead.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">git clone https://github.com/Kami/aws-lambda-event-to-stackstorm.git
<span class="nb">cd </span>aws-lambda-event-to-stackstorm
<span class="c"># Install python-lambda package which takes care of creating and deploying</span>
<span class="c"># Lambda bundle for your</span>
pip <span class="nb">install </span>python-lambda
<span class="c"># Edit config.yaml file and make sure all the required environment variables</span>
<span class="c"># are set - things such as StackStorm Webhook URL, API key, etc.</span>
<span class="c"># vim config.yaml</span>
<span class="c"># Deploy your Lambda function</span>
<span class="c"># For that command to work, you need to have awscli package installed and</span>
<span class="c"># configured on your system (pip install --upgrade --user awscli ; aws configure)</span>
lambda deploy
<span class="c"># You can also test it locally by using the provided event.json sample event</span>
lambda invoke</code></pre></figure>
<p>You can confirm that the function has been deployed by going to the AWS console
or by running AWS CLI command:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">aws lambda list-function
aws lambda get-function <span class="nt">--function-name</span> send_event_to_stackstorm</code></pre></figure>
<p>And you can verify that it’s running by tailing the function logs:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">LAMBDA_FUNCTION_NAME</span><span class="o">=</span><span class="s2">"send_event_to_stackstorm"</span>
<span class="nv">LOG_STREAM_NAME</span><span class="o">=</span><span class="sb">`</span>aws logs describe-log-streams <span class="nt">--log-group-name</span> <span class="s2">"/aws/lambda/</span><span class="k">${</span><span class="nv">LAMBDA_FUNCTION_NAME</span><span class="k">}</span><span class="s2">"</span> <span class="nt">--query</span> logStreams[<span class="k">*</span><span class="o">]</span>.logStreamName | jq <span class="s1">'.[0]'</span> | xargs<span class="sb">`</span>
aws logs get-log-events <span class="nt">--log-group-name</span> <span class="s2">"/aws/lambda/</span><span class="k">${</span><span class="nv">LAMBDA_FUNCTION_NAME</span><span class="k">}</span><span class="s2">"</span> <span class="nt">--log-stream-name</span> <span class="s2">"</span><span class="k">${</span><span class="nv">LOG_STREAM_NAME</span><span class="k">}</span><span class="s2">"</span></code></pre></figure>
<h4 id="2-create-aws-eventbridge-rule-which-runs-your-lambda-function">2. Create AWS EventBridge Rule Which Runs Your Lambda Function</h4>
<p>Now we need to create AWS EventBridge rule which will match the events and
trigger AWS Lambda function.</p>
<div class="imginline">
<a href="/images/2019-07-13-consuming-aws-eventbridge-events-in-stackstorm/eventbridge_rule.png" target="_blank"><img src="/images/2019-07-13-consuming-aws-eventbridge-events-in-stackstorm/eventbridge_rule.png" class="inline" /></a>
<span class="image-caption">AWS EventBridge Rule Configuration</span>
</div>
<p>As you can see in the screenshot above, I simply configured the rule to send
every event to Lambda function.</p>
<p>This may be OK for testing, but for production usage, you should narrow this
down to the actual events you are interested in. If you don’t, you might get
surprised by your AWS Lambda bill - even on small AWS accounts, there are tons
of events being being constantly generated by various services and account
actions.</p>
<h4 id="3-monitor-your-stackstorm-instance-for-new-aws-eventbridge-events">3. Monitor your StackStorm Instance For New AWS EventBridge Events</h4>
<p>As soon as you configure and enable the rule, new AWS EventBridge events
(trigger instances) should start flowing into your StackStorm deployment.</p>
<p>You can monitor for new instances using <code class="language-plaintext highlighter-rouge">st2 trace list</code> and
<code class="language-plaintext highlighter-rouge">st2 trigger-instance list</code> commands.</p>
<div class="imginline">
<a href="/images/2019-07-13-consuming-aws-eventbridge-events-in-stackstorm/st2_trace_list.png" target="_blank"><img src="/images/2019-07-13-consuming-aws-eventbridge-events-in-stackstorm/st2_trace_list.png" class="inline" /></a>
<span class="image-caption">AWS EventBridge event matched StackStorm rule
criteria and triggered an action execution.</span>
</div>
<p>And as soon as a new EC2 instance is launched, your action which was defined in
the StackStorm rule above will be executed.</p>
<h4 id="conclusion">Conclusion</h4>
<p>This post showed how easy it is to consume AWS EventBridge events inside
StackStorm and tie those two services together.</p>
<p>Gaining access to many thousand of different AWS and AWS partner events
inside StackStorm opens up many new possibilities and allows you to apply
cross-domain automation to many new situations.</p>
Our airport setup (weather station, cameras and LiveATC audio feed)https://www.tomaz.me/2017/09/18/our-airport-setup-weather-station-cameras-liveatc-feed.html2017-09-18T00:00:00+02:00Tomaz Muraus<h2 id="our-airport-setup-weather-station-cameras-and-liveatc-audio-feed"><a href="/2017/09/18/our-airport-setup-weather-station-cameras-liveatc-feed.html">Our airport setup (weather station, cameras and LiveATC audio feed)</a></h2>
<p>In this post I describe how I set up weather station, live camera feed and
liveatc.net audio feed for our little airstrip.</p>
<div class="imginline">
<a href="https://www.aeroklub-zagorje.si/" target="_blank"><img src="/images/2017-09-18-our-airport-setup-weather-station-cameras-liveatc-feed/ak_website.jpg" class="inline" /></a>
<span class="image-caption">Real-time camera feed and weather information displayed on the club website.</span>
</div>
<h3 id="background">Background</h3>
<p>A while back I moved from Ljubljana to a small town around 30 km East of
Ljubljana. Before I moved here, I used to fly small single engine planes (for
fun) out of <a href="https://en.wikipedia.org/wiki/Ljubljana_Jo%C5%BEe_Pu%C4%8Dnik_Airport">Ljubljana Airport</a> and <a href="https://en.wikipedia.org/wiki/Portoro%C5%BE_Airport">Portoroz Airport</a>.</p>
<p>This means that both of those two airports are now too far to regularly fly
out of them. With no traffic, it would take around 1 hour to get to Ljubljana
Airport and around 2 hours to get to Portoroz Airport. Those two hours can
easily turn into 3 hours during peak summer time when highway is full of holiday
travelers (done that once, have no plans to repeat it).</p>
<p>Regular flying is very important to stay current and be a safe pilot so I needed
to find a new airport which is closer by. Luckily there are two airports in
the 30 minutes driving distance - <a href="https://www.aeroklub-zagorje.si/">Airport Zagorje ob Savi</a><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> and <a href="https://www.lk-sentvid.com/">Airport
Sentvid Pri Sticni</a>.</p>
<div class="imginline">
<a href="https://www.aeroklub-zagorje.si/" target="_blank"><img src="/images/2017-09-18-our-airport-setup-weather-station-cameras-liveatc-feed/zagorje_steza.png" class="inline" /></a>
<span class="image-caption">Zagorje ob Savi Airport (Vzletisce Ruardi).</span>
</div>
<p>Both of themare small, un-towered general aviation airports with a relatively
short grass runway.</p>
<p>For some reason, I decided to visit Zagorje ob Savi Airport and club first. There I
met a bunch of dedicated, friendly and welcoming people. They were the main
reason I decided to make this my new “home airport”.</p>
<p>Both of the airports I used to fly from, were bigger towered international
airports (for Slovenian standard, but in global scheme of things, they are still
really small and low traffic airports). Compared to those airports, this small
airport also feels a lot more casual, social and homey[^2].</p>
<p>As a big DIY person, I really love connecting / automating / improving things
with technology, so I also wanted to bring some of those improvements to the
airport.[^7]</p>
<p>In this post you can learn a little about our setup - notably our weather
station, web camera feed and LiveATC audio feed.</p>
<h3 id="internet-and-networking-setup">Internet and Networking Setup</h3>
<p>To be able to add some technological improvements to the airport, we first
needed an internet connection. Sadly there is no fixed internet connection at
the airport, but there is good 3G / 4G coverage so I decided to re-purpose one
of my old Android phones to act as an access point.</p>
<p>I purchased a local prepaid plan with 50 GB monthly data cap and utilized
tethering functionality on an Android phone. So far, the whole setup has been
online for almost a month and we haven’t encountered any issues yet.</p>
<p>Based on the data usage calculations I did[^4], 50 GB should also be plenty.</p>
<p>Because the devices are behind NAT and all the incoming connections and ports
are blocked, I set up an SSH tunnel and port forwarding from a Rasperry Pi
which is behind NAT to an outside cloud-server to which I have access. This way
I can tweak and change various settings, without needing to be physically
present at the airfield.</p>
<h3 id="weather-station">Weather Station</h3>
<p>Weather is one of the most important factors for flying (especially for small
single engine planes in VFR conditions) so getting a weather station was the
first thing I did.</p>
<p>It’s also worth noting that the airfield already had a wind sock before. Since
wind is a very important factor in flying, windsock is one of the many mandatory
things a place needs to be officially classified as an airfield or an airport.</p>
<div class="imginline">
<img src="/images/2017-09-18-our-airport-setup-weather-station-cameras-liveatc-feed/windsock.jpg" class="inline" />
</div>
<p>I did some research and I decided to purchase a <a href="http://www.foshk.com/Wifi_Weather_Station/WH2900.html">Fine Offset WH2900</a> weather
station.</p>
<p>Here is a list of minimum featuresthat a weather station needs to have for me to
consider the purchase.</p>
<ul>
<li>temperature, dew point data</li>
<li>weather data (direction, speed)</li>
<li>pressure data</li>
<li>rainfall data - nice to have, but not mandatory</li>
<li>ability to easily fetch data via USB or similar, or ability to automatically
send data to the internet</li>
</ul>
<div class="imginline">
<a href="http://www.foshk.com/Wifi_Weather_Station/WH2900.html" target="_blank"><img src="/images/2017-09-18-our-airport-setup-weather-station-cameras-liveatc-feed/WH2900-1.jpg" class="inline" /></a>
<span class="image-caption">Fine Offset WH2900 weather station, sold under various different brands and names.</span>
</div>
<p>This weather station offered all these features and based on the specifications and
reviews I read online, it offered the best price / performance ratio in this range
(there is of course no upper limit for the price and you could easily spend more than
1000$ for a more professional weather station).</p>
<p>It’s also worth noting that Fine Offset is a Chinese OEM and I purchased the station
from a German company on Amazon under a different name (there are many
companies which sell those weather stations under their own brand).</p>
<p>Luckily, the tower at the airport already had an antenna pole on which we could
install the station. The location of the weather station is one of the most
important factors if you want accurate data. Trees, houses and other obstacles
in the proximity can all affect the readings.</p>
<p>weathe station installing data tower</p>
<p>The pole itself offered a good location, because it’s quite high (around 6
meters) and relatively far from things which could affect the readings.</p>
<p>The setup itself was simple, because the weather station only needs to be
connected to the wireless access point and configured to send data to <a href="https://www.wunderground.com/personal-weather-station/dashboard?ID=IZAGORJE4">Weather
Underground</a>.</p>
<p>Sending data directly to WU is good because it’s easy to set up, but it also
means you are locked-in to an online service and there is no official way for
the weather station to send data to your personal server or directly grab data
from it.</p>
<p>As a big open-source proponent and believer that everyone should have access to
their own data, this was especially a big downside for me. This and lack of
security are also one of the main reasons why I’m so skeptical and worried about
the whole IoT movement.</p>
<p>Luckily, there is a great open-source project called <a href="http://www.weewx.com/">weewx</a> out there which
allows you to fetch data from the station. This project retrieves data from the
weather station by sniffing and reverse engineering the network traffic sent to
WU by the weather station.</p>
<p>Now that the station was set up, we can easily see this data in real-time on
the LCD display which also acts as a base station and online on
Weather Underground.</p>
<div class="imginline">
<a href="https://www.wunderground.com/personal-weather-station/dashboard?ID=IZAGORJE4" target="_blank"><img src="/images/2017-09-18-our-airport-setup-weather-station-cameras-liveatc-feed/wu_data.png" class="inline" /></a>
<span class="image-caption">Real-time weather data as shown on Weather Underground.</span>
</div>
<p>It would also come handy for this data to be available in concise form via text
message and over the phone so I decided to implement “poor man’s ATIS / AWOS”.</p>
<p>For that, I utilized Weather Underground API, some Python and <a href="https://www.plivo.com/">Plivo telephony
service</a>. Python script fetches real-time data using the WU API, caches it
and process it so it works with Plivo text-to-speech API.</p>
<p><span class="x-frame soundcloud" data-width="100%" data-height="166" data-video="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/342724385&color=%23ff5500&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false"></span></p>
<p>I originally tried to use <a href="https://www.twilio.com">Twilio</a> as I had experience using it in the past,
but it couldn’t get it to work with a Slovenian inbound phone number. I was
able to reach the phone number if I used Skype call or foreign phone number,
but none of the Slovenian telephony providers were able to route phone calls to
it.</p>
<p>In addition to this data (wind, pressure, temperature / dew point) being useful
to flying, we can now also observe various short and long term weather patterns
and trends. If you are a bit of a weather geek like myself, that’s quite cool.</p>
<div class="imginline">
<img src="/images/2017-09-18-our-airport-setup-weather-station-cameras-liveatc-feed/temp_graph.png" class="inline" />
<span class="image-caption">Very small temperature and dew point spread? Very likely there is fog and low clouds at the airfield.</span>
</div>
<h3 id="web-cameras">Web Cameras</h3>
<p>Ability to view real-time weather information is useful, but it would be even
better if you could also see in real-time what is going on at the airport.</p>
<p>And that’s where the web cameras come in.</p>
<p>In addition to allowing you to see what (if anything) is going on at the
airfield, cameras are a great addition to real-time weather information when
you are evaluating weather. They allow you to see real-life conditions and
things such as cloud coverage, fog / mist, etc.</p>
<p>I already had a bunch of <a href="https://foscam.com/C1.html">Foscam C1</a> wide-angle HD cameras at home so I
decided to lease / move two of them to the airfield.</p>
<div class="imginline">
<a href="https://foscam.com/C1.html" target="_blank"><img src="/images/2017-09-18-our-airport-setup-weather-station-cameras-liveatc-feed/foscam_c1.png" class="inline" /></a>
<span class="image-caption">Foscam C1. A relatively in-expensive IP camera which offers decent image quality.[^5]</span>
</div>
<p>Since the cameras are meant to be used indoors, we installed them inside the
tower to protect them from bad weather and vandalism. The picture is not crystal
clear because there is a privacy foil on the glass which makes the picture look
a bit darker then normal and there is also some glare from the sun depending on
the sun position, but it’s good enough for now.</p>
<p>Now that the cameras were installed, they needed to be configured to stream /
send images to the internet. This camera and many other IP cameras allow you to
connect to the camera to view stream in real-time or to periodically upload
image to a FTP server.</p>
<div class="imginline">
<img src="/images/2017-09-18-our-airport-setup-weather-station-cameras-liveatc-feed/webcam1.jpg" class="inline" />
<span class="image-caption">Example image from one of the cameras.</span>
</div>
<p>It’s worth noting that this and most other IP cameras are very insecure by
default and in many cases, can’t be made more secure (no SSL, many security
vulnerabilities, etc.).</p>
<p>This is not so problematic if camera is located in a public place such as is
the case here, but I still decided to make it as secure as possible.</p>
<p>The cameras are behind NAT and all the incoming connections and ports are
blocked which solves part of a problem. Second part is solved by utilizing
“periodically upload image” to FTP functionality. Sadly the camera only
supports FTP (insecure) and not FTPS (secure) so I needed to get a bit
creative.</p>
<p>I configured the cameras to send images to an FTP server which is running inside a
local network on a Rasperry Pi 3 based Linux server. Rasperry Pi then
periodically sends those images to a cloud server over FTPS.</p>
<p>This cloud server is also used to automatically process those images (create
thumbnails, optimize image size, create time-lapse gifs, add watermark, etc.)
and serve them over HTTPS.[^6]</p>
<p>Those real-time camera images can be accessed <a href="https://www.aeroklub-zagorje.me/ipcam/zagorje/">on the website</a>. In addition
to the real-time images, you can also view time-lapse videos for the past hour
on the website.</p>
<p>Time-lapse videos are generated from raw camera images every 10 minutes for
the period of now - 60 minutes. Serving the time-lapse videos presents some unique
challenges. A lot of people will access the website through a phone which can’t
(easily) play videos so instead of generating a video file, I decided to
generate an animated .gif image.</p>
<p>The problem with that is that animated gif is really a poor storage format for
what is basically a video, so I needed to experiment a bit with ImageMagick
command line options to find the combination which offers the best quality / size
ratio.</p>
<p>By default, generated .gif size was around 28-45 MB, but I managed to get it
down to 5-12MB with a reasonable quality.</p>
<p>In addition to the time-lapse videos which are generated automatically for the past
hour, raw camera images can be used to generate time-lapse videos of the
various interesting time frames (e.g. when a storm passes, etc.). An example
of such time-lapse video can be found below.</p>
<p><span class="x-frame youtube" data-width="100%" data-height="315" data-caption="Timelapse video of storm passing the airport." data-video="https://www.youtube-nocookie.com/embed/jFD1rxMcOJE"></span></p>
<h3 id="liveatc-audio-feed">LiveATC audio feed</h3>
<p>Now that the weather station and cameras were in place, the only thing missing
was real-time audio feed from the airport.</p>
<p>The airport is un-towered which means there is no ATC, but we do have UNICOM
/ CTAF station on a local sport frequency of <code class="language-plaintext highlighter-rouge">123.500</code> MHz. This station is
active when there are multiple people flying and offers traffic and runway
condition advisories.[^3]</p>
<p>Nowadays it’s very easy very to set up software based radio receiver by utilizing a
USB DVB-T TV tunner dongle based on the RTL2832U chipset. The chip covers a
very wide frequency range (24 - 1766 MHz), but if you want good sound signal
quality you need to use an antenna specifically optimized for a frequency range
you want to receive.</p>
<p>In this case, that is the airband frequency range (108 - 140 MHz). I purchased
an inexpensive 30$ airband whip antenna which seems to be doing a good job of
receiving transmission from the airplanes in vicinity. It’s also worth
noting that VHF frequencies signal follows line of sight propagation which means
the antenna itself should be mounted as high as possible and away from any
obstructions.</p>
<p>To receive the signal and stream it to the internet, I connected USB dongle to
Rasperry Pi model 3, installed <a href="http://sdr.osmocom.org/trac/wiki/rtl-sdr">rtl-sdr</a> library and <a href="https://github.com/szpajder/RTLSDR-Airband">RTLSDR-Airband</a>.</p>
<div class="imginline">
<img src="/images/2017-09-18-our-airport-setup-weather-station-cameras-liveatc-feed/rpi_stuff.jpg" class="inline" />
<span class="image-caption">Some of the used equipement.</span>
</div>
<p>RTLSDR-Airband is an open-source Linux program optimized for AM voice channel
reception and sending this data to online services such as liveatc.net.</p>
<p>Before setting up Rasperry Pi, I connected USB dongle to my computer and used
open source <a href="http://gqrx.dk/">Gqrx SDR</a> program which allows you to visualize received
signal (strength, noise, etc.) and tweak various parameters.</p>
<div class="imginline">
<a href="http://gqrx.dk/" target="_blank"><img src="/images/2017-09-18-our-airport-setup-weather-station-cameras-liveatc-feed/gqrx.jpg" class="inline" /></a>
<span class="image-caption">Gqrx is a great open-source program for software defined radio.</span>
</div>
<p>This was an important step, because simply connecting dongle to the Rasperry Pi
and setting the frequency in rtl airband config usually doesn’t yield good
results out of the box.</p>
<p>Gqrx allows you to find the optimal frequency and other settings (gain, squelch,
etc.) where the signal is the strongest and audio quality is the best.</p>
<p>If the whole thing wouldn’t work out well, I also had a backup plan. Backup
plan involved using a USB sound card and connecting headphones output on the
handheld airband radio / transceiver directly to the line-in on the sound card.</p>
<p>Once I found the best settings, I plugged dongle to Rasperry Pi, configured
rtl_airband settings and configured rtl_airband to stream audio to my self
hosted Icecast server to test it out.</p>
<p>After I confirmed everything was working, I contacted people at <a href="https://www.liveatc.net/">LiveATC</a>
for settings for their Icecast server so our feed would also be included on
LiveATC.</p>
<p>There were some issues because our airfield was the first one without an ICAO
identifier, but they were very accommodating and in the end we managed to find
a good compromise. The compromise was listing our feed under Ljubljana / LJLJ
page.</p>
<div class="imginline">
<a href="https://www.liveatc.net/search/?icao=ljlj" target="_blank"><img src="/images/2017-09-18-our-airport-setup-weather-station-cameras-liveatc-feed/liveatc_feed.png" class="inline" /></a>
<span class="image-caption">Our feed (including the archives) is available on LiveATC.net.</span>
</div>
<h3 id="future-improvements">Future Improvements</h3>
<p>The whole setup has been running less than a month so far and I already have
some ideas for future improvements and additions.</p>
<h4 id="traffic-feed">Traffic Feed</h4>
<p>Now that we have live audio feed, the only thing missing is a life traffic
feed - ability to see which airplanes are flying in the area.</p>
<p>There are already services such as flightradar24 out there which allow you
to track aircraft movements in real time, but mostly, they only track
commercial traffic via ADS-B (they also cover some GA traffic, but because of
their receiver locations, they only cover airplanes flying at higher
altitudes).</p>
<p>There are a couple of ways to track aircraft movements:</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/Automatic_dependent_surveillance_%E2%80%93_broadcast">ADS-B</a></li>
<li>Via transponder (<a href="https://en.wikipedia.org/wiki/Aviation_transponder_interrogation_modes#Mode_A_with_Mode_C">Mode C / Mode S</a>)</li>
<li><a href="https://en.wikipedia.org/wiki/FLARM">FLARM</a> (proprietary protocol, mostly used in gliders)</li>
</ul>
<p>ADS-B provides the most information, but it’s not widely used here in Slovenia
(yet).</p>
<p>Mode C and Mode S transponders are more common here. Mode C is not all that
useful because it only provides altitude, but Mode S also provides other data.</p>
<p>Both of those approaches use the same frequency (1090 MHz) so it’s possible to
receive those signals using the same RTL-SDLR USB dongle which is used to
receive audio signal for LiveATC feed.</p>
<p>So in theory, all it takes is a USB RTL-SDLR dongle plugged to Rasperry Pi,
antenna optimized for this frequency range and some software on Rasperry Pi
which interacts with RTL-SDLR and processes the received signal.</p>
<p>In fact, there are already some projects such as <a href="http://www.pilotaware.com/">PilotAware</a> out there so
we might be able to utilize existing software and won’t need to write our own.</p>
<p>pilotaware radar screen / traffic screen</p>
<h4 id="power-bank-based-ups">Power Bank Based UPS</h4>
<p>All the devices are connected to the electrical network and haven’t had any
power outage yet, but outages can happen so it would be good to be able to
connect all the devices to a UPS.</p>
<p>Luckily, all the devices consume very little power so it should be possible to
make a simple UPS a USB power bank which supports pass through charging and has
a capacity of at least 15.000 mAh.</p>
<h3 id="equipment-list-and-costs">Equipment List and Costs</h3>
<p>For the reference, here is a full list of the physical equipment used:</p>
<ul>
<li>Fine Offset 2900 Weather Station - $200$</li>
<li>2x Foscam C1 - 2x ~70$ (used to be around 100$ when I first got it,
but now you can get it for much less)</li>
<li>Rasperry Pi model 3, enclosure, micro SD card - ~80$</li>
<li>Whip airband antenna - ~25$</li>
<li>NooElec USB RTL-SDR dongle - ~30$</li>
<li>USB sound card - ~30$ (optional and only needed if you want to use airband
transceiver / scanner instead of the DVB-T tunner for receiving audio)</li>
<li>MCX to BNC adapter - ~10$</li>
</ul>
<p>In addition to that, other costs include 10$/month for a cloud server and
17$/month for a 4G data plan.</p>
<p>doesn’t meet all the local Civil Aviation Authority rules and standards for an
airport, most notably, the minimum runway length.
[^2]: “casual” and “homey” don’t need to mean unsafe. For me, safety is the
most important factor and always comes first so I never compromise on it.
[^3]: Being a non-ATC facility means it can only offers advisories and
recommendations, but no clearances or similar. Pilots perform their actions
based on their discretion at their own risk. It would be foolish to do so, but
pilots can also chose to ignore those recommendations if they wish so.
[^4]: New images are uploaded from both of the cameras every 10 seconds, and
image size is around 80-150 KB, the audio stream is 16kbps and weather station
uses negligible amount of data. For the worst case scenario, this gives a
consumption of less than 1.1 GB per day.
[^5]: As noted before, most of the popular IP cameras and other IoT devices have
a very poor security record so you should be very careful when you install them
in a private place and connect them to the internet. Ideally they should be deployed
in an isolated network which doesn’t have access to the internet.
[^6]: Free <a href="https://letsencrypt.org/">Let’s Encrypt</a> SSL certificate is used and CloudFlare is used
as a reverse proxy to act as a caching layer in-front of the webserver.
[^7]: Flying itself is rewarding and a lot of fun, but being a tech person myself,
working on side projects like this one is even more satisfactory and fun than
flying :)</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Well, technically it’s an airstrip (vzletisce) and not an airport. It <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Libcloud now supports OpenStack Identity (Keystone) API v3https://www.tomaz.me/2014/08/23/libcloud-now-supports-openstack-identity-keystone-api-v3.html2014-08-23T00:00:00+02:00Tomaz Muraus<h2 id="libcloud-now-supports-openstack-identity-keystone-api-v3"><a href="/2014/08/23/libcloud-now-supports-openstack-identity-keystone-api-v3.html">Libcloud now supports OpenStack Identity (Keystone) API v3</a></h2>
<p>I have recently pushed support for <a href="http://developer.openstack.org/api-ref-identity-v3.html">OpenStack Identity API v3</a> to Libcloud
trunk. In this blog post I’m going to have a look at the motivation for that,
changes which were involved and show some examples of how you can utilize those
changes and newly available features.</p>
<h3 id="what-is-openstack-keystone--identity-service">What is OpenStack Keystone / Identity service?</h3>
<p><a href="http://docs.openstack.org/developer/keystone/">OpenStack Keystone</a> is an OpenStack project that provides identity and
authentication related features to OpenStack projects such as Nova.</p>
<p>The project started as a simple service which only provided basic
authentication features, but it has since grown into a fully fledged and
powerful identity management service.</p>
<p>The latest version supports advanced user management, multiple projects,
complex ACLs and more.</p>
<p>Future release will also include a <a href="https://github.com/openstack/keystone-specs/blob/master/specs/juno/keystone-to-keystone-federation.rst">Keystone to Keystone federation</a> feature
which will makes things such as a seamless cross-cloud authorizations possible.</p>
<h3 id="motivation">Motivation</h3>
<p>Support for OpenStack Nova was first added to Libcloud back in 2011. First
version only included support for a simple token based authentication.</p>
<p>Since then a lot has changed and new (and more flexible) OpenStack Keystone
versions have been released. We have been pretty good at following those
changes and support for authenticating against Keystone API v2.0 has been
available in Libcloud for a long time.</p>
<p>Those changes worked fine, but the problem was that not much thinking went
into them and support for multiple Keystone versions was added after the fact.
This means that the code was hacky, inflexible, hard to re-use and extend.</p>
<p>Luckily, those things were (mostly) hidden from the end user who just wanted
to connect to the OpenStack installation. They only became apparent if you
wanted to talk directly to the Keystone service or do anything more complex
with it.</p>
<p>For one of the features we are working on at <a href="http://www.divvycloud.com/">DivvyCloud</a>, we needed support
authenticating and talking to OpenStack Keystone API v3. Since Libcloud
didn’t include support for this version yet, I decide to go ahead and add it.</p>
<p>All of the “hackiness” of the existing code also became very apparent when I
wanted to add support for API v3. Because of that, I have decided to spend
more time on it, do it “the right way” and refactor the existing code to make
it more re-usable, extensible and maintainable.</p>
<h3 id="refactoring-the-existing-code">Refactoring the existing code</h3>
<p>Before my changes, all of the logic for talking to Keystone, handling of the
token expiration, re-authentication, etc. was contained in a single class
(<code class="language-plaintext highlighter-rouge">OpenStackAuthConnection</code>).</p>
<p>To authenticate, there was one method per Keystone API version (<code class="language-plaintext highlighter-rouge">authenticate_1_0</code>,
<code class="language-plaintext highlighter-rouge">authenticate_1_1</code>, <code class="language-plaintext highlighter-rouge">authenticate_2_0_with_apikey</code>,
<code class="language-plaintext highlighter-rouge">authenticate_2_0_with_password</code>). This means there was a lot of duplicated
code, the code was hard to extend, etc.</p>
<p>I went ahead and moved to a “base class with common functionality” + “one class
per Keystone API version” model. This approach has multiple advantages over the
old one:</p>
<ul>
<li>the code is easier to re-use, maintain and extend</li>
<li>version specific functionality is available via methods on the version
specific class</li>
<li>less coupling</li>
</ul>
<p>Some other notable changes are described bellow.</p>
<h3 id="identity-related-code-has-been-moved-to-a-separate-independent-module">Identity related code has been moved to a separate (independent) module</h3>
<p>All of the identity related code has been moved from <code class="language-plaintext highlighter-rouge">libcloud.common.openstack</code> to
a new <code class="language-plaintext highlighter-rouge">libcloud.common.openstack_identity</code> module.</p>
<p>This module reduces coupling between general OpenStack and Identity related
code and makes code re-use and other things easier.</p>
<h3 id="improvements-in-the-service-catalog-related-code">Improvements in the service catalog related code</h3>
<p>Before my changes, parsed service catalog entries were stored in an
unstructured dictionary on the <code class="language-plaintext highlighter-rouge">OpenStackServiceCatalog</code> class. To make
things even worse, the structure and the contents of the dictionary differed
based on the Keystone API version.</p>
<p>Dynamic nature of Python can be a huge asset and can make development and
prototyping faster and easier. The problem is that when it’s abused /
overused it makes code hard to use, maintain and reason about. Sadly, that’s
pretty common in the Python world and many times, people tend to over-use
dictionaries and base their APIs around passing around unstructured
dictionaries.</p>
<p>I refactored the code to store service catalog entries in a structured format
(a list of <code class="language-plaintext highlighter-rouge">OpenStackServiceCatalogEntry</code> and
<code class="language-plaintext highlighter-rouge">OpenStackServiceCatalogEntryEndpoint</code> objects).</p>
<p>Now only the code which parses service catalog responses needs to know
about the response structure. The user itself doesn’t need to know anything
about the internal structure and the code for retrieving entries from the
service catalog is API version agnostic.</p>
<h3 id="addition-of-the-administrative-related-functionality">Addition of the administrative related functionality</h3>
<p>In addition to the changes mentioned above, <code class="language-plaintext highlighter-rouge">OpenStackIdentity_3_0_Connection</code>
class now also contains methods for performing different administrative related
tasks such as user, role, domain and project management.</p>
<h3 id="examples">Examples</h3>
<p>This section includes some examples which show how to use the newly available
functionality. For more information, please refer to the docstrings in the
<a href="https://github.com/apache/libcloud/blob/trunk/libcloud/common/openstack_identity.py">openstack_identity</a> module.</p>
<h3 id="authenticating-against-keystone-api-v3-using-the-openstack-compute-driver">Authenticating against Keystone API v3 using the OpenStack compute driver</h3>
<p>This example shows how to authenticate against Keystone API v3 using the
OpenStack compute driver (for the time being, default auth version used
by the compute driver is 2.0).</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span>
<span class="kn">from</span> <span class="nn">libcloud.compute.types</span> <span class="kn">import</span> <span class="n">Provider</span>
<span class="kn">from</span> <span class="nn">libcloud.compute.providers</span> <span class="kn">import</span> <span class="n">get_driver</span>
<span class="n">cls</span> <span class="o">=</span> <span class="n">get_driver</span><span class="p">(</span><span class="n">Provider</span><span class="p">.</span><span class="n">OPENSTACK</span><span class="p">)</span>
<span class="n">driver</span> <span class="o">=</span> <span class="n">cls</span><span class="p">(</span><span class="s">'<username>'</span><span class="p">,</span> <span class="s">'<password>'</span><span class="p">,</span>
<span class="n">ex_force_auth_version</span><span class="o">=</span><span class="s">'3.x_password'</span><span class="p">,</span>
<span class="n">ex_force_auth_url</span><span class="o">=</span><span class="s">'http://192.168.1.100:5000'</span><span class="p">,</span>
<span class="n">ex_force_service_type</span><span class="o">=</span><span class="s">'compute'</span><span class="p">,</span>
<span class="n">ex_force_service_region</span><span class="o">=</span><span class="s">'regionOne'</span><span class="p">,</span>
<span class="n">ex_tenant_name</span><span class="o">=</span><span class="s">'<my tenant>'</span><span class="p">)</span>
<span class="n">pprint</span><span class="p">(</span><span class="n">driver</span><span class="p">.</span><span class="n">list_nodes</span><span class="p">())</span></code></pre></figure>
<h3 id="obtaining-auth-token-scoped-to-the-domain">Obtaining auth token scoped to the domain</h3>
<p>This example show how to obtain a token which is scoped to a domain and not
to a project / tenant which is a default.</p>
<p>Keep in mind that most of the OpenStack services don’t yet support tokens which
are scoped to a domain, so such tokens are of a limited use right now.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span>
<span class="kn">from</span> <span class="nn">libcloud.common.openstack_identity</span> <span class="kn">import</span> <span class="n">OpenStackIdentity_3_0_Connection</span>
<span class="kn">from</span> <span class="nn">libcloud.common.openstack_identity</span> <span class="kn">import</span> <span class="n">OpenStackIdentityTokenScope</span>
<span class="n">driver</span> <span class="o">=</span> <span class="n">OpenStackIdentity_3_0_Connection</span><span class="p">(</span><span class="n">auth_url</span><span class="o">=</span><span class="s">'http://<host>:<port>'</span><span class="p">,</span>
<span class="n">user_id</span><span class="o">=</span><span class="s">'admin'</span><span class="p">,</span>
<span class="n">key</span><span class="o">=</span><span class="s">'<key>'</span><span class="p">,</span>
<span class="n">token_scope</span><span class="o">=</span><span class="n">OpenStackIdentityTokenScope</span><span class="p">.</span><span class="n">DOMAIN</span><span class="p">,</span>
<span class="n">domain_name</span><span class="o">=</span><span class="s">'Default'</span><span class="p">,</span>
<span class="n">tenant_name</span><span class="o">=</span><span class="s">'admin'</span><span class="p">)</span>
<span class="n">driver</span><span class="p">.</span><span class="n">authenticate</span><span class="p">()</span>
<span class="n">pprint</span><span class="p">(</span><span class="n">driver</span><span class="p">.</span><span class="n">auth_token</span><span class="p">)</span></code></pre></figure>
<h3 id="talking-directly-to-the-openstack-keystone-api-v3">Talking directly to the OpenStack Keystone API v3</h3>
<p>This example shows how to talk directly to OpenStack Keystone API v3 and
perform administrative tasks such as listing users and roles.</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">from</span> <span class="nn">pprint</span> <span class="kn">import</span> <span class="n">pprint</span>
<span class="kn">from</span> <span class="nn">libcloud.common.openstack_identity</span> <span class="kn">import</span> <span class="n">OpenStackIdentity_3_0_Connection</span>
<span class="kn">from</span> <span class="nn">libcloud.common.openstack_identity</span> <span class="kn">import</span> <span class="n">OpenStackIdentityTokenScope</span>
<span class="n">driver</span> <span class="o">=</span> <span class="n">OpenStackIdentity_3_0_Connection</span><span class="p">(</span><span class="n">auth_url</span><span class="o">=</span><span class="s">'http://<host>:<port>'</span><span class="p">,</span>
<span class="n">user_id</span><span class="o">=</span><span class="s">'admin'</span><span class="p">,</span>
<span class="n">key</span><span class="o">=</span><span class="s">'<key>'</span><span class="p">,</span>
<span class="n">token_scope</span><span class="o">=</span><span class="n">OpenStackIdentityTokenScope</span><span class="p">.</span><span class="n">PROJECT</span><span class="p">,</span>
<span class="n">tenant_name</span><span class="o">=</span><span class="s">'admin'</span><span class="p">)</span>
<span class="c1"># This call doesn't require authentication
</span><span class="n">pprint</span><span class="p">(</span><span class="n">driver</span><span class="p">.</span><span class="n">list_supported_versions</span><span class="p">())</span>
<span class="c1"># The calls bellow require authentication and admin access
# (depends on the ACL configuration)
</span><span class="n">driver</span><span class="p">.</span><span class="n">authenticate</span><span class="p">()</span>
<span class="n">users</span> <span class="o">=</span> <span class="n">driver</span><span class="p">.</span><span class="n">list_users</span><span class="p">()</span>
<span class="n">roles</span> <span class="o">=</span> <span class="n">driver</span><span class="p">.</span><span class="n">list_roles</span><span class="p">()</span>
<span class="n">pprint</span><span class="p">(</span><span class="n">users</span><span class="p">)</span>
<span class="n">pprint</span><span class="p">(</span><span class="n">roles</span><span class="p">)</span></code></pre></figure>
<h3 id="a-quick-note-on-backward-compatibility">A quick note on backward compatibility</h3>
<p>If you only use OpenStack compute driver, those changes are fully backward
compatible and you aren’t affected.</p>
<p>If you use <code class="language-plaintext highlighter-rouge">OpenStackAuthConnection</code> class to talk directly to the Keystone
installation, you need to update your code to either use the new
<code class="language-plaintext highlighter-rouge">OpenStackIdentityConnection</code> class or a version specific class since
<code class="language-plaintext highlighter-rouge">OpenStackAuthConnection</code> class has been removed.</p>
Libcloud at ApacheCon NA, 2014 in Denver, Coloradohttps://www.tomaz.me/2014/03/07/libcloud-at-apachecon-na-2014.html2014-03-07T00:00:00+01:00Tomaz Muraus<h2 id="libcloud-at-apachecon-na-2014-in-denver-colorado"><a href="/2014/03/07/libcloud-at-apachecon-na-2014.html">Libcloud at ApacheCon NA, 2014 in Denver, Colorado</a></h2>
<p>This is just a quick heads up that I will be attending <a href="http://events.linuxfoundation.org/events/apachecon-north-america">ApacheCon North
America, 2014</a> in Denver, Colorado next month.</p>
<p>I will be giving two talks. First one is titled <a href="http://events.linuxfoundation.org/events/apachecon-north-america">5 years of Libcloud</a>. This
is retrospective talk where I will tell a story of how Libcloud grew from a
small project originally developed for the needs of <a href="https://en.wikipedia.org/wiki/Cloudkick">Cloudkick</a> product into
a fully fledged and relatively popular Apache project.</p>
<div class="imginline">
<a href="http://events.linuxfoundation.org/events/apachecon-north-america" target="_blank">
<img src="/images/apachecon_denver_2014.png" class="inline" /></a>
</div>
<p>I will go into details on some of the challenges we faced, what we learned
from them and how we grew the community and the project.</p>
<p>Second one is titled <a href="http://apacheconnorthamerica2014.sched.org/event/3dd818c4a382e5c32aee514f024b2e4e">Reducing barriers to contribution to an Apache
project</a>.</p>
<p>To give you some context, I first need to say that I’m a person who loves to
move fast and loves lean and efficient approaches and teams. On top of that,
I also have zero tolerance for unnecessary / useless processes and deep and
mostly useless hierarchies.</p>
<p>All of that means I don’t see myself a big company person, where having useless
processes, which among many other things, slow innovation down is usually the
norm.</p>
<p>Apache is a relatively big organization which means it has it’s own fair share
of (useless) proceses. A lot of “new era” developers who grew up with Github
also consider Apache as slow, inflexible and place where projects go to die<sup id="fnref:fn1" role="doc-noteref"><a href="#fn:fn1" class="footnote" rel="footnote">1</a></sup>.</p>
<p>In this talk I will go into details why this is not totally true and how Apache
is (slowly) becoming more flexible and changing to adopt those new work-flows.</p>
<p>On top of that, I will also give some examples on how you can adopt those new
work-flows, iterate fast and still receive all the benefits from being an
Apache project. Those examples will be taken directly from the things we have
learned at the <a href="https://libcloud.apache.org/">Apache Libcloud</a> project.</p>
<p>Depending on how many people will attend the talk, I think it would also be
very interesting to turn this into a panel where other people can contribute
their ideas and we can discuss how to reduce barriers even further and make
Apache more attractive for “new-era projects”.</p>
<p>Besides my talks, <a href="https://twitter.com/sebgoa">Sebastien Goasguen</a>, a long time contributor to the
project who has <a href="https://libcloud.apache.org/blog/2014/02/17/sebastien-goasguen-joins-our-team.html">recently joined the PMC</a> is also giving a talk titled
<a href="http://apacheconnorthamerica2014.sched.org/event/d6f5edb63d65ca64923450993cb6c223">Apache Libcloud</a>.</p>
<p>If you are around, you should stop by to listen to those talks and contribute
your ideas to my second talk.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:fn1" role="doc-endnote">
<p>Citation needed. I’m kinda lazy and tired atm, but you can Google it up. <a href="#fnref:fn1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Libcloud Google Summer of Code 2014 Call for Participationhttps://www.tomaz.me/2014/02/11/libcloud-gsoc-cfp.html2014-02-11T00:00:00+01:00Tomaz Muraus<h2 id="libcloud-google-summer-of-code-2014-call-for-participation"><a href="/2014/02/11/libcloud-gsoc-cfp.html">Libcloud Google Summer of Code 2014 Call for Participation</a></h2>
<p>This is call for participation / proposals for all the students who are
interested in working on <a href="https://libcloud.apache.org">Apache Libcloud</a> project during the <a href="http://google-melange.appspot.com/gsoc/homepage/google/gsoc2014">Google
Summer of Code 2014 program</a>.</p>
<div class="imginline">
<a href="http://libcloud.apache.org" target="_blank">
<img src="/images/2013-12-11-libcloud-update-key-pair-management-methods-are-now-part-of-the-base-api/libcloud.png" class="inline" /></a>
</div>
<p>Before diving further, I just want to give a short disclaimer. We (Apache
Software Foundation and Libcloud as a project) haven’t been accepted into
Google Summer of Code 2014 yet, but we will apply, cross our fingers and
hope we get a spot.</p>
<p>We will know if we have been accepted on February 24th when Google publishes
a list of the accepted mentoring organizations.</p>
<h3 id="what-is-google-summer-of-code">What is Google Summer of Code?</h3>
<div class="imginline">
<a href="http://google-melange.appspot.com/gsoc/homepage/google/gsoc2014" target="_blank">
<img src="/images/2014-02-11-libcloud-gsoc-cfp/gsoc2014.jpg" class="inline" /></a>
</div>
<p><a href="http://google-melange.appspot.com/gsoc/homepage/google/gsoc2014">Google Summer of Code</a> is a program where Google sponsors students from
around the world to spend their summer working on open-source projects. Student
is paid 5500$ if they successfully complete all of their evaluations. More
information about the program can be found on the <a href="http://google-melange.appspot.com/gsoc/homepage/google/gsoc2014">project website</a>.</p>
<p>This year is also special, because it’s a tenth anniversary of the program. To
celebrate the anniversary, Google is, among other things, giving out 5500$ for
successfully completed projects instead of the usual 5000$.</p>
<h3 id="google-summer-of-code-and-libcloud">Google Summer of Code and Libcloud</h3>
<p><a href="https://www.apache.org/">Apache Software Foundation</a> is not a stranger to Google Summer of Code
since it has already participated in this program <a href="https://community.apache.org/mentoring/mentoring/experiences.html">multiple times over the past
years</a>.</p>
<p>We (as in Libcloud project) have also participated in GSoC 2014 with one
project. I have mentored a student Ilgiz Islamgulov from Russia who has worked
and successfully completed (yeah, software is never really done, but in this case
completed refers to him passing all the GSoC evaluations) a project called
<a href="https://github.com/islamgulov/libcloud.rest">Libcloud REST interface</a>.</p>
<p>If you want to know more about his project and our GSoC 2012 participation, you
should check the following links:</p>
<ul>
<li><a href="https://libcloud.apache.org/gsoc-2012.html">Libcloud GSoC 2013 page</a></li>
<li><a href="https://github.com/islamgulov/libcloud.rest">Github repository</a></li>
<li><a href="https://docs.google.com/document/d/1P9fIxILn-WdgpkXDPydHB_dghGs-BYuoSmkFwh0Y36w">Strategic plan</a></li>
<li><a href="http://islamgulov.blogspot.com/2012/08/gsoc-experience.html">Ilgiz’s post about his GSoC experience</a></li>
</ul>
<h3 id="why-should-i-participate--what-do-i-get-out-of-participating-in-gsoc">Why should I participate / What do I get out of participating in GSoC?</h3>
<p>Before looking at our call for proposals, let’s have a look at why you might
want to participate in Google Summer of Code program.</p>
<p>For a moment, lets forget about the money and have a look at a couple of other
reasons why participating in GSoC is great for you:</p>
<ul>
<li>Instead of spending your summer flipping burgers (or similar) you can spend
summer flipping bits (fun!)</li>
<li>You will learn how open source projects and communities work</li>
<li>You will get experience with working in distributed / remote teams</li>
<li>You will get experience with working across different timezones</li>
<li>You will make new friends in the open source community</li>
<li>It’s a great thing for your C.V. You will be able to show potential
employers some concrete things you have worked on.</li>
</ul>
<h3 id="libcloud-google-summer-of-code-2014-call-for-proposals">Libcloud Google Summer of Code 2014 Call For Proposals</h3>
<p>This is the part where you, dear students come in. We would like to invite
all the students who are interested in participating to start thinking about
the things they could work on and start reaching out to the community.</p>
<p>It doesn’t matter if you are already using or contributing to the project or
not, everyone is welcome (people who have or are already contributing to the
project have a slight advantage though since we already know what they are
capable of)!</p>
<p>Only pre-requisite is a good knowledge of Python, HTTP, REST APIs and a basic
familiarity with different cloud services such as Amazon EC2 and others.</p>
<p>As noted in the opening paragraph, we haven’t been accepted yet and the
student applications will open in about a month. The reason I’m already posting
this is because we want to give potential candidates plenty of time to get
familiar with the project and the community.</p>
<p>If you would like to participate, now is the right time to start <a href="http://s.apache.org/lcgsoc2014">exploring the
existing ideas</a> (you are also more than welcome to propose your own ideas),
start thinking about the things you could work, getting familiar with the code
base and start reaching out to the community.</p>
<p>Other links you might find useful:</p>
<ul>
<li><a href="http://google-melange.appspot.com/gsoc/homepage/google/gsoc2014">Official GSoC 2014 page</a></li>
<li><a href="http://en.flossmanuals.net/GSoCStudentGuide/">GSoC student guide</a></li>
<li><a href="https://libcloud.apache.org/gsoc-2012.html">Libcloud GSoC 2012 page</a></li>
<li><a href="https://libcloud.apache.org/gsoc-2014.html">Libcloud GSoC 2014 page</a></li>
<li><a href="http://s.apache.org/lcgsoc2014">Libcloud GSoC project ideas</a></li>
</ul>
New Libcloud website is now livehttps://www.tomaz.me/2014/01/26/new-libcloud-website-is-now-live.html2014-01-26T00:00:00+01:00Tomaz Muraus<h2 id="new-libcloud-website-is-now-live"><a href="/2014/01/26/new-libcloud-website-is-now-live.html">New Libcloud website is now live</a></h2>
<p>I’m happy to announce that a new Libcloud website is now live at
<a href="https://libcloud.apache.org">libcloud.apache.org</a>.</p>
<p>Design and layout wise, previous website hasn’t really changed since 2009 so
a makeover was long overdue.</p>
<div class="imginline">
<a href="https://libcloud.apache.org" target="_blank">
<img src="/images/2014-01-26-new-libcloud-website-is-now-live/libcloud_website.png" class="inline" />
</a>
<span class="image-caption">New website.</span>
</div>
<p>The new website includes many new features and improvements. One of the more
important ones is a new and fully responsive design which means that
the content can now also more easily be consumed on devices with smaller
resolutions such as mobile phones and tablets.</p>
<p>On top of that the new website is now powered by Jekyll (same as my blog) which
makes adding content and many other things easier.</p>
<p>Without further ado, I encourage you to go <a href="https://libcloud.apache.org">check out the new website</a> and
read the <a href="https://libcloud.apache.org/blog/2014/01/23/welcome-to-the-new-website.html">announcement blog post</a>.</p>
Say hello to Živahttps://www.tomaz.me/2014/01/24/say-hello-to-ziva.html2014-01-24T00:00:00+01:00Tomaz Muraus<h2 id="say-hello-to-živa"><a href="/2014/01/24/say-hello-to-ziva.html">Say hello to Živa</a></h2>
<p>This week I made a big decision. I finally decided to get a dog. In this
blog post I’m going to introduce you to my new 24kg / 52 lbs heavy fluffy
meatball friend called Živa.</p>
<div class="imginline">
<a href="/images/2014-01-24-say-hello-to-ziva/IMG_20140120_125424.jpg" class="fancybox" rel="dog">
<img src="/images/2014-01-24-say-hello-to-ziva/IMG_20140120_125424_thumb.jpg" class="inline" />
</a>
<span class="image-caption">Živa on the way to her new home.</span>
</div>
<h3 id="why-get-a-dog-in-the-first-place">Why get a dog in the first place?</h3>
<p>First I need to say that I do like cats, but I was always more of a dog person.
I always wanted my own dog, but haven’t really had a chance before.</p>
<p>When I still lived at my parents place, I couldn’t do it because we lived in an
apartment and my parents didn’t allow it.</p>
<p>Later on when I moved to the US I was thinking of getting a dog again, but after
some more thought I decided it would be irresponsible for me to do it. I did
have a nice apartment with a patio, but the problem is that dogs need a lot of
attention and I wasn’t really planning in staying in the US for the long term.
I was working a lot and leaving the dog at home by itself for a half day or
more just seems irresponsible to me.</p>
<p>On top of that, finding an apartment and landlord in San Francisco which allows
larger dogs is quite hard and makes moving very painful.</p>
<p>Another opportunity showed itself again recently when I moved back to Slovenia.
I decided that I want to stay in Europe for the foreseeable future and work
from home. On top of that, my landlord here in Ljubljana has no problem with me
having a dog inside the apartment.</p>
<p>Working from home means I can dedicate enough attention and love to the dog and
that is also one of the main reasons why I decided to get it.</p>
<h3 id="adopting-a-dog-from-a-shelter">Adopting a dog from a shelter</h3>
<p>A lot of people today buy puppies from local dog breeders. The problem is that
all puppies are cute and a lot of people don’t realize that dogs are a big
responsibility, especially when they grow up. Usually this leads to a lot of
dogs being dumped once they grow up.</p>
<p>I’m not like that so I decided to adopt a slightly older dog. In top of that,
I believe that dogs living in the shelters need more help so I decided to adopt
one from a <a href="/2014/01/24/say-hello-to-ziva.html">local shelter</a>.</p>
<p>Before visiting the shelter in person, I have visited the website and immediately
fell in love with a lovely young female called <a href="http://zavetisce-ljubljana.si/ziva-1301032106122013/default.aspx">Živa</a>. I know that dogs are
very similar to people and outside look is not everything so I didn’t keep my
hopes too high.</p>
<p>Luckily, when I visited the shelter in person it turned out that she’s one of
the friendliest dogs there. Unlike other dogs, she was very friendly, didn’t
bark and simply stuck her nose through the fence to greet me.</p>
<p>After getting to know her a little better, I came back on Monday and decided
to adopt her.</p>
<h3 id="živa">Živa</h3>
<p>Without further ado, lets get to know this lovely (not so) little creature.</p>
<p>Živa is a mixed breed (looks very much like a German shepherd) ~10 months old
female which weighs around 24 kg / 52 lbs. She was found on the street and
arrived in the shelter about two months ago. Her name is in Slovenian and
stands for lively / playful / cheery. She got this name at the shelter and
I decided to keep it since it describes her temperament really well.</p>
<div class="imginline">
<a href="/images/2014-01-24-say-hello-to-ziva/IMG_20140121_112244.jpg" class="fancybox" rel="dog">
<img src="/images/2014-01-24-say-hello-to-ziva/IMG_20140121_112244_thumb.jpg" class="inline" />
</a>
<span class="image-caption">Resting.</span>
</div>
<p>She is a very friendly, quiet and a kind dog, but she also has almost unlimited
energy and needs a tons of play time.</p>
<div class="imginline">
<a href="/images/2014-01-24-say-hello-to-ziva/IMG_20140121_115810.jpg" class="fancybox" rel="dog">
<img src="/images/2014-01-24-say-hello-to-ziva/IMG_20140121_115810_thumb.jpg" class="inline" />
</a>
<span class="image-caption">Tummy rub time.</span>
</div>
<div class="imginline">
<a href="/images/2014-01-24-say-hello-to-ziva/IMG_20140121_120532.jpg" class="fancybox" rel="dog">
<img src="/images/2014-01-24-say-hello-to-ziva/IMG_20140121_120532_thumb.jpg" class="inline" />
</a>
<span class="image-caption">Tummy rub time.</span>
</div>
<p>Even though she is very friendly she is currently still afraid of a lot of
things including moving vehicles, bridges, stairs and so on. Interesting
enough, she is not afraid of other people and dogs. She loves to jump on
and kiss people :-)</p>
<div class="imginline">
<a href="/images/2014-01-24-say-hello-to-ziva/vlcsnap-2014-01-22-20h28m23s241.png" class="fancybox" rel="dog">
<img src="/images/2014-01-24-say-hello-to-ziva/vlcsnap-2014-01-22-20h28m23s241_thumb.png" class="inline" />
</a>
<span class="image-caption">Dat look.</span>
</div>
<p>I don’t know her history, but being afraid of so many thing probably means
that her previous owner didn’t socialize her much / enough.</p>
<h3 id="first-day-with-a-dog">First day with a dog</h3>
<p>First day was very happy, but also very stressful for both of us. We both
didn’t get a lot of sleep (she was constantly checking on me and I was
checking on her) and there were some potty accidents. Potty accidents were
mostly my fault because I didn’t recognize that she needs to go to the toilet
(I thought she just wants to play).</p>
<div class="imginline">
<a href="/images/2014-01-24-say-hello-to-ziva/IMG_20140122_093219.jpg" class="fancybox" rel="dog">
<img src="/images/2014-01-24-say-hello-to-ziva/IMG_20140122_093219_thumb.jpg" class="inline" />
</a>
<span class="image-caption">Deer antler is nom nom.</span>
</div>
<div class="imginline">
<a href="/images/2014-01-24-say-hello-to-ziva/IMG_20140122_093227.jpg" class="fancybox" rel="dog">
<img src="/images/2014-01-24-say-hello-to-ziva/IMG_20140122_093227_thumb.jpg" class="inline" />
</a>
<span class="image-caption">Mmm, bone marrow...</span>
</div>
<h3 id="plans-for-the-future">Plans for the future</h3>
<p>First couple of days were great, but there is still a tons of things to do
in the future. She needs more socializing, I need to train her not to pull
and she needs to be trained to not be so afraid of many things.</p>
<p>As far as the pulling goes, I tried some manual approaches without much
success. I’ve ordered an anti pull harness which goes around her front legs and
doesn’t hurt her. Hopefully that will help.</p>
<p>On top of that, I also plan to take her to the dog school. The dog school is
actually more for me than her (this is my first dog).</p>
Migrating from Zerigo to Rackspace Cloud DNS using Libcloudhttps://www.tomaz.me/2014/01/18/migrating-from-zerigo-to-rackspace-cloud-dns-using-libcloud.html2014-01-18T00:00:00+01:00Tomaz Muraus<h2 id="migrating-from-zerigo-to-rackspace-cloud-dns-using-libcloud"><a href="/2014/01/18/migrating-from-zerigo-to-rackspace-cloud-dns-using-libcloud.html">Migrating from Zerigo to Rackspace Cloud DNS using Libcloud</a></h2>
<p>In this blog post I’m going to describe how to migrate from <a href="http://www.zerigo.com/managed-dns">Zerigo DNS</a>
to <a href="http://www.rackspace.com/cloud/dns/">Rackspace Cloud DNS</a> using a ~80 lines long Python script which utilizes
<a href="https://libcloud.apache.org/">Libcloud</a>.</p>
<div class="imginline">
<a href="http://libcloud.apache.org" target="_blank">
<img src="/images/2013-12-11-libcloud-update-key-pair-management-methods-are-now-part-of-the-base-api/libcloud.png" class="inline" /></a>
</div>
<h3 id="background-and-motivation">Background and Motivation</h3>
<p>In September of the last year, I wrote how to <a href="/2013/09/07/exporting-libcloud-dns-zone-to-bind-zone-file-format-and-migrating-between-dns-providers.html">export a Libcloud zone to the
BIND zone format</a> and use the BIND zone file to migrate between DNS
providers.</p>
<p>At that time, my motivation for migrating away from Zerigo was mostly fueled
by a very unreliable service which was a consequence of DDoS attacks and less
than ideal service architecture.</p>
<p>I have a paid Zerigo plan, so back then, I only migrated the most important
domains to a different provider. Not long after I have done this, Zerigo
announced that they have <a href="http://www.zerigo.com/article/akamai-dns-partnership">partnered with Akamai</a> and that going forward,
they will outsource running of the DNS infrastructure to Akami and as such,
the service should be way more stable and reliable.</p>
<p>I thought great, I won’t need to migrate rest of the domains away, but an
unplesant surprise came earlier this month, when Zerigo announced pricing
changes (see <a href="http://www.zerigo.com/news/notice-zerigo-dns-change-of-plans">1</a>, <a href="http://www.zerigo.com/news/zerigo-price-increase-facts">2</a>, <a href="http://www.zerigo.com/news/on-grandfathering-pre-paid-dns-accounts">3</a> & <a href="https://gist.github.com/Kami/5199908f006383dbfdcc">4</a>).</p>
<p>Previously, I have paid <strong>19$ years per year</strong>, but with a new plan which
matches my current one, I would need to pay <strong>25$ per month</strong>. That’s with
an existing customer loyalty discount. New customers will need to pay
<strong>38$ per month</strong> (what a great deal, instead of paying 24 times more,
now I need to pay <strong>just</strong> 15 times more!). Yes, you have read this correctly,
that’s more than one order of magniture per year more than I used to pay
before.</p>
<p>I honestly don’t mind paying for a great software and services and I wouldn’t
mind paying a little more if the service improved, but that kind or price
increase is simply too much. That is especially true, because all of the ~15
domains that I still have at Zerigo are used to host non-profit and community
websites and paying 25$ per month is simply too much.</p>
<h3 id="why-rackspace-cloud-dns">Why Rackspace Cloud DNS?</h3>
<p><em>Disclamer: I used to work at Rackspace, but I don’t work there anymore and
I’m not affiliated with them in any way.</em></p>
<p>Before I dive further, lets have a look at why you might want to use Rackspace
Cloud DNS.</p>
<p>The main reason for me to migrate to Rackspace is that they have a decent
API, they are supported in Libcloud and best of all, the service is totally
free for the existing cloud servers customers. On top of that, the service is
supposed to use Anycast.</p>
<p>All of that made it a good fit for hosting my non-profit domains there.</p>
<p>I also need to add that I haven’t used the service a lot before, so I can’t
really talk much about the service relablitity at this point. Only time and
monitoring will tell how reliable the service really is.</p>
<h3 id="migrating-from-zerigo-dns-to-rackspace-cloud-dns-using-libcloud">Migrating from Zerigo DNS to Rackspace Cloud DNS using Libcloud</h3>
<p>Instead of using Libcloud’s export to BIND zone file functionality, this script
works by talking directly to both of the provider APIs.</p>
<p>The reason for that is that this approach is more robust and makes
performing partial migrations and synchronizations easier. On top of that it
also works with other providers which don’t support importing a BIND zone file.</p>
<p>It’s also important to note that the script relies on some Libcloud fixes which
are currently only available in trunk. As such, you should use <code class="language-plaintext highlighter-rouge">pip</code> to
install latest version from Git inside a virtual environment:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">pip <span class="nb">install</span> <span class="nt">-e</span> git+https://github.com/apache/libcloud.git@trunk#egg<span class="o">=</span>libcloud</code></pre></figure>
<p>After you have done this, you can use the script bellow to migrate all of
your zones from Zerigo to Rackspace:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="kn">import</span> <span class="nn">hashlib</span>
<span class="kn">from</span> <span class="nn">libcloud.dns.types</span> <span class="kn">import</span> <span class="n">Provider</span><span class="p">,</span> <span class="n">RecordType</span>
<span class="kn">from</span> <span class="nn">libcloud.dns.providers</span> <span class="kn">import</span> <span class="n">get_driver</span>
<span class="n">ZERIGO_USERNAME</span> <span class="o">=</span> <span class="s">''</span>
<span class="n">ZERIGO_API_KEY</span> <span class="o">=</span> <span class="s">''</span>
<span class="n">RACKSPACE_USERNAME</span> <span class="o">=</span> <span class="s">''</span>
<span class="n">RACKSPACE_API_KEY</span> <span class="o">=</span> <span class="s">''</span>
<span class="n">CONTACT_EMAIL</span> <span class="o">=</span> <span class="s">''</span> <span class="c1"># Rackspace requires a valid email for every domain
</span>
<span class="n">ZONE_TTL</span> <span class="o">=</span> <span class="mi">30</span> <span class="o">*</span> <span class="mi">60</span> <span class="c1"># Default zone TTL (in seconds) which should be used
</span><span class="n">MIN_TTL</span> <span class="o">=</span> <span class="mi">300</span> <span class="c1"># Minim TTL supported by the target provider
</span><span class="n">IGNORED_RECORD_TYPES</span> <span class="o">=</span> <span class="p">[</span><span class="n">RecordType</span><span class="p">.</span><span class="n">NS</span><span class="p">,</span> <span class="n">RecordType</span><span class="p">.</span><span class="n">PTR</span><span class="p">]</span>
<span class="n">source_cls</span> <span class="o">=</span> <span class="n">get_driver</span><span class="p">(</span><span class="n">Provider</span><span class="p">.</span><span class="n">ZERIGO</span><span class="p">)(</span><span class="n">ZERIGO_USERNAME</span><span class="p">,</span> <span class="n">ZERIGO_API_KEY</span><span class="p">)</span>
<span class="n">destination_cls</span> <span class="o">=</span> <span class="n">get_driver</span><span class="p">(</span><span class="n">Provider</span><span class="p">.</span><span class="n">RACKSPACE</span><span class="p">)(</span><span class="n">RACKSPACE_USERNAME</span><span class="p">,</span>
<span class="n">RACKSPACE_API_KEY</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">get_record_hash</span><span class="p">(</span><span class="n">record</span><span class="p">):</span>
<span class="s">"""
Return a hash for the provided record. This is used to determine if the
record already exists.
"""</span>
<span class="n">record_hash</span> <span class="o">=</span> <span class="n">hashlib</span><span class="p">.</span><span class="n">md5</span><span class="p">(</span><span class="s">'%s-%s-%s'</span> <span class="o">%</span> <span class="p">(</span><span class="n">record</span><span class="p">.</span><span class="n">name</span><span class="p">,</span> <span class="n">record</span><span class="p">.</span><span class="nb">type</span><span class="p">,</span>
<span class="n">record</span><span class="p">.</span><span class="n">data</span><span class="p">)).</span><span class="n">hexdigest</span><span class="p">()</span>
<span class="k">return</span> <span class="n">record_hash</span>
<span class="n">source_zones</span> <span class="o">=</span> <span class="n">source_cls</span><span class="p">.</span><span class="n">list_zones</span><span class="p">()</span>
<span class="n">destination_zones</span> <span class="o">=</span> <span class="n">destination_cls</span><span class="p">.</span><span class="n">list_zones</span><span class="p">()</span>
<span class="n">destination_domains</span> <span class="o">=</span> <span class="p">[</span><span class="n">zone</span><span class="p">.</span><span class="n">domain</span> <span class="k">for</span> <span class="n">zone</span> <span class="ow">in</span> <span class="n">destination_zones</span><span class="p">]</span>
<span class="c1"># 1. Create zones
</span><span class="k">for</span> <span class="n">zone</span> <span class="ow">in</span> <span class="n">source_zones</span><span class="p">:</span>
<span class="k">if</span> <span class="n">zone</span><span class="p">.</span><span class="n">domain</span> <span class="ow">in</span> <span class="n">destination_domains</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Zone "%s" already exists, skipping...'</span> <span class="o">%</span> <span class="p">(</span><span class="n">zone</span><span class="p">.</span><span class="n">domain</span><span class="p">))</span>
<span class="k">continue</span>
<span class="n">extra</span> <span class="o">=</span> <span class="p">{</span><span class="s">'email'</span><span class="p">:</span> <span class="n">CONTACT_EMAIL</span><span class="p">}</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Creating zone: %s'</span> <span class="o">%</span> <span class="p">(</span><span class="n">zone</span><span class="p">.</span><span class="n">domain</span><span class="p">))</span>
<span class="n">destination_cls</span><span class="p">.</span><span class="n">create_zone</span><span class="p">(</span><span class="n">domain</span><span class="o">=</span><span class="n">zone</span><span class="p">.</span><span class="n">domain</span><span class="p">,</span> <span class="n">ttl</span><span class="o">=</span><span class="n">ZONE_TTL</span><span class="p">,</span>
<span class="n">extra</span><span class="o">=</span><span class="n">extra</span><span class="p">)</span>
<span class="n">destination_zones</span> <span class="o">=</span> <span class="n">destination_cls</span><span class="p">.</span><span class="n">list_zones</span><span class="p">()</span>
<span class="n">supported_record_type</span> <span class="o">=</span> <span class="n">destination_cls</span><span class="p">.</span><span class="n">list_record_types</span><span class="p">()</span>
<span class="c1"># 2. Create records
</span><span class="k">for</span> <span class="n">source_zone</span> <span class="ow">in</span> <span class="n">source_zones</span><span class="p">:</span>
<span class="n">destination_zone</span> <span class="o">=</span> <span class="p">[</span><span class="n">zone</span> <span class="k">for</span> <span class="n">zone</span> <span class="ow">in</span> <span class="n">destination_zones</span>
<span class="k">if</span> <span class="n">zone</span><span class="p">.</span><span class="n">domain</span> <span class="o">==</span> <span class="n">source_zone</span><span class="p">.</span><span class="n">domain</span><span class="p">][</span><span class="mi">0</span><span class="p">]</span>
<span class="n">source_records</span> <span class="o">=</span> <span class="n">source_zone</span><span class="p">.</span><span class="n">list_records</span><span class="p">()</span>
<span class="n">destination_records</span> <span class="o">=</span> <span class="n">destination_zone</span><span class="p">.</span><span class="n">list_records</span><span class="p">()</span>
<span class="k">for</span> <span class="n">source_record</span> <span class="ow">in</span> <span class="n">source_records</span><span class="p">:</span>
<span class="c1"># Rackspace doesn't have a special SPF record type
</span> <span class="k">if</span> <span class="n">source_record</span><span class="p">.</span><span class="nb">type</span> <span class="o">==</span> <span class="n">RecordType</span><span class="p">.</span><span class="n">SPF</span><span class="p">:</span>
<span class="n">source_record</span><span class="p">.</span><span class="nb">type</span> <span class="o">=</span> <span class="n">RecordType</span><span class="p">.</span><span class="n">TXT</span>
<span class="n">record_hash</span> <span class="o">=</span> <span class="n">get_record_hash</span><span class="p">(</span><span class="n">source_record</span><span class="p">)</span>
<span class="n">destination_record_hashes</span> <span class="o">=</span> <span class="p">[</span><span class="n">get_record_hash</span><span class="p">(</span><span class="n">record</span><span class="p">)</span> <span class="k">for</span> <span class="n">record</span>
<span class="ow">in</span> <span class="n">destination_records</span><span class="p">]</span>
<span class="k">if</span> <span class="n">source_record</span><span class="p">.</span><span class="n">name</span><span class="p">:</span>
<span class="n">fqdn</span> <span class="o">=</span> <span class="s">'%s.%s'</span> <span class="o">%</span> <span class="p">(</span><span class="n">source_record</span><span class="p">.</span><span class="n">name</span><span class="p">,</span> <span class="n">source_zone</span><span class="p">.</span><span class="n">domain</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">fqdn</span> <span class="o">=</span> <span class="n">source_zone</span><span class="p">.</span><span class="n">domain</span>
<span class="k">if</span> <span class="n">record_hash</span> <span class="ow">in</span> <span class="n">destination_record_hashes</span><span class="p">:</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Record "%s" already exists, skipping...'</span> <span class="o">%</span> <span class="p">(</span><span class="n">fqdn</span><span class="p">))</span>
<span class="k">continue</span>
<span class="k">if</span> <span class="n">source_record</span><span class="p">.</span><span class="nb">type</span> <span class="ow">in</span> <span class="n">IGNORED_RECORD_TYPES</span><span class="p">:</span>
<span class="k">print</span><span class="p">((</span><span class="s">'Encountered ignored record type (type=%s,name=%s) '</span>
<span class="s">'skipping...'</span><span class="p">)</span> <span class="o">%</span> <span class="p">(</span><span class="n">source_record</span><span class="p">.</span><span class="nb">type</span><span class="p">,</span> <span class="n">fqdn</span><span class="p">))</span>
<span class="k">continue</span>
<span class="k">if</span> <span class="nb">type</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">supported_record_type</span><span class="p">:</span>
<span class="k">print</span><span class="p">((</span><span class="s">'Encountered unsupported record type (type=%s,name=%s)'</span>
<span class="s">', skipping...'</span><span class="p">)</span> <span class="o">%</span> <span class="p">(</span><span class="n">source_record</span><span class="p">.</span><span class="nb">type</span><span class="p">,</span> <span class="n">fqdn</span><span class="p">))</span>
<span class="k">continue</span>
<span class="n">extra</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">ttl</span> <span class="o">=</span> <span class="n">source_record</span><span class="p">.</span><span class="n">extra</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'ttl'</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span>
<span class="n">priority</span> <span class="o">=</span> <span class="n">source_record</span><span class="p">.</span><span class="n">extra</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">'priority'</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span>
<span class="k">if</span> <span class="n">ttl</span><span class="p">:</span>
<span class="k">if</span> <span class="n">ttl</span> <span class="o"><</span> <span class="n">MIN_TTL</span><span class="p">:</span>
<span class="n">ttl</span> <span class="o">=</span> <span class="n">MIN_TTL</span>
<span class="n">extra</span><span class="p">[</span><span class="s">'ttl'</span><span class="p">]</span> <span class="o">=</span> <span class="n">ttl</span>
<span class="k">if</span> <span class="n">priority</span><span class="p">:</span>
<span class="n">extra</span><span class="p">[</span><span class="s">'priority'</span><span class="p">]</span> <span class="o">=</span> <span class="n">priority</span>
<span class="n">name</span> <span class="o">=</span> <span class="n">source_record</span><span class="p">.</span><span class="n">name</span>
<span class="nb">type</span> <span class="o">=</span> <span class="n">source_record</span><span class="p">.</span><span class="nb">type</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">source_record</span><span class="p">.</span><span class="n">data</span>
<span class="k">print</span><span class="p">(</span><span class="s">'Creating a record: %s'</span> <span class="o">%</span> <span class="p">(</span><span class="n">fqdn</span><span class="p">))</span>
<span class="n">destination_zone</span><span class="p">.</span><span class="n">create_record</span><span class="p">(</span><span class="n">name</span><span class="o">=</span><span class="n">name</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">type</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="n">data</span><span class="p">,</span>
<span class="n">extra</span><span class="o">=</span><span class="n">extra</span><span class="p">)</span></code></pre></figure>
<p>Before proceeeding it’s worth knowing that there are some differences between
the providers and some limitations you should be aware of:</p>
<ul>
<li>Zerigo supports more record types. If you use more advanced record types
which are not supported by Rackspace, then Rackspace might not be a good
fit for you.</li>
<li>Rackspace only allows you to create <code class="language-plaintext highlighter-rouge">PTR</code> records for resources (cloud
servers & load balancers) which are hosted in their data centers.</li>
<li>Rackspace doesn’t support <code class="language-plaintext highlighter-rouge">SPF</code> record type. This is not a big deal since
this record type has been deprecated anyway and <code class="language-plaintext highlighter-rouge">TXT</code> can be used instead.
This script transparently handled remapping of <code class="language-plaintext highlighter-rouge">SPF</code> to <code class="language-plaintext highlighter-rouge">TXT</code> for you.</li>
<li>Minimum supported TTL by Zerigo is <code class="language-plaintext highlighter-rouge">180</code> seconds and the minimum supported
TTL by Rackspace is <code class="language-plaintext highlighter-rouge">300</code> seconds. If during the migration the script
encounteres a TTL smaller than 300 seconds, it simply uses the smallest
possible TTL which is 300 seconds.</li>
</ul>
<p>To use it, simply plug in your API credentials and run it:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">python migrate_dns_providers.py</code></pre></figure>
<div class="imginline">
<a href="/images/2014-01-18-migrating-from-zerigo-to-rackspace-cloud-dns-using-libcloud/zerigo.png" class="fancybox"><img src="/images/2014-01-18-migrating-from-zerigo-to-rackspace-cloud-dns-using-libcloud/zerigo.png" class="inline" /></a>
<span class="image-caption">Zerigo control panel.</span>
</div>
<p>If the script for some reason fails half-way through (bad connectivity, API
issues, etc.), it’s safe to run it again since all the operations are
idempotent.</p>
<div class="imginline">
<a href="/images/2014-01-18-migrating-from-zerigo-to-rackspace-cloud-dns-using-libcloud/rax.png" class="fancybox"><img src="/images/2014-01-18-migrating-from-zerigo-to-rackspace-cloud-dns-using-libcloud/rax.png" class="inline" /></a>
<span class="image-caption">Rackspace Cloud DNS control panel after the migration.</span>
</div>
<p>After you have run the script, you should check if everything looks OK and if
it does, you can go ahead and change the DNS records for your domains to point
to the Rackspace Cloud DNS servers (<code class="language-plaintext highlighter-rouge">dns1.stabletransit.com</code> &
<code class="language-plaintext highlighter-rouge">dns2.stabletransit.com</code>).</p>
Programatically detecting type / platform of the Amazon Machine Imageshttps://www.tomaz.me/2014/01/12/programatically-detecting-type-platform-of-the-amazon-machine-images.html2014-01-12T00:00:00+01:00Tomaz Muraus<h2 id="programatically-detecting-type--platform-of-the-amazon-machine-images"><a href="/2014/01/12/programatically-detecting-type-platform-of-the-amazon-machine-images.html">Programatically detecting type / platform of the Amazon Machine Images</a></h2>
<p>Yesterday I was talking with one of the Libcloud users on our IRC channel. The
user was trying to figure out if there is a programmatic way to detect type of
the image used (also called a platform) by an EC2 instance (e.g. Linux, RHEL,
Windows, Windows with SQL server, etc.).</p>
<p>This information is important because the EC2 instance pricing depends on the
type of the image used (more on that bellow).</p>
<p>I was already looking into this in the past while trying to extend pricing
information which is available in Libcloud. I didn’t have much luck back then,
but I decided to look into it again and dig deeper this time.</p>
<p>After a lot of research and poking with the API, it turned out that there
still seems to be no programmatic and reliable way to determine that (if I
missed something out, please let me know).</p>
<p>In this post I’m going to have a quick look at how EC2 instance pricing works
and at some of the less than ideal approaches which can be used to determine
the image type.</p>
<h3 id="how-ec2-instance-pricing-works">How EC2 instance pricing works</h3>
<p>First lets have a quick look at how the whole EC2 instance pricing works.</p>
<p>Compared to a lot of other cloud providers, EC2 pricing is very complex and
depends on multiple factors:</p>
<ol>
<li>Region (us-east-1 us-west-1, eu-west-1, …)</li>
<li>Instance type (t1.micro, m1.small, m1.xlarge, …)</li>
<li>Image type (Linux, RHEL, SLES, Windows, Windows with SQL Server standard, …)</li>
<li>Is the instance EBS optimized</li>
<li>Is the instance on-demand, reserved or spot</li>
<li>Volume discounts</li>
<li>Data transfer</li>
<li>Other resources associated with this instance (e.g. EBS volumes)</li>
</ol>
<p>If you want to calculate an accurate instance pricing information, you need to
take into account all the factors mentioned above.</p>
<h3 id="amazon-ec2-pricing-information">Amazon EC2 pricing information</h3>
<p>Amazon offers all the pricing information in a human readable format on their
<a href="http://aws.amazon.com/ec2/pricing/">pricing page</a>, but they don’t offer a documented API which could be used
to consume this information programatically.</p>
<p>Luckily, the pricing page reads JSON files (e.g.
http://aws.amazon.com/ec2/pricing/json/linux-od.json) which can also be
consumed programatically.</p>
<div class="imginline">
<img src="/images/2014-01-12-programatically-detecting-type-platform-of-the-amazon-machine-images/bad_time.jpg" class="inline" />
</div>
<p>Those JSON files are undocumented and the bad thing with any undocumented
feature is that it could be changed or removed at any time without any prior
notice.</p>
<p>Sadly that’s the best we’ve get so far so we need to stick with it for now.</p>
<h3 id="programatically-detecting-the-image-type--platform">Programatically detecting the image type / platform</h3>
<p>I’ve spent a bunch of time researching and poking with the API and the web
interface, but I had no luck with finding an API method which would return that
information.</p>
<p><a href="http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeImages.html">DescribeImages</a> API method does return <code class="language-plaintext highlighter-rouge">platform</code> attribute, but only for
Windows based images. This means you still need to use a different approach
to detect RHEL, SLES and other type of Windows images.</p>
<p>EC2 api has some undocumented features like the undocumented <code class="language-plaintext highlighter-rouge">max-instances</code>,
<code class="language-plaintext highlighter-rouge">max-elastic-ips</code> and <code class="language-plaintext highlighter-rouge">vpc-max-elastic-ips</code> value for the <code class="language-plaintext highlighter-rouge">AttributeName</code>
filter used by the <a href="http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ApiReference-query-DescribeAccountAttributes.html">DescribeAccountAttributes</a> API method. Because of that,
I also tried a bunch of undocumented things and filter values, but I had no
luck with retrieving a <code class="language-plaintext highlighter-rouge">platform</code> attribute for all the images or retrieving
only RHEL based images.</p>
<p>The interesting thing is that the web interface does show an image type /
platform, but it seems to use a private method to obtain this information.</p>
<div class="imginline">
<a href="/images/2014-01-12-programatically-detecting-type-platform-of-the-amazon-machine-images/web_interface_platform.png" class="fancybox">
<img src="/images/2014-01-12-programatically-detecting-type-platform-of-the-amazon-machine-images/web_interface_platform_thumb.png" class="inline" />
</a>
<span class="image-caption">Image platform as displayed in the web
interface.</span>
</div>
<div class="imginline">
<a href="/images/2014-01-12-programatically-detecting-type-platform-of-the-amazon-machine-images/inspector_network_request.png" class="fancybox">
<img src="/images/2014-01-12-programatically-detecting-type-platform-of-the-amazon-machine-images/inspector_network_request_thumb.png" class="inline" />
</a>
<span class="image-caption">Web interface calls a private API method which
returns information which is not available via the public one.</span>
</div>
<h3 id="1-inferring-platform-from-the-image-details">1. Inferring platform from the image details</h3>
<p>Each image has name a name, description and a bunch of other attributes
associated with it.</p>
<p>This information can be used to infer the platform from it or to build a
static list which maps image id to a platform.</p>
<p>Inferring platform from the name and description should work reasonably
well for the standard images, but it breaks down for private or copied
images with custom names and descriptions.</p>
<p>On the other hand, the problem with a static list approach is that it doesn’t
scale and it’s time consuming and error prone to keep it up to date.</p>
<h3 id="2-scrapping-the-cloud-market-website">2. Scrapping The Cloud Market website</h3>
<p><a href="http://thecloudmarket.com/">The Cloud Market</a> website provides details (including platform / image
type) for every publicly available Amazon Machine Image.</p>
<p>This approach basically just builds on the static list approach, but instead of
putting the burden of keeping this list up to date on you, it puts it on the
Cloud Market team.</p>
<p>The Cloud Market website provides an API, but you can only retrieve details for
the images which you are owner of. This means that to retrieve a platform for
a particular image, you need to scrape the website which again is very hacky
and far from ideal.</p>
<h3 id="conclusion">Conclusion</h3>
<p>As you can see, all of the approaches I have describes are hacky and far from
ideal, but sadly that’s the best we have so far.</p>
<p>Let’s just hope Amazon will pick their stuff together and finally provide an
official API for this in the near future.</p>