Who am I?

  • Tomaz Muraus (Kami)
  • Engineer at StackStorm
  • Previously: Cloudkick, Rackspace, DivvyCloud
  • Likes: Distributed systems, open standards and open source systems, startups, tinkering / "hacking" 80's music, working out

DevOps

What is DevOps?

Description
source: http://blog.sei.cmu.edu/post.cfm/devops-agile-317

What is DevOps?

Description
source: http://www.edureka.co/blog

What is DevOps?

  • Bridging the gap between developers and operations
  • Cultural and organization shift
  • A lot of shared values and principles with agile
  • In the end (final goal), it's all about making customers happy

DevOps benefits

  • Faster iteration
  • Smaller changes, less impact
  • Happier customers / user

StackStorm

What is StackStorm?

  • Platform for integration and automation across services and tools
  • Allows you to take actions in response to events
  • IFTTT for operations
  • Open source, Apache 2.0 licensed
  • Primarily targeted towards DevOps and SREs
  • It's all code! (and some YAML)

What is StackStorm?

Description
source: http://expertintegratedsystemsblog.com

StackStorm Use Cases

Advanced CI / CD

Description

Advanced CI / CD

Description

Advanced CI / CD

Description

Facilitated Troubleshooting

Description

Closed Loop Remediation

Description

Using StackStorm for Security Automation

Description

StackStorm - Terminology and Concepts

  • Sensors - Inbound integrations (events)
  • Triggers - Contain information about a particular event
  • Actions - Outbound integrations
  • Workflows - Allow you to perform complex bussines logic
  • Rules - Wire together triggers and actions
  • Timers - Allow you to run action on a pre-defined time interval or on a specific date

StackStorm - Terminology and Concepts

Description

StackStorm - Sample Sensor

...
    def _handle_pubmsg(self, connection, event):
        trigger = 'irc.pubmsg'
        payload = {
            'source': {
                'nick': event.source.nick,
                'host': event.source.host
            },
            'channel': event.target,
            'timestamp': event.timestamp,
            'message': event.arguments[0]
        }
        self._sensor_service.dispatch(trigger=trigger, payload=payload)
...
{
    "source": {
        "nick": "Kami__",
        "host": "gateway/web/irccloud.com/x-uvv"
    },
    "channel": "#test989",
    "timestamp": 1419166748,
    "message": "this is a test message"
}

StackStorm - Sample Action

import httplib
import requests
from st2actions.runners.pythonrunner import Action

SEND_EMAIL_API_URL = 'https://api.mailgun.net/v2/%(domain)s/messages'

class SendEmailAction(Action):
    def run(self, sender, recipient, subject, text, html=None):
        domain = self.config['domain']
        api_key = self.config['api_key']

        data = {
            'from': sender,
            'to': recipient,
            'subject': subject,
            'text': text
        }

        if html:
            data['html'] = html

        api_url = SEND_EMAIL_API_URL % {'domain': domain}
        response = requests.post(api_url, auth=('api', api_key), data=data)
        return response.status_code == httplib.OK

StackStorm - Sample Rule

---
  name: "relayed_matched_irc_message"
  description: "Relay IRC message to Slack if the message contains word StackStorm"
  enabled: true

  trigger:
    type: "irc.pubmsg"
    parameters: {}

  criteria:
      trigger.message:
          type: "icontains"
          pattern: "StackStorm"

  action:
    ref: "slack.post_message"
    parameters:
        message: "{{trigger.source.nick}} on {{trigger.channel}}: {{trigger.message}}"
        channel: "#irc-relay"

StackStorm - Existing Integrations

Description

StackStorm - Existing Integrations

Description

StackStorm - Integrations

Description
source: http://expertintegratedsystemsblog.com

Remediation Demo

Remediation Demo - Setup

  • Linux Cloud Server running Ubuntu
  • Running "standard" services (Apache HTTPD, MySQL)
  • Running a "rouge" service which allows remote command execution

Remediation Demo - Setup

Description

Remediation Demo - Used Packs

  • core
  • linux
  • st2
  • libcloud
  • rackspace
  • mailgun

Remediation Demo - Workflow

Description

Remediation Demo - Workflow

  • Collect forensic information
  • Put server in a lockdown mode
  • Notify administrators

Remediation Demo - Workflow - Information Collection

  • Retrieve logged users
  • Retrieve open files
  • Retrieve listening ports
  • Retrieve established connections
  • Retrieve running processes
  • Retrieve file hashes
  • Retrieve logs from /var/log/*
  • Upload information to Rackspace CloudFiles

Remediation Demo - Workflow - Lockdown

Description

Remediation Demo - Workflow - Lockdown

  • Associate "lockdown" tag / metadata with the server
  • Firewall server from the outside world

Remediation Demo - Workflow - Notify Administrators

Description

Remediation Demo - Workflow - Notify Administrators

  • Post a message to the Slack channel
  • Send an email to the administrators

Remediation Demo - Workflow - Notify Administrators

Description

Remediation Demo - Workflow - Administrators

Description

Remediation Demo

Remediation Demo - Possible Improvements

  • Use syslog / write logs to append only datastore
  • Mount root partition as read-only
  • Clone the server (create an image)
  • Put server in a special security group
  • Use VLAN / VPC
  • Open JIRA ticket
  • Accounting on FreeBSD / Solaris / SmartOS
  • ...
  • Sky is the limit!

Go Check it Out

Thank You!