DevOps on OVH's cloud with Saltstack

Introduction

OVH is a french company that provides cheap virtual instances. I've selected a VPS, a Virtual Private Server, on Ubuntu 13.10. Its price is less than 2€ per month.

In this tutorial, I'll expose my way of using these instances using a powerful DevOps tool named Saltstack. With a single command, I can ping all the instances that I manage, ensure that all the servers run my latest NGINX configuration, etc, ... Everything is done from my Mac at home. It is reproducible. Therefore on risky configuration settings, I can recreate a virtual machine reproducing an existing environment (as described in my previous articles: Virtualize your servers and Fill up your servers automatically with goodies), checking my new configuration before applying it on a production server.

Note : For the story, I'm a simple customer of OVH. I don't own any share in this company. This is not an advertisement for their service. I only share it as they provide a good service and as VPS may be used to deploy almost any kind of software that you need. This tutorial could be applied to other cloud service.

Installing Saltstack master on my personal computer

Installation of Saltstack is done on OSX with Homebrew:

brew install saltstack

Now, we are going to increase the opened sockets capacity of OSX:

sudo launchctl limit maxfiles 4096 8192

When installed from Homebrew, Saltstack doesn't come with the default configuration files. Theses files are described in the Saltstack documentation pages: 21.6. Configuration file examples. Saltstack expects to see at least 2 files, master and minion into the uncreated /etc/salt directory. Let's fix that:

sudo mkdir /etc/salt

And copy the content of master and minion into this freshly created directory.

Before customizing your master file, you should know your IP address in the LAN. Here's a simple command that analyse your current configuration (note that I've setup it as an alias in my personal dotfiles as ips command).

ifconfig -a | perl -nle'/(\d+\.\d+\.\d+\.\d+)/ && print $1'

Now edit the /etc/master file to reflects the opened sockets capacity and as I dont' like sudoing each time I have to launch a command, add your username to the allowed users (here it's PEM, of course):

max_open_files: 8192
user: PEM
interface: 192.168.1.30
file_roots:
  base:
    - /Users/PEM/Projects/SolutionsM3/DevOps/states
pillar_roots:
  base:
    - /Users/PEM/Projects/SolutionsM3/DevOps/pillar

Note that I've setup my Saltstack master so that its all the formulas that I deploy are stored in my personal directory. This allows me to modify every deployed configuration and save them with Git once I've finished working on them. This is what DevOps is for: your infrastructure and administration as simple script files with formulas reproducible, idempotent, evolving, without connecting manually to every servers each time you need to adjust a simple variable.

Prepare your SSH configuration

First, I ensure that I can connect myself to my VPS without password. I don't like sudoing nor I don't like being asked for password when my security has been tightened. OVH sets up an OpenSSH server on all the instances.  OSX comes already bundled with OpenSSH, the client and the server. Though, there is one step that you need to do, if you haven't done it before : generating your personal SSH keychain. This is done with a simple bundled command:

ssh-keygen

This command generate the following files in your home directory:

/Users/PEM/.ssh
├── id_rsa
├── id_rsa.pub
└── known_hosts

We are going to authorize ourselves on our VPS with your public key:

cat ~/.ssh/id_rsa.pub | ssh root@vpsXXXXX.ovh.neta 'cat >> .ssh/authorized_keys'

Where XXXXX is the VPS's number that OVH has provided you.
Now connect yourself to your VPS using a simple ssh command and without password anymore:

ssh root@vpsXXXXX.ovh.net

Declaring your VPS as a minion

Normally, your Saltstack master should be visible from your minions, the distant servers that you need to manage. Using this configuration, Saltstack is able to handle thousands of servers in the blink of an eye by relying on secured AES tunnel relying on ZeroMQ.

But in my case, I'm on my personal Mac, at home, thus, in my LAN. Even if I modify my gateway, my ISP may change my IP whenever it wants it. I could setup a DynDNS service but each time I'll travel to another one location, I would be forced to set it up again. Thankfully, last year, Saltstack added an SSH transportation capabilities. It's a bit slower but it is as powerful as a real master / minion configuration using ZeroMQ. All you have to do is to create a list (a roster) of the servers that you want to manage in your /etc/salt/roster file with this info:

vpsXXXXX:
  host: vpsXXXXX.ovh.net

Now, whenever I want to check by a ping all my servers, I use a single command:

salt-ssh '*' test.ping

If I want to target a specific one:

salt-ssh 'vpsXXXXX' test.ping

With simple naming scheme, I'm able to achieve deployment of a specific package on a specific group of servers. Nifty.

We install remotely the appropriate Saltstack packages on every server:

salt-ssh '*' -r 'apt-get install -y salt-minion'

A simple example

Deploying the tree command on all my servers or checking that it has already been deployed from the comfort of my coach is done like this. In my DevOps project, I've set up 2 files:

├── pillar
└── states
    ├── top.sls
    └── tree.sls

The states/top.sls file declare all the available formulas that I want to apply on every servers. In this simple example it contains only a basic rule to install the tree command:

base:
  '*':
    - tree

And for the states/tree.sls file, just a simple call to the Saltstack's module pkg, which is able to handle almost every Linux packaging tools that I've been playing with:

tree:
  pkg:
    - installed

Time for the installation. As the Saltstack's state are idempotent, I can run this command every time I want. It will only execute it where it is required:

tree:
salt-ssh '*' -c /etc/salt state.highstate

With this installation, I'm capable of checking the tree of files exposed by NGINX on all my servers with a single command:

salt-ssh '*' -r 'tree /var/www'