rubba.ch - The perfect dev server

I had the need to create a dev server, with a single public IP address, cheap hardware, SSL support and many web applications to be installed. I found out a good way to achieve that (at least it suits my needs) so I share this how-to.

The instructions are for Ubuntu Server 14.04 but the principles are the same for other distributions. Also this how-to is meant to describe how to setup a server to easily create and manage LXC containers on it. The management and provisioning of the containers are not covered but you can still read and follow the links in this article. Also, I assume you have some experience typing commands into a shell.

PART I: Setup the server and configure networking

First get yourself a (virtual of physical) server with Ubuntu Server 14.04 installed and accessible via SSH. You must be able to get root privileges on this system. Then install the required software with:

apt-get install lxc libvirt-bin iptables-restore nginx fail2ban

Then enable IP forwarding by adding the following line in /etc/sysctl.conf:

net.ipv4.ip_forward = 1

To make the change effective you have to type

sysctl -p /etc/sysctl.conf

Then to configure iptables you can copy the following config and past it into /etc/iptables/rules.v4, don't forget to adapt the POSTROUTING directive so it matches your configuration:

# Generated by iptables-save v1.4.21
*mangle
:PREROUTING ACCEPT [2968:266152]
:INPUT ACCEPT [2668:241540]
:FORWARD ACCEPT [656:141380]
:OUTPUT ACCEPT [2064:663031]
:POSTROUTING ACCEPT [2720:804411]
COMMIT
# Completed
# Generated by iptables-save v1.4.21
*nat
:PREROUTING ACCEPT [4:584]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [10:1052]
:POSTROUTING ACCEPT [11:1380]
-A POSTROUTING -s 192.168.122.0/24 -o eth0 -j SNAT --to-source [PUBLIC_IP]
COMMIT
# Completed
# Generated by iptables-save v1.4.21
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [4:1312]
:OUTPUT ACCEPT [21:1854]
-A INPUT -i lo -j ACCEPT
-A INPUT -i virbr0 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "[netfilter] " --log-level 7
-A INPUT -j DROP
COMMIT
# Completed

About fail2ban: it should already be configured correctly, nevertheless you may want to customize it under /etc/fail2ban/.

At this point everything is ready to create containers, the server have all the software installed and it's able to route the traffic correctly.

PART II: Manage and provision LXC containers

I strongly recommends to use unprivileged containers as described in this article but with a more generic user called "lxc". When you could successfully create a dummy container you can continue with the steps below. Unprivileged containers may not work on all distributions, even if you're running LXC >= 1.0. In this case you should use qemu/kvm or standard privileged containers but in this case make sure you don't host anything critical.

Don't forget to assign static IP to each controller, it will be needed to reach the containers afterward.

PART III: Accessing and working with the container

I assume you can create DNS entries pointing to the host IP for each service, if you don't have DNS servers you can also just add the entries in your local host file.

First you want to access the container via ssh for convenience, the problem is that you have to jump on your host before reaching it. Here's a little trick to make that transparent, in your .ssh/config write this:

Host [HOST_FQDN]
    User [HOST_SSH_USER]
    Hostname [HOST_INTERNAL_IP]
    ForwardAgent yes

Host [CONTAINER_FQDN]
    User [CONTAINER_SSH_USER]
    HostName [CONTAINER_INTERNAL_IP]
    ProxyCommand  ssh [HOST_FQDN] netcat -w 120 %h %p

Then you can access your container by typing ssh [CONTAINER_FQDN]. To avoid typing passwords, install your public key to both the host and the container.

HTTP(S)

Maybe you noticed that we installed nginx on our host and setup two directives in iptables to allow traffic over HTTP(S). In fact it will be used as a reverse proxy in order to reach the applications running in the containers. I recommend using a SSL certificate to encrypt the traffic. You can easily find on internet how to buy or generate a self-signed one.

Here's an example of nginx HTTP proxy with a redirection from HTTP to HTTPS

server {
    listen 80 spdy;
    listen [::]:80 spdy;

    server_name .[CONTAINER_FQDN];
    location / {
        rewrite     ^(.*)   https://[CONTAINER_FQDN]$1 permanent;
    }
}
server {
        listen 443 ssl spdy;
        listen [::]:443 ssl spdy;

        server_name .[CONTAINER_FQDN];

        ssl_certificate     [HOST_PATH_TO_SSL_CERTIFICATE];
        ssl_certificate_key [HOST_PATH_TO_SSL_CERTIFICATE_key];
        include /etc/nginx/security.conf;

        location / {
                proxy_pass http://[CONTAINER_INTERNAL_IP]/;
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $remote_addr;
        }
}

And a little "bonus", the file /etc/nginx/security.conf:

# taken from https://cipherli.st/

ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
ssl_session_tickets off; # Requires nginx >= 1.5.9
ssl_stapling on; # Requires nginx >= 1.3.7
ssl_stapling_verify on; # Requires nginx => 1.3.7
resolver 208.67.222.222 208.67.220.220 valid=300s;
resolver_timeout 5s;

# https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html

ssl_dhparam /etc/ssl/certs/dhparam.pem;

Restart nginx and try to visit http://[CONTAINER_FQDN]. If you see a nginx proxy error double check that your container have a web server started on port 80.