logo CoolAJ86

adventures in haproxy: tcp, tls, https, ssh, openvpn

Published 2015-6-24

DRAFT

This document is not complete

Config

This is my current config for this demo, in whole: https://gist.github.com/coolaj86/2faa07aa535e6dc04639

wget https://gist.githubusercontent.com/coolaj86/2faa07aa535e6dc04639/raw/haproxy.cfg

Installation

HAProxy v1.5 on Ubuntu 12.04 through 15.04 (and beyond?).

sudo apt-get install --yes software-properties-common

sudo apt-add-repository ppa:vbernat/haproxy-1.5
sudo apt-get update

sudo apt-get install --yes haproxy

Configuration

default /etc/haproxy/haproxy.cfg:

TODO: this config was taken from 1.4 - oops. Needs to be updated to v1.5 default config

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        daemon

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client 50000
        timeout server 50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

A simple HTTPS server

We need a simple HTTPS server that we can test to see that our haproxy config works as expected.

We can install server-https from npm:

npm install --global serve-https
serve-https -p 1443 -c 'Default Server on port 1443' &

And once it has printed the Listening message we can test that it works

curl https://localhost.example.com:1443

Note that it listens on localhost.example.com, which points to localhost, and we don't need to pass the --insecure option to curl because the test certificates bundled with the server are actually valid for localhost.example.com.

A simple HTTPS passthrough

Add this to the end

frontend foo_ft_https
        mode tcp
        option tcplog
        bind 0.0.0.0:64443
        default_backend foo_bk_https

backend foo_bk_https
        mode tcp
        option tcplog
        server foo_srv_default 0.0.0.0:1443

Testing simple HTTPS passthrough

Now if we request directly to port 1443 we should get a response directly from serve-https.

If we request through port 64443 we will hit haproxy, but it should behave almost as if it had happened to port 1443 directly.

sudo killall haproxy
sudo haproxy -db -f /etc/haproxy/haproxy.cfg

Note: you must have already started the server on 1443 as mentioned in the previous section.

curl https://localhost.example.com:64443

Note: if you see (35) Unknown SSL protocol error in connection it's probably because you don't have the "backend" server running on 1443.

Adding SNI routing

We want to test that we can have multiple HTTPS servers running and that we can switch them based on their Virtual Host, which will be presented as Server Name Indication (SNI) during the TLS (SSL) handshake.

frontend foo_ft_https
        mode tcp
        option tcplog
        bind 0.0.0.0:64443

        tcp-request inspect-delay 5s
        tcp-request content accept if { req.ssl_hello_type 1 }

        acl foo_app_bar req.ssl_sni -i bar.example.com
        acl foo_app_baz req.ssl_sni -i baz.example.com

        use_backend foo_bk_bar if foo_app_bar
        use_backend foo_bk_baz if foo_app_baz

        default_backend foo_bk_default

backend foo_bk_default
        mode tcp
        option tcplog

        server foo_srv_default 127.0.0.1:1443

backend foo_bk_bar
        mode tcp
        option tcplog

        server foo_srv_bar 127.0.0.1:2443

backend foo_bk_baz
        mode tcp
        option tcplog

        server foo_srv_baz 127.0.0.1:3443

serve-https 2443 -c 'bar.example.com Serving on port 2443' &
serve-https 3443 -c 'baz.example.com Serving on port 3443' &

Next we need to alter /etc/hosts to include our fake domains (because the 'Host' header has no effect on cURL's handling of SNI).

sudo vim /etc/hosts:

127.0.0.1       foo.example.com
127.0.0.1       bar.example.com
127.0.0.1       baz.example.com
127.0.0.1       localhost

Now we should be able to make requests.

# this should work just as before
curl https://localhost.example.com:64443

# this should return the message for our 'default' site
curl --insecure https://foo.example.com:64443

# this should return the message for our 'bar' site
curl --insecure https://bar.example.com:64443

# this should return the message for our 'baz' site
curl --insecure https://baz.example.com:64443

Detecting standard SSH

If that worked great for you, let's continue on. Now detect ssh and separate it from the https connections.

We could do this a few ways, but the best is probably to look for the hard-coded string SSH-2.0 as the first bits of the packet, which is what the spec requires:

frontend foo_ft_https
        mode tcp
        option tcplog
        bind 0.0.0.0:64443

        tcp-request inspect-delay 5s
        tcp-request content accept if { req.ssl_hello_type 1 }

        #
        # SSH
        # match the hex version of 'SSH-2.0'
        #
        acl foo_proto_ssh payload(1,7) -m bin 5353482d322e30
        tcp-request content accept if foo_proto_ssh

        acl foo_app_bar req.ssl_sni -i bar.example.com
        acl foo_app_baz req.ssl_sni -i baz.example.com

        #
        # SSH
        #
        use_backend foo_bk_ssh if foo_proto_ssh

        use_backend foo_bk_bar if foo_app_bar
        use_backend foo_bk_baz if foo_app_baz

        default_backend foo_bk_default

backend foo_bk_ssh
        mode tcp
        option tcplog

        # add a nice, long timeout
        # (so that the ssh client doesn't get disconnected every few minutes)
        timeout server 2h
        server foo_srv_ssh 127.0.0.1:22

Alternatively we could just match anything that doesn't send SNI, but that's nowhere near as clean of a solution, IMNSHO.

        #
        # Alternative Method (frontend)
        # match anything that doesn't send SNI
        # (not a great idea)
        #
        acl foo_has_sni req.ssl_sni -m found
        use_backend ssh if !foo_has_sni

Anyway, now we can test after restarting the server.

# this should return the message for our 'default 1443' site
curl --insecure https://foo.example.com:64443

# this should return the message for our 'baz 3443' site
curl --insecure https://baz.example.com:64443

# this should give us an ssh prompt (ctrl+c to cancel)
ssh localhost -p 64443

Potential Problem: If the SSH client you use waits for the server's "connection string" before sending its own (with the SSH-2.0) then it would fallthru to the default backend.

If we use a SSH-over-HTTPS proxy, we can avoid that (more to come in a moment).

See also http://blog.chmd.fr/ssh-over-ssl-episode-4-a-haproxy-based-configuration.html

Detecting SSH over HTTPS

Okay, now let's say that we want to be super tricky and run ssh over https (and, y'know, defeat that crazy library, school, or corporate firewall 'n' stuff).

/etc/hosts:

127.0.0.1 shell.example.com

If we want to run this in conjuction with our exist haproxy service, we'll actually need to either run a separate instance of haproxy or set up internal proxies in this one.

/etc/haproxy/haproxy.cfg:

frontend foo_ft_https
        ...

        # in the frontend we add another SNI 'vhost'
        acl foo_proto_sslh req.ssl_sni -i shell.example.com

        use_backend foo_proxy_sslh if foo_proto_sslh

        ...

frontend foo_ft_sslh
        bind 127.0.0.1:22443 ssl crt /etc/haproxy/certs/shell.example.com.bundle.pem accept-proxy

        use_backend foo_bk_sslh


backend foo_bk_sslh
        mode tcp
        option tcplog

        #timeout server 2h
        server foo_srv_ssl 127.0.0.1:22

backend foo_proxy_sslh
        mode tcp
        option tcplog

        #timeout server 2h
        server foo_srv_sslh 127.0.0.1:22443 send-proxy

Believe it or not, this actually works!

We can test it with openssl s_client:

openssl s_client \
  -quiet \
  -connect shell.example.com:64443 \
  -servername shell.example.com \
  -CAfile /etc/haproxy/certs/ca/my-root-ca.crt.pem

The two perhaps non-obvious things are

  • -servername sets SNI
  • -CAfile is where you specify your self-signed Root CA

Note: You cannot serve a self-signed cert directly. A self-signed cert is considered a "Root Certificate Authority" and is not allowed to be used as a certificate. You may use the Root CA that you create to sign another certificate, however, and this is valid.

TODO: make testing certificates available and link to other article

Now we can test with actual ssh.

~/.ssh/config:

Host shell.example.com
  ProxyCommand openssl s_client -quiet -connect shell.example.com:64443 -servername shell.example.com -CAfile /etc/haproxy/certs/ca/my-root-ca.crt.pem

On Windows you need to follow these insructions: http://blog.chmd.fr/ssh-over-ssl-episode-4-a-haproxy-based-configuration.html

If your domain has a valid certificate that is already listed in your operating system's certificate chain, you can exclude -CAfile, otherwise you'll need to find the Root CA chain from your SSL issuer's website and create a PEM bundle for openssl to use.

I've found that most of the certificates I've bought this past year from name.com work perfectly fine in every browser, but they haven't yet made it in with the system updates to OS X's Yosemite's system chain.

Adding OpenVPN to the mix

There are two easy ways to acheive connectian OpenVPN. There are several medium to hard ways.

Easiest: Don't use haproxy. Run OpenVPN on udp 1194. Or, if you want to get past a firewall that is tricky but not that tricky, use udp 53 (normally DNS).

Easy: Use OpenVPN over SSH

~/.ssh/config:

Host shell.example.com
  ProxyCommand openssl s_client -quiet -connect shell.example.com:64443 -servername shell.example.com -CAfile /etc/haproxy/certs/ca/my-root-ca.crt.pem
  DynamicForward 1080

That sounds a bit crazy because you'd now have 3 layers of encryption, but you can also run OpenVPN listening only to localhost, with encryption turned off.

You would need to use the socks option with OpenSSL

Fairy Easy:

I haven't tested this yet, but I just realized that socat may be able to do the work of found out that

socat TCP-LISTEN:61194 OPENSSL:vnet.example.com:443,verify=0

# or
socat stdio openssl-connect:vnet.example.com:433,cert=$HOME/etc/client.pem,cafile=$HOME/etc/server.crt

See OpenSSL Client at http://www.dest-unreach.org/socat/doc/socat-openssltunnel.html

The ovpn config would change to connect to localhost tcp 61194.

If that's correct, this would be a better solution than the SSH SOCKS proxy.

Not Hard: Use OpenVPN with SNI

I haven't tested this yet, but it looks like the tls-remote option allows you to specify which certificate you would like to request - using servername indication (SNI), I would assume.

This is equivalent to what we did with SSH over HTTPS, except that OpenVPN is not actually running over HTTPS. It's running over it's own custom form of TLS.

More Difficult: stunnel and company

There are a number of tools to allow you to tunnel OpenVPN through https, but unlike SSH, there's no simple line of openssl s_client to throw into your ovpn or tunnelblick config file.

DRAFT

This post is not finished, but what I have so far works.

Where to turn for help

Well... I am available for hire, but you can also try the well-kept secret of haproxy's mailing list:

mailing list

To subscribe: send an empty message to haproxy+subscribe@formilux.org

Note: you will not get a confirmation

You can immediately begin to post to haproxy@formilux.org

It will immediately show up in the archives at https://marc.info/?l=haproxy

You can stop receiving list messages by sending an empty email to haproxy+unsubscribe@formilux.org

haproxy tag on stackoverflow

https://stackoverflow.com/questions/tagged/haproxy

Errors and Such

it sometimes works

haproxys default behavior is to run as a daemon with SO_REUSEADDR. If you were playing around and at some point started a server that you didn't kill (very easy to do by mistake), you will have incredibly random results.

sudo killall haproxy

In fact I'd reccommend trying this:

#!/bin/bash

sudo killall haproxy
sudo rm -f /var/log/haproxy*
sudo service rsyslog restart
sleep 0.3
echo Starting HAProxy
sudo haproxy -db -f /etc/haproxy/haproxy.cfg

Or, if you prefer the one-liner

sudo bash -c "killall haproxy; rm -f /var/log/haproxy*; service rsyslog restart; sleep 0.3; echo Starting HAProxy; haproxy -db -f /etc/haproxy/haproxy.cfg

it never works in curl

You're on OS X and you have that version of curl that doesn't send SNI - but only when the --insecure option is used.

See Create your own certificate authority (for testing).

(35) Unknown SSL protocol error in connection

Did you start a server on the port that you're using as a backend? If haproxy is listening on 64443, you're forwarding to 1443, but nothing is listeneng, you'll get this error.

error detected while parsing

error detected while parsing - you're using an old version of haproxy. Double check haproxy --help and upgrade to 1.5 or later.

Also, there's also the off chance that you mistyped something.

connection reset or empty reply

If you see any of these, know that they all pretty much translate to "you forgot to put the 'https://' in front".

  • The connection was reset
  • (52) Empty reply from server
  • the server unexpectedly dropped the connection

Chrome will sometimes actually try for https even when you forget to put it in front, but it's quite common for Firefox, Safari, and cURL to assume the insecure protocol when you don't specify.

sendto logger #1 failed

Right after installing haproxy I would always get this error when I ran it:

[ALERT] 182/212643 (27154) : sendto logger #1 failed: No such file or directory (errno=2)

First try restarting the log daemon and then restarting haproxy.

sudo service rsyslog restart
sudo killall haproxy

sudo haproxy -db -f /etc/haproxy/haproxy.cfg

If you still get the error you might have to adjust the default settings.

TODO: I should extrapolate this blog

Take a look at the posts above and check on and or edit these files accordingly

  • /etc/haproxy/haproxy.cfg
  • /dev/log
  • /etc/rsyslog.d/49-haproxy.conf
  • /var/lib/haproxy/dev/log
  • /var/log/haproxy.log

By AJ ONeal

If you loved this and want more like it, sign up!


Did I make your day?
Buy me a coffeeBuy me a coffee  

(you can learn about the bigger picture I'm working towards on my patreon page )