Let's Encrypt with HAProxy

Watch on YouTube: youtu.be/1kBk97UJM5E

You may also be interested in:

• A QuickStart Guide to LetsEncrypt
• Adventures in HAProxy

The Port 443 Problem

Right now there's still a very important debate with ACME / Let's Encrypt - whether or not to only allow DVSNI traffic on ports other than 443 in production.

Here's the problem:

Let's say you run your nginx, apache, caddy, node, go, or whatever webserver on port 443...

Hmm... 'nuff said, actually.

You probably don't want to have to take down your webserver to run letsencrypt to issue, or reissue certificates - and you obviously can't use the same port for two different applications.... or can you?

Enter HAProxy

HA-Proxy or H-A-Proxy or haproxy or however you say it... anyway, it's got a neat little trick up it's sleeve when it comes to handling port 443 - it's not a webserver. It's a tcp proxy that happens to also handle http.

Using its low-level (but simple) TCP and TLS features we can divert traffic intended for *.acme.invalid (meaning <nonce>.acme.invalid) to letsencrypt and let all other traffic go to our webserver.

Installing HAProxy 1.5+

Because I'm using v1.5+ for other things, that's what I have installed and that's what I've tested with, so I know it works.

That said, the config below may work perfectly with 1.4, but I haven't tried it.

Also, I've only set this up on Ubuntu (and Raspbian on Raspberry Pi), but I imagine you can find out how to install on your distro of choice.

sudo apt-get install --yes software-properties-common

sudo apt-add-repository ppa:vbernat/haproxy-1.5
sudo apt-get update

sudo apt-get install --yes haproxy

# you must restart the logging service manually
sudo service rsyslog restart

Overload 443 for your WebServer and LetsEncrypt

NOTE: I'm only providing the whole file in case you want to try my configuration exactly. Everything in the global and defaults section is just default config that came with haproxy when I installed it.

Three important components here:

• haproxy listens on 0.0.0.0:443 - every network interface on port 443
• you'll change your current webserver to listen on 127.0.0.1:8443
• letsencrypt will run dvsni on 127.0.0.1:63443

Aside from port 443, which is the standard for HTTPS, the I picked the ports arbitrarily.

/etc/haproxy/haproxy.cfg:

###################################
 DEFAULTS (unchanged boring stuff)
###################################

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

#####################
 THE GOOD STUFF (TM)
#####################


frontend foo_ft_https
    log     global
    mode tcp
    option tcplog
    bind 0.0.0.0:443

    # wait up to 5 seconds from the time the tcp socket opens
    # until the hello packet comes in (otherwise fallthru to the default)
    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 }

    # wildcard match all SNI servernames for *.acme.invalid
    acl foo_app_letsencrypt req.ssl_sni -m end .acme.invalid
    use_backend foo_bk_letsencrypt if foo_app_letsencrypt

    # send everything that doesn't match *.acme.invalid to the default
    default_backend foo_bk_default

backend foo_bk_letsencrypt
    log     global
    mode tcp
    option tcplog

    # all letsencrypt traffic goes to 63443 locally
    server foo_srv_letsencrypt 127.0.0.1:63443

backend foo_bk_default
    log     global
    mode tcp
    option tcplog

    # all normal traffic goes to 8443 locally
    server foo_srv_default 127.0.0.1:8443

IMPORTANT: haproxy defaults to running as a daemon with SO_REUSEADDR and SO_REUSEPORT. This means that you don't get an error if an instance is already running and when you were testing you may have accidentally caused servers to run. Always do both of the following when testing:

sudo service haproxy stop
sudo killall haproxy

Test it all out

• some /etc/hosts entries pointing to localhost (or your remote server)
• a test https "main" server on 8443
• haproxy accepting on 443
• letsencrypt running at will on 63443

modify /etc/hosts

We'll explicitly want to test for localhost.acme.invalid and you'll probably want to test other things too, so go ahead and make some simple additions to your hostfile, like-a-so:

/etc/hosts:

127.0.0.1 localhost.example.com
127.0.0.1 localhost.acme.invalid

127.0.0.1 localhost

A dinky webserver

If you want a quick and dirty https server to test with, I got your back, bro:

curl -fsSL bit.ly/iojs-min | bash
npm install -g serve-https

serve-https -p 8443 -c "<h1>I'm a little teapot</h1>"

This will serve a certificate for https://localhost.example.com:8443.

However, you could certainly edit your /etc/hosts to make change it to point to your server if you wanted to.

HAProxy in the foreground

Always kill all existing instances of haproxy first - just like an "unloaded" gun, never trust your believe that you "already did it".

sudo service haproxy stop
sudo killall haproxy

sudo haproxy -db -f /etc/haproxy/haproxy.cfg

• -db for "don't background" (no daemon)
• -f for "config file"

And you can watch the log if you want, it'll probably be helpful.

sudo tail -f /var/log/haproxy.log

Test that HAProxy is Proxying

And just a quick test should reveal that the dinky server is getting forwarded properly. Try accessing both of these:

• https://localhost
• https://localhost.example.com

For both of those you should see the Teapot message.

Now you can also test https://localhost.acme.invalid, which should just hang up (because we don't have anything listening there). Just to double check you can start a server on 63443 and see if you get the correct message

serve-https -p 63443 -c "These *are* the droids you're looking for."

If that doesn't work for you then either you or I did something wrong, but I've got this config working for me, so you probably copy and pasted like a first class nincompoop, tbh.

Anyway, be sure to ctrl+c that server.

letsencrypt

Follow my quickstart or the official guide to install letsencrypt.

Then change the email and domain in the script below to your email and a real domain that points to your server.

sudo ./venv/bin/letsencrypt \
  --email john.doe@gmail.com \
  --domains foo.example.com \
  --authenticator standalone \
  --dvsni-port 63443 \
  --text \
  --agree-eula \
  auth

NOTE: letsencrypt is still in the early stages, so some of the options may change. If that happens, Don't Panic - just run with --help and figure it out - or run with no options and a terribly slow and ugly gui will pop up and "help" you (can you tell that I'm a gui hater?).

If you don't have a real domain easy accessible, but you want to play around, try using FreeDNS or similar.

Once you run it, you should see that it works. Yay!

curl gotchas

curl is great for playing around, testing SNI, etc, but be warned, it has issues on OS X (even from brew).

WARNING: cURL on OS X doesn't support SNI with the -k (--insecure) option. You can overcome this is some situations by using --capath.

Celebrate

You should have been successful in your test and now you can issue yourself certificates for this server, at will, without having to restart your server.

sudo apt-get install --yes tree

sudo tree /etc/letsencrypt

Zero Downtime in Production

You can acheive perfect 0 downtime:

• test your haproxy configuration on another machine (or port 3443 on the same machine)
• leave your webserver running on 443
• start another instance of it on 8443
• start haproxy on 443, (it'll force a double bind)
• stop your webserver on 443

If you ever need to restart haproxy itself to reread the config, you can do so like this:

haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

See http://www.mgoff.in/2010/04/18/haproxy-reloading-your-config-with-minimal-service-impact/

By AJ ONeal


Did I make your day?
Buy me a coffee  

(you can learn about the bigger picture I'm working towards on my patreon page )

---

---