Deploying HTTPS everywhere using Let's Encrypt, Docker, Cron, and a dash of Bash
HTTPS (aka HTTP over TLS) is the industry-standard in securing communications between servers and clients in the context of the web. It relies on cryptographic certificates, which are issued by an authority (CA) that’s trusted by browsers. Up until now, getting certificates was costly and/or took several hours at the very least. Let’s Encrypt is a new non-profit CA that lets certificates be issued for free, in a matter of minutes, and in a completely automated fashion, without compromising security.
Prologue
There are several things that need to be understood prior to getting started with Let’s Encrypt (from here on, “LE”):
- LE certificates expire after 90 days. (This is a feature, not a bug.)
- Obtaining, revoking, and renewing certificates is done through an API. (“Traditional” GUIs exist, but are not the primary mean of interaction with this CA.)
- LE does not offer wildcard certificates. (But you probably don’t need those anymore.)
- LE verifies that the domain you’re asking the certificate for is pointing to a web server you control. (Semantics!)
Short-lived certificates are better for security, easier than revocation, and more efficient than OCSP. In case of breach, a short-lived certificate can simply stop being issued instead of needing to revoke the certificate. After the certificate expires, browsers will immediately display an error, without having to check revocation lists or OCSP servers (which can take time or even simply fail, leaving the insecure certificate looking “valid” for some clients).
The issue with short-lived certificates is that you need to update your certificates regularly. So far, only large companies with high-value domains were able to do so, presumably through special arrangements with their CA. LE solves this by issuing certificates very quickly and in an automated way for everyone.
They do trade off some security for convenience, though, as their certificates have 3 months expiry, instead of, say, 7 days. An attacker getting hold of one of your certificates can still use it for a few weeks without the expiry warning kicking in, and cause some amount of damage. LE does provide revocation services, though, so you can use both approaches as needed. Consult their documentation and/or security experts for more details.
LE verifies that you can obtain certificates for a domain in much the same way that services verify that an email belongs to you. Basically:
- Query the LE servers for a token
- Place that token in a webserver listening on port 443 at the path
/.well-known/acme-challenge/... - Tell LE to perform domain verification (DV)
- LE issues a request to
abc.example.com(or whatever domain a DV was requested for), looking for the path as above - Hopefully DNS resolves, pointing to your server, and LE validates your request.
The actual process is more strictly specified, involves a lot more security considerations, and is formalised under a protocol named ACME, which is currently undergoing editing to be published as an IETF standard. But a basic understanding helps when troubleshooting issues.
LE generates certificates with as many SANs as you specify, creating multi-domain certificates. These can be completely unrelated domains, so you can have the same certificate for foobar.com, lookatmy.cat, and some.other.unrelated.site.
Wildcard certificates are often used when you have many subdomains pointing at the same server. In our case, we have various subdomains on mckay.co.nz. Securing these domains would have required as many certificates as there were subs, and the need to issue a new certificate every time we set up a new sub. A wildcard certificate, although costly, would make the process of adding subs more convenient.
Another example is GitHub Pages and Heroku. Both offer a subdomain on their namespace for hosting a static site and routing traffic to your app, respectively. There are more examples in the wild; these are generally referred to as “vanity” domains. Both are presumably implemented using wildcards, for *.github.io and *.herokuapp.com.
LE certificates can be issued in about a minute and can cover many domains. In both cases, these may be used to replace wildcards. In the convenience case, the speed and ease of getting and installing LE certificates negates the need of splurging for a wildcard simply for added convenience. In the vanity case, it is not unreasonable to imagine an infrastructure that would request a brand new certificate whenever a new user signs up or a new app is created, then a TLS termination layer that selects the correct certificate based on SNI.
Discussing this with my good friend Alice, another potentially awesome possibility came up: enabling HTTPS on custom domains for these ‘vanity’ services. Due to how LE works, it should not take too much more effort for the service to automagically request a certificate for your own custom domain (e.g. passcod.name instead of passcod.github.io) without you having to do anything (except buying the domain and pointing it to the service’s nameservers, of course).
TL;DR: Actually Doing This
The Let’s Encrypt documentation has various methods for installing and using the client used to request certificates. The client can actually magically configure your Apache server to use the certificates it gets, but we don’t use Apache everywhere. We use various things, mostly Nginx and HAProxy, and only sometimes Apache. So I chose to manage certificates myself. One thing that all our servers have in common? They run Docker. So using the official Let’s Encrypt container is a natural choice. It also makes updating the client a simple docker pull.
The documentation suggests running the container like this:
$ docker run -it --rm -p 443:443 -p 80:80 --name letsencrypt \
-v "/etc/letsencrypt:/etc/letsencrypt" \
-v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
quay.io/letsencrypt/letsencrypt OPTIONS
I found, however, that for troubleshooting I wanted to look at the logs in case of failure, but this container discards them all when it finishes running the command! Thus, I also bound the log directory to the outer filesystem.
The next thing is that the client has to bind to ports 80 and 443, which are obviously used by our webserver. Now, if we had more traffic coming in to our servers, I would either have to schedule a maintenance outage, or run everything at night or whenever traffic dies down, or modify our front door’s policy to let traffic coming from LE servers go to the LE container without disrupting other traffic at all. (Using a custom LE client, a webserver with built-in LE support, or just HAProxy. That last one is closer to what I would do, but I would take advantage of Docker to make it simpler. A future blog post, maybe!)
We don’t have a lot of traffic on most servers, so taking the webserver down for a minute is fine. I put everything in a script at /usr/local/bin/letsencrypt so it’s easier to invoke:
The first thing to do after setting this up is to decide what domains you want a certificate for. I made the mistake of issuing a few certificates before I nailed it down, and the clutter is annoying. Once you’ve got a list, choose which one should be the name of the certificate. LE will create a single certificate for all these domains, but it will name it as the first one. This will also be important when renewing. Pick a first domain and stick with it.
Then prepare a comma-separated list in a text editor somewhere, ready for copy-and-paste. That saves a lot of time and uncertainty. Again, I say that because I made this mistake on the first deploy. So I learned!
$ letsencrypt auth
Once first run, there are a couple screens that need to be completed, like approving the T&Cs and providing an email address for recovery. Then you get to the interesting part:
Just copy-paste the prepared string of domains, hit enter, and waaaait. The domain verification process takes time. Sometimes, with more than a few domains, it will fail to verify some domains, which will fail the entire thing. Retry. Often this is a temporary failure. If it fails consistently, you’ve got an issue. Check spelling and DNS. Re-read the Prologue and documentation for details on DV.
Server configuration
At this point we’ve got certificates. Where? Here: /etc/letsencrypt. More specifically, the only folder in there you want is /etc/letsencrypt/live/. This is helpfully generated by the LE client and contains symlinks pointing to the latest certificates and related files, so once you’ve configured your server to point at these “live” certificates, you won’t have to change the paths again.
This is where the “first domain is the name of the certificate” thing comes in. In the case of one of our servers, the main domain for this server is mckay.co.nz. So the certificates live in /etc/letsencrypt/live/mckay.co.nz/.
There are several files in there. The documentation helpfully points to what file should be used with what directive for Apache and Nginx.
The top server block is a sample of how to transform existing HTTP server blocks into HTTPS ones. The server block at the bottom catches all plain-HTTP requests and permanently redirects them to the HTTPS version. When doing the initial set up, it’s a good idea to use a 302 redirect instead, only switching over when you’re content with it all.
The ssl.inc.conf file (separated out for clarity and so multiple server blocks all can use the same thing) contains, predictably, the SSL configuration:
ssl on;
ssl_certificate /etc/letsencrypt/live/{certificate name}/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/{certificate name}/privkey.pem;
ssl_session_timeout 5m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA";
ssl_prefer_server_ciphers on;
This is all pretty straightforward. The big list of ciphers is taken from Mozilla’s Security guidelines for their own servers. (If it’s good for Mozilla, it’s good enough for us, right?) It comes in three flavours: Modern, Intermediate, and Old. Don’t use Old unless you really need to. In an ideal world, we should all use the Modern list, but unfortunately we have clients with older somewhat browsers and the Intermediate list is safest.
global
# ...
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-G
CM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDH
E-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:A
ES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-
DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
ssl-default-bind-options no-sslv3
tune.ssl.default-dh-param 2048
defaults
mode http
# ...
frontend www-https
bind 0.0.0.0:443 ssl crt /etc/letsencrypt/live/{certname}/haproxycert.pem
reqadd X-Forwarded-Proto:\ https
reqadd X-Forwarded-Port:\ 443
default_backend www-backend
backend www-backend
server www-1 localhost:80 check
This is a HAProxy config for one of our servers. This is a hybrid HAProxy-Nginx setup (Nginx config not shown), but it should be a good base for straight HAProxy setups, or other kinds of hybrid setups. The key point with HAProxy is it requires the fullchain.pem to be concatenated to the privkey.pem (in that order) into one file: that’s the config’s haproxycert.pem file.
Someone has asked how to set up LE certificates on cPanel. While I don’t have cPanel experience, and we don’t use cPanel-based hosting anywhere at McKay, this thread indicates the CABUNDLE file needs to be extracted from the fullchain.pem, it’s the second certificate in that file. To get the certificates you can either use https://gethttpsforfree.com or use LE in manual mode. There may be an automated way to set up LE within the cPanel GUI in the future.
Renewing certificates
If you’ve forgotten, right at the beginning this article states:
LE certificates expire after 90 days.
It is therefore necessary to renew certificates before that occurs. We do that on the first day of every month. This gives us two months to notice the certificate hasn’t updated, with an automatic retry at the halfway point. Do not renew right before the deadline. That’s a surefire way to mess something up and have your clients see big bad warnings.
This is a script that’s at /usr/local/bin/letsencrypt-renew on every machine with an LE setup:
-d ... lines and the $base variable are different for every server.For HAProxy setups, the service line at the bottom is changed to the relevant one, and preceded by this, to regenerate the haproxycert.pem combined file:
cd "/etc/letsencrypt/live/{certificate name}"
cat fullchain.pem privkey.pem > haproxycert.pem
Then all that’s needed is to add an entry to the Crontab, with $ crontab -e:
@monthly /usr/local/bin/letsencrypt-renew
And we’re done. Rinse, test, repeat on every server as needed, apply conditioner enjoy.