Let's Encrypt Tiny

I think all HTTP communication on the internet should be encrypted – and thanks to Let’s Encrypt, we are now much closer to this goal than we were two years ago. However, when I set up Let’s Encrypt on my server (which is more than a year ago by now), I was not very happy with the official client: The client manages multiple certificates with different sets of domains per certificate, but I found it entirely unclear which commands would replace existing certificates or create a new one. Moreover, I have some special needs: I’ve set up DNSSEC with TLSA records containing hashes of my certificates, so replacing a certificate has to also update DNS and deal with the fact that DNS entries get cached. Lucky enough, Let’s Encrypt is based on open standards, so I was not forced to use their client!

To make a long story short, I decided to write my own Let’s Encrypt client, which I describe in this post.

Let’s Encrypt Tiny

The client is based on acme-tiny, a beautifully small Python library (<200 lines) speaking the ACME protocol. That’s the protocol developed by Let’s Encrypt to communicate with an automated CA. I duly called my client “Let’s Encrypt Tiny”, and with less than 250 lines I think that name is still fair. For now, Let’s Encrypt Tiny resides in my server-scripts repository, and it will stay there until anyone else has an interesting in using it. ;)

Update: Let’s Encrypt Tiny now has its own repository. /Update

The central concept of Let’s Encrypt Tiny is a “certificate line” – a sequence of certificates, possibly for different private keys, that “belong together” in the sense that each is considered (by their owner) the successor of the previous. Services like apache are configured to use a particular certificate line for a particular domain (well, in my case, it’s the same line for all domains, but one could imagine different setups). Each certificate line has a separate config file, whose most important job is to configure the set of domains in the latest certificate of the line. The key operations on a certificate line are to create a fresh key and obtain a certificate for it, and to obtain a new certificate for the existing key. Let’s Encrypt certificates expire after 90 days, so we have to perform renewal at least that often, but there is no reason to always generate a fresh private key. One reason to keep the previous key is that the TLSA record in DNS can be configured to be a hash of just the key, so a renewal for an existing key can be done without changing DNS.

The other important concept in Let’s Encrypt Tiny is the idea of a “staging key”, as opposed to the “live key”. This is needed to properly support DNSSEC+TLSA. Just briefly, the idea of TLSA is to not use certificate authorities to determine the correct certificate for a domain (CAs have proven untrustworthy again and again, and a single failing CA undermines the security of the entire system), but instead use DNS. If DNS is secured with DNSSEC – which is much more resilient against a single entity failing than the CA system – then we can just put a hash of the certificate, or a hash of the public key, into the DNS, side-stepping the entire CA system. Unfortunately, TLSA has not seen widespread adoption, though a Firefox extension is available (and extensions for some other browsers as well). Still, I like this technology, so I have deployed it on my server.

However, one consequence of TLSA records is that a freshly generated key cannot immediately be used (i.e., by the web server): The DNS still contains the old key’s data, and that data gets cached! In Let’s Encrypt Tiny, this key first gets “staged”. Next, we have to update the DNS zone to contain TLSA records for both the old and the new key. Then we have to wait until the TTL (time-to-live) of that record passes, to make sure that no caches still contain only the old key. Finally, we “unstage” the key so all the servers (web server, jabber server, and so on) to use the new certificate and key, which are now “live”.

Configuration

Let’s look at an example: Here is the configuration file for this server, ralfj.de, with comments explaining the purpose of the various options:

# The domains currently making up this certificate line:
domains =
  ralfj.de
  www.ralfj.de
  lists.ralfj.de
  git.ralfj.de
  ns.ralfj.de
  ipv4.ns.ralfj.de
  ipv6.ns.ralfj.de
  jabber.ralfj.de
  conference.jabber.ralfj.de
# ... this list goes on.

# The size of the RSA secret key (in bits).
key-length = 4096

[timing]
# Max. age of the private key before we generate a new one.
max-key-age-days = 256
# How long a new private key is "staged" before it is used as the live key.
# 0 disables staging.
staging-hours = 0
# How many days before the certificate expires should be request a new one?
renew-cert-before-expiry-days = 15

[hooks]
# Script to execute after the certificate changed.
post-certchange = /root/letsencrypt/cert-hook
# Script to execute after the key changed.  Only needed when using DNSSEC+TLSA.
#post-keychange = /root/letsencrypt/key-hook

[acme]
# File storing the ACME account private key (created if missing).
account-key = /etc/ssl/private/letsencrypt/account.key
# Directory to put the ACME challenges into.  Must be mapped to
# /.well-known/acme-challenge on all domains listed above.  For example, in
# apache, put the following directive somewhere global:
#   Alias /.well-known/acme-challenge/ /srv/acme-challenge/
challenge-dir = /srv/acme-challenge/

[dirs]
# Directory for certificates.
certs = /etc/ssl/mycerts/letsencrypt
# Directory for private keys.
keys = /etc/ssl/private/letsencrypt
# A place to put old certificates and keys.
backups = /etc/ssl/old/letsencrypt

[files]
# Filename prefix (without extension) for the live key and certificate.
live = live
# Filename prefix (without extension) for the staging key and certificate.
staging = staging

With this configuration, Let’s Encrypt Tiny creates files /etc/ssl/mycerts/letsencrypt/live.crt and /etc/ssl/private/letsencrypt/live.key. These files are always the “tip” of the certificate line and should be configured in the various servers – however, most servers will need these files to be massaged a bit. First of all, we also need a key chain, and the intermediate CA used by Let’s Encrypt actually changes over time. Moreover, some servers want certificate and key in one file, while others want the certificates to be bundled with the keychain and expect the private key in a separate file. Sometimes, the Diffie-Hellman parameters are also expected in the same file as the certificate – every SSL-supporting server seems to handle this slightly differently.

This is all handled by the certificate hook, which creates the various derived files:

#!/bin/sh
cd /etc/ssl/mycerts/letsencrypt
export PATH="/usr/sbin/:/sbin/:$PATH"

# Determine the intermediate CA used by this certificate.  We expect the
# intermediate certificates to be stored in files in /etc/ssl/chains/,
# e.g. /etc/ssl/chains/letsencrypt-X3.crt.
ISSUER=$(openssl x509 -issuer -in live.crt -noout | sed 's/.*Authority \(X[0-9]\+\).*/\1/')
ISSUER_FILE=/etc/ssl/chains/letsencrypt-"$ISSUER".crt
if ! [ -f "$ISSUER_FILE" ]; then
    echo "Cannot find certificate for issuer $(openssl x509 -issuer -in live.crt -noout)"
    exit 1
fi

# Create derived files.  We expect /etc/ssl/dh2048.pem to contain the DH
# parameters, generated with
#   openssl dhparam -out /etc/ssl/dh2048.pem 2048
cat "$ISSUER_FILE" > live.chain # just the chain
cat live.crt /etc/ssl/dh2048.pem > live.crt+dh # Certificate plus DH parameters
cat live.crt live.chain > live.crt+chain # Certificate plus chain

# Fill in here:  the code to restart/reload all relevant services.

With this, the apache SSL configuration looks as follows:

# Certificate, Key, and DH parameters
SSLCertificateFile      /etc/ssl/mycerts/live.crt+dh
SSLCertificateKeyFile   /etc/ssl/private/live.key
SSLCertificateChainFile /etc/ssl/mycerts/live.chain

# configure SSL ciphers and protocols
SSLProtocol All -SSLv2 -SSLv3
# TODO: Once OpenSSL supports GMC with more than just AES, revisit this
# NOTE: The reason we support non-FS ciphers is stupid middleboxes that don't support FS
SSLCipherSuite 'kEECDH+AESGCM:kEDH+AESGCM:kEECDH:kEDH:AESGCM:ALL:!3DES:!EXPORT:!LOW:!MEDIUM:!aNULL:!eNULL'
SSLHonorCipherOrder     on	

(My cipher suite is deliberately not the one from bettercrypto.org because I prefer to not update it with every change in OpenSSL’s supported ciphers.)

Obtaining the First Certificate

You can now run letsencrypt-tiny -c letsencrypt.conf init to perform the initial setup.

In the future, to change the set of domains, first edit the config file and then run letsencrypt-tiny -c letsencrypt.conf -k renew. The -k tells Let’s Encrypt Tiny to also run the certificate hook.

Automation Via Cron

Let’s Encrypt certificates expire after 90 days, so we want renewal to be automated. To this end, just make sure that letsencrypt-tiny -c letsencrypt.conf -k cron gets run regularly, like once a day. I have the following in root’s crontab (sudo crontab -e):

32  6  *   *   *     /root/server-scripts/letsencrypt-tiny -c /root/letsencrypt/conf -k cron

This will check the time intervals you configured above, and act accordingly. If any action is taken, the script will print that information on standard output; if you have email set up on your server, this means you will get an email notification.

DNSSEC and TLSA

Everything described so far should give you a working SSL setup if you do not use DNSSEC+TLSA. If you do use DNSSEC+TLSA, like I do on this server, you need to enable the post-keychange hook and have it regenerate your DNS zone, and you need to increase staging-hours. The zone should always contain the hash of the live key, and, if a staging key exists, also the hash of the staging key.

I am managing my DNS zones with zonemaker, and wrote some Python code to automatically generate TLSA records using the tlsa tool:

def TLSA_from_crt(protocol, port, crtfile):
    crtfile = "/etc/ssl/mycerts/"+crtfile
    open(crtfile).close() # check if the file exists
    # make sure we match on *the key only*, so that we can renew the certificate without harm
    zone_line = subprocess.check_output(["tlsa", "--selector", str(TLSA.Selector.SubjectPublicKeyInfo), "--certificate", crtfile, "example.org"]).decode("utf-8")
    m = re.match("^[0-9a-zA-Z_.-]+ IN TLSA ([0-9]+) ([0-9]+) ([0-9]+) ([0-9a-zA-Z]+)$", zone_line)
    assert m is not None
    assert int(m.group(1)) == TLSA.Usage.EndEntity
    assert int(m.group(2)) == TLSA.Selector.SubjectPublicKeyInfo
    return TLSA(protocol, port, TLSA.Usage.EndEntity, TLSA.Selector.SubjectPublicKeyInfo, int(m.group(3)), m.group(4))

def TLSA_for_LE(protocol = Protocol.TCP, port = 443):
    # add both the live and (potentially) staging certificate to the letsencrypt TLSA record set
    r = [TLSA_from_crt(protocol, port, "letsencrypt/live.crt")]
    try:
        r.append(TLSA_from_crt(protocol, port, "letsencrypt/staging.crt"))
    except IOError:
        pass
    return r

Now I add TLSA_for_LE(port = 443) to the records of my domains. Finally, the key hook just runs zonemaker and has bind reload the zone (it will automatically also be resigned). Now, whenever a staging key is created, it is automatically added to my zone. At least 25h later (I have the TTL set to 24h), the key gets unstaged, and the old TLSA record is removed from the zone.

That’s it! If you have any questions, feel free to report issues at GitHub.

Posted on Ralf's Ramblings on Dec 26, 2017.
Comments? Drop me a mail!