With the go-live of https://letsencrypt.org/ its time to transition from the pricy and manual standard SSL cert issuing model to a fully automated process using the ACME protocol. Most orgs have numerous usages of CA purchased certs, this post will cover hosts running apache/nginx and AWS ELBs, all of these usages are to be replaced with automated provisioning and renewal of letsencrypt signed certs.
Provisioning and auto-renewing Apache and nginx TLS/SSL certs
For externally accessible sites where Apache/Nginx handles TLS/SSL termination moving to letsencrypt is quick and simple:
1 – Install the letsencrypt client software (there are RHEL and Centos rpms – so thats as simple as adding the package to puppet policies or
yum install letsencrypt
2 – Provision the keys and certificates for each of the required virtual hosts. If a virtual host has aliases, specify multiple names with the -d arg.
letsencrypt certonly --webroot -w /var/www/sites/static -d static.mwclearning.com -d img.mwclearning.com
This will provision a key and certificate + chain to the letsencrypt home directory (defaults /etc/letsencrypt). The /etc/letsencrypt/live directory contains symlinks to the current keys and certs.
3 – Update the apache/nginx virtualhost configs to use the symlinks maintained by the letsencrypt client, ie:
# static Web Site
ServerName static.mwclearning.com
ServerAlias img.mwclearning.com
ServerAlias registry.mwclearning.ninja # <<-- dummy alias for internal site
ServerAdmin webmaster@mwclearning.ninja
DocumentRoot /var/www/sites/static
DirectoryIndex index.php index.html
AllowOverride all
Options +Indexes
ErrorLog /var/log/httpd/static_error.log
LogLevel warn
CustomLog /var/log/httpd/static_access.log combined
ServerName static.mwclearning.com
ServerAlias img.mwclearning.com
ServerAlias img.mwclearning.ninja
ServerAdmin webmaster@mchost
DocumentRoot /var/www/sites/static
DirectoryIndex index.php index.html
AllowOverride all
Options +Indexes
ErrorLog /var/log/httpd/static_ssl_error.log
LogLevel warn
CustomLog /var/log/httpd/static_ssl_access.log combined
SSLEngine on
SSLCipherSuite ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS:!RC4
SSLHonorCipherOrder on
SSLInsecureRenegotiation off
SSLCertificateKeyFile /etc/letsencrypt/live/static.mwclearning.com/privkey.pem
SSLCertificateFile /etc/letsencrypt/live/static.mwclearning.com/cert.pem
SSLCertificateChainFile /etc/letsencrypt/live/static.mwclearning.com/chain.pem
4 – Create a script for renewing these certs, something like:
#!/bin/bash
# Vars
PROG_ECHO=$(which echo)
PROG_LETSENCRYPT=$(which letsencrypt)
PROG_FIND=$(which find)
PROG_OPENSSL=$(which openssl)
#
# Main
#
${PROG_ECHO} "Current expiries: "
for x in $(${PROG_FIND} /etc/letsencrypt/live/ -name cert.pem); do ${PROG_ECHO} "$x: $(${PROG_OPENSSL} x509 -noout -enddate -in $x)";done
${PROG_ECHO} "running letsencrypt certonly --webroot .. on $(hostname)"
${PROG_LETSENCRYPT} renew --agree-tos
LE_STATUS=$?
systemctl restart httpd
if [ "$LE_STATUS" != 0 ]; then
${PROG_ECHO} Automated renewal failed:
cat /var/log/letsencrypt/renew.log
exit 1
else
${PROG_ECHO} "New expiries: "
for x in $(${PROG_FIND} /etc/letsencrypt/live/ -name cert.pem); do echo "$x: $(${PROG_OPENSSL} x509 -noout -enddate -in $x)";done
fi
# EOF
5 – Run this script automatically everyday with cron or jenkins
6 – Monitoring the results of the script and externally monitor the expiry dates of your certificates (something will go wrong one day)
Provisioning and auto-renewing AWS Elastice Load Balancer TLS/SSL certs
This has been made very easy by Alex Gaynor with a handy python script: https://github.com/alex/letsencrypt-aws. This is a great use-case for docker and Alex has created a docker image for the script: https://hub.docker.com/r/alexgaynor/letsencrypt-aws/. To use this with ease I created a layer on top creating a new Dockerfile:
#
# mwc letsencrypt-aws image
#
FROM alexgaynor/letsencrypt-aws:latest
MAINTAINER Mark
ENV LETSENCRYPT_AWS_CONFIG="{\"domains\": \
[{\"elb\":{\"name\":\"TestExtLB\",\"port\":\"443\"}, \
\"hosts\":[\"test.mwc.com\",\"test-app.mwc.com\",\"test-api.mwc.com\"], \
\"key_type\":\"rsa\"}, \
{\"elb\":{\"name\":\"ProdExtLb\",\"port\":\"443\"}, \
\"hosts\":[\"app.mwc.com\",\"show.mwc.com\",\"show-app.mwc.com\", \
\"app-api.mwc.com\",\"show-api.mwc.com\"], \
\"key_type\":\"rsa\"}], \
\"acme_account_key\":\"s3://config-bucket-abc123/config_items/private_key.pem\"}"
ENV AWS_ACCESS_KEY_ID="<AWS_ACCESS_KEY_ID>"
ENV AWS_SECRET_ACCESS_KEY="<AWS_SECRET_ACCESS_KEY>"
ENV AWS_DEFAULT_REGION="ap-southeast-2"
# EOF
The explanation of these values can be found at https://hub.docker.com/r/alexgaynor/letsencrypt-aws/. Its quite important to create a specific IAM User to conduct the required Route53/S3 and ELB actions. This images need to be build on changes:
sudo docker build -t registry.mwc.ninja:5000/syseng/ao-letsencrypt-aws .
sudo docker push registry.mwc.ninja:5000/syseng/ao-letsencrypt-aws
With this image built another cron or jenkins job can be run daily executing something like:
sudo docker pull registry.mwc.ninja:5000/syseng/ao-letsencrypt-aws
sudo docker run registry.mwc.ninja:5000/syseng/ao-letsencrypt-aws
sleep 10
sudo docker rm $(sudo docker ps -a | grep registry.mwc.ninja:5000/syseng/ao-letsencrypt-aws | awk '{print $1}')
Again, the job must be monitored along with external monitoring of certificates. See a complete SSL checker at https://github.com/markz0r/tools/tree/master/ssl_check_complete.
