Let's Encrypt

Getting an SSL certificate for your web server has traditionally been a something of an effort.  You need to correctly generate a weird thing called a certificate signing request (CSR), submit it to the web page of your chosen Certificate Authority (CA), wait for them to sign and generate a certificate, work out where to put the certificate to configure it for your web server—making sure you also configure any required intermediate CA certificates—and then restart the web server.  If you got all that right, you then need to enter a calendar entry so you'll remember to go through the process again in (say) a year's time. Even some of the biggest names in IT can mess up this process.

With new CAs like Let's Encrypt, along with some supporting software, the rigmarole around SSL certificates becomes a thing of the past.  The technology behind this revolution is Automatic Certificate Management Environment (ACME), a new IETF standard (RFC 8555) client/server protocol which allows TLS certificates to be automatically obtained, deployed, and renewed. In this protocol, an "agent" running on the server that needs an SSL certificate will talk to to the CA's ACME server over HTTP.

A popular method for using ACME on your Red Hat Enterprise Linux 7 server is certbot. Certbot is a standalone ACME agent that is configured out-of-the-box to work with Let's Encrypt and can work with Apache httpd, Nginx, and a wide variety of other web (and non-web!) servers.  The certbot authors have an excellent guide describing how to set up certbot with httpd on RHEL7.

In this tutorial, I'll show an alternative method—the mod_md module—which is an ACME agent implemented as a module for Apache httpd, tightly integrated with mod_ssl, and is supported today in Red Hat Enterprise Linux 7.  The mod_md module was implemented by Stefan Eissing—a prolific developer who also added HTTP/2 support to httpd—and contributed to the Apache Software Foundation, becoming a standard part of any new installation since httpd version 2.4.30.


I'm using a virtual machine running Red Hat Enterprise Linux 7 in Amazon EC2, and to get going I'll install Apache httpd from the Red Hat Software Collections repository:

# yum-config-manager --enable rhui-REGION-rhel-server-rhscl > /dev/null
# yum install -y httpd24 httpd24-mod_ssl httpd24-mod_md
  httpd24.x86_64 0:1.1-18.el7 httpd24-mod_md.x86_64 0:2.4.34-7.el7.1 httpd24-mod_ssl.x86_64 1:2.4.34-7.el7.1

Dependency Installed:
  httpd24-httpd.x86_64 0:2.4.34-7.el7.1 httpd24-httpd-tools.x86_64 0:2.4.34-7.el7.1 httpd24-libcurl.x86_64 0:7.61.1-2.el7 
  httpd24-libnghttp2.x86_64 0:1.7.1-7.el7 httpd24-runtime.x86_64 0:1.1-18.el7


The Software Collections repository is not enabled by default, so the first step is to enable it.  Note that mod_md is installed just like any other httpd module from the httpd24 collection.


Now, to configure the server.  The minimal changes are required in the SSL configuration /opt/rh/httpd24/root/etc/httpd/conf.d/ssl.conf, so fire up your editor and adjust the default configuration as follows:

MDomain mytestsslserver.site
ServerAdmin jorton@redhat.com

<VirtualHost _default:443>
# General setup for the virtual host, inherited from global configuration
#DocumentRoot "/var/www/html"
ServerName mytestsslserver.site:443
#SSLCertificateFile /etc/pki/tls/certs/localhost.crt
#SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

Since mod_md is going to manage the certificates for this virtual host, SSLCertificateFile and SSLCertificateKeyFile must be either removed or commented out. I've also added MDomain to tell mod_md to manage the domain, which must match the name used in the VirtualHost, and added my email address in ServerAdmin.

If we now start up the server, we should get some errors from mod_md:

# systemctl start httpd24-httpd
# tail -4 /var/log/httpd24/error_log 
[Mon Jul 22 10:05:09.679997 2019] [mpm_prefork:notice] [pid 5395] AH00163: Apache/2.4.34 (Red Hat) OpenSSL/1.0.2k-fips configured -- resuming normal operations
[Mon Jul 22 10:05:09.680018 2019] [core:notice] [pid 5395] AH00094: Command line: '/opt/rh/httpd24/root/usr/sbin/httpd -D FOREGROUND'
[Mon Jul 22 10:05:11.083123 2019] [md:error] [pid 5397] (70008)Partial results are valid but processing is incomplete: mytestsslserver.site: the CA requires you to accept the terms-of-service as specified in <https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf>. Please read the document that you find at that URL and, if you agree to the conditions, configure "MDCertificateAgreement url" with exactly that URL in your Apache. Then (graceful) restart the server to activate.
[Mon Jul 22 10:05:11.083160 2019] [md:error] [pid 5397] (70008)Partial results are valid but processing is incomplete: AH10056: processing mytestsslserver.site

mod_md uses the Let's Encrypt service by default—but if you configure a different ACME server via the MDCertificateAuthority directive, you'll get a different error message here.  After reading the terms of service for your CA, indicate acceptance via the MDCertificateAgreement directive—edit /opt/rh/httpd24/root/etc/httpd/conf.d/ssl.conf one more time:

MDomain mytestsslserver.site
MDCertificateAgreement https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
ServerAdmin jorton@redhat.com

If you'd read the terms of service already, you can add the directive to start with and skip a reload.  I now restart the server two more times, allowing a short delay for certificate issuance to take place:

# systemctl reload httpd24-httpd
# sleep 60
# tail -1 /var/log/httpd24/error_log 
[Mon Jul 22 10:08:19.344969 2019] [md:notice] [pid 5480] AH10059: The Managed Domain mytestsslserver.site has been setup and changes will be activated on next (graceful) server restart.
# systemctl reload httpd24-httpd
# tail -1 /var/log/httpd24/ssl_error_log 
[Mon Jul 22 10:09:00.341603 2019] [ssl:info] [pid 5395] AH02568: Certificate and private key mytestsslserver.site:443:0 configured from /opt/rh/httpd24/root/etc/httpd/state/md/domains/mytestsslserver.site/pubcert.pem and /opt/rh/httpd24/root/etc/httpd/state/md/domains/mytestsslserver.site/privkey.pem

Bingo! That last message tells us that mod_md has provided the SSL certificate and private key for the virtual host. I can now load my new SSL site via https://mytestsslserver.site/ and get the familiar RHEL "welcome" page, and start adding actual content to /opt/rh/httpd24/root/var/www/html/.

More information

In the current generation of the ACME protocol, the Let's Encrypt servers will use an SSL request to this httpd server to confirm that I am the owner of the domain for which I'm requesting an SSL certificate.  I can see this in the logs here:

# tail /var/log/httpd24/access_log - - [22/Jul/2019:10:08:17 +0000] "GET /.well-known/acme-challenge/z2xetCt0LMwehxmGUerh9GFdkg9aqlIXAPWRb_8PEJg HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

For this challenge/response exchange to work, your SSL server must be accessible—not firewalled—via port 443 on a public network.  That means you can't use this configuration for private, internal servers.  There are ways around this, notably DNS challenge validation, which are available in certbot and also in future versions of mod_md, but those require a more complex configuration.

For more information on using mod_md, you can read the upstream documentation. For more information on getting started with Red Hat Software Collections, see the release notes.

Last updated: July 29, 2019