LibreLAMP Apache Configuration

Some people are under the impression that properly setting up a secure Apache web server is a difficult task that requires a lot of skill. If you are under that impression and you have a lot of money and you are willing to pay me to do it for you, then yes, it is an extremely difficult task.

For everyone else, as long as you have a basic understand of the UN*X shell, it really is not that difficult. This page gives basic instructions for completing the task.

I do recommend that you read the ‘SSL/TLS Deployment Best Practices’ guide that is available at https://www.ssllabs.com/projects/best-practices/index.html.

This document (not that linked document) assumes that you are running LibreLAMP on RHEL/CentOS 7 with the mod_ssl package installed.


TLS Private Key and Certificate Generation

A TLS web server works on the principles of Public-key Cryptography.

With Public-key Cryptography, two keys exist. One key has a secret and is kept private, and the other key is a public key generated from that secret. You can always re-create the public key from the private key, but the function is one way, you can not figure out the private key from the public key.

When used just for authentication of a message and not encryption, a signature is created using the private key and attached to the message. The recipient can then use the public key of the sender to verify from the signature that the message has not been altered since it was signed by the private key. It also verifies it was sent by someone who has access to the private key.

When used for encryption, the sender of the message uses the public key provided by the recipient and their own private key to encrypt the message. This is called asymmetric encryption. Once encrypted, unless the ciphers are broken or the private keys are really weak, it can only be decrypted using the private key of the recipient and the public key of the sender. This is what TLS does.

To implement a TLS server, first we must generate a strong TLS private key.

From that private key, we have to generate a Certificate Signing Request. This CSR will contain the public key associated with the private key, as well as some meta-data about the web site, such as the domain name and the company.

The generated CSR is then sent to a Certificate Authority. The Certificate Authority does some verification that the owner of the domain listed in the CSR is in fact the person sending the CSR. If they are satisfied, the Certificate Authority signs it themselves using their own private key. Any browser that trusts that particular Certificate Authority will then trust the certificate for the website is valid, and allow a TLS connection.

Generate the TLS Private Key

There are two different types of private keys you can generate: ECDSA and RSA.

ECDSA has better performance and clearly is the future, but there are still some clients that do not support it. RSA is supported by every TLS client, so that is what these instructions will primarily focus on.

It is possible to use both but the only logical reason for doing this is if you wish to use an ECDSA certificate while at the same continuing to support some archaic old clients. Warning: My personal experience is that some clients that are not capable of using a ECDSA certificate also fail to work with servers that provide both ECDSA and RSA certificates. The back-end content scraper Tumblr uses is one such example. For that reason, I no longer recommend dual-certificate configurations. Just use an RSA certificate if you need to support clients not capable of dealing with ECDSA, and use ECDSA when you do not need to support those few clients that do not support ECDSA.

An RSA private key should be at least 2048 bits and certificate authorities will rightfully refuse to issue a signed certificate if it is not. 2048 bit is the default for LibreSSL, we do not need to specify any options to achieve it. On RHEL/CentOS 7 systems, private keys are traditionally kept in the directory /etc/pki/tls/private/.

To generate a 2048-bit RSA key in LibreLAMP:

pushd /etc/pki/tls/private
DUMASK=`umask`
umask 0277
/usr/bin/libressl genpkey -algorithm RSA -out example.org-20180620.key
umask ${DUMASK}
popd

The purpose behind temporarily changing the umask is to make sure that from the start, only the root user can read the file. Obviously use your domain name instead of example.org and use the current YYYYMMDD instead of 20180620.

Having the date of generation in the filename makes it easy to see when a private key is over a year old. It is best practice to regenerate any private keys that are over a year old, just in case it was compromised without you being aware.

If you do wish to use an ECDSA certificate, instructions for generating private keys are at the bottom of this page in the section titled ECDSA Private Keys.

Generate a Certificate Signing Request

A Certificate Signing Request is what you send to a Certificate Authority to obtain a signed certificate. You do not send them your private key, you do not send anyone your private key. Private means private. The CSR is what you send the certificate authority. I emphasize this because I have seen Certificate Authorities before that give you the option of sending them your private key. It breaks my heart when I see something that wrong.

When creating a CSR you will be asked several questions. Most of them are self-explanatory. A few of them are not.

For Organization Name (eg, company) — if this is not for a company website, just use the name of the website.

For Organizational Unit Name (eg, section) — you can just leave that blank.

For Common Name — be sure to use the Fully Qualified Domain Name that the certificate is being issued for (e.g. example.org) except when the FQDN begins with www. — then leave the starting www. off of the Common Name field.

For Email Address — you can leave that blank but I usually use the same e-mail address I used when registering the domain.

For A challenge password — it is very important that you leave that blank, or you will need to enter it every time you start the web-server.


Okay, now that those fields have been clarified, here is how to generate a CSR in LibreLAMP:

pushd /etc/pki/tls/csr
/usr/bin/libressl req -new -key ../private/example.org-20180620.key \
  -out example.org-20180620.csr

After answering the questions, the CSR will have been created. The contents of the file will look something like this:

(Screen Readers: skip past long base 64 encoded output text)

-----BEGIN CERTIFICATE REQUEST-----
MIIC0TCCAbkCAQAwgYsxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlh
MQ4wDAYDVQQHDAVSb2RlbzEYMBYGA1UECgwPSnVuayBUaXJlcywgTExDMRQwEgYD
VQQDDAtleGFtcGxlLm9yZzEnMCUGCSqGSIb3DQEJARYYbnNhaGFja2VyQHdoaXRl
aG91c2UuZ292MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEArDTOMxtd
U9/yrmaX57+opvq43aTvFFP+yzckfA6cBdY/TGBvkZudvKPqY/UACSWlj9Kyzxib
/1vtRmp/b00hSJVScbMoZHrdNVYuq9a8kk2nJedRQ2x+eyF2zJ/yW3Tp5PG+Leba
DNvlwMKof/kwMAJ6CJfaw6qeEPKDHY1SIZJq+Rf2wgeGCzVNoTd5IXRPbWOIwgD4
yxypgp5csv6e+hlojfoR4uB14zoVlSJ+6QRSOWOMFIjR6ut71VOVNxLJch2aG5Kr
MwYsyHIcXm7t6qr9BgSC72dozT2BiSO01wX4biwaz2uv2UhtM36mTuGvONQnjSjU
hFjGuhXLymvSgwIDAQABoAAwDQYJKoZIhvcNAQELBQADggEBAF+zrfkmo3YaAMY/
FgJl/F4CRi8dq7rsvYz62g7mXjTw+qVB3Md7wSVS0YAzhWsDMaiIeCJ1gGPKE5Dc
xFS9c3TBsrSMoTtCSBtpqJOce5k/aecB7SH1cR1yyc2g+TRxDYNBV6L8xs/DLMRj
JuCtx1QU/mJjjQf8PpIbktMNVZWbcbMF3WS2QNBKWN7Uy4UXp4aVVCrccVvArIQL
dYsetBD1+S2C2sJ0YFqVmVaYYvOzO9Wm9y79N60NYXCNSybEqhZ1E6kxgpLp6J2T
KwM5DaEOdDUFuidvaG9h/NcFrcAc2KBN5LKZoLq/dd1DSFWmwY9izthVDx6GTJJH
VOroT3s=
-----END CERTIFICATE REQUEST-----

“But there’s way too much information to decode the Matrix. You get used to it. I don’t even see the code. All I see is blonde, brunette, redhead.” — Cypher

Certificate Types

Now that we have a private key and a CSR generated from that key, we can get a signed certificate. There are four types of signed certificates:

  • Self-Signed
  • Domain Validation (DV)
  • Organization Validation (OV)
  • Extended Validation (EV)

Do not let some slick salesman con you into thinking the more expensive certificates provide better encryption. They do not. The quality of the encryption is strictly determined by your private key and the cipher suite negotiated between the server and the client, the certificate has absolutely nothing to do with the actual encryption.

What a valid certificate does is offer the user a reason why they should trust that you are who say you are. It gives the browser (and the user) some degree of confidence that a MITM attack is not taking place.

A self-signed certificate does not meet that purpose and should not be used on a web server with a public IP address. This is a common flaw I see with web hosting companies, by the way. They often use self-signed certificates for pages that users are suppose to use to administer their accounts. If a web-hosting service does not understand the security implications of a self-signed certificate then they should not be trusted to keep the hosts secure, it is that simple.

One of the major problems with a self-signed certificate is that there is not a mechanism by which a browser can determine if the certificate has been revoked. If a hacker manages to get the private key, the hacker can pull off a MITM attack and any browser that accepted the certificate will continue to accept it because it can not effectively be revoked.

Do not use self-signed certificates on public web servers. Just don’t.

With a DV certificate, a Certificate Authority validates that the CSR is coming from an administrator of the domain. This is usually done by sending an e-mail to an administrative e-mail address at the domain (e.g. admin@example.org) or to the e-mail address in the WHOIS record, if available.

With an OV certificate, the checking a Certificate Authority does is a little more extensive before they issue the certificate. They validate that the website actually is associated with the organization named in the certificate and that the CSR is coming from an appropriate person in the organization.

On the surface that sounds better, but in reality it just costs more. The browser does not care and the user only knows if they actually look at the certificate. Almost no one ever does.

With an EV certificate, extensive validation is done before the Certificate Authority will issue the certificate. If your website is a financial institution or other high target for MITM attacks or phishing attacks, then many users will expect you to have an EV certificate.

Many modern web browsers will give the user a visual indication that the web site has an EV certificate. Typically this is done by placing the company name and country in green letters between the lock and url in the browser bar:

[Image showing the green company name and country displayed between the lock and URL]

Nutshell: Use a DV certificate unless your website is the kind of website likely to be targeted for MITM attacks or phishing attacks. Then spend the extra money for an EV certificate.

Certificate Authority

For purchasing a signed TLS certificate (and for domain name registration), I highly highly recommend Namecheap.

Not only is their customer service the best I have ever experienced with any registrar, but they will save you money. Most places that claim to save me money have pathetic customer service, that is not the case with Namecheap. Their customer service is exceptional.

On , a PositiveSSL certificate purchased from Comodo was priced at $49.00 (US). The same certificate purchased from Namecheap, $9.00 (US).

Very often, companies like Comodo actually prefer a re-seller like Namecheap handle the customer service, hence the price difference.

Unless you need an EV certificate, Comodo PositiveSSL is the certificate I recommend. It is inexpensive, and they automatically add the www sub-domain to the domains the certificate is valid for.

Comodo PositiveSSL at Namecheap

The rest of this section is image heavy and shows the process of purchasing a PositiveSSL certificate from Namecheap. If that is not of interest to you, you can skip past it.

Also, the rest of this section was created in 2015. Namecheap has changed the process. I will re-write this next time I buy a certificate through them.

After you have purchased the Comodo PositiveSSL certificate, you will need to go through a web interface at Namecheap to submit your CSR.

Where the form says Select web server it can be a little confusing. One of the options is for Apache + mod_ssl. That is for old crusty Apache 1.x servers. Modern Apache 2.x, if linked against OpenSSL or LibreSSL, should select the option Apache + OpenSSL.

In the large text field that is labeled Enter csr, that is where you put the contents of the CSR. Copy and paste everything in the file starting with the -----BEGIN CERTIFICATE REQUEST----- and ending with the -----END CERTIFICATE REQUEST----- lines, including those lines.

[Screenshot showing web server type and CSR data]
Webpage screenshot at Namecheap showing selection of web server type and entered CSR data

After you have entered the required data, click on the Next >> button below the CSR text area.

The next page will present information extracted from the CSR and ask you to select an e-mail address for the Domain Validation.

Double-check that the information extracted from the CSR is valid. If it is valid, select an e-mail address to use for Domain Validation.

You must be able to receive e-mail at one of the addresses shown. That is how they verify that the domain is yours. If you do not have a mail server set up on the domain, make sure the WHOIS information for your domain has a valid e-mail address associated with it, so that you can use that address.

[Screenshot showing extracted CSR details, e-mail address selection]
Webpage screenshot at Namecheap showing extracted CSR details and selection of e-mail address for Domain Validation

After you selected the e-mail address you want to use, click on the Next >> button.

That will bring you to a page that tells you an e-mail has been sent:

[Screenshot showing order process]
Webpage screenshot at Namecheap showing flowchart of order process

There is nothing for you to actually do on that page, it just shows where you currently are in the process.

The next thing that will happen, you will be sent an e-mail to the address you selected. That e-mail will contain a code and a link to a page where you need to enter the code to verify that you both asked for the certificate to be signed and have access to the e-mail address used for validation.

[Screenshot showing Domain Control Validation form]
Webpage screenshot at Comodo showing Domain Control Validation form

Once you have entered the proper validation code, Comodo will e-mail you a zip archive bundle including your signed certificate and supporting certificates from Comodo. You are done with that part, you now have a signed TLS certificate.

Certificate Installation

Your Certificate Authority should have e-mailed you a zip archive containing your signed certificate as well as supporting certificates. Nothing in that zip archive needs to be kept secret, the Apache web server will send them to any client that connects. The zip archive is probably named something like example_org.zip.

Upload the bundle to your web server and put it in the /etc/pki/tls/cert_bundle/ directory. I like to rename it to the same scheme I use for the private key and the CSR, and then unpack it:

pushd /etc/pki/tls/cert_bundle
mv example_org.zip example.org-20180620.zip
unzip example.org-20180620.zip

In the case of a Comodo PositiveSSL bundle, this results in two files:

  • example_org.ca-bundle
  • example_org.crt

The last one, example_org.crt, is the actual certificate for the site. Move it into /etc/pki/tls/certs/ and rename it to match the same scheme and date stamp as your private key:

mv example_org.crt example.org-20180620.crt
mv example.org-20180620.crt ../certs/

The first one, with the name bundle in it, is what is sometimes called the Certificate Chain. I prefer to call it the Certificate Authority Bundle. Rename it to the same scheme and date as your private key but noting that it is the bundle:

mv example_org.ca-bundle example.org-cab-20180620.crt
mv example.org-cab-20180620.crt ../certs/

You now have all three TLS related files in place that are needed to configure Apache:

  1. /etc/pki/tls/certs/example.org-20180620.crt
  2. /etc/pki/tls/private/example.org-20180620.key
  3. /etc/pki/tls/certs/example.org-cab-20180620.crt

Basic Apache TLS Configuration

A fresh install of the Apache web server in LibreLAMP is virtually identical to a fresh install of the Apache web server in RHEL/CentOS 7.

As such, you could just put your web pages in /var/www/html/ but I highly discourage that practice.

The best thing to do, in my opinion, is to create Name Based Virtual Hosts for every domain name Apache will be serving. Each domain should have its own configuration file in /etc/httpd/conf.d/ making it much easier to migrate a configuration from one server to another.

For these instructions, we will continue with the use of example.org and create a fresh Apache Name Based Virtual Host configuration file for that domain called /etc/httpd/conf.d/example.org.conf.

Create the Apache Webroot

As the root user:

pushd /srv
mkdir -p example.org/www
mkdir example.org/phpinclude
chown -R alice:alice example.org
popd

That gives the user alice and the group alice ownership of the /srv/example.org directory so that root access is not needed to put content where it will be served.

The directory /srv/example.org/www is the Apache web root we will use for the domain.

The directory /srv/example.org/phpinclude is a directory outside the Apache web root where PHP include files can go, such as database authentication files, that should never be where the web server can accidentally serve them in plain text.

Redirect Virtual Hosts

When running a secure web server, it is considered bad practice to also serve content on the same domain without encryption. So to begin our Apache configuration, we will start with Port 80 redirects to the secure protocol.

For the purpose of brevity, we will only cover IPv4 hosts. However I highly encourage you to run a dual-stack server that supports IPv6 as well.

We will be using example.org as the domain we serve from, redirecting client requests to www.example.org to example.org. Of course you are free to do the reverse on your server if you prefer the reverse.

The start of our /etc/httpd/conf.d/example.org.conf file:

#Port 80 Redirect

<VirtualHost 192.168.0.12:80>
ServerName example.org
Redirect permanent / https://example.org/
</VirtualHost>

<VirtualHost 192.168.0.12:80>
ServerName www.example.org
Redirect permanent / https://example.org/
</VirtualHost>

The Redirect permanent directive tells Apache to send a 301 HTTP redirect header. You can see this with the curl utility:

[alice@localhost ~]$ curl -I http://librelamp.com/ApacheConfiguration
HTTP/1.1 301 Moved Permanently
Date: Wed, 07 Feb 2018 17:38:12 GMT
Server: Apache/2.4.29 (LibreLAMP) LibreSSL/2.6.4 PHP/7.1.14
Location: https://librelamp.com/ApacheConfiguration
Content-Type: text/html; charset=iso-8859-1

[alice@localhost ~]$

Additionally, we need a redirect on port 443 to redirect client requests to https://www.example.org/ to https://example.org/.

Since the request will use TLS, we will need to turn on the SSLEngine for this virtual host, and tell it where the key and certificate files are located.

There are four directives we need to turn on TLS:

SSLEngine on
This directive turns the TLS engine on.
SSLCertificateFile
This directive tells Apache where the signed certificate is located.
SSLCertificateKeyFile
This directive tells Apache where the private key associated with the certificate is located.
SSLCACertificateFile
This directive tells Apache where the Certificate Authority bundle certs are located.
# Port 443 Redirect

<VirtualHost 192.168.0.12:443>
ServerName www.example.org
Redirect permanent / https://example.org/
SSLEngine on
SSLCertificateFile            /etc/pki/tls/certs/example.org-20180620.crt
SSLCertificateKeyFile       /etc/pki/tls/private/example.org-20180620.key
SSLCACertificateFile      /etc/pki/tls/certs/example.org-cab-20180620.crt
</VirtualHost>

This is why is important for the certificate to be valid for the domain both with and without the www. prefix. The redirect from one to the other needs to validate in the certificate as well, or else the user will get a scary message about a server mismatch.

As you can see, with virtual hosts that only redirect, they really do not need very much. You do not even really need to add log file directives. For the port 80 redirects, you just need the ServerName and Redirect permanent directives. For port 443 you just need to add the the SSL* directives.

Live Server Virtual Host

With the Virtual Hosts that redirect out of the way, now we can configure the Virtual Host they redirect to:

#Live Host

<VirtualHost 192.168.0.12:443>
ServerName example.org
DocumentRoot "/srv/example.org/www"
SSLEngine on
SSLCertificateFile            /etc/pki/tls/certs/example.org-20180620.crt
SSLCertificateKeyFile       /etc/pki/tls/private/example.org-20180620.key
SSLCACertificateFile      /etc/pki/tls/certs/example.org-cab-20180620.crt
ErrorLog logs/example.org.error_log
CustomLog logs/example.org.access_log combined
</VirtualHost>

The DocumentRoot directive points to the /srv/example.org/www directory we created earlier. We need to tell Apache what it is allowed to do with that directory:

<Directory "/srv/example.org/www">
  Options FollowSymlinks
  AllowOverride All
  Require all granted
  php_value include_path "/srv/example.org/phpinclude:/usr/share/pear"
</Directory>

I prefer to initially set the Options directive for a directory to just FollowSymlinks. That actually is the default value in Apache 2.4 if it is left out, but earlier versions of Apache defaulted to All if it was left out, which was not very secure. Even though I am specifying the default, I think that is better as it may change again in the future resulting in unintended results when upgrading Apache.

For other options, see the Options Directive in the Apache manual.

The AllowOverride directive specifies what a .htaccess file in within the web root is allowed to do. When setting up a new server I prefer to set it to All so that I have some flexibility without needing to frequently restart the web server. It can be locked down later when the needs of the host are better known.

The Require all granted directive tells Apache that it is okay to serve the contents of the directory to any requesting client.

The php_value directive should only be used on servers that have PHP enabled.

It sets the initial PHP include path for scripts served from that directory to something sane and secure. I do not personally like the common practice of including the current directory in the default PHP include path.


With any luck, your Apache server should now be properly configured to serve content securely over a TLS connection.

HTTP Strict Transport Security

HTTP Strict Transport Security (abbreviated HSTS) is a mechanism by which your web server can tell web browsers that they should always use HTTPS when connecting to your server.

When running a web server, you should have all requests to Port 80 redirect to the TLS enabled secure server on Port 443 but that is not good enough by itself.

Let us say you own the domain SecureExampleBank.com. You have requests to Port 80 redirect to port 443 but you do not send the HSTS header.

You have a customer named Billy Joe. Billy Joe is a happy camper, loves your bank, he is a really nice guy.

A hacker named Dirnt creates a phishing website, SecureExampleBamk.com, that he wants to use to steal Billy Joe’s login credentials.

Dirnt knows that Billy Joe always looks carefully at the domain name in any e-mail, so what Dirnt does is send Billy Joe an e-mail pretending to be from his bank telling him there is an issue that needs his attention. The e-mail contains a hyperlink to http://SecureExampleBank.com/.

The domain is correct, but it goes to Port 80 and not Port 443. Your web server at SecureExampleBank.com will redirect Billy Joe to the secure Port 443, as it should, but the redirect itself is not sent over a secure connection because the request came over Port 80.

This allows Dirnt to easily pull a MITM attack on the un-secure plain text redirect, changing it to instead redirect Billy Joe to the server controlled by Dirnt located at https://SecureExampleBamk.com.

Now Billy Joe’s Day is Blue because Dirnt stole all his Green and now he has to move back to Rodeo Boredeo.

If the real server had made a practice of sending the HSTS header, then from a previous visit Billy Joe’s browser would know to only use a secure connection with that domain and would have changed the insecure http://SecureExampleBank.com/ request to the secure https://SecureExampleBank.com/ request from the start, foiling Dirnt’s evil plan so the nice guy doesn’t finish last.

The point is every secure server should implement HSTS. It protects the users.

Sending the HSTS Header

Add the following directive to the beginning of your Apache virtual host configurations:

Header always set Strict-Transport-Security "max-age=63072000; preload"

If every single sub-domain also uses TLS then you can indicate it by using the following directive instead:

Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"

The max-age is in seconds, 730 days in the example above. That tells the browser how long to remember that any requests to the domain should use TLS.

I personally only add the directive to the virtual hosts running on Port 443. Clients that try to connect to Port 80 will get the header as soon as they are redirected, and if there is already a MITM attack taking place on Port 80 redirects, the attack would just filter that header out. So there is really no functional benefit to adding the header to Port 80 virtual hosts.

SSLCipherSuite

Without proper configuration, the cipher suite used by the TLS connection may not be the best choice for proper security.

Without proper configuration, it may be possible for a MITM attack to influence the selected cipher used, resulting in a cipher that is easier for the attacker to break. That exact kind of attack has happened in the past.

It is also important that all cipher suites use Forward Secrecy (sometimes called Perfect Forward Secrecy). Forward Secrecy makes it very difficult for a session key to be compromised if in the future one of the long-term keys is compromised.

This is important because even though TLS traffic is encrypted, it can still be logged. We know that the U.S. government NSA has done this in hopes of being able to break the encryption in the future, and they probably are not the only one. Forward Secrecy removes one of the methods such an attacker could use if they managed to capture one of the long-term encryption keys. A properly configured secure web server will only allow cipher suites that properly support Forward Secrecy.

Selecting the proper cipher suites can seem like VooDoo. This section is intended to clarify how the process was done for my own needs, and the resulting configuration should be a good general configuration that you can use.

Cipher suite selection is specific to the virtual host configuration. Each virtual host that runs on port 443 (the SSL/TLS port) needs to have the allowed ciphers specified to really be secure.

I generally place it directly after the SSLEngine on directive:

SSLEngine On
SSLHonorCipherOrder on
SSLCipherSuite "cipher-spec1 cipher-spec2 !cipher-spec3"

The SSLHonorCipherOrder on is important, or the cipher selected may not be the preferred cipher to use for best security and performance.

A cipher-spec is a group of parameters that ciphers are matched against. The parameters are separated by a + delimiter.

For example EECDH+aRSA+AESGCM is a single cipher-spec with three parameters.

All ciphers that match the cipher-spec will be added to the list of ciphers that can be used unless the specified cipher is also in a list of ciphers to exclude.

Ciphers to exclude are put at the end of the SSLCipherSuite directive, and use the same cipher-spec format but the string is preceded by a ! operator.

The mod_ssl package in LibreLAMP always excludes all ciphers that match the following cipher-spec directives automatically:

!aNULL !eNULL !EXP !RC4 !LOW

Those automatic exclusions give protection against known insecure cipher suites when the system administrator makes a configuration mistake (or fails to make any configuration) but they are not a substitute for a proper SSLCipherSuite directive.

To properly configure the SSLCipherSuite directive, we must select the parameters carefully and logically.

Forward Secrecy

For forward secrecy, ciphers supporting either ECDHE key echange or DHE key exhange must be used.

The ciphers that use RSA key exchange should not be used unless you absolutely must support very old clients that do not have forward secrecy support. Those old clients are no longer maintained by their vendor and no longer receive security updates, so supporting them is not secure anyway.

Only ECDHE and DHE key exchange ciphers should be allowed on a secure server.

The not yet finalized TLS 1.3 specification only allows ciphers that support Forward Secrecy, the ECDHE and DHE ciphers. We should not wait to implement that restriction ourselves.

Between the two, ECDHE is superior for a variety of reasons. ECDHE ciphers are much faster, and are not vulnerable to the Logjam Attack. DHE ciphers should only be used for compatibility with clients that do not support ECDHE and if used should be specified at the end of the cipher list.

The cipher-spec parameter to specify ECDHE capable cipher suites is EECDH. The cipher-spec parameter to specify DHE capable cipher suites is EDH.

Private Key Types

There are two types of private key certificates that are suitable for use on a TLS capable web server:

  • >= 2048 bit RSA
  • >= 256 bit ECDSA

ECDSA is the superior of the two and is the future, but a very few clients do not support it. All clients currently support RSA.

Your choice in Certificate Authorities is also currently more limited with ECDSA.

My personal preference on my servers is to just use ECDSA unless it is a website I want to promote via sharing on Tumblr. Tumblr will make use of OpenGraph data in a website but it seems to be using a crusty old version of libcurl that breaks with ECDSA certs, so to share a site on Tumblr and have the OpenGraph data made available, one still needs to use RSA certificates.

The parameter to specify ECDSA capable cipher suites is ECDSA. The parameter to specify RSA capable cipher suites is aRSA. Unless you are using both types of certificates, you do not need to worry about specifying which, the server will only select ciphers that match your certificate.

When you are using both, you should specify that ECDSA capable ciphers are used for ECDHE key exchange.

AEAD Ciphers

AEAD is a mode of operation where the cipher encrypts the confidential data and provides authentication for it during the same operation that it adds non-encrypted associated data to the content. It does this with a simple programming interface.

This is important, when encryption fails very often the failure is because the encryption was bypassed rather than actually broken. With an easier to use programming interface, it is far less likely for there to be bugs in the cipher implementation that allow the encryption to be bypassed.

This is so important that it appears the TLS 1.3 specification (not yet finalized) will only allow AEAD ciphers for TLS 1.3 connections.

The AEAD ciphers in LibreLAMP are the ciphers that use ChaCha with POLY1305 and AES with GCM.

All other things being equal, AEAD ciphers should be given precedence in the cipher order.


With those concepts in mind, we can now create our SSLCipherSuite directive based on the level of TLS and client support we wish to achieve.

TLS 1.2 Support

To properly cover all (or at least most) TLS 1.2 clients, we need to take the following parameter combinations into consideration:

  • ChaCha 20 (CHACHA20)
  • AES with GCM (AESGCM)
  • AES with SHA384 (AES+SHA384)
  • AES with SHA256 (AES+SHA256)

Both the CHACHA20-POLY1305 and AESGCM selected suites are AEAD cipher suites and should be used before any other cipher suites if the client supports them.

With clients that have hardware acceleration for AES-NI the AESGCM selected ciphers give better performance (most modern Intel based desktops and laptops) but for clients without that hardware acceleration (most mobile platforms) the ChaChA20 ciphers give better performance.

So what we do, we list CHACHA20 first in our cipher list, followed by AESGCM. Clients on hardware that support AES-NI acceleration ideally would not report ChaCha20 support during the handshake so that those clients will benefit from the hardware acceleration. I do not think that is the case, what I have seen is they do report ChaCha20, but that is a client bug.

For TLS 1.2 clients that do not support either of those, and there are a few, the AES selected ciphers with a SHA384 hash are preferable to those with a SHA256 hash.

Every TLS 1.2 client supports ECDHE so for TLS 1.2 support we do not need to worry about including any DHE ciphers.

Thus our SSLCipherSuite directive would look like this:

SSLCipherSuite "EECDH+CHACHA20 EECDH+AESGCM EECDH+AES+SHA384 EECDH+AES+SHA256"

On a server with an RSA private key that directive would expand to the list of the following 5 ciphers:

  1. TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcc13)
  2. TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
  3. TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
  4. TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
  5. TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)

I believe that short list covers virtually every TLS 1.2 client out there.

TLS 1.0 Support

In the real world, unfortunately there are still many clients that only support TLS 1.0.

To support them, we can add a cipher-spec containing EECDH+AES to our directive.

That will add any ECDHE AES capable ciphers not already listed. The added ciphers:

  1. TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
  2. TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)

Our SSLCipherSuite now looks like:

SSLCipherSuite "EECDH+CHACHA20 EECDH+AESGCM EECDH+AES+SHA384 EECDH+AES+SHA256 EECDH+AES"

That directive gives sane cipher selection for TLS 1.2 clients and for most TLS 1.0 clients.

DHE Cipher Support

They are becoming scarce, but there are still some TLS 1.0 clients that do not support ECDHE but can achieve Forward Secrecy with DHE ciphers. Also, some sites like Tumblr are still restricted to DHE when retrieving information about links.

If you wish to support those clients, you can add EDH+AES256 to the mix. That will add a few ciphers those clients support. The SSLCipherSuite directive now looks like this:

SSLCipherSuite "EECDH+CHACHA20 EECDH+AESGCM EECDH+AES+SHA384 EECDH+AES+SHA256 EECDH+AES EDH+AES256"

The following three ciphers were added:

  1. TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x9f)
  2. TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x6b)
  3. TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x39)

In modern Apache, for the DHE key exchange ciphers just added, Apache will default to using predefined DH parameters corresponding to the number of bits in the RSA certificate.

That means with a 2048-bit RSA certificate, by default you will be using 2048-bit DH parameters from a well-known group used on a lot of different servers.

Apache as packages by LibreLAMP has been patched to default to 3072-bit DH parameters when custom DH parameters are not specified.

Optional Paranoid Tightening

The first two ciphers added by the EDH+AES256 cipher-spec allow for potential ECDHE to DHE key exchange downgrade attack with TLS 1.2 clients. It is very unlikely but their very presence makes it possible.

You can exclude those two ciphers by adding the following to the end of your SSLCipherSuite:

!EDH+AESGCM !EDH+SHA256

Those clients might still be vulnerable to a downgrade attack to the third cipher, the cipher we want, but that third cipher only uses an SHA1 hash so some TLS 1.2 clients that support the first two already do not support the third and thus are only potentially vulnerable to a downgrade attack to the first two.

Adding the exclusions is paranoid, but the presence of those two ciphers does not add support for any clients not already supported by the better ECDHE key exchange ciphers, so the presence of those two ciphers is simply not needed.

HTTP Public Key Pinning

I am not a fan of HPKP and do not use it myself, it is too complicated to maintain with a severe risk of bricking your website for some users under certain circumstances. However instructions for how to do it if you want to are provided here.

HTTP Public Key Pinning (abbreviated as HPKP) is a mechanism to help protect your users from future MITM attacks involving a fraudulent TLS certificate.

In a nutshell, fingerprints of the public keys associated with your private keys (plural) are sent in a header to the client, along with instructions on how long to store them.

When connecting to a secure site that uses HPKP, browsers that support HPKP will check to see if they have a stored copy of the fingerprint from a previous visit to that site, sent with the header during an earlier visit. If they do, the browser will then check that the fingerprint for the certificate used with the current connection matches one of the stored fingerprints.

If it does not match, then the connection will be refused. This protects the user from cases where a hacker manages to obtain a fraudulently signed certificate that matches the domain and then poisons DNS server cache to point requests to the legitimate server to the hacker’s server.

For this to work, you need to have two private keys for your server. One private key that is actually on the server and used with the signed certificate, and a second private key that is not on the server that you can use to generate a new certificate if the private key on your server is compromised or revoked.

If your backup key is on the server and the server is compromised, then the backup is no good. Do not implement HPKP unless you have a secure place, preferably not on a networked computer, to generate and store your backup private key.

Extract Fingerprint

The fingerprint can be extracted from the private key itself, from the signing request, or from the signed certificate. The method shown here generates the fingerprint from the private key, as that is likely the only method available for the backup key (you do not need to get a signed certificate for the backup until it is needed, if ever.)

To extract the fingerprint from your private key:

/usr/bin/libressl pkey -pubout -outform der -in /path/to/YOUR_PRIVATE_KEY.key |\
  /usr/bin/libressl dgst -sha256 -binary |\
  /usr/bin/libressl enc -base64

In the event you are not familiar with UN*X shell commands, the | is called a pipe. It takes the output of the command before it and sends it to the command after it. The \ is a backslash, an escape character. It escapes the return character so that you can continue the command on a new line.

The output of the command above should look something like this:

2QYrZaFL/cwf2kqKINuaX4gkeOxmw8jzEybObYMpFYU=

The fingerprint is a base64 encoded string that will always end with an equal sign.

For the rest of the document, KEY1 refers to the base64 fingerprint for the key you use with your signed certificate, and BACKUP refers to the base64 fingerprint for the key you have offline as a backup.

Configure Apache to Send the Header

To send a header in Apache you can use the Header directive:

Header always set Public-Key-Pins "pin-sha256=\"KEY1\"; pin-sha256=\"BACKUP\"; max-age=5184000"

The max-age part of the directive tells the browser how long it should store the pins, in seconds. 5184000 seconds is 60 days. That gives a decent period of protection time to your visitors after their last visit to your site.

The directive should be placed at the beginning of each virtual host that uses the certificate associated with the KEY1 fingerprint.

Certificate Rotation

Approximately 70 days before your signed certificate expires, generate a new private key. Extract the fingerprint from that key and add it to your header, so the header now looks like this:

Header always set Public-Key-Pins "pin-sha256=\"KEY1\"; pin-sha256=\"NEWKEY\"; pin-sha256=\"BACKUP\"; max-age=5184000"

Make sure to do it for each virtual host that uses the certificate associated with KEY1 and restart the Apache web server.

At least 60 days later but before your existing certificate expires, generate a CSR for the new key and obtain a signed certificate. Update the Apache configuration to use the new certificate.

We have to start sending the pin for the new private key at least 60 days before we start using a certificate based on it so that when we start using the new certificate, the pin will be already have been pre-loaded in any clients that connected within the last 60 days. Otherwise some of them may reject the new certificate.

After you have verified the new signed certificate is working properly, remove the pin associated with the old private key from the directive, so the directive will now look like:

Header always set Public-Key-Pins "pin-sha256=\"NEWKEY\"; pin-sha256=\"BACKUP\"; max-age=5184000"

Backup Key Rotation

It is not absolutely necessary to rotate your backup private key, but I personally recommend it. In a corporate environment where more than just one person may have access to the backup key, it is more important to rotate it in the case of key theft.

If an employee manages to steal the backup key, they may be able to obtain a fraudulently signed certificate using it and pull off a MITM attack using it. Rotating reduces the time frame for which that is possible.

To keep things simple, do not rotate the backup key at the same time the signed private key is being rotated.

Generate a new private key to be stored on a system other than server. If at all possible, generate the new private key on a separate system from your server. Extract the fingerprint from that new private key. We will refer to this fingerprint as NEWBACKUP. Change the header sent to include the new backup pin:

Header always set Public-Key-Pins "pin-sha256=\"KEY1\"; pin-sha256=\"NEWBACKUP\"; pin-sha256=\"BACKUP\"; max-age=5184000"

After 60 days, remove the key pin for the old backup private key:

Header always set Public-Key-Pins "pin-sha256=\"KEY1\"; pin-sha256=\"NEWBACKUP\"; max-age=5184000"

In the event the private key associated with your certificate is compromised during a backup key rotation, use the old backup for generating a new certificate as the pin for the new backup may not yet be stored in the browsers of some clients.

Server Compromise Recovery

Note: The advice in this subsection needs review and feedback.

If your server has been compromised by a hacker, it is possible the private key associated with your certificate has been stolen. Take the following steps:

  1. Shut down your web server.
  2. Notify your Certificate Authority that your private key may have been compromised. This allows them to revoke the certificate.
  3. Change any and all system passwords and SSH keys.
  4. Figure out how the attacker got in and fix the issue.
  5. Check for any back-doors the attacker may have left behind. Remove them if found.
  6. Generate a CSR from your backup private key and get a new certificate issued by your Certificate Authority. Generate a new backup key, the old backup key is now no longer your backup key.
  7. Order a pizza. I recommend Italian Sausage, Black Olives, and Green Peppers. If you live where they are called pies, get some Buffalo Wings too.
  8. Notify your users, but word it so they feel empathy towards you and anger at the attacker, not anger at your system administration skills. Lock their accounts until they reset their password. Of course you should be using individually salted hashes for each password, but still you should require a password change for each user of your web application.

Final Configuration

If implementing everything described above on an IPv4 server using a 2048-bit RSA private key, your virtual host configuration will look something like the following:

#Port 80 Redirect

<VirtualHost 192.168.0.12:80>
ServerName example.org
Redirect permanent / https://example.org/
</VirtualHost>

<VirtualHost 192.168.0.12:80>
ServerName www.example.org
Redirect permanent / https://example.org/
</VirtualHost>

# Port 443 Redirect

<VirtualHost 192.168.0.12:443>
Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
ServerName www.example.org
Redirect permanent / https://example.org/
SSLEngine on
SSLHonorCipherOrder on
SSLCipherSuite "EECDH+CHACHA20 EECDH+AESGCM EECDH+AES+SHA384 EECDH+AES+SHA256 EECDH+AES EDH+AES256 !EDH+AESGCM !EDH+SHA256
SSLCertificateFile            /etc/pki/tls/certs/example.org-20180620.crt
SSLCertificateKeyFile       /etc/pki/tls/private/example.org-20180620.key
SSLCACertificateFile      /etc/pki/tls/certs/example.org-cab-20180620.crt
</VirtualHost>

#Live Host

<VirtualHost 192.168.0.12:443>
Header always set Strict-Transport-Security "max-age=63072000; includeSubdomains; preload"
ServerName example.org
DocumentRoot "/srv/example.org/www"
SSLEngine on
SSLHonorCipherOrder on
SSLCipherSuite "EECDH+CHACHA20 EECDH+AESGCM EECDH+AES+SHA384 EECDH+AES+SHA256 EECDH+AES EDH+AES256 !EDH+AESGCM !EDH+SHA256
SSLCertificateFile            /etc/pki/tls/certs/example.org-20180620.crt
SSLCertificateKeyFile       /etc/pki/tls/private/example.org-20180620.key
SSLCACertificateFile      /etc/pki/tls/certs/example.org-cab-20180620.crt
ErrorLog logs/example.org.error_log
CustomLog logs/example.org.access_log combined
</VirtualHost>

<Directory "/srv/example.org/www">
  Options FollowSymlinks
  AllowOverride All
  Require all granted
  php_value include_path "/srv/example.org/phpinclude:/usr/share/pear"
</Directory>

IPv6 Instructions

For IPv6 support, just copy each IPv4 Virtual Host but change the IPv4 address to an IPv6 address.

Note that in Apache, due to a : being used to separate the IP address from the TCP port it listens on, IPv6 addresses must be encapsulated within [] square brackets.

Server Testing

I highly recommend frequently testing the quality of your TLS configuration at Qualys® SSL Labs:

Qualys® SSL Labs SSL Server Test

The server for Naughty.Audio is configured according to the instructions presented here. To see how it performs:

https://www.ssllabs.com/ssltest/analyze.html?d=naughty.audio

Unsupported Clients

With the above configuration, the following three clients in the Qualys SSL Labs test are not supported:

  1. Android 2.3.7
  2. Internet Explorer 8 / XP
  3. Java 6u45

None of those clients support the TLS SNI extension, severely limiting their usefulness as real world clients. None of those clients still even receives security updates from the vendor. They are dead clients and are not safe for users to be using.


ECDSA Private Keys

The future of x.509 certificates is clearly with ECDSA certificates. Using them is actually very similar to using RSA certificates, the differences are mostly taken care of for you by the software.

At this time, unless I need Tumblr support, I personally only use ECDSA certificates and only support TLS 1.2 (1.3 is not yet final).

For those who want to start playing with ECDSA certificates now, hopefully this section will be of some benefit.


Selection of which Elliptic Curve to use with an ECDSA certificate is the hard part, and it is not very difficult.

To see all the curves available as part of LibreSSL 2.6.4 run the following command:

/usr/bin/libressl ecparam -list_curves

The list is way too long to replicate here. However most of those curves are useless to us.

In the context of an x.509 certificate for a website, we have to use a curve that is supported by the Certificate Authority that will sign the certificate. With Certificate Authorities that support ECDSA certificates, that list is usually fairly short. With Comodo, I believe it is limited to the following curves:

secp384r1
NIST/SECG curve over a 384 bit prime field
secp521r1
NIST/SECG curve over a 521 bit prime field
prime256v1
X9.62/SECG curve over a 256 bit prime field
sect163k1
NIST/SECG/WTLS curve over a 163 bit binary field

That list may change, and other Certificate Authorities may have support for additional curves.

From that list, we can eliminate sect163k1 because 163-bit is below the minimum 256-bit currently considered to be Deployment Best Practices for ECDSA certificates. It also does not appear to be supported by common browsers.

From that list, we can eliminate secp521r1 because several browsers, including current versions of Google Chrome and some versions of Internet Explorer, do not support it.

Between the two remaining options, secp384r1 is probably more difficult to crack simply because the prime field is larger. So that is the option I presently am going with.

I am assuming that prime256v1 is the same curve as secp256r1 that most browsers list as supported, several Internet queries give that impression, but you should double-check before using it.

Curve Paranoia

Not all curves are the same. In cryptography it is very important to know why the parameters for a given curve were chosen.

Without that knowledge, it is possible that the parameters were chosen because they create a weakness to the curve that is not obvious. There just is no way to know.

What we do know, is that in the past the NSA has influenced cryptography for the purpose of making it easier for them to break it in the future. That is why it is very important that decisions like the parameters chosen when developing a specific curve must be transparent for the curve to really be considered secure.

This is not just a hypothetical paranoia, they have done it before.

Both secp384r1 and prime256v1 lack that transparency. It is possible that the parameters chosen in developing those curves were chosen because it was known that certain characteristics might make it possible to crack them in the future. We just do not know.

Private Key Generation

To generate your ECDSA private key using the secp384r1 curve:

pushd /etc/pki/tls/private
DUMASK=`umask`
umask 0277
/usr/bin/libressl ecparam -name secp384r1 -genkey -out example.org-20180620.key
umask ${DUMASK}
popd

Optional Paranoid Strict TLS 1.2 Support

When I use ECDSA I choose to only support TLS 1.2 clients and use a very limited SSLCipherSuite directive that only includes a small set of ciphers:

SSLCipherSuite "EECDH+CHACHA20 EECDH+AES256 -SHA"

Currently that setting results in only three ciphers being available, two that are AEAD and one that is CBC. That is enough to cover most TLS 1.2 capable clients.

It bothers me a little bit that Android 5 and Android 6 are listed as not being supported at Qualys SSL Labs with that configuration. Both were supported with that configuration with LibreSSL 2.5.5.

The failure is because there was a preview version of the ChaCha20 cipher suite and now a final version of the ChaCha20 cipher suite. Those Android clients only support the preview version. LibreSSL 2.5.5 supported both preview and final, but 2.6.4 only supports the finalized version. It is possible that Android will update their clients that support the preview version, and it possible other brothers (e.g. FireFox on Android) are supported, I just do not know.

If you remove the -SHA from the above SSLCipherSuite directive, an additional TLS 1.0 cipher will be available that is supported by those versions of Android.

That is all there is to it. From this point on, you can generate a CSR and proceed to configure Apache the same way you would if using an RSA private key, as discussed in the section Private Key and Certificate Generation.

Live Example

To see how a Paranoid Strict TLS 1.2 LibreLAMP powered server with an ECDSA certificate using a secp384r1 curve measures up:

https://www.ssllabs.com/ssltest/analyze.html?d=media.librelamp.com