Apache TLS Certificate Generation

NOTE: If you use the shell scripts on this page to generate a certificate or CSR that you will be using outside the content of Apache, you must read the OCSP Must Staple Note.

Terms Used

Before I discuss key types, I have a list of terms I hopefully use correctly. The list is long, if you feel you understand them, you can skip past them or not open the details node if you are in a graphical browser.

Terms Used (click to open)
Quantization Hash
Normally just called a hash. What it is, in a nutshell, is an algorithm that takes arbitrary data and turns it into an integer number within a specified integer range (the quantization range). The range is usually described in terms of computer bits. A one-bit hash would have an integer range of 0 to 1 (two possible values) which uses a single bit. A two-bit hash would have an integer range of 0 to 3 (four possible values — 00, 01, 10, and 11) which uses two bits. An 8-bit hash has 256 possible values (28), etc.
With a quantization hash function, the same input data will always result in the same hash. The hash is one way, you can always generate the hash from the input data but you can not determine the input data from the hash. Sometimes you can determine input data that results in the same hash as a given hash, but that does not guarantee the input data was identical.
A collision is where two sets of input data that are different from each other result in an identical hash value. With a one bit hash where we use a 0 to represent any even integer value and 1 to represent any odd integer value, the hash of 122 and 62638 would both be 0 since they are both even, so that is a collision.
With any quantization hash there is an infinite set of data that can be hashed but a finite set of hashes that the algorithm can produce. Therefore mathematically collisions must exist.
Butterfly Effect
A quantization hash algorithm is said to possess the butterfly effect when even the smallest change to the input data results in a quantized hash value that is radically different in an unpredictable fashion.
A salt is a piece of data added to the input data before it is hashed. If the hashing algorithm has the butterfly effect, this salt will result in a hash that is radically different than it would have been without the salt or with a different salt.
Password hashes usually use a unique salt for this reason. If twenty-five users all use the same password, and either a salt is not used or the same salt is used for all passwords, an attacker who gets a-hold of the hashes will see that they all have the same hash and can deduce the password for the accounts by trying lists of commonly used passwords. However if each password hash has its own unique salt, the hashes will all be different and the attacker does not know who uses common passwords.
With password hashing algorithms, it is common to create a random string to use as the salt when the password is first hashed. That salt is then stored in the database with the password hash itself. It does not matter if an attacker who steals the database knows the salts as long as all the salt for each password hash is different.
Cryptographic Hash Algorithm
A cryptographic hash algorithm is a quantization hash algorithm where the quantization range is large enough that statistically, collisions will only be found if there is a flaw in the algorithm that either biases the hash to certain values within the range or allows intentional changes to the data to result in predictable changes to the hash.
A cryptographic hash algorithm will usually have a quantization range of at least 128 bits. 160 bits gives even better confidence. 256 bits is what most cryptography applications use.
It is estimated there are 1024 stars in the known universe. That is approximately equivalent to 280 stars. If every star represented data and we took a 128 bit hash of that data, it is unlikely we would have a collision. With a 256 bit hash, it is inconceivable. A collision with a 256 bit hash will only occur if the hashing algorithm itself is biased in the values it outputs. Unfortunately that happens, it is a difficult math task to come up with a hash algorithm that does not have flaws. As soon as any collision is found in a cryptographic hash algorithm, you know the algorithm is flawed and should not be used.
A cryptographic hash algorithm will have a proper butterfly effect. Given two hashes, it will be impossible to determine how similar the different sets of input data behind those hashes are. If just one byte of input data has changed, the resulting hash should be as different as if the two inputs are completely unrelated to each other.
The hash from a cryptographic hash algorithm will not reveal any information about the original data. Amount (size) of data being hashed, repetition of patterns within the data being hashed, file formats of the data being hashed, none of that information can be determined from the output of a cryptographic hash function.
Broken Cryptographic Hash Algorithm
Any hash algorithm that is intended to meet the requirements of a cryptographic hash algorithm but does not meet those requirements is called ‘broken’. They should not be used.
An example is the MD5 hash algorithm. MD5 is so broken, it is now possible to craft a digital image of a hash where the image has the same hash as what it visually shows in the image. MD5 no longer has any valid use. Another example is the SHA1 algorithm. The SHA1 algorithm is a 160-bit algorithm, but once a collision was found, it became clear the algorithm itself had to have a flaw. That flaw has now been detailed. It is more difficult to intentionally create a collision than with MD5 but it is still possible to do so. If you must use a 160-bit hash, use RipeMD 160.
Digital Signature
A digital signature combines a cryptographic hash with public / private key cryptography. Please note that digital signatures do not encrypt anything.
The entity (Alice) vouching for the authenticity of the input data will take a cryptographic hash of that data. The hash is what is then signed using their private key. When someone else (Bob) wants to verify the data has not changed since Alice vouched for it, Bob will use the same hashing algorithm to find the hash of the data. Bob can then uses Alice’s public key to check that the signature matches what it should match if the identical hash was signed by Alice’s private key.
If the signature does not match, then Bob knows either the data was modified, the private key that signed it does not match the public key for Alice that he has, or both.
As long as the signature matches, the cryptographic hash algorithm used is not broken, and Bob trusts that Alice and only Alice has access to the private key used to sign the data, then Bob can have confidence that Alice vouches for the integrity of the data.
Chain of Trust
If Bob knows Alice on a personal level, he can trust the public key is hers by talking to her directly. Usually we do not have that luxury.
The Chain of Trust starts with one entity that you implicitly trust to tell you what public keys they have verified as belonging to the entity that claims they own them.
For web servers, this is usually done with an x.509 certificate. Alice will get a certificate that contains her public key, her website address, and other details. That certificate in turn has a digital signature from a Certificate Authority that Bob’s web browser trusts.
Usually Alice’s certificate is actually signed by an intermediary that is trusted because the intermediary itself is signed by a root anchor that Bob’s browser implicitly trusts. That is called the Chain of Trust.
Asymmetric Encryption
Asymmetric Encryption is where both parties have a private key and a public key, and they do not have a shared private key between them. When Alice sends a message to Bob, she uses Bob’s public key to encrypt the message. Only Bob can decrypt the message, as that requires his private key. When Bob replies, he uses Alice’s public key to encrypt the message sent to Alice.
The strength of Asymmetric Encryption, the two parties do not have to be directly connected. The message can pass through intermediaries, as often happens with e-mail where Alice and Bob never actually connect to each other.
The weakness of Asymmetric Encryption, the private keys are stored long term making them potentially susceptible to theft or cracking. The NSA allegedly has terabytes of logged asymmetric encrypted messages just waiting for the possibility of a captured private key that will allow them to decrypt the message.
Symmetric Encryption
With symmetric encryption, both parties use a shared secret for the communication. As both parties share the secret, they do not use public keys for the encryption.
In the context of a TLS connection, the two parties negotiate a shared secret when the session first connects. By using either DHE or ECDHE key exchange, it is possible for both parties to derive the same shared secret without the secret itself ever being sent over the network.
This allows the shared secret to be a short term ‘ephemeral’ secret that is discarded and forgotten by both parties after the session. This is called Forward Secrecy (sometimes Perfect Forward Secrecy) because if the encrypted communication is logged, capturing the private keys of either party at a later (forward) date is useless in decryption of the session because the shared secret is not derived from the private keys.
With this encryption scheme, the public and private key pairs are still used (usually only the server) but only for authentication that the server is who it claims to be when the connection is initially negotiated. The private and public keys are not used for encryption or decryption, they are only used for digital signatures.
The limitation of this encryption method, the two parties must be connected to each other so they can negotiate the shared secret. This pretty much limits it to TLS, it is not useful for Alice encrypting an e-mail message that can only be decrypted by Bob.
The weakness of this encryption method, if you use DHE key exchange and your DH parameters are too small, an attacker can figure out the shared secret. The solution is to use ECDHE key exchange if you can. If you must use DHE key exchange, do not use DH parameters smaller than 2048-bit. I personally recommend 4096 bit.
RSA Key Pair
RSA is a Public / Private key cryptography system invented in 1978. The British actually invented the same system earlier in 1973 but it was classified. It uses the mathematical difficulty of factoring the product of two very large prime numbers for security.
RSA is a slower algorithm, but it is useful for digital signatures as well as for asymmetric encryption. This is why it is often used with S/MIME and PGP.
RSA is the most common key pair scheme used by web servers, but that is mostly due to tradition. At this point in time, everyone should switch to ECDSA key pairs.
An RSA private key MUST be at least 2048-bit if you use it at all. 2048-bit is large enough. I do use 4096-bit when I have to use an RSA key but that is simply to silence those who are vocal and think bigger is better even when 2048-bit is big enough.
ECDSA Key Pair
ECDSA is a Public / Private key cryptography system that is an updated version of the older DSA system. DSA was invented in 1991.
ECDSA (and its predecessor DSA) are only useful for digital signatures, they are not useful for asymmetric encryption. You can not use a ECDSA public key to encrypt a message that can later be decrypted with the private key.
What you can do however is use ECDSA to authenticate the server to the client (and optionally the client to the server) so they can use ECDHE key exchange to negotiate a shared secret for use with symmetric encryption. This is how modern TLS encryption is done.
Any graphical web browser that can not handle ECDSA is at this point several years too old to be safe. The only reason for not using ECDSA for your server certificate is a specialty web server that provides web services for non-human users, and even the vast majority of those use cases have supported ECDSA for several years now.
This is the type of Private / Public key pair I recommend for a web server.
Revoked Trust, CRL and OCSP
Sometimes a Certificate Authority revokes the trust they have previously vouched for. This can happen because either they discover they were tricked into vouching for the information in the certificate, or because they have become aware the private key that corresponds with the public key in the certificate has been compromised.
Smaller Certificate Authorities (such as a corporate CA) will sometimes use a Certificate Revocation List (CRL) that identifies the serial number of certificates they signed but no longer trust. CRLs are not a practical solution for Certificate Authorities that sign large quantities of certificates, they do not scale well.
Online Certificate Status Protocol (OCSP) is a more scalable solution, it allows the client to query the status of a specific certificate without needing to download a huge list of revoked certificates to see if a particular certificate is listed as revoked.
OCSP Stapling
An OCSP response is trusted by the client because it contains the digital signature of the Certificate Authority. As they have a relatively short period of time they are valid for (typically 24 hours), the server can streamline the client verification of the certificate by requesting the OCSP response, caching it, and including the response every time it sends its certificate to a requesting client as part of the TLS negotiation.
This is called OCSP stapling, and every web server should do it. Apache for LibreLAMP is configured to do it by default.
With servers that are capable of OCSP stapling, security in increased by using a certificate that requires OCSP stapling. This tells the client it should reject the validity of the certificate if the certificate is not accompanied by a valid OCSP response. It solves the problem of a MITM attack where the attacker has a stolen private key, the certificate has been revoked as a result, but the attacker prevents the client from querying the OCSP server to find out the certificate has been revoked.
Whenever using a TLS server that supports OCSP stapling, a certificate that tells the client a valid OCSP response MUST accompany the certificate should be used.
In the context of things like code signing or S/MIME, OCSP stapling does not make sense because the OCSP response will likely have expired by the time the certificate is verified by the client. Stapling only makes sense for TLS connections. Outside the context of TLS, the client needs to perform the OCSP query itself.

TLS Key Pair for Apache

There are two types of private / public key pairs you can use with Apache: ECDSA is what is currently recommended, RSA can still be used if you think legacy technology is warranted.

With HTTPS in 2019 we should only be using cipher suites that provide Forward Secrecy, so the private / public key pair only serves the purpose of authentication, it is not used in the actual encryption of the traffic. ECDSA is much more efficient at that task, and ECDSA has been widely supported in browsers for several years now. I do not believe it makes sense to continue using RSA on web servers.

The certificate for a web server should be signed by a Certificate Authority with a root anchor that is trusted by the major browser vendors by default. The major browser vendors include Mozilla, Microsoft, Opera, Google, and Apple. The minor browser vendors tend to use the operating system for their root anchor trust. GNU/Linux and *BSD distributions tend to use the Mozilla root anchors. Other operating systems are also major browser vendors and use the same root anchors that their browsers trust.

Let’s Encrypt meets that requirement and is free, but the downside is that the certificates are only valid for ninety days. If you need a certificate with a longer life, I recommend buying a Comodo certificate from Namecheap.

Let’s Encrypt

There are two strategies with Let’s Encrypt. Full automation of renewal, which is what they recommend, and manual renewal, which is what I recommend.

I prefer manual renewal because I want to know right away if there was a problem. I want to know right away if the certificate they delivered has a problem. Maybe I am paranoid, but with automated systems, it seems when they fail is when I am out for a weekend or otherwise unable to deal with the issue. I do not like automated renewals. The claim is automated renewals reduce expired certificated but I have seen several Let’s Encrypt sites with expired certificates because something broke the automated renewal process and the webmaster did not realize it until the certificate expired.

If you want to use an automated renewal method, visit Let’s Encrypt Community Support for help in setting that up if you need help. The instructions that follow in this section are for manual renewal.


To use Let’s Encrypt, you need to have the certbot utility installed. This utility is part of EPEL for CentOS 7:

[root@host ~]# yum install certbot

That will likely bring in a few Python dependencies as well. If you want automated renewals, this is also the utility that does it, but I do not have instructions for that.

Manual Renewal Shell Script

I wrote the following bash shell script to make manual renewal of Let’s Encrypt certificates easier for me:

It does several things.

If necessary, it generates a private secp384r1 ECDSA key. There are only two ECDSA curves that are commonly supported by web browsers:

  • secp384r1
  • prime256v1 (also called secp256r1)

The secp384r1 curve is the strongest of the two, which is why I use it. Most browsers also support secp521r1 however Google Chrome dropped support for that curve, I am not aware of why. As secp384r1 provides stronger cryptography than 4096-bit RSA and few people were using secp521r1, it is possible they simply found it unnecessary, especially since it only provides authentication and not encryption.

The script only generates a fresh private key first time it is run for a particular domain during the calendar year. Subsequent renewals will use the same private key until the next calendar year. This is done so that those using DANE do not have to update their TLSA records every couple of months.

The very first time it is run, and possibly whenever the terms of use change, running the script will ask you if you agree to the Let’s Encrypt terms of service. You can read them here:

The shell script generates a CSR request and then sends the CSR via the certbot utility to obtain a signed certificate.

You need to edit the shell script to change some details about you before you run it. The line numbers are specified at the top of the shell script but this is the part that needs editing:

C                      = US
ST                     = Washington
L                      = Seattle
O                      = Your Company Name
CN                     = ${FQDN}
emailAddress           = admin@example.net

At a minimum you must change the emailAddress line to use an e-mail address that actually belongs to you. It is a good idea to also update the C (Country), ST (State or Province), L (City or Location), and O (Organization or Company) fields though honestly I do not think Let’s Encrypt cares if they are accurate. Still, they really should be, they will be part of your certificate and your visitors may lose confidence in your service if they are not accurate. Do not change the CN (Common Name) field, it is set by the bash shell script.

Execution of Shell Script

The shell script takes at least one argument: the domain the certificate is to be obtained for. Additional domain names the issued certificate is valid for can be specified, however please note that each domain must resolve to the same IP address the script is being run on or the certificate will not be issued.

In many respects, the script is a ‘dumb’ script. It does no sorting of the specified domains. It assumes you will specify the primary domain name the certificate is being issued to first.

You can use the same certificate for many different domain names, but generally speaking, it better to only use the same private key and certificate for domains that are related to each other, e.g. all sub-domains of the same domain.

So for example, if you were to run a server for example.org, www.example.org, forum.example.org, themes.elsewhere.com, and shopping.elsewhere.com — you would probably want two private keys and certificates. One for example.org, www.example.org, and forum.example.org and the second for themes.elsewhere.com and shopping.elsewhere.com.

You could also have individual private keys and certificates for all five in that example, but if they run on the same physical server that is waste of resources. Apache needs to load each certificate and perform OCSP queries for each certificate, so I only recommend separate certificates for domains that are not related to each other.

The script uses the first domain specified as part of the filename for the private key and certificate. I highly recommend you make it the shortest domain name the certificate will be good for.

An example execution of the script:

[root@librelamp ~]# systemctl stop httpd.service
[root@librelamp ~]# sh LE-HTTPS-Keygen.sh librelamp.com www.librelamp.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Performing the following challenges:

...[redacted output]...

TLSA from PubKey:
3 1 1 FF24B8FAF28DB0998F594B6CA8C4F8A513A56AE4508745A7724825581D205BBA

Private Key:    /etc/pki/tls/eff_private/librelamp.com-HTTPS-20190124.key
x.509 Cert:     /etc/pki/tls/eff_certs/librelamp.com-HTTPS-20190124.crt
CA Bundle:      /etc/pki/tls/eff_certs/librelamp.com-HTTPS-CAB-20190124.pem
DANE TLSA Data: /etc/pki/tls/tlsa/librelamp.com-HTTPS-20190124.tlsa

New Private Key Generated.
Be sure to update your DNS TLSA record(s) if you use DANE
[root@librelamp ~]# systemctl start httpd.service

A few things to notice about the above script execution.

First, it ran as root. The Let’s Encrypt certbot utility needs to open up Port 443 for the challenge and response that Let’s Encrypt uses to make sure certbot is being run by an administration user on the IP address for the hostname(s) the certificate is being issued for. This is how Let’s Encrypt makes sure it is not issuing a fraudulent certificate.

Second, I stopped the Apache server. This important, otherwise Port 443 is in use already and certbot can not use it for validation.

Third, when I executed the script, I used both librelamp.com and www.librelamp.com as arguments. I have the www. variant redirect to librelamp.com but that redirect needs to be secure, so I need a certificate for that variant. Some certificate authorities automatically add the www. variant, Let’s Encrypt (rightfully in my opinion) does not, you need to specify it if you want it.

Fourth, the output includes the RDATA portion of TLSA record needed for DANE support (see RFC 6698). DANE is well supported with popular SMTP MTA client implementations, but at this point in time, no mainstream HTTPS client supports DANE. You do not need to do anything with that data, it is there simply for you to use if you want to be an early adopter of DANE on HTTPS to encourage browser support for it.

Fifth, running the script will generate three or four files:

Private Key
The first time in a calendar year the script is run for a specific domain, it generates a new private key. This is what you use with the SSLCertificateKeyFile Apache directive to specify your private key.
Note that the next time you run the script in the same calendar year with the same domain as the first argument, it will re-use this private key.
x.509 Cert
This is the full path to the certificate file issued by Let’s Encrypt. This is what you use with the SSLCertificateFile directive.
When you run this script again with the same domain as the first argument, a new filename is created with a different YYYYMMDD encoded at the end of the filename.
CA Bundle
This is the full path the intermediate certificates between the Let’s Encrypt Certificate Authority Trust Anchor and the your signed certificate. This is what you use with the SSLCACertificateFile Apache directive.
When you run this script again with the same domain as the first argument, a new filename is created with a different YYYYMMDD encoded at the end of the filename. Most of the time the contents of the file does not change, but it is always possible it will.
This is a file that includes the current TLSA record data. The contents only change when a new private key is generated.
When it changes, if you implement DANE, make sure to create a new TLSA record and give the new record time to propagate in the DNS system before you update the Apache configuration to use the new private key. Once you are using the new private key, delete the TLSA record(s) that point to the old fingerprint.

Sixth, notice it informs me that a new private key was generated. When that notice is NOT there, the only directives in the Apache configuration file that should be updated are the SSLCertificateFile and the SSLCACertificateFile directives. Only when that notification is there, also update the SSLCertificateKeyFile directive.

Seventh, notice that I start the web server right away. That way it will continue to serve clients with the old certificate while I (if needed) update the DNS TLSA record and update the apache configuration files. Once the apache configuration files are updated, I can just restart it at that point.

Certificate Renewal Schedule

A certificate issued by Let’s Encrypt is only valid for 90 days. Most of the time that works out to be just under three months.

Some people who manually renew just renew every month, and that works fine. I personally renew every other month, always renewing during an odd month (January, March, May, July, September, November). Either schedules ensures you get a new certificate before the current one is 90 days old, with enough cushion time to deal with issues that prevent certificate renewal.

Make sure you have Postfix properly set up at least as a Null Client so that the operating system can send you messages if the certificates used by Apache are close to expiring.

Traditional Commercial Certificate Authority

If either you have a lot of domains or very limited budget, you really should just use Let’s Encrypt. Even with the manual renewal I describe, it becomes fast and painless with a regular schedule. However with just a few domains and a decent budget, there are advantages to using a traditional certificate authority:

  1. You do not need to shut down the web server regularly for renewals
  2. You have the option of generating your private key on your local workstation and upload it to your server along with the signed certificate. Okay technically you can generate the private key on your local workstation with Let’s Encrypt too, but it is a bit more complicated to do so.
  3. It is best practice to generate a fresh private key once a year, but you can (optionally) buy a two year certificate so that if you are a week or three or seventeen late in your yearly renew schedule, your server still has a valid certificate.
  4. For the paranoid, you can have a firewall that restricts outgoing connections from the server and still renew your certs. Many WordPress and other malware operate by making an external connection and downloading code to execute. Such outbound firewalls are often a good idea, but they make Let’s Encrypt difficult to use.

If you wish to use it, I have written a shell script that automates the generation of a CSR request that you can send to a commercial Certificate Authority. The script is very similar to the Let’s Encrypt with a few key differences:

  1. It places the private key in /etc/pki/tls/private instead of /etc/pki/tls/eff_private
  2. It always generates a fresh private key, unless one has been generated already with the same file name (generated on same day) in which case the script exits.
  3. It only generates the CSR, you have to send that to your Certificate Authority to get the certificate.

The script is here: CSR-HTTPS-Keygen.sh

The first argument is the primary domain name the certificate will be issued for. You can add additional domain names as arguments, however unlike Let’s Encrypt, you will have to pay more if you choose to do that. When buying a cert, make sure you specifically buy one that covers the total number of domains you want the certificate to cover.

Also note with Comodo, they will automatically add the www. sub-domain. Many seem to do that without you need to specify it during CSR generation.

An example execution of the script:

[root@host ~]# sh CSR-HTTPS-Keygen.sh example.org


TLSA from PubKey:
3 1 1 F7E7915506F6034D8270ACD3F953379453E6B10E0A7D76DAF274218C7F68DE09

Private Key:    /etc/pki/tls/private/example.org-HTTPS-20190124.key
CSR File:       /etc/pki/tls/csr/example.org-HTTPS-20190124.csr
DANE TLSA Data: /etc/pki/tls/tlsa/example.org-HTTPS-20190124.tlsa

That is everything you need to get an secp384r1 curve ECDSA certificate.

Most Certificate Authorities will have a form where you simply copy and paste the CSR data into the form (including the opening and closing

OCSP Must Staple Note

When generating the configuration file for the CSR, both shell scripts add the following line to the configuration:     = DER:30:03:02:01:05

That is done for security reasons. It tells clients to reject the certificate if a valid OCSP response is not stapled to the certificate. The Apache server is capable of OCSP stapling and the default configuration in LibreLAMP is to OCSP staple.

The problem being solved: If someone (e.g. NSA) manages to steal your private key, they can use your private key and the certificate to potentially perform a MITM attack.

Once you know your private key has been compromised, you can (and should) notify your Certificate Authority so they can revoke the certificate. The attacker however can still use the certificate, and simply block the victim’s attempt to contact the OCSP server. Their client will then still accept the certificate as valid. However when the certificate requires OCSP stapling, clients will then reject the certificate if it does not have a valid OCSP response sent with it, thwarting the attack.

If you intend to use the certificate with server software other than Apache, make sure that server software is both capable of OCSP stapling and is configured to do so.

Otherwise, you need to remove that line from the shell script, or clients will reject the certificate.

Postfix and Dovecot are examples of servers that do not (yet) support OCSP stapling.

For Postfix, you should use the certificate generation script on the Postfix TLS page. Postfix also still should use RSA rather than ECDSA, which the script on that page does.

For Dovecot, at present use the Postfix script. I will soon be working on Dovecot specific instructions, but at the present, just use the SMTP key generation or if you want ECDSA with Dovecot, use the scripts on this page but remove the line from the script that causes the certificate to require OCSP stapling.