Configuring Apache for Client Certificates (such as DoD CAC cards) on Red Hat Linux/CentOS

David A. Wheeler

2012-06-26

Here is how to configure the Apache web server on Red Hat Enterprise Linux (RHEL)/CentOS so that it will accept either client-side certicates (particularly U.S. DoD CAC cards) or username/passwords. This is simply a combination of material available elsewhere, in a convenient form. This focuses on U.S. DoD CAC cards, but most of this material is the same for any client-side certificate. With some variation it would apply to many similar circumstances. The latest versions of the encryption protocol SSL is actually named TLS, but I'll call it SSL here.

NOTE: This page does not discuss how to configure a client (such as a web browser or email client) to use client certificates. Other pages go through the details; a key is that you need to configure them to load the DoD root certificates.

First, you need to set up the Certificate Authority (CA) certificates so that Apache can validate that client certificates (such as CAC cards) are signed by the proper authorities. Then, you need to configure the web pages so that they require the right kind of authentication, in this case, to accept either client side certificates (CAC cards) or username/password.

But first, if you use VirtualHosts (multiple web sites on a single system), you should be aware of important SSL-related issues.

VirtualHosts and SSL

A single Apache server can serve many virtual hosts, using its VirtualHost block directive. If you are using virtual hosts, read this section first, because there are important issues involving VirtualHosts and SSL.

If you use SSL, in practice you need to have a separate IP address for each virtual host that uses SSL. The Apache 2.2 documents baldly say that "name-based virtual hosting cannot be used with SSL secure servers because of the nature of the SSL protocol." The problem is the original SSL (TLS) specification itself; when using SSL, Apache must quickly switch to using SSL before it finds out the name of the webserver, but it needs to know what certificate to use before it can switch to SSL.

There is relief on the way. The IETF RFC 3546 specification section 3.1 adds the "Server Name Indication (SNI)" extension to the TLS security protocol, and this extension makes it possible to configure named-based virtual hosts. Problem is, both the web server and every web client that uses that web server must support SNI for this to work. The web server side isn't so bad; Apache version 2.2.12 and later include built-in support for SNI (you also have to use OpenSSL 0.9.8f or later, and have support for it compiled into Apache). For older versions of Apache (e.g., the one in RHEL5), you can instead use the Apache module mod_gnutls (here's a discussion on mod_gnutls and why you might use it).

Many web browsers support SNI, but not all, and that is the real problem. Web browsers that support SNI include Firefox 2.0 or later, Opera 8.0 or later, Internet Explorer 7.0 or later (unfortunately, only on Vista), Google Chrome, and Safari 3.2.1 (unfortunately only on OS X 10.5.6 or later). But note that this does not include many browsers still in use. What's worse, users will not understand why they can't access the server. Server Name Indication (SNI) is a great improvement, but I suggest not depending on it for now. Techrepublic says that "In practical terms, this means that for a serious e-commerce Web site or one that needs to have broad appeal, this solution won’t work — yet."

So presuming that you have a different IP address for every web address that uses SSL, how can you avoid repeating information? You don't want to have to repeat all the configurations in each case, so use "include" files to avoid repeating yourself. E.G., do something like this, so that MY_HOSTNAME.conf has all the common configuration stuff that would be true for users using either http (port 80) or https (port 443):

<VirtualHost MY_IP_ADDRESS:80>
    Include conf/MY_HOSTNAME.conf
</VirtualHost>
<VirtualHost MY_IP_ADDRESS:443>
    Include conf/MY_HOSTNAME.conf
    Include conf/ssl-common.conf
    SSLCertificateFile    [Filename for server certificate]
    SSLCertificateKeyFile [Filename for server certificate private key]
    SSLCertificateChainFile /etc/pki/tls/certs/verisign-intermediate.crt
    SSLCARevocationFile /etc/pki/tls/certs/CRL-bundle.crl
    ... and other SSL-specific commands, if any.
</VirtualHost>

The sections below will talk more about how to set up some of this, especially the CRL-bundle.

Setting up certificates

First, some absolutely necessary terminology. Server certificates authenticate the server to the client. Client certificates authenticate the client to the server. Both server and client certificates can be signed by third parties, which also have certificates; a third party that signs certificates is a Certificate Authority (CA). In practice, there is often a chain of CA certificates. In most cases you want both server certificates and client certificates signed by at least one CA. There's no requirement that server certificates and client certificates be signed by the same CA, in fact, they are often different.

Loading Certificate Authority (CA) certificates into Apache

There are various instructions for configuring Apache web servers so you can use U.S. Department of Defense (DoD) CAC cards, and these will tell you how to get the DoD certificate authority (CA) certificates (you need this information so that you can determine if client certificates were actually signed by a DoD CA). Here is documentation for Ubuntu, but it doesn't directly apply to CentOS. Forge.mil's info on CAC cards, including CAC enabling Apache, also has some useful info.

Red Hat Enterprise Linux (RHEL)/CentOS instructions are available, and are close, but as of 2011-01-19 they are a little old. To use them, you need to make many changes (change "/etc/ssl/certs" to "/etc/pki/tls/certs" (its current name), note that private keys are stored in "/etc/pki/tls/private/" instead, beware that the "greater than" symbol is corrupted in the directions, and add support for the new dodeca2.cac certificate).

An updated sequence (based on that page) for loading the DoD root Certificate Authority (CA) certificates is:

TDIR=~/CAC
rm -fr "$TDIR" ; mkdir "$TDIR" ; cd "$TDIR"
# Get certificates files, one at a time, from: https://crl.gds.disa.mil/
# wget http://dodpki.c3pki.chamb.disa.mil/rel3_dodroot_1024.p7b
# wget http://dodpki.c3pki.chamb.disa.mil/rel3_dodroot_2048.p7b
# wget http://dodpki.c3pki.chamb.disa.mil/dodeca.p7b
# wget http://dodpki.c3pki.chamb.disa.mil/dodeca2.p7b
# Handle p7b format
for f in *.p7b ; do # Convert .p7b to PEM:
  openssl pkcs7 -inform DER -outform PEM -in $f -out ${f%.p7b}.pem -print_certs
done
# Handle .cer format
for f in *.cer ; do
 openssl x509 -inform DER -outform PEM -in "$f" -out "${f%.cer}.pem"
done
rm -f dod-root-certs.pem
# Old way:
# cat rel3_dodroot_1024.pem rel3_dodroot_2048.pem \
#    dodeca.pem dodeca2.pem > dod-root-certs.pem
# /bin/cp -p -f *.pem /etc/pki/tls/certs/
cat *.pem > dod-roots
mv dod-roots /etc/pki/tls/certs/dod-root-certs.pem

The "wget" commands above always work, but because they use unsecured http:, they risk a man-in-the-middle replacing them. Thus, you need to verify these files (or get them from another more trustworthy source). One way is to compare these certificates from a source you can trust. Another is to use the DoD PKI Management site. If you have a CAC card you can go to the DoD PKI Certificate Manager, select Retrieval, and then use "Import CA Certificate Chain" to get them. You can even use multiple methods to verify them.

Getting the CA Revocation List

You should also get the CA revocation list (CRL); this is a list of certificates that a CA has signed but are no longer valid. You need to update this (which will repeat some steps). For the DoD, the DoD PKI Management site has a CRL you can get. Let's say you get "ALLCRLZIP.zip" (which zips up a bunch of .crl files) and put it in the same $TDIR directory (you can get the ALLCRLZIP.zip file directly).

Again, you need to convert them to PEM, put them into a bundle, and put the bundle in a useful place:

unzip ALLCRLZIP.zip
rm -f CRL-bundle.crl # Remove any pre-existing one.
for f in *.crl ; do  # Convert to PEM format.
  openssl crl -inform DER -outform PEM -in "$f" -out "${f%.crl}.pem_crl"
done
cat *.pem_crl > CRL-bundle.crl
mv CRL-bundle.crl /etc/pki/tls/certs/

Note: This will produce a big file as the bundle! Also, note that the option in openssl below is "crl"; if you incorrectly give "pkcs7" you'll get weird errors.

Modify the Apache configuration (under /etc/httpd/conf/) so that it will check against this file, by including this line in the relavant SSL-specific section:

 SSLCARevocationFile /etc/pki/tls/certs/CRL-bundle.crl

Unfortunately, because this is such a large CRL bundle, this can slow starting the web server (it can take 5-10 seconds for it to load this file during web server startup).

Setting up the server certificate

Eperezdesigns' CAC info also shows how to create a certificate request, so that you could get your server certificate signed by the DoD certificate authority. If you're trying to get your server certificate signed by the DoD, it basically needs to be a DoD machine (.mil/.smil); there is a procedure that the DoD uses to ensure that you're a legitimate requestor. If you're not configuring such a machine, then don't bother, DoD will not sign server certificates of other machines (ones outside .mil or .smil). You can still use the setup above to confirm DoD CAC cards, though. To get your server certificate signed, simply use a standard commercial server certificate to prove to users that you are who you say you are (that works!).

To set up your server certificate, make one as described in the Apache documents and set up your configuration like this (again, in the SSL-specific section for the entire host):

    # Server Certificate:
    SSLCertificateFile /etc/pki/tls/certs/MYCERT.crt
    #   Server Private Key - do NOT let anyone else see this:
    SSLCertificateKeyFile /etc/pki/tls/private/MYCERT.key
    #   Server Certificate Chain (file that contenates Certificate Authority
    #   certificates), e.g. for Verisign:
    SSLCertificateChainFile /etc/pki/tls/certs/verisign-intermediate.crt

If you're going to accept client-side certificates such as CAC cards, you should also configure Apache to log each client username (technically SSL_CLIENT_S_DN_CN) with each request. You can do this by modifying /etc/httpd/conf/ssl-common.conf (or some other configuration file) to record them, like this:

CustomLog logs/ssl_request_log \
 "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%{SSL_CLIENT_S_DN_CN}x\" \"%r\" %b"

Configure Apache so either client-side certificate or username/password works

Now configure Apache to authenticate with client-side certificates (such as CAC cards). The Apache SSL Howto has some nice examples.

Generally, you modify the Apache configuration file to identify areas that are to be private, like this:

   <Directory "/var/www/private">
        AllowOverride None
        Options None
        AuthName "Restricted Access"
        ... [commands that give the rules for access]
    </Directory>

Now, where should we put these commands? If you're using a virtual host as described above, you should probably put them in "conf/MY_HOSTNAME.conf" or equivalent. That way, the rules will apply regardless of whether they try to access it using SSL or not.

You now need to give the specific authentication and authorization rules, where the "..." are above. If you only need them once, just put the rules at the "..." place. However, if you need the same rules many times, create yet another file to be included. Then, for each "...", replace that with "Include conf/SEPARATE_FILENAME" That will eliminate repetition.

Here are rules that allow either username/password or specific CAC cards (if the users are listed):

# Require either (1) Client certificates (e..g, CAC card) for *specific*
# CAC card users, OR (2) Username/password.
# NOTE: If a card user forgets to insert their card, they will
# be presented with a username/password query instead.
# If this happens, the user just needs to stick in the card and try again.
# For more info, see http://httpd.apache.org/docs/2.2/ssl/ssl_howto.html

# SSL is required, in part because it protects username/password,
# but also because anything with these restrictions
# probably shouldn't be visible to the Internet while in transit.
SSLRequireSSL

SSLVerifyClient      optional
SSLVerifyDepth       5
SSLCACertificateFile /etc/pki/tls/certs/dod-root-certs.pem

SSLOptions           +FakeBasicAuth
SSLRequire           %{SSL_CIPHER_USEKEYSIZE} >= 128

# Couple with HTTP Basic Authentication via FakeBasicAuth
# so that only certain users are authorized:
AuthType             Basic
AuthBasicProvider    file
AuthUserFile         /etc/httpd/userlist.txt
Require              valid-user

Adding a user

You can add a regular username/password using the usual approach:

 htpasswd -b /etc/httpd/userlist.txt USERNAME PASSWORD
Be sure to replace USERNAME and PASSWORD as appropriate (don't use "password" or other easy passwords as the PASSWORD).

To add a CAC user, you can ask them to run a program to get their distinguished name (DN) aka CAC username as described in the document above. If you don't want to do that, an alternative approach is to have them first log into the system (this will fail, since they don't have an account). Then look at the logged attempt, e.g.,

 tail -40 /var/log/httpd/ssl_request_log

Look for their distinguished name (DN) - which is basically their CAC username - and check that the username, time, and IP address are correct. Check to make sure no one is trying to spoof that username.

Once you know the DN, you can Now add them to the list of users using:

 htpasswd -b /etc/httpd/userlist.txt DN password
Replace "DN" with the user's DN surrounded by double-quotes (you must enclose the DN with double-quotes because it embeds a space). The DN must begin with "/"; do not add a client certificate username if it does not begin with "/". Also, do NOT replace "password" with some special password; it must be the literal word "password". Thus, CAC card authentications will look like this:
 htpasswd -b /etc/httpd/userlist.txt "/C=US/O=U.S. Government/OU=DoD/OU=PKI/OU=CONTRACTOR/CN=LASTNAME.FIRSTNAME.MIDDLENAME.NUMBER" password

Doesn't this enable spoofing?

You might think that this is insecure from a spoofing attack. After all, CAC card users all have the password set to the literal "password", we also allow username/password entry, and there's only one database of usernames with their passwords. At first it appears that someone could reuse a known client certificate name and the password "password", and get in. Client certificate names (as reported in %{SSL_CLIENT_S_DN_CN}) aren't secret, after all.

However, Apache username/password authentication implements a special check that makes this okay for security. When FakeBasicAuth is used, Apache will automatically reject the password "password" for a user-entered username/password entry if the certificate username (SSL_CLIENT_S_DN_CN) begins with "/" (as they do on CAC cards). This special check makes it possible to securely have both username/password and CAC card authentication, even when using a single simple username/password database... so while it feels a little hackish, there is a certain simplicity to it too. Jared Jennings showed in a MIL-OSS posting (on 1/18/2011 6:47PM) that this check is implemented in httpd-2.2.3-45.el5.src.rpm, httpd-2.2.3/modules/ssl/ssl_engine_kernel.c, around line 877:

   /*
    * Make sure the user is not able to fake the client certificate
    * based authentication by just entering an X.509 Subject DN
    * ("/XX=YYY/XX=YYY/..") as the username and "password" as the
    * password.
    */
   [...]
           if ((username[0] == '/') && strEQ(password, "password")) {
               ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
                   "Encountered FakeBasicAuth spoof: %s", username);
               return HTTP_FORBIDDEN;
           }

The Apache documents don't mention this at this time. Nevertheless, this is why it's okay to mix CAC certificates and normal usernames into a single user database and use "password" as the CAC user passwords; this password cannot be used for the same usernames in the typical username/password log in.

Fixing clients

If CAC cards do not work on some client machine, but work elsewhere, then clearly that client machine has a misconfiguration. MilitaryCAC has lots of hints.

However, here are two steps that are likely to help (at least on Windows):

  1. Re-install the client CAC card driver software (e.g., ActivClient Agent on Windows). I think ActivClient is terrible software; it's always getting patches (this kind of software should never need to change).
  2. Reconfigure Internet Explorer (this will also fix Chrome). This is especially likely to help if Firefox works but the other two do not. See Making AKO work with Internet Explorer for how to do so. The key is on slide 14, which removes some intermediate certificate authorities (CAs). If the client software has intermediate CAs installed (such as "Common Policy" issued by "Common Policy"), these intermediate CAs are added to the certificate chain sent to the server, potentially confusing the server. By removing certain intermediate CAs from the client, the confusion disappears.
  3. Firefox uses a slightly different CAC/ECA configuration approach from Internet Explorer and Chrome, which means it is quite possible for Firefox to work (while Chrome does not), or vice-versa. The key is to configure Firefox to use the correct library (DLL in Windows) to access the CAC card or ECA certificate. This library may change (and need to be reconfigured) if you update the CAC driver software. The Firefox information page on MilitaryCAC.com and the Firefox Tech Note give step-by-step instructions for doing this. For CAC card access, these involve setting the configuration under tools/options/advanced tab/security devices.

If you want to see if extraneous intermediate CAs on the client are causing trouble, here are some symptoms:

  1. On the client end, check the certificate path. For Internet Explorer, once you see the "Windows Security" window saying "Select a Certificate", click on "Click here to view certificate properties" and select the "Certification Path" tab. If you see a short, simple 3-level path like "DoD Root CA 2" to "DOD CA-25" to name-of-person, you're fine. If you see a long path like "Common Policy" (representing the whole US), then "SHA-1 Federal Root CA" (representing the US government), then "DoD Interoperability Root CA 1" (representing the connection between the federal government and DoD), and then the rest as before (e.g., "DoD Root CA 2", "DOD CA-25", and name-of-person), then it may not work properly.
  2. On the server end, the log files may note that re-negotiation was attempted, and that in the end, "re-negotiation handshake failed: Not accepted by client!?".

WARNING: Many kinds of updates can mess this up. Microsoft will sometimes update its certificate authority (CA) list as a patch; if this update is installed, these problematic intermediate CAs get re-installed. Updating browsers can screw up their connection with the underlyind drivers, too. Thus, you may have to re-do the steps above after some software updates.

Logging usernames: Apache bug

Apache, at least version 2.2.3, has a minor bug in how it writes logs. If the username has a space (which it does for all SSL certificates signed by the "U.S. Government" including CAC cards), then that space isn't escaped in the log. The result is weirdness in the logs, because the log fields are space-separated. I've reported this Apache bug (failure to escape spaces in log files) as Apache bug 51543. If you are hit by this bug, you can work around this by removing the space in "U.S. Government" - a space cannot occur in recorded URLs, because those are escaped (as %20).

The DISA page on External and Federal PKI Interoperability shows how various US federal government PKIs work together.

This HTML page is released to the public domain.