Apache Configuration System

OpenSSL Bug Info

← Go Back

Problems with Apache 2.0 and 2.2 mod_ssl on Certain Systems

Certain older versions of OpenSSL contain a bug that causes Apache to fail while starting on systems that have a large number of Virtual Hosts, at least one of which is used for SSL access. The problem stems from a bug in the OpenSSL version provided in CentOS/RedHat Enterprise Linux 4 and older. Some systems experience the problem with as few as 800 Virtual Hosts, while others do fine until ~1100 Virtual Hosts. RedHat has released an updated OpenSSL package for RedHat Enterprise Linux 5 that fixes this problem, however they have not yet released one for RedHat Enterprise Linux 4 or RedHat Enterprise Linux 3. You should contact Redhat to request an estimate as to when they will be released.

https://bugzilla.redhat.com/show_bug.cgi?id=236164

Apache 1.3 is not affected by this issue.

Detailed Analysis

The core problem is that versions of OpenSSL before 0.9.7k and 0.9.8c have a fault that appears when certain functions are used after FD_SETSIZE (generally 1024) filedecriptors have been opened:

http://marc.info/?l=openssl-dev&m=114301777619250&w=2

With Apache 1.3, initialization of mod_ssl occurs before httpd.conf is processed. With Apache 2.0 and 2.2, most initialization of mod_ssl occurs after httpd.conf has been processed, meaning that all the VirtualHost logfiles will be open when ssl_init_Module() is executed. The functions in OpenSSL affected by this issue at present are:

RAND_status(), called by ssl_init_Module() -> ssl_rand_seed()

RSA_generate_key(), called by ssl_init_Module() -> ssl_tmp_keys_init() -> ssl_tmp_key_init_rsa()

It does not appear that there are any faults once initialization of mod_ssl is completed on Apache 2.0 and 2.2. Starting Apache with a trimmed down httpd.conf file, copying the complete httpd.conf file over it and then restarting Apache does not cause a segfault because ssl_init_Module() is not executed again.

How Can This Bug Be Fixed?

Unfortunately, the fix provided by Red Hat is only in CentOS/RHEL 5 and Fedora at this time. Older versions are left with the following options:

  1. Update Your Operating System If Possible - Granted this is no trivial task but then you'll have the latest openssl without this openssl bug.
  2. Use Only Apache 1.3 - Tests with Apache 1.3 show it is not susceptible to this bug, due to the way it performs some module initialization before processing httpd.conf, while Apache 2.x does the module initialization after httpd.conf processing.
  3. Install A Custom Version of OpenSSL In /opt/openssl - Support for custom installs of OpenSSL were improved in EA3 to work around this bug. If /opt/openssl exists, EA3 will use it for all OpenSSL linking, whether for Apache, PHP, or other applications built by EA3. When executing the OpenSSL configure script, the following flags must be provided: --prefix=/opt/openssl shared
  4. Patch Apache 2.x - At this time, cPanel patches Apache 2.0 and 2.2 mod_ssl to avoid the call to RAND_status() for older versions of OpenSSL. On some systems, this is sufficient to avoid any problems. Reordering the call to RSA_generate_key() is more involved since that function must not run before other initialization functions have executed. It is possible that other function calls during mod_ssl's initialization also cause segmentation faults under some circumstances.

How Do You Know Whether An Install of Apache 2.x Suffers From This Problem?

A quick way of checking for this specific problem is by capturing the of httpd startup via the strace program. The OpenSSL bug manifests at the end of the output as:

  1. Reading from /dev/urandom
  2. Closing /dev/urandom
  3. Get the current time
  4. Seg Fault

A more positive identification can be done by disabling all SSL Vhosts in httpd.conf and restarting Apache. If Apache restarts without error, then the problem lies with mod_ssl/OpenSSL. To pinpoint the exact function call causing the segmentation fault, Apache must be recompiled with debugging symbols so that a usable backtrace can be created with GDB.

What Problems May Arise From Having A Different OpenSSL In /opt/openssl?

Although cPanel has not received any reports of problems resulting from the use of a custom OpenSSL library installed into /opt, it does create binaries that are linked against both the system's version of OpenSSL and the custom version. It is often problematic for a program to link to two versions of the same library since they may interfere with each other or cause the program to communicate using the wrong ABI.

For example: assume PHP is linked to mysql which is already linked to the system openssl, will compiling PHP and Apache against the other path cause conflicts? Testing should be done to find out.