News

The release of 11.25.0-RELEASE_42399 and 11.25.0-CURRENT_42399 introduced a configuration issue that may cause Apache configuration syntax errors on servers with no accounts when the mod_userdir tweak is enabled. The condition will present itself upon addition of the first account to the server. This issue is quickly addressed by rebuilding the Apache configuration file.

Symptoms:
The following error can be seen in the cpanel error_log and when attempting to restart Apache:
   
    UserDir "enable" keyword requires a list of usernames
   
Newly created websites would be sent to defaultwebpage.cgi instead of the appropriate document root.
   
Resolutions:
1.) New builds, 11.25.0-CURRENT_42400 and 11.25.0-RELEASE_42400, have been published to address the issue. This issue will not be present on subsequent new installations. Upgrading a currently affected system will resolve this issue.
   
2.) Rebuilding the Apache configuration after the addition of the first account will permanently resolve this issue.
   
    From the command line:
    /scripts/rebuildhttpdconf
    /scripts/restartsrv_httpd

        or

    /scripts/autorepair userdir_enable_fix
    /scripts/restartsrv_httpd

   
    From the WHM:
        Service Configuration -> Apache Configuration -> Global Configuration ->
        Save -> Rebuild Configuration and Restart Apache

Note:
The Apache configuration may be rebuilt as a result of other changes made in the WHM. Your new installation may not experience this issue as a result the the Apache configuration being rebuilt automatically.

The Quality Assurance team discovered a bug within the SpamAssassin ruleset that will mark messages sent in the year 2010 (that's today) and beyond with a higher spam score than expected.     This bug can result in legitimate mail being flagged as spam.

The cPanel Development team has issued a hot fix that will address this issue and will automatically update the SpamAssassin ruleset to resolve this issue.   If you have automatic cPanel updates enabled, no further action is required.

If you do not have automatic cPanel updates enabled, you can manually update  the SpamAssassin ruleset by executing the following commands in a root shell:

/scripts/autorepair spamd_y2010_fix

For a more detailed explanation and information on resolving this problem on a non-cPanel environment, please review:

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6269

As always, the entire cPanel team has pulled together to provide a rapid response to this issue as we realize proper email delivery is mission critical.

If you need any assistance, our 24x7 Technical Analyst team is standing by to provide you with further instructions, answers, and solutions to this bug.   To reach them please submit a ticket via:

https://tickets.cpanel.net/submit/index.cgi?step=2&reqtype=tickets&product=cpanel

You can also join the discussion on the SpamAssassin bug  at http://forums.cpanel.net/f43/spamassassin-fh_date_past_20xx-0-0-rule-bug-142725.html

In cPanel 11.25 we have made a few feature improvements to the bandwidth tracking system built into WHM and cPanel. While these new features grants you a much more accurate and granular view of client bandwidth usage, customers with inadequately sized /var partitions may run into issues related to partition size and disk space consumption.

It should also be noted that the creation of these files occurs during the first upcp that results in an upgrade to 11.25 and can require significantly more time than normal upgrades.

The notice for this change was in our release notes for 11.25 which can be found at http://docs.cpanel.net/twiki/pub/AllDocumentation/ReleaseNotes/1125releasenotes.pdf

Change: Bandwidth graph RRD files are split off for each individual service per domain and subdomain and stored in /var/cpanel/bandwidth/ .

Potential Issue: On some machines, depending on the partitioning schema and the number of domains on a server, there may be a disk space issue. This will only effect customers with large numbers of domains and a relatively small /var/ partition, i.e., 8-10G. In these situations, the /var/ partition may become full.

Workaround: For the time being, if you are affected by this issue, we are suggesting the following temporary fixes.
 

• Disabling Bandwidth Stats Generation: ‘touch /etc/rrdtooldisable ’ . This will disable generation of the rrdfiles altogether. Removing this file will result in the graphs being generated again.

or

• Symlinking /var/cpanel/bandwidth to a partition with more space.

mkdir /home/bwtmp; tar -czvf /root/cp-bandwidth-backup.tar.gz /var/cpanel/bandwidth

mv /var/cpanel/bandwidth/* /home/bwtmp/; rm -rf /var/cpanel/bandwidth

ln -s /home/bwtmp /var/cpanel/bandwidth

The above command will create a /home/bwtemp directory, copy the contents of /var/cpanel/bandwidth to it, create a backup in /root/cp-bandwidth-backup.tar.gz, and then symlink the old directory to the new one where there should be more diskspace.

If you have issues executing the fix for this change, please submit a ticket via https://tickets.cpanel.net and we will assist you with this fix.

Problem

Certain older versions of OpenSSL contain a bug that causes Apache to fail while starting on systems that have a large number of VirtualHosts, at least one of which is used for SSL access.

Red Hat® and CentOS 4.6+ and 5.x have been patched for this issue. If your system is running one of those distributions and is up-to-date, you should not encounter this bug.

If you are running a different distribution and encounter segmentation faults when starting up Apache 2.0 or 2.2, you may need to contact your Linux distribution vendor and inquire about this problem.

Apache 1.3 is not affected by this issue.

Detailed Analysis

The core problem is that versions of OpenSSL before 0.9.7k and 0.9.8c have a fault that appears when certain functions are used after FD_SETSIZE (generally 1024) file descriptors have been opened. For more information, see:

http://marc.info/?l=openssl-dev&m=114301777619250&w=2

With Apache 1.3, initialization of mod_ssl occurs before httpd.conf is processed. With Apache 2.0 and 2.2, most initialization of mod_ssl occurs after httpd.conf has been processed, meaning that all the VirtualHost logfiles will be open when ssl_init_Module() is executed. The functions in OpenSSL affected by this issue currently are:

RAND_status(), called by ssl_init_Module() -> ssl_rand_seed()

RSA_generate_key(), called by ssl_init_Module() -> ssl_tmp_keys_init() -> ssl_tmp_key_init_rsa()

It does not appear that there are any faults once initialization of mod_ssl is completed on Apache 2.0 and 2.2. Starting Apache with a trimmed down httpd.conf file, copying the complete httpd.conf file over it, and then restarting Apache does not cause a segfault because the ssl_init_Module() is not executed again.

How do you know whether an install of Apache 2.x suffers from this problem?

A quick way of checking for this specific problem is by capturing the output of the httpd startup via the strace program. The OpenSSL bug manifests at the end of the output as:

1. Reading from /dev/urandom.
2. Closing /dev/urandom.
3. Getting the current time.
4. Seg Fault.

A more positive identification can be done by disabling all SSL Vhosts in httpd.conf and restarting Apache. If Apache restarts without error, then the problem lies with mod_ssl/OpenSSL.

To pinpoint the exact function call causing the segmentation fault, Apache must be recompiled with debugging symbols so that a usable backtrace can be created with GDB. In order to recompile Apache with debugging symbols, you must simply set the appropriate CFLAGS and then run EasyApache, like so:

• export CFLAGS="-g"
    /scripts/easyapache --build

How Can This Bug Be Fixed?

 Unfortunately, the fix provided by Red Hat is only in CentOS/RHEL 5 and Fedora® at this time. Older versions are left with the following options:

1. Update Your Operating System, if Possible -- Granted, this is no trivial task. But it will give you the latest OpenSSL without this OpenSSL bug.
2. Use Only Apache 1.3 - Tests with Apache 1.3 show that it is not susceptible to this bug, due to the way it performs some module initialization before processing httpd.conf, while Apache 2.x does the module initialization after processing httpd.conf.
3. Patch Apache 2.x - At this time, cPanel patches Apache 2.0 and 2.2 mod_ssl to avoid the call to RAND_status() for older versions of OpenSSL. On some systems, this is sufficient to avoid any problems. Reordering the call to RSA_generate_key() is more involved, since that function must not run before other initialization functions have executed. It is possible that other function calls during mod_ssl’s initialization also cause segmentation faults under some circumstances.

After applying a patch, if you choose that option, please be sure to test Apache again to see whether it is still suffering from the issue.

A note about the /opt/openssl workaround

We previously recommended that you use /opt/openssl to work around this issue. However, that method does not ensure that you have the latest patched version of OpenSSL, as /opt/openssl is not updated by either EasyApache or the system package manager.

This can be a security concern, and requires you to manually track and update the OpenSSL installation in /opt/openssl. In addition, when using the /opt/openssl workaround, multiple copies of libssl can be loaded into memory. When a function is called by a program looking to interact with libssl, there may be two potentially different copies of that function loaded into memory.

There's no real way to determine which function is being called, and therefore, troubleshooting issues with a /opt/openssl implementation is difficult to manage.

For these reasons, we do not recommend using /opt/openssl to work around this bug.

Bug: Apache 2.2 Child Processes in “G” Status

Problem

Apache 2.2 has a bug affecting some systems, wherein a graceful restart does not fully clean up any running processes.

The result is that child processes are stuck in "G" status, and more file descriptors are used, which can push Apache over its limit and crash it.

Fix

Apache is aware of this bug and patches have been submitted; however, they are all problematic. For more information, see http://issues.apache.org/bugzilla/show_bug.cgi?id=42829.

Apache is attempting to deal with the problem upstream, but the fix is still not finalized. For more information, see http://svn.apache.org/viewvc/httpd/httpd/trunk/
server/mpm/prefork/prefork.c?view=log&pathrev=613260.

If you experience this issue on a server, your best and only supported option is to run Apache 2.0 instead of 2.2. This is because the patches are not ready for production, as they have known issues and need further refining by Apache.