Pure Digital Thought

I kept running into a problem where I would try to restart apache and in error_log I would get a message stating:

[notice] Digest: generating secret for digest authentication …

Apache would just appear to hang for minutes. Stopping and restarting would result in the same thing. I looked around on google and others had similar issues. I finally found a site that has apparently gone away but I was able to get the text intact thanks to Google’s cache (Apparently it’s still there but has a funky connection). So, thanks to the guys at raptorized.com. Below is the text as found in Google’s cache to preserve this for future admins who hit this issue. Personally, I used the rng-tools package and that worked for me so I just stuck with that.

==================================
Below is the unaltered text copied verbatim
==================================

I have a machine on my network that is very special. It’s a rather old Quad Xeon, an HP LH4 that I scavenged out of the proverbial dumpster of a buisness that didn’t want it anymore. In fact, they were about to trash six of them.

I decided it was a crime to shitcan such beautiful machines, so with the help of my friend Mike “I mangle french words” Le Blanc, we drove there, armed with a large truck and patience. Carrying them down the three stories with no elevator was an interesting experience. I scavenged lots and lots of interesting hardware there.

But i’m getting ahead of myself here. The point is, lots of screwy things start occuring when you have a Quad SMP machine, such a timers and clock drift. I started having a lot of problems with Apache recently… once in a blue moon, on restart, it would spawn a single process with no PID file, and hang there. Checking out error_log pointed out that apache would apparently hang while generating the Secret seed for Digest authentication (mod_digest). Disabling mod_digest would have worked, but sadly, I use it. This could be found in error_log:

[notice] Digest: generating secret for digest authentication …
[notice] Digest: done
[notice] Apache configured — resuming normal operations
[notice] caught SIGTERM, shutting down
[notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec2)
[notice] Digest: generating secret for digest authentication …

Fortunately, I cobbled up a fix. Updated! Read on for a more elegant fix.

So it hangs while generating the secret for digest authentication. My gut feeling was that it had to do with the system running out of entropy (random data, that is). Probably waits for apr_generate_random_bytes() to complete but that never happens. I immediatly checked out the kernel for available entropy from /dev/random and friends.

# cat /proc/sys/kernel/random/entropy_avail
16

Now, I have no clue why, but entropy ran out (not quite, but close). Probably another whacky quad SMP issue. I’m starting to drive the Gentoo devs and maintainers crazy with my strange hardware.

Note: Before proceeding, please read the updated section below.

So, to remedy it, I fetched and installed rngd from the rng-tools package in Gentoo Portage. Check out your distro’s package repository for it or something similar. This tool allegedy gathers random data from I/O transactions and hardware stuff around. I compiled it, installed it and launched it. The Gentoo way:

# emerge -va rng-tools
# rc-update add rngd default && /etc/init.d/rngd start

Once that was taken care of, I checked the available entropy by querying the kernel once again.

# cat /proc/sys/kernel/random/entropy_avail
6854

That’s much better. To my surpise, Apache finished loading instantly the split second rngd started up. Schweet. Sounds like that fixes the problem. I have no idea why it occurs right now, but this works around it elegantly.

UPDATE! I figured out what the greasy poop was going on. Installing rngd will gather entropy using /dev/urandom if no hardware RNG is found by default. I didn’t pay much attention to that detail at the time. However, you will notice that when installing the Apache Portable Runtime on Gentoo (dev-libs/apr) the use flag urandom is available. Using euse (from gentoolkit) to get info about it returns the following:

valkyrie ~ # euse -i urandom
global use flags (searching: urandom)
************************************************************
no matching entries found

local use flags (searching: urandom)
************************************************************
[- ] urandom (dev-libs/apr):
Use /dev/urandom instead of /dev/random

This will make apache fetch randomness from /dev/urandom directly, therefore giving the same result as rng-tools. The proper solution would be to just re-emerge dev-libs/apr with the urandom use flag enabled.

I tried it and it works beautifully, and looks a lot more elegant to me.