On some occasions apache server will just fail, and become crashed, failing to restart with an error message like
# tail /var/log/httpd/error_log [Sun Dec 22 10:05:27.006998 2013] [core:emerg] [pid 15001:tid 140145945569216] (28)No space left on device: AH00023: Couldn't create the fcgid-proctbl mutex [Sun Dec 22 10:05:38.000403 2013] [core:emerg] [pid 15018:tid 140368783902656] (28)No space left on device: AH00023: Couldn't create the proxy mutex [Sun Dec 22 10:05:38.000456 2013] [proxy:crit] [pid 15018:tid 140368783902656] (28)No space left on device: AH02478: failed to create proxy mutex
If you see an error similar to the bellow, it could indicate that your server has run out of semaphores. To see how many semaphores are being used, type the following command
# ipcs -s ------ Semaphore Arrays -------- key semid owner perms nsems 0x00000000 38797312 apache 600 1 0x00000000 38731777 apache 600 1 0x00000000 38830082 apache 600 1 0x00000000 38961155 apache 600 1 0x00000000 38895620 apache 600 1 0x00000000 38993925 apache 600 1 0x00000000 39026694 apache 600 1 0x00000000 39059463 apache 600 1 0x00000000 39092232 apache 600 1 0x00000000 39125001 apache 600 1 0x00000000 39157770 apache 600 1 0x00000000 39288843 apache 600 1 0x00000000 39223308 apache 600 1 0x00000000 39321613 apache 600 1 0x00000000 39354382 apache 600 1 0x00000000 39387151 apache 600 1 0x00000000 39419920 apache 600 1 0x00000000 39452689 apache 600 1 0x00000000 39485458 apache 600 1
To fix it and get Apache server started again, we must clean the semaphores. Run the following command to flush them:
for whatever in `ipcs -s | awk '{print $2}'`; do ipcrm -s $whatever; done
If that command may not work on older server then you may need to do the following
# /etc/init.d/httpd stop
# ipcs -s | grep nobody | gawk '{ print $2 }' | xargs -n 1 ipcrm sem
# /etc/init.d/httpd start
Final, you may want to increase the semaphore limits on your server. You can do that by adding the following to the /etc/sysctl.conf
kernel.msgmni = 512 kernel.sem = 250 128000 32 512
After, type the following command to update new setting for kernel
# sysctl -p