Apache 意外關閉
我必須說,我不是設置伺服器的人,而是負責找出暫時發生的事情的可憐的 sap。我對apache和linux只有粗略的了解,所以請多多包涵……
問題
我們的 apache 伺服器關閉(主要是在夜間)並且無法恢復並重新啟動,而沒有人真正告訴它這樣做。
據我所知,這裡有趣的是 apache 錯誤日誌中的以下行:
[Wed Apr 15 03:43:02.114879 2015] [mpm_prefork:notice] [pid 25778] AH00171: Graceful restart requested, doing restart
之後是一個很長的堆棧跟踪,這裡是前幾行:
*** Error in `/usr/sbin/httpd': free(): invalid pointer: 0x00007f581d5c13c0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x7d19d)[0x7f583b69519d] /etc/httpd/modules/libphp5.so(php_module_shutdown+0x2b)[0x7f58301d255b] /etc/httpd/modules/libphp5.so(php_module_shutdown_wrapper+0x9)[0x7f58301d2619] [...]
回溯會持續一段時間,但有趣的是,在這之間,它會
Error in /usr/sbin/httpd [...]
重複 5 次,直到最後一條消息[Wed Apr 15 03:43:02.269626 2015] [core:notice] [pid 25778] AH00060: seg fault or similar nasty error detected in the parent process
下一條消息是在我再次啟動伺服器的第二天:
[Wed Apr 15 08:14:46.200884 2015] [core:notice] [pid 30326] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0 [Wed Apr 15 08:14:46.215410 2015] [suexec:notice] [pid 30326] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Wed Apr 15 08:14:46.235346 2015] [auth_digest:notice] [pid 30326] AH01757: generating secret for digest authentication ... [Wed Apr 15 08:14:46.236045 2015] [lbmethod_heartbeat:notice] [pid 30326] AH02282: No slotmem from mod_heartmonitor [Wed Apr 15 08:14:46.280992 2015] [core:warn] [pid 30326] AH00098: pid file /run/httpd/httpd.pid overwritten -- Unclean shutdown of previous Apache run? [Wed Apr 15 08:14:46.284919 2015] [mpm_prefork:notice] [pid 30326] AH00163: Apache/2.4.6 (CentOS) PHP/5.4.16 configured -- resuming normal operations [Wed Apr 15 08:14:46.284939 2015] [core:notice] [pid 30326] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
想法
正如我所看到的,prefork 模組以某種方式請求 apache 程序(或其中之一?)關閉。這失敗了,作為回報,整個事情都可怕地崩潰了。
問題
首先也是最重要的:我的分析——導致伺服器崩潰的問題是prefork模組——正確嗎?
我應該禁用 prefork 模組嗎?我看不到如何調試/幫助軟體本身內部的記憶體錯誤。
當我禁用此模組時,它有什麼影響?
版本資訊
Linux version 3.10.0-123.13.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GCC) ) Server version: Apache/2.4.6 (CentOS) Server built: Jan 12 2015 13:22:31 Server's Module Magic Number: 20120211:23 Server loaded: APR 1.4.8, APR-UTIL 1.5.2 Compiled using: APR 1.4.8, APR-UTIL 1.5.2 Architecture: 64-bit Server MPM: prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=256 -D HTTPD_ROOT="/etc/httpd" -D SUEXEC_BIN="/usr/sbin/suexec" -D DEFAULT_PIDLOG="/run/httpd/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf"
對評論的回應
cron
沒有
/etc/cron.d/dailyjobs
,只有一個0hourly
腳本,它執行每小時的腳本0anacron
,0yum-hourly.cron
並且dellrda.cron
- 似乎都沒有做任何與 apache 相關的事情(恕我直言)/etc/logrotate.d/httpd
/var/log/httpd/*log { missingok notifempty sharedscripts delaycompress postrotate /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true endscript }
手動重新載入
/bin/systemctl reload httpd.service
帶來以下結果
Job for httpd.service failed. See 'systemctl status httpd.service' and 'journalctl -xn' for details.
並在與
error_log
上面相同的消息中顯示。快速
systemctl status httpd.service
揭示:httpd.service - The Apache HTTP Server Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled) Active: failed (Result: signal) since Fri 2015-04-17 12:26:36 CEST; 8s ago Process: 8828 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS) Process: 8826 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS) Process: 8767 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=killed, signal=ABRT) Main PID: 8767 (code=killed, signal=ABRT) Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
apachectl -M
Loaded Modules: core_module (static) so_module (static) http_module (static) access_compat_module (shared) actions_module (shared) alias_module (shared) allowmethods_module (shared) auth_basic_module (shared) auth_digest_module (shared) authn_anon_module (shared) authn_core_module (shared) authn_dbd_module (shared) authn_dbm_module (shared) authn_file_module (shared) authn_socache_module (shared) authz_core_module (shared) authz_dbd_module (shared) authz_dbm_module (shared) authz_groupfile_module (shared) authz_host_module (shared) authz_owner_module (shared) authz_user_module (shared) autoindex_module (shared) cache_module (shared) cache_disk_module (shared) data_module (shared) dbd_module (shared) deflate_module (shared) dir_module (shared) dumpio_module (shared) echo_module (shared) env_module (shared) expires_module (shared) ext_filter_module (shared) filter_module (shared) headers_module (shared) include_module (shared) info_module (shared) log_config_module (shared) logio_module (shared) mime_magic_module (shared) mime_module (shared) negotiation_module (shared) remoteip_module (shared) reqtimeout_module (shared) rewrite_module (shared) setenvif_module (shared) slotmem_plain_module (shared) slotmem_shm_module (shared) socache_dbm_module (shared) socache_memcache_module (shared) socache_shmcb_module (shared) status_module (shared) substitute_module (shared) suexec_module (shared) unique_id_module (shared) unixd_module (shared) userdir_module (shared) version_module (shared) vhost_alias_module (shared) dav_module (shared) dav_fs_module (shared) dav_lock_module (shared) lua_module (shared) mpm_prefork_module (shared) proxy_module (shared) lbmethod_bybusyness_module (shared) lbmethod_byrequests_module (shared) lbmethod_bytraffic_module (shared) lbmethod_heartbeat_module (shared) proxy_ajp_module (shared) proxy_balancer_module (shared) proxy_connect_module (shared) proxy_express_module (shared) proxy_fcgi_module (shared) proxy_fdpass_module (shared) proxy_ftp_module (shared) proxy_http_module (shared) proxy_scgi_module (shared) systemd_module (shared) cgi_module (shared) php5_module (shared)
我的猜測是您遇到了這個錯誤:http : //bugs.centos.org/view.php?id=7669,但我沒有在您載入的模組中看到 mod_gnutls。不過,對我來說,您似乎遇到了某種錯誤。如我所見,您沒有使用最新的 httpd 版本,因此您應該首先嘗試更新您的 httpd 包:
yum update
請稍後重新啟動您的 httpd,然後嘗試重新載入問題是否仍然存在。如果它仍然存在,我建議在 CentOS 上為此打開一個錯誤報告。為避免同時在半夜崩潰,我建議編輯您的 logrotate 腳本以使用
/bin/systemctl restart httpd.service
而不是它的
reload
對應物,直到問題得到解決。編輯:
在 CentOS 上打開錯誤報告之前,您應該確保您只使用標準的 CentOS apache 包和模組。如果您使用任何自編譯或從 3rd 方儲存庫安裝的 apache 模組,他們可能不會接受這一點。
要顯示所有已安裝的軟體包及其來自的 repo,您可以使用命令
rpm -qa --qf '%{NAME} %{VENDOR}\n'