Mysql

Percona slave 因“意外的 PLT reloc 類型”而崩潰

  • September 3, 2014

我有 2 個數據中心,每個數據中心都有一個執行 Percona 伺服器 5.5.23 的主/從數據庫。由於某種原因,其中一個從數據庫因以下錯誤而崩潰:

03:00:40 UTC - mysqld got signal 11 ;  
This could be because you hit a bug. It is also possible that this binary  
or one of the libraries it was linked against is corrupt, improperly built,  
or misconfigured. This error can also be caused by malfunctioning hardware.  
We will try our best to scrape up some info that will hopefully help  
diagnose the problem, but since we have already crashed,   
something is definitely wrong and this may fail.  
key_buffer_size=16777216  
read_buffer_size=131072  
max_used_connections=1  
max_threads=151  
thread_count=0  
connection_count=0  
It is possible that mysqld could use up to   
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 346809 K  bytes of         memory  
Hope that's ok; if not, decrease some variables in the equation.  
Thread pointer: 0x0  
Attempting backtrace. You can use the following information to find out  
where mysqld died. If you see no messages after this, something went  
terribly wrong...  
stack_bottom = 0 thread_stack 0x80000  
/usr/local/Percona-Server-5.5.23-rel25.3-   240.Linux.x86_64/bin/mysqld(my_print_stacktrace+0x35)[0x7d4c85]  
/usr/local/Percona-Server-5.5.23-rel25.3-240.Linux.x86_64/bin/mysqld(handle_fatal_signal+0x3e1)[0x690cb1]  
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7fd2b94c7cb0]  
/usr/local/Percona-Server-5.5.23-rel25.3-240.Linux.x86_64/bin/mysqld[0x8f2df2]  
/usr/local/Percona-Server-5.5.23-rel25.3-240.Linux.x86_64/bin/mysqld[0x81d607]  
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fd2b94bfe9a]  
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fd2b86a93fd]  
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.  
140802 03:00:41 mysqld_safe Number of processes running now: 0  
140802 03:00:41 mysqld_safe mysqld restarted  
/usr/local/Percona-Server-5.5.23-rel25.3-240.Linux.x86_64/bin/mysqld: error while loading shared libraries: unexpected PLT reloc type 0x00  
140802 03:00:41 mysqld_safe mysqld from pid file /dbdata1/mysqld.pid ended  

每次嘗試重新啟動該從站都會導致上一個錯誤日誌的最後 3 行。它沒有說明它在載入哪個共享庫時遇到問題。我對“意外的 PLT reloc 類型”進行了一些搜尋,但沒有具體解釋它是什麼。我確實看到了一些關於損壞的二進製文件是如何導致它的,所以我在我的所有 4 個數據庫伺服器上的 mysqld 上執行了校驗和。原來我所有的工作數據庫都顯示相同的校驗和:

sha256sum mysqld
0b42e4625a87de52e5f51f2eb74fb7f2db63116e2b78f51d2897c1938a0e03d1  mysqld

正如我損壞的數據庫顯示的那樣:

sha256sum mysqld
7bfd58d1c1948a36cf4602c697dadd60e422d61ff75eeb4a0344f8ec395b03ea  mysqld

因此二進製文件似乎已損壞,儘管奇怪的是,所有二進製文件都具有相同的修改日期和相同的字節數。我不確定正在執行的伺服器的二進製文件可能會發生什麼損壞它。

我可以嘗試重新安裝 percona 以獲得工作二進製文件,但我想知道這裡發生了什麼,以便我可以防止它在未來再次發生。

多虧了這篇博文才弄清楚:https ://blogs.oracle.com/ksplice/entry/attack_of_the_cosmic_rays1

因此二進製文件也具有與其他工作伺服器相同的字節數和修改時間(2012 年),所以奇怪的是磁碟上的二進製文件沒有更改,但校驗和仍然不同。似乎二進製文件完全記憶體在 RAM 中,這就是損壞的地方。清除記憶體有效!

引用自:https://serverfault.com/questions/625825