“oom-killer”之後“Mem-Info”轉儲的特定列標題是什麼

我正在執行 Debian 8 Jessie 開發伺服器。16GB 記憶體。

我看過其他關於 oom-killer 的文章，但沒有一個與我在這裡問的內容真正相關

我正在嘗試追踪導致oom-killer事件的過程。有一個記憶體轉儲，但沒有列標題 - 所以我不知道這些數字是什麼意思。我沒有從Google搜尋中找到此資訊。

有幾行看起來像潛在的問題，但我不知道，直到我可以解釋資訊。

我特別好奇是什麼-999 docker-containe意思….

傾倒

Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.996382] sshd invoked oom-killer: gfp_mask=0x2000d0, order=2, oom_score_adj=-1000
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.996458] sshd cpuset=/ mems_allowed=0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.996589] CPU: 5 PID: 943 Comm: sshd Tainted: G         C    3.16.0-6-amd64 #1 Debian 3.16.56-1+deb8u1
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.996661] Hardware name: System manufacturer System Product Name/P8H67-M PRO, BIOS 3904 04/27/2013
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997200]  0000000000000000 ffffffff81533b49 00000000002000d0 0000000000000000
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997417]  ffffffff815326ab ffffffff81537f8e 0000000000000200 ffffffff8106b413
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997634]  ffffffff810c91a4 ffffffff8115c7f3 ffff8800083ea210 ffffffff8172faca
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997849] Call Trace:
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997905]  [<ffffffff81533b49>] ? dump_stack+0x5d/0x78
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.997962]  [<ffffffff815326ab>] ? dump_header+0x95/0x1fd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998019]  [<ffffffff81537f8e>] ? mutex_lock+0xe/0x30
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998078]  [<ffffffff8106b413>] ? put_online_cpus+0x23/0x90
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998135]  [<ffffffff810c91a4>] ? rcu_oom_notify+0xc4/0xe0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998195]  [<ffffffff8115c7f3>] ? do_try_to_free_pages+0x483/0x520
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998254]  [<ffffffff8114a5fd>] ? oom_kill_process+0x21d/0x370
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998312]  [<ffffffff8114a19d>] ? find_lock_task_mm+0x3d/0xa0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998369]  [<ffffffff8114adbe>] ? out_of_memory+0x4be/0x4f0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998427]  [<ffffffff81151776>] ? __alloc_pages_nodemask+0xca6/0xcf0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998486]  [<ffffffff8106875a>] ? copy_process.part.25+0x11a/0x1ae0
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998544]  [<ffffffff8106a2d3>] ? do_fork+0xc3/0x400
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998601]  [<ffffffff8105a4cb>] ? __do_page_fault+0x1ab/0x470
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998659]  [<ffffffff81539ef9>] ? stub_clone+0x69/0x90
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998715]  [<ffffffff81539b5c>] ? system_call_fast_compare_end+0x1c/0x21
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016369.998783] Mem-Info:
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079179] [20093]  1006 20093   306152     5057     107     1873             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079247] [20120]     0 20120    13507       38      32       73             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079315] [20123]  1006 20123     1085        7       8       15             0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079383] [20124]  1006 20124   306153     4858     110     2030             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079452] [20557]     0 20557    13507       91      32       20             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079520] [20560]  1006 20560     1085       22       8        0             0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079588] [20562]  1006 20562   306109     4393     110     2423             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079656] [21447]     0 21447    13507       93      32       18             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079725] [21450]  1006 21450     1085       23       8        0             0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079793] [21452]  1006 21452   305668     6278     108      261             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079861] [21489]     0 21489    13507       93      32       18             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.079929] [21492]  1006 21492     1085       22       8        0             0 sh

                                                                                        (snip ... a bazillion node, sh, cron)

Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176046] [23607]     0 23607    23318      237      49        0             0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176114] [ 2963]  1010  2963     8982      203      23        0             0 systemd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176183] [ 3198]  1010  3198    49655      573      31      154             0 (sd-pam)
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176251] [ 3843]  1010  3843    23318      240      46        0             0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176319] [ 4188]  1010  4188     6510      867      18        0             0 bash
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176387] [ 6721]     0  6721    13507       97      32       14             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176455] [28824]     0 28824    23319      237      50        0             0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176523] [ 9005]   110  9005     9560      142      21        0             0 pickup
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176591] [ 9898]  1006  9898     1085       22       8        0             0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176659] [ 9900]  1006  9900   300400     6495      94        0             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176727] [21244]  1010 21244    23319      242      47        0             0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176795] [22039]  1010 22039     3177       42      12        0             0 sftp-server
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.176864] [31202]     0 31202    23318      238      48        0             0 sshd

                                                                                        (snip ... a bazillion node, sh, cron)

Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.182092] [13192]     0 13192     1849      209       8        0          -999 docker-containe
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.182163] [13208]     0 13208   352255    84126     319        0             0 zcashd

                                                                                        (snip ... a bazillion node, sh, cron)

Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184048] [13334]     0 13334     1085       30       8        0             0 sessionclean
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184118] [13335]  1006 13335    11718       53      19        0             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184185] [13336]     0 13336     9884       49      18        0             0 php
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184255] [13342]     0 13342     1085       30       8        0             0 sessionclean
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184324] [13344]     0 13344     6447       29      11        0             0 sort
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184392] [13345]     0 13345     6447       29      11        0             0 sort
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184461] [13346]     0 13346     1085       28       8        0             0 sessionclean
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184530] [13347]     0 13347    12216      139      28        0             0 sshd
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184599] [13362]     0 13362    11068       74      19        0             0 docker
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184668] [13363]     0 13363     1085       28       8        0             0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184736] [13364]  1006 13364     1085       28       8        0             0 sh
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184806] [13365]     0 13365     8808       55      23       25             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184875] [13366]     0 13366     8808       55      23       25             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.184945] [13372]   111 13372     5874       67      16       76             0 nrpe
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185015] [13373]     0 13373     9884       49      18        0             0 php7.2
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185083] [13374]  1006 13374    11728      197      15        0             0 node
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185153] [13375]     0 13375     9884       48      20        0             0 php
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185221] [13378]     0 13378     7185       33      18       39             0 cron
Oct 26 10:41:09 Debian-89-jessie-64-minimal kernel: [6016370.185289] [13380]     0 13380     7185       33      18       39             0 cron

使用源碼！在 Debian 原始碼中搜尋片語“invoked oom-killer”會找到 mm/oom_kill.c，它具有應該列印的 9 列標題。

* Dumps the current memory state of all eligible tasks.  Tasks not in the same
* memcg, not in the same cpuset, or bound to a disjoint set of mempolicy nodes
* are not shown.
* State information includes task's pid, uid, tgid, vm size, rss, nr_ptes,
* swapents, oom_score_adj value, and name.
*/
static void dump_tasks(const struct mem_cgroup *memcg, const nodemask_t *nodemask)
{
   struct task_struct *p;
   struct task_struct *task;

   pr_info("[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name\n");

這已在 UNIX SE 上討論過，請參閱問題Debug out-of-memory with /var/log/messages

-999 對您來說很突出，因為該程序（按名稱 docker 命名）是這組行中唯一一個其 oom 分數調整得很低的程序。（這對我來說似乎是一個貪婪的初始化腳本。）但它的常駐記憶體（rss）很小，所以它可能不是一個巨大的記憶體消耗者。

還要尋找一條Killed process線，它有多大。OOM 會殺死將釋放最多記憶體資源的任務。

監控系統範圍的記憶體消耗，並確定主機是否有足夠的記憶體。/proc/meminfo基礎是隨著時間的推移擷取併計算任務的數量。監控工具將有助於收集數據。如果您沒有這樣的工具，請考慮從netdata的可用記憶體、每個使用者的記憶體和程序計數圖開始。

你有相當多的東西正在執行，cron 和 php 以及 node 和 docker… 驗證任何應該執行的計劃任務和應用程序都在正常執行。

引用自：https://serverfault.com/questions/937401