Hard-Drive

OfflineUncorrectableSector 和 CurrentPendingSector 電子郵件 - 我該怎麼辦?

  • December 13, 2018

執行 Centos 7:

我剛剛在我的一個伺服器上執行yum update了 a reboot,當它恢復時我收到了兩封電子郵件:

第一的:

OfflineUncorrectableSector
Device: /dev/sdb [SAT], 28 Offline uncorrectable sectors

第二:

CurrentPendingSector
Device: /dev/sdb [SAT], 24 Currently unreadable (pending) sectors

這是smartctl -x /dev/sdb

smartctl 6.5 2016-05-07 r4318 [x86_64-linux-3.10.0-957.1.3.el7.x86_64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     HGST Ultrastar 7K6000
Device Model:     HGST HUS726060ALE610
Serial Number:    K1KLXSNN
LU WWN Device Id: 5 000cca 255f2e12a
Firmware Version: APGNT907
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Thu Dec 13 14:58:00 2018 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM level is:     254 (maximum performance)
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                       was completed without error.
                                       Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                       without error or no self-test has ever
                                       been run.
Total time to complete Offline
data collection:                (  113) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                       Auto Offline data collection on/off support.
                                       Suspend Offline collection upon new
                                       command.
                                       Offline surface scan supported.
                                       Self-test supported.
                                       No Conveyance Self-test supported.
                                       Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                       power-saving mode.
                                       Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                       General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 898) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                       SCT Error Recovery Control supported.
                                       SCT Feature Control supported.
                                       SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
 1 Raw_Read_Error_Rate     PO-R--   100   100   016    -    0
 2 Throughput_Performance  P-S---   062   062   054    -    2796
 3 Spin_Up_Time            POS---   100   100   024    -    0
 4 Start_Stop_Count        -O--C-   100   100   000    -    3
 5 Reallocated_Sector_Ct   PO--CK   100   100   005    -    25
 7 Seek_Error_Rate         PO-R--   100   100   067    -    0
 8 Seek_Time_Performance   P-S---   128   128   020    -    18
 9 Power_On_Hours          -O--C-   100   100   000    -    5790
10 Spin_Retry_Count        PO--C-   100   100   060    -    0
12 Power_Cycle_Count       -O--CK   100   100   000    -    3
192 Power-Off_Retract_Count -O--CK   099   099   000    -    1257
193 Load_Cycle_Count        -O--C-   099   099   000    -    1257
194 Temperature_Celsius     -O----   120   120   000    -    50 (Min/Max 25/61)
196 Reallocated_Event_Count -O--CK   100   100   000    -    25
197 Current_Pending_Sector  -O---K   100   100   000    -    24
198 Offline_Uncorrectable   ---R--   100   100   000    -    28
199 UDMA_CRC_Error_Count    -O-R--   200   200   000    -    0
                           ||||||_ K auto-keep
                           |||||__ C event count
                           ||||___ R error rate
                           |||____ S speed/performance
                           ||_____ O updated online
                           |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      1  Comprehensive SMART error log
0x03       GPL     R/O      1  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  SATA NCQ Queued Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x12       GPL     R/O      1  SATA NCQ NON-DATA log
0x15       GPL,SL  R/W      1  SATA Rebuild Assist log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    256  Current Device Internal Status Data log
0x25       GPL     R/O    256  Saved Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
Device Error Count: 66 (device log contains only the most recent 4 errors)
       CR     = Command Register
       FEATR  = Features Register
       COUNT  = Count (was: Sector Count) Register
       LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
       LH     = LBA High (was: Cylinder High) Register    ]   LBA
       LM     = LBA Mid (was: Cylinder Low) Register      ] Register
       LL     = LBA Low (was: Sector Number) Register     ]
       DV     = Device (was: Device/Head) Register
       DC     = Device Control Register
       ER     = Error register
       ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 66 [1] occurred at disk power-on lifetime: 5009 hours (208 days + 17 hours)
 When the command that caused the error occurred, the device was active or idle.

 After command completion occurred, registers were:
 ER -- ST COUNT  LBA_48  LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 40 -- 53 00 38 00 00 14 e0 43 c8 40 00  Error: UNC 56 sectors at LBA = 0x14e043c8 = 350241736

 Commands leading to the command that caused the error were:
 CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
 -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
 25 00 00 00 38 00 00 14 e0 43 c8 e0 08  7d+20:36:03.771  READ DMA EXT
 25 00 00 00 70 00 00 14 e2 10 00 e0 08  7d+20:36:00.941  READ DMA EXT
 25 00 00 04 00 00 00 14 e2 0c 00 e0 08  7d+20:36:00.938  READ DMA EXT
 25 00 00 04 00 00 00 14 e2 08 00 e0 08  7d+20:36:00.936  READ DMA EXT
 25 00 00 04 00 00 00 14 e2 04 00 e0 08  7d+20:36:00.934  READ DMA EXT

Error 65 [0] occurred at disk power-on lifetime: 5009 hours (208 days + 17 hours)
 When the command that caused the error occurred, the device was active or idle.

 After command completion occurred, registers were:
 ER -- ST COUNT  LBA_48  LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 40 -- 53 00 38 00 00 14 e0 43 c8 40 00  Error: UNC 56 sectors at LBA = 0x14e043c8 = 350241736

 Commands leading to the command that caused the error were:
 CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
 -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
 25 00 00 04 00 00 00 14 e0 40 00 e0 08  7d+20:36:00.421  READ DMA EXT
 25 00 00 04 00 00 00 14 e0 3c 00 e0 08  7d+20:35:57.616  READ DMA EXT
 25 00 00 04 00 00 00 14 e0 38 00 e0 08  7d+20:35:57.609  READ DMA EXT
 25 00 00 04 00 00 00 14 e0 34 00 e0 08  7d+20:35:57.601  READ DMA EXT
 25 00 00 04 00 00 00 14 e0 30 00 e0 08  7d+20:35:57.593  READ DMA EXT

Error 64 [3] occurred at disk power-on lifetime: 5009 hours (208 days + 17 hours)
 When the command that caused the error occurred, the device was active or idle.

 After command completion occurred, registers were:
 ER -- ST COUNT  LBA_48  LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 40 -- 53 03 60 00 00 13 f9 4b 10 40 00  Error: UNC 864 sectors at LBA = 0x13f94b10 = 335104784

 Commands leading to the command that caused the error were:
 CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
 -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
 25 00 00 03 60 00 00 13 f9 4b 10 e0 08  7d+20:35:00.738  READ DMA EXT
 25 00 00 00 d0 00 00 13 fa f2 70 e0 08  7d+20:34:57.902  READ DMA EXT
 25 00 00 04 00 00 00 13 fa ee 70 e0 08  7d+20:34:57.900  READ DMA EXT
 25 00 00 04 00 00 00 13 fa ea 70 e0 08  7d+20:34:57.898  READ DMA EXT
 25 00 00 04 00 00 00 13 fa e6 70 e0 08  7d+20:34:57.896  READ DMA EXT

Error 63 [2] occurred at disk power-on lifetime: 5009 hours (208 days + 17 hours)
 When the command that caused the error occurred, the device was active or idle.

 After command completion occurred, registers were:
 ER -- ST COUNT  LBA_48  LH LM LL DV DC
 -- -- -- == -- == == == -- -- -- -- --
 40 -- 53 03 60 00 00 13 f9 4b 10 40 00  Error: UNC 864 sectors at LBA = 0x13f94b10 = 335104784

 Commands leading to the command that caused the error were:
 CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
 -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
 25 00 00 04 00 00 00 13 f9 4a 70 e0 08  7d+20:34:57.413  READ DMA EXT
 25 00 00 04 00 00 00 13 f9 46 70 e0 08  7d+20:34:54.625  READ DMA EXT
 25 00 00 04 00 00 00 13 f9 42 70 e0 08  7d+20:34:54.622  READ DMA EXT
 25 00 00 04 00 00 00 13 f9 3e 70 e0 08  7d+20:34:54.619  READ DMA EXT
 25 00 00 04 00 00 00 13 f9 3a 70 e0 08  7d+20:34:54.614  READ DMA EXT

SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
   1        0        0  Not_testing
   2        0        0  Not_testing
   3        0        0  Not_testing
   4        0        0  Not_testing
   5        0        0  Not_testing
Selective self-test flags (0x0):
 After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       256 (0x0100)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    50 Celsius
Power Cycle Min/Max Temperature:     44/61 Celsius
Lifetime    Min/Max Temperature:     25/61 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -40/70 Celsius
Temperature History Size (Index):    128 (100)

Index    Estimated Time   Temperature Celsius
101    2018-12-13 12:51    50  *******************************
...    ..(126 skipped).    ..  *******************************
100    2018-12-13 14:58    50  *******************************

SCT Error Recovery Control:
          Read: Disabled
         Write: Disabled

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 2) ==
0x01  0x008  4               3  ---  Lifetime Power-On Resets
0x01  0x018  6      6820807272  ---  Logical Sectors Written
0x01  0x020  6        20477840  ---  Number of Write Commands
0x01  0x028  6    444004265414  ---  Logical Sectors Read
0x01  0x030  6       448331816  ---  Number of Read Commands
0x01  0x038  6     20846500900  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4            5586  ---  Spindle Motor Power-on Hours
0x03  0x010  4            5586  ---  Head Flying Hours
0x03  0x018  4            1257  ---  Head Load Events
0x03  0x020  4              25  ---  Number of Reallocated Logical Sectors
0x03  0x028  4         1610568  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4              66  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4              18  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              50  ---  Current Temperature
0x05  0x010  1              49  N--  Average Short Term Temperature
0x05  0x018  1              50  N--  Average Long Term Temperature
0x05  0x020  1              61  ---  Highest Temperature
0x05  0x028  1              25  ---  Lowest Temperature
0x05  0x030  1              58  N--  Highest Average Short Term Temperature
0x05  0x038  1              25  N--  Lowest Average Short Term Temperature
0x05  0x040  1              53  N--  Highest Average Long Term Temperature
0x05  0x048  1              25  N--  Lowest Average Long Term Temperature
0x05  0x050  4              34  ---  Time in Over-Temperature
0x05  0x058  1              60  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               0  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4              71  ---  Number of Hardware Resets
0x06  0x010  4              28  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
                               |||_ C monitored condition met
                               ||__ D supports DSN
                               |___ N normalized value

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS
0x0008  2            0  Device-to-host non-data FIS retries
0x0009  2           70  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           71  Device-to-host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within host-to-device FIS
0x000d  2            0  Non-CRC errors within host-to-device FIS

我覺得它說的很奇怪PASSED,但我想這意味著驅動器仍然大部分都可以,但是Reallocated_Sector_Ct25 表明它可能很快就會失敗?

我以前從未遇到過任何此類錯誤(在執行幾個專用盒子的 10 年中),並且不想搞砸。我現在該怎麼辦?

這是一個 Hetzner 儲存盒,有 10 個 6GB 驅動器,我在 RAID 6 中設置了大約 48TB 的空間。

最好只是給他們發消息,他們會為我處理這個問題,還是我應該先做些什麼……

我已經有一個備份,每天晚上執行到Google云服務。我寧願不必使用它,因為我知道恢復起來會很慢。

我已經登錄到 Hetzner 的支持系統,對於驅動器故障,他們給出了這個建議(這是在送出票證之前,我還沒有完成):

Please give us the following information so that we can better understand your HDD/SSD defect:

 * Serial number(s) of the defective and/or intact HDD(s)/SSD(s)
 * Evidence of the defect (entire SMART log, less than one week old)

The following assistance is available:

 * [Instructions on establishing the serial numbers as well as information on defective HDDs/SSDs][1]
 * [Instructions on exchanging an HDD/SSD with software RAID][1]
 * [Instructions on creating a complete SMART log][1]

Please note that we can only exchange your defective drive for an empty drive. We do not carry out any data exchange or backups.

最後一行特別引起了我的注意。當他們將空驅動器放入時會發生什麼,我需要做什麼來重建它,還是會自動發生?

謝謝!

所以看起來你可能有一個失敗的磁碟。RAID6 為您提供了發生兩個磁碟故障的可能性,所以如果這個磁碟死了,您的數據應該仍然完好無損。使用 RAID 的全部意義在於,您通常可以通過使用寫入陣列中剩余健康磁碟的數據來重建一個故障(隨後被替換)磁碟的內容。

如果/當更換磁碟時,您將需要對其進行初始化並將其添加到現有陣列中以便可以使用它。這將取決於陣列的建構方式,如果它是軟體 RAID,那麼您可能需要創建正確類型的單個分區,然後將其添加到陣列mdadm中,此時陣列應自行修復/重建。查看陣列中現有磁碟之一的分區將顯示您需要複製的內容。

此頁面應向您展示您需要做的基礎知識:https ://www.thegeekdiary.com/replacing-a-failed-mirror-disk-in-a-software-raid-array-mdadm/

硬體 RAID 通常有它自己的供應商特定的方式來做同樣的事情,但我假設您使用的是軟體 RAID,因為您仍然可以看到各個硬碟。

引用自:https://serverfault.com/questions/944195