Error messages for one of my hard drives in syslog "Unhandled error code" "I/O error"

Question

I am seeing some strange errors in syslog. I am running a debian 7.9. Has anybody seen these before and/or know what could be happening? Is my hard drive failing? I did an fsck the other day on boot and there were no messages saying so. It is an external USB encrypted hard drive (LUKS), which is decrypted/mounted at boot. There are no RAIDs or anything of the sort set up on the computer. I have two other drives set up similarly with no problems.

I looked at the smart data for the drive and it says that it passed the "overall health test". Is there anything else I can check/do to get an idea of its condition? Why would I be getting these errors?

Sep 24 02:49:03 centaur kernel: [ 1429.377929] sd 2:0:0:0: [sdb] Unhandled error code
Sep 24 02:49:03 centaur kernel: [ 1429.377933] sd 2:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Sep 24 02:49:03 centaur kernel: [ 1429.377939] sd 2:0:0:0: [sdb] CDB: Read(10): 28 00 38 80 16 08 00 00 08 00
Sep 24 02:49:03 centaur kernel: [ 1429.377950] end_request: I/O error, dev sdb, sector 947918344
Sep 24 11:41:19 centaur kernel: [32752.064706] sd 2:0:0:0: [sdb] Unhandled error code
Sep 24 11:41:19 centaur kernel: [32752.064710] sd 2:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Sep 24 11:41:19 centaur kernel: [32752.064715] sd 2:0:0:0: [sdb] CDB: Read(10): 28 00 38 40 f6 08 00 00 08 00
Sep 24 11:41:19 centaur kernel: [32752.064727] end_request: I/O error, dev sdb, sector 943781384
Sep 24 13:00:15 centaur kernel: [37487.599035] sd 2:0:0:0: [sdb] Unhandled error code
Sep 24 13:00:15 centaur kernel: [37487.599040] sd 2:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Sep 24 13:00:15 centaur kernel: [37487.599045] sd 2:0:0:0: [sdb] CDB: Read(10): 28 00 38 40 e1 68 00 00 08 00
Sep 24 13:00:15 centaur kernel: [37487.599056] end_request: I/O error, dev sdb, sector 943776104
Sep 24 13:55:44 centaur kernel: [40816.471962] sd 2:0:0:0: [sdb] Unhandled error code
Sep 24 13:55:44 centaur kernel: [40816.471967] sd 2:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Sep 24 13:55:44 centaur kernel: [40816.471972] sd 2:0:0:0: [sdb] CDB: Read(10): 28 00 38 40 6b e0 00 00 08 00
Sep 24 13:55:44 centaur kernel: [40816.471984] end_request: I/O error, dev sdb, sector 943746016
Sep 24 14:26:37 centaur kernel: [42670.078961] sd 2:0:0:0: [sdb] Unhandled error code
Sep 24 14:26:37 centaur kernel: [42670.078965] sd 2:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Sep 24 14:26:37 centaur kernel: [42670.078971] sd 2:0:0:0: [sdb] CDB: Read(10): 28 00 38 00 25 20 00 00 08 00
Sep 24 14:26:37 centaur kernel: [42670.078983] end_request: I/O error, dev sdb, sector 939533600
Sep 24 14:57:41 centaur kernel: [44533.796981] sd 2:0:0:0: [sdb] Unhandled error code
Sep 24 14:57:41 centaur kernel: [44533.796987] sd 2:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Sep 24 14:57:41 centaur kernel: [44533.796992] sd 2:0:0:0: [sdb] CDB: Read(10): 28 00 38 40 76 08 00 00 08 00
Sep 24 14:57:41 centaur kernel: [44533.797003] end_request: I/O error, dev sdb, sector 943748616

Info from smartctl:

smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.2.0-4-amd64] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda LP
Device Model:     ST32000542AS
Serial Number:    6XW135W4
LU WWN Device Id: 5 000c50 01f01e7ee
Firmware Version: CC34
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Fri Sep 25 00:18:47 2015 MDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:        (  623) seconds.
Offline data collection
capabilities:            (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:    (   1) minutes.
Extended self-test routine
recommended polling time:    ( 255) minutes.
Conveyance self-test routine
recommended polling time:    (   2) minutes.
SCT capabilities:          (0x103f) SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   118   099   006    Pre-fail  Always       -       178542268
  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   085   085   020    Old_age   Always       -       15977
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   065   060   030    Pre-fail  Always       -       81661499813
  9 Power_On_Hours          0x0032   045   045   000    Old_age   Always       -       48686
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       69
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       4295032833
189 High_Fly_Writes         0x003a   083   083   000    Old_age   Always       -       17
190 Airflow_Temperature_Cel 0x0022   052   040   045    Old_age   Always   In_the_past 48 (0 174 52 30)
194 Temperature_Celsius     0x0022   048   060   000    Old_age   Always       -       48 (0 14 0 0)
195 Hardware_ECC_Recovered  0x001a   042   020   000    Old_age   Always       -       178542268
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       208202834670158
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       1107416570
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       361340835

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I also looked at this site and one of the posters change his sysctl.conf like what is pasted below, but mine is already set like this...

/etc/sysctl.conf

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

score 1 · Answer 1 · edited Apr 13 '17 at 12:37

Back Up First

Before troubleshooting, make sure all the data you need on the affected hard drive (/dev/sdb) is backed up.

Trouble Backing Up?

If you are encountering I/O errors that are not the result of filesystem corruption (you indicated that your fsck results were clean) and your data is actually important to you, the safest way for you to recover what you can would be to use gddrescue (or dd_rescue, if you prefer).

Here's an example with gddrescue:

ddrescue -Svv /dev/sdb /somewhere/else/rescue.img /somewhere/else/ddrescue.log

… where /somewhere/else is the path to a mount that has more free space than the size of /dev/sdb.

gddrescue's log file at /somewhere/else/ddrescue.log can also help you identify what areas of your hard drive are unreadable.

Once a full image of the drive has been created, you can recover most of what should be recoverable using other tools like TestDisk or kpartx -av /somewhere/else/rescue.img and cryptsetup luksOpen, but these are out of scope of your question.

Answering Your Questions

Has anybody seen these before and/or know what could be happening?

I've seen a variety of hardware issues in hard drives and my relatively short time working with them. Though I'm not an expert, it's still difficult to pinpoint what kind of failures may be happening with your specific drive.

Is my hard drive failing?

Well, your drive is not aware that it's failing.

It could discover or sometimes cause more problems the more you try to read from it, and write operations can sometimes make conditions worse, depending on what kind of damage is present.

I did an fsck the other day on boot and there were no messages saying so.

Hardware read errors can affect the contents of files but not the filesystem itself.

Is there anything else I can check/do to get an idea of its condition?

Yeah, I tried to write a comprehensive guide for you below.

Why would I be getting these errors?

There are many causes of hard drive I/O problems.

A common one is dropping or moving the drive too forcefully while it's in use. This happens a lot with external hard drives like yours.

Another one is that you have a Seagate Barracuda, which tends to be unreliable:

^{_{(Click on the image above to see a review of hard drive reliability.)}}

It could also be just random issues that may affect just a small part of the drive or eventually grow into a write-off.

Troubleshooting

The commands in this section are non-destructive.

Read Individual Sectors

Your first indication of a problem was the kernel logging I/O errors. Try seeing if those sectors are readable:

time hdparm --read-sector 947918344 /dev/sdb

… where 947918344 is the sector number recorded in the kernel log and /dev/sdb is the suspected problematic drive.

Most hard drives will try to read the sector, so it may take some time. If you get an I/O error immediately, it means that the drive knew for sure that the sector is off-limits and won't even try to read it. If it takes some seconds to get an I/O error, the drive probably tried and failed to read the sector. Here is an example of an I/O error that happened right away:

root@node51 [~]# time hdparm --read-sector 947918344 /dev/sdb

/dev/sdb:
reading sector 947918344: FAILED: Input/output error

real    0m0.038s
user    0m0.000s
sys     0m0.002s

Here is an example of a successful read:

root@node51 [~]# time hdparm --read-sector 947918344 /dev/sdb

/dev/sdb:
reading sector 947918344: succeeded
c193 76f6 7ae7 2dfa f121 b9ef 04da 2f82
4f6f 90af 6593 f215 d098 8857 2cfb d191
c6b1 245a 746c c1b8 bde9 f8f2 b3d2 ea68
3740 935d ce98 7fc6 bfef 41a5 f6b5 94e8
0833 6d39 3137 9b38 717d d310 f33d d508
d72d b504 77e5 7c8b 4496 6ac1 20f9 f49e
1224 8def df01 1163 d855 ae2b c6a8 eee5
3a22 04ae 012e bdd2 b5e9 94a4 3454 16ff
6000 81bc 356c a412 4b11 a875 588a 8394
37e5 310b f9eb afb6 4be1 0307 74f2 fce4
0f82 da8b 954f 4181 a544 ebff bfb5 4f1a
bcf9 e0be a912 fd78 e68d 9f76 424a 563f
0cc8 70ab 187e b7b1 b45d 732d 734d 0d7b
bb79 1e0c 8406 3fd9 5608 a6d4 9cd7 4167
ddb2 1972 7283 57bb 175e 1724 b3e4 9c07
0ba1 058d d8c9 b873 3e60 17e8 ab38 fad6
e9da b4e6 82f5 fb04 2316 b2ac 6cb3 05e1
7158 5a91 e21e 6333 f73e d839 e174 53bc
2f12 47dd 5dd2 d6d0 3d6d 7a6f a61c 7431
9e5f e3a2 55c5 1542 6efd 8b34 c40c 5382
a297 4627 2b64 1dc1 23bd 1442 f863 b1b1
767b c463 1936 ab96 b74c b899 7cae 632d
262d cef2 3088 2958 4d27 0f1b 9c44 ed30
f4ce 1d20 9a7e 2dca e99e 65bd 4686 2a8f
d3f4 f74a 35d3 e034 8473 9006 08d0 2372
0053 78fa d7ef 3f1a bcfd f702 376b c000
6517 3b41 5fb0 ff4c 87bb f79b 40ba 7bdf
c597 0246 663b 0ece 256d 26bd f806 e0f9
70b2 a1c8 ba4a 365d 69c6 e551 44d3 c840
7f9e 006d c1bc 07e5 c7df 5ca7 b37c 1de8
8d86 f118 e594 ede7 04b0 cc10 fac5 4070
a2a2 cd43 9ac6 fbd1 5790 eb60 fe65 f32d

real    0m0.014s
user    0m0.000s
sys     0m0.000s

You have a problem when sectors that exist return an input/output error. It could mean data corruption.

S.M.A.R.T. Extended Self Test

You can run a S.M.A.R.T. extended offline self test with this command:

smartctl -t long /dev/sdb

See how to check the progress of a test here.

Once the test finishes, you can check its status with this command:

smartctl -l selftest /dev/sdb

Here is an example from a hard drive that I "repaired" (explanation upcoming):

root@node51 [~]# smartctl -l selftest /dev/sdb
smartctl 6.4 2014-10-07 r4002 [x86_64-linux-3.16.0-4-amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     23316         -
# 2  Extended offline    Completed: read failure       30%     22068         1475120744
# 3  Extended offline    Completed without error       00%     19943         -
# 4  Short offline       Completed without error       00%     19940         -
1 of 1 failed self-tests are outdated by newer successful extended offline self-test # 1

In the past, sector 1475120744 could not be read, and the test failed right then, but some time later, the sector was fixed? Read on for how this can be.

"Repairing" Your Drive

You may lose data if you attempt these commands. Make sure you have adequate backups.

Remap Bad Sectors

If repeated attempts at hdparm --read-sector keep slowing down your drive and returning an I/O error after some time, the data probably won't ever come back. You can tell the hard drive to remap the affected sector to a spare one, which will completely blank out the sector from your perspective:

hdparm --yes-i-know-what-i-am-doing --write-sector 947918344 /dev/sdb

… where 947918344 is the sector number that you want to overwrite.

It is possible to run out of spare sectors, but if you do, your drive has gone bad, and you should replace it. (Example)

Scan and Repair

A drive can only know about bad sectors if it sees them. You can scan for bad sectors using badblocks:

badblocks -b 512 -s /dev/sdb

Failed reads would be logged into the kernel log as you saw before and outputted by badblocks. You can use this information to remap sectors using hdparm --write-sector.

ATA Secure Erase

If you don't want to wait for badblocks or script something up to remap sectors, you can wipe out all the bad sectors (as well as all your other data) in one go.

Note: These instructions might not work over a USB interface.

hdparm --security-set-pass password /dev/sdb
time hdparm --security-erase password /dev/sdb

Remap and Wipe with `dd`

You can also remap sectors by wiping your drive with dd:

dd conv=noerror if=/dev/zero of=/dev/sdb bs=2M

Remap and Wipe with `dd` and Preserve LUKS

Alternatively, if your LUKS device is decrypted at /dev/mapper/your-luks-device:

dd conv=noerror if=/dev/zero of=/dev/mapper/your-luks-device bs=2M