EXT4 filesystem suddenly becomes read-only

[pmjobin]

Hi,

I have been experiencing a strange issue with a brand new PC on which I'm running Fedora 16 x64. After using the system for a while, my secondary HDD (unto which my /home partition is located) suddenly becomes read-only! I can hear a strange noise coming from the drive when this happens. After restarting the system, the drive seems to be OK for a while and then, all of a sudden, it fails again. The HDD is a ST1000DM003 connected to a Marvell 88SE9128 SATA controller.

Any idea what might be causing this issue? I'd like to know if there is anything I can do in order to test whether the drive is faulty or if it's a software problem.

If it can be of any help, here are the relevant entries that appear into /var/log/messages just after the HDD fails:

Dec 5 20:23:29 phobos kernel: [ 385.949107] ata7.00: exception Emask 0x0 SAct 0x7f SErr 0x0 action 0x6 frozen
Dec 5 20:23:29 phobos kernel: [ 385.949117] ata7.00: failed command: WRITE FPDMA QUEUED
Dec 5 20:23:29 phobos kernel: [ 385.949129] ata7.00: cmd 61/08:00:08:08:80/00:00:2a:00:00/40 tag 0 ncq 4096 out
Dec 5 20:23:29 phobos kernel: [ 385.949131] res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout)
Dec 5 20:23:29 phobos kernel: [ 385.949135] ata7.00: status: { DRDY }
Dec 5 20:23:29 phobos kernel: [ 385.949138] ata7.00: failed command: WRITE FPDMA QUEUED
Dec 5 20:23:29 phobos kernel: [ 385.949145] ata7.00: cmd 61/08:08:80:08:80/00:00:2a:00:00/40 tag 1 ncq 4096 out
Dec 5 20:23:29 phobos kernel: [ 385.949146] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 5 20:23:29 phobos kernel: [ 385.949149] ata7.00: status: { DRDY }
Dec 5 20:23:29 phobos kernel: [ 385.949152] ata7.00: failed command: WRITE FPDMA QUEUED
Dec 5 20:23:29 phobos kernel: [ 385.949158] ata7.00: cmd 61/10:10:30:09:80/00:00:2a:00:00/40 tag 2 ncq 8192 out
Dec 5 20:23:29 phobos kernel: [ 385.949159] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 5 20:23:29 phobos kernel: [ 385.949162] ata7.00: status: { DRDY }
Dec 5 20:23:29 phobos kernel: [ 385.949165] ata7.00: failed command: WRITE FPDMA QUEUED
Dec 5 20:23:29 phobos kernel: [ 385.949171] ata7.00: cmd 61/00:18:00:a5:c0/04:00:0f:00:00/40 tag 3 ncq 524288 out
Dec 5 20:23:29 phobos kernel: [ 385.949172] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 5 20:23:29 phobos kernel: [ 385.949175] ata7.00: status: { DRDY }
Dec 5 20:23:29 phobos kernel: [ 385.949178] ata7.00: failed command: WRITE FPDMA QUEUED
Dec 5 20:23:29 phobos kernel: [ 385.949184] ata7.00: cmd 61/08:20:a0:0a:81/00:00:2a:00:00/40 tag 4 ncq 4096 out
Dec 5 20:23:29 phobos kernel: [ 385.949185] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 5 20:23:29 phobos kernel: [ 385.949188] ata7.00: status: { DRDY }
Dec 5 20:23:29 phobos kernel: [ 385.949190] ata7.00: failed command: WRITE FPDMA QUEUED
Dec 5 20:23:29 phobos kernel: [ 385.949196] ata7.00: cmd 61/08:28:00:08:00/00:00:02:00:00/40 tag 5 ncq 4096 out
Dec 5 20:23:29 phobos kernel: [ 385.949198] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 5 20:23:29 phobos kernel: [ 385.949201] ata7.00: status: { DRDY }
Dec 5 20:23:29 phobos kernel: [ 385.949203] ata7.00: failed command: WRITE FPDMA QUEUED
Dec 5 20:23:29 phobos kernel: [ 385.949209] ata7.00: cmd 61/08:30:a8:08:00/00:00:02:00:00/40 tag 6 ncq 4096 out
Dec 5 20:23:29 phobos kernel: [ 385.949210] res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 5 20:23:29 phobos kernel: [ 385.949213] ata7.00: status: { DRDY }
Dec 5 20:23:29 phobos kernel: [ 385.949221] ata7: hard resetting link
Dec 5 20:23:29 phobos NetworkManager[903]: <info> (wlan0): supplicant interface state: associated -> 4-way handshake
Dec 5 20:23:29 phobos NetworkManager[903]: NetworkManager[903]: <info> (wlan0): supplicant interface state: associated -> 4-way handshake
Dec 5 20:23:29 phobos NetworkManager[903]: <info> (wlan0): supplicant interface state: 4-way handshake -> completed
Dec 5 20:23:29 phobos NetworkManager[903]: NetworkManager[903]: <info> (wlan0): supplicant interface state: 4-way handshake -> completed
Dec 5 20:23:34 phobos kernel: [ 391.289221] ata7: link is slow to respond, please be patient (ready=0)
Dec 5 20:23:39 phobos kernel: [ 395.967329] ata7: COMRESET failed (errno=-16)
Dec 5 20:23:39 phobos kernel: [ 395.967338] ata7: hard resetting link
Dec 5 20:23:44 phobos kernel: [ 401.306092] ata7: link is slow to respond, please be patient (ready=0)
Dec 5 20:23:49 phobos kernel: [ 405.981781] ata7: COMRESET failed (errno=-16)
Dec 5 20:23:49 phobos kernel: [ 405.981790] ata7: hard resetting link
Dec 5 20:23:54 phobos kernel: [ 411.321108] ata7: link is slow to respond, please be patient (ready=0)
Dec 5 20:24:24 phobos kernel: [ 440.914703] ata7: COMRESET failed (errno=-16)
Dec 5 20:24:24 phobos kernel: [ 440.914713] ata7: limiting SATA link speed to 3.0 Gbps
Dec 5 20:24:24 phobos kernel: [ 440.914716] ata7: hard resetting link
Dec 5 20:24:29 phobos kernel: [ 445.949629] ata7: COMRESET failed (errno=-16)
Dec 5 20:24:29 phobos kernel: [ 445.949639] ata7: reset failed, giving up
Dec 5 20:24:29 phobos kernel: [ 445.949643] ata7.00: disabled
Dec 5 20:24:29 phobos kernel: [ 445.949648] ata7.00: device reported invalid CHS sector 0
Dec 5 20:24:29 phobos kernel: [ 445.949651] ata7.00: device reported invalid CHS sector 0
Dec 5 20:24:29 phobos kernel: [ 445.949654] ata7.00: device reported invalid CHS sector 0
Dec 5 20:24:29 phobos kernel: [ 445.949659] ata7.00: device reported invalid CHS sector 0
Dec 5 20:24:29 phobos kernel: [ 445.949662] ata7.00: device reported invalid CHS sector 0
Dec 5 20:24:29 phobos kernel: [ 445.949665] ata7.00: device reported invalid CHS sector 0
Dec 5 20:24:29 phobos kernel: [ 445.949667] ata7.00: device reported invalid CHS sector 0
Dec 5 20:24:29 phobos kernel: [ 445.951634] ata7: EH complete
Dec 5 20:24:29 phobos kernel: [ 445.951662] sd 6:0:0:0: [sdb] Unhandled error code
Dec 5 20:24:29 phobos kernel: [ 445.951665] sd 6:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Dec 5 20:24:29 phobos kernel: [ 445.951669] sd 6:0:0:0: [sdb] CDB: Write(10): 2a 00 02 00 08 a8 00 00 08
Dec 5 20:24:29 phobos kernel: [ 445.953197] EXT4-fs (sdb2): Remounting filesystem read-only
Dec 5 20:24:29 phobos kernel: [ 445.953201] EXT4-fs (sdb2): previous I/O error to superblock detected
Dec 5 20:24:29 phobos kernel: [ 445.953228] sd 6:0:0:0: [sdb] Unhandled error code
Dec 5 20:24:29 phobos kernel: [ 445.953231] sd 6:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Dec 5 20:24:29 phobos kernel: [ 445.953235] sd 6:0:0:0: [sdb] CDB: Write(10): 2a 00 02 00 08 00 00 00 08 00
Dec 5 20:24:29 phobos kernel: [ 445.953249] end_request: I/O error, dev sdb, sector 33556480
Dec 5 20:24:29 phobos kernel: [ 445.953253] Buffer I/O error on device sdb2, logical block 0
Dec 5 20:24:29 phobos kernel: [ 445.953256] lost page write due to I/O error on sdb2
Dec 5 20:24:29 phobos kernel: [ 445.960606] sd 6:0:0:0: [sdb] Unhandled error code
Dec 5 20:24:29 phobos kernel: [ 445.960609] sd 6:0:0:0: [sdb] Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Dec 5 20:24:29 phobos kernel: [ 445.960613] sd 6:0:0:0: [sdb] CDB: Write(10): 2a 00 3b 04 08 00 00 00 08 00
Dec 5 20:24:29 phobos kernel: [ 445.960621] end_request: I/O error, dev sdb, sector 990119936
Dec 5 20:24:29 phobos kernel: [ 445.960633] JBD2: I/O error detected when updating journal superblock for sdb2-8.

Thanks for your help,
PM

[Gareth Jones]

This looks like hardware.

Ext4 will typically remount read-only as a safety feature when errors occur.

Run the "Disk Utility" application and look at the SMART data.

Gareth

[pmjobin]

Thanks for your help.

But after some additionnal research, I discovered I'm not the only one experiencing issues with a Marvell SATA controller under Linux:

https://bugzilla.redhat.com/show_bug.cgi?id=549981

[acorlato]

Yes, my 64-bit Fedora switches the ext4 root filesystem to read-only mode, too, during normal operation, almost every 10 days. It doesn't happen on my i686 notebook. There are no bad blocks on the harddisk.

I react always by rebooting the system, pressing Ctrl-D when it comes to boot problem and fsck hint, then going to single user mode by command "systemctl isolate rescue.target" and calling "umount -a" before "fsck -y /" and "init 6".

My great hope is that the problem won't happen with Fedora 17. But I hate the work of upgrades.

[bbigby]

The same thing is happening to me. I ran Ubuntu 10.04 on my laptop for 3 years and never had this problem. I recently backed up my data and did a fresh installation of Fedora 18 beta. Suddenly, about after a few days, I started to see weird problems like Firefox not launching (couldn't obtain secure socket) and Evolution not being able to load e-mail. When I tried to remove files in my home directory, the shell complained that I didn't have permission because the file system was read-only! What the heck???!!! Even after repairing the problem with fsck and updating to Fedora 18 final with the latest updates, the problem reappeared.

It turns out that both of my partitions were read-only /dev/mapper/fedora-root and /dev/mapper/fedora-home. I did some digging and found some info about resetting the ext4 journal, and I just did that, but only time will tell whether it worked. This problem is REALLY disappointing. I know that Fedora 18 is bleeding edge but these types of problems just shoudn't occur ...free or not. A file system that does this is a useless file system ... again ... free or not. Perhaps, I should convert everything back to ext3. If the problem happens again, that's exactly what I'm going to try.

Here's my lspci output:

00:00.0 Host bridge: Intel Corporation Mobile 4 Series Chipset Memory Controller Hub (rev 07)
Subsystem: CLEVO/KAPOK Computer Device 0806
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
Latency: 0
Capabilities: [e0] Vendor Specific Information: Len=0a <?>

00:01.0 PCI bridge: Intel Corporation Mobile 4 Series Chipset PCI Express Graphics Port (rev 07) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00002000-00002fff
Memory behind bridge: f5000000-f7ffffff
Prefetchable memory behind bridge: 00000000d0000000-00000000dfffffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [88] Subsystem: CLEVO/KAPOK Computer Device 0806
Capabilities: [80] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Address: fee0300c Data: 41a1
Capabilities: [a0] Express (v1) Root Port (Slot+), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #2, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug+ Surprise-
Slot #1, PowerLimit 0.000W; Interlock- NoCompl+
SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
Control: AttnInd Off, PwrInd On, Power- Interlock-

[mikee]

If you can boot a 3.6.* kernel, that should work if it's that bug.

Originally Posted by pmjobin

Thanks for your help.

But after some additionnal research, I discovered I'm not the only one experiencing issues with a Marvell SATA controller under Linux:

https://bugzilla.redhat.com/show_bug.cgi?id=549981

[bbigby]

Originally Posted by pmjobin

Thanks for your help.

But after some additionnal research, I discovered I'm not the only one experiencing issues with a Marvell SATA controller under Linux:

https://bugzilla.redhat.com/show_bug.cgi?id=549981

I don't think that it's your controller necessarily. I have an Intel SATA controller and I've run into the same probem. I was not having this problem when I was running Ubuntu 10.04, which I did for the past 3+ years. I only encountered this problem after performing a fresh installation of Fedora 18.

[Patrick Wiens]

I'm having the same problem here on Fedora 17. I tried booting with kernel 3.7 and 3.6, same result.

How did you reset ext4, can you send the link to it?

[acorlato]

On my computer the problem didn't occur after I separated /home from root filesystem an put it on a good old jfs filesystem.

And on last weekend I installed a ssd disk in combination with Fedora 18. As filesystem I choosed the new btrfs in the hope not to miss the ugly ext period.

[george_toolan]

I'm having the same problem here on Fedora 17. I tried booting with kernel 3.7 and 3.6, same result.

What exactly is the error message? Please check /var/log/messages for errors.

A file system might become read-only when it contains errors to prevent it from further damage.

In this case you might have to boot from a DVD and check the file system manually with e2fsck

[bbigby]

Originally Posted by george_toolan

What exactly is the error message? Please check /var/log/messages for errors.

A file system might become read-only when it contains errors to prevent it from further damage.

In this case you might have to boot from a DVD and check the file system manually with e2fsck

It turns out that my laptop's HDD was indeed unhealthy ... aka bad. I replaced it recently with a Samsung 840 Pro Series 128 GB SSD and all seem stable now.