[SOLVED] boot can't access ext4 partitions on LVM logical volumes on RAID array

[bbobbo]

i have a fedora 11 server which can't access the ext4 partitions on lvm logical volumes on a raid array during boot-up.

the problem manifested itself after a failed preupgrade to fedora 12; however, i think the attempt at upgrading to fc12 might not have anything to do with the problem, since i last rebooted the server over 250 days ago (sometime soon after the last fedora 11 kernel update). prior to the last reboot, i had successfully rebooted many times (usually after kernel updates) without any problems.

i'm pretty sure the fc12 upgrade attempt didn't touch any of the existing files, since it hung on the dependency checking of the fc12 packages. when i try to reboot into my existing fedora 11 installation, though, i get the following screen:


(click for full size)

a description of the server filesystem (partitions may be different sizes now due to the growing of logical volumes):

   - 250GB system drive
	250MB		/dev/sdh1			/boot			ext3
	lvm partition rest of drive	VolGroup_System
		10240	VolGroup_System-LogVol_root	/			ext4
		6144	VolGroup_System-LogVol_swap				swap

   - 1.5TB backup drive
	entire drive	/dev/sdg1			/backup			ext4

   - 6x1TB RAID5
	lvm partition entire RAID	VolGroup_RAID
		204800	VolGroup_RAID-LogVol_home	/home			ext4
		61440	VolGroup_RAID-LogVol_mysql	/var/lib/mysql		ext4
		30720	VolGroup_RAID-LogVol_install	/home/mnt/install	ext4
		409600	VolGroup_RAID-LogVol_media	/home/mnt/media		ext4
		1572864	VolGroup_RAID-LogVol_video	/home/mnt/media/video	ext4

now, when i boot into rescue mode using a fedora 14 dvd and mount the filesystem, everything appears fine under /mnt/sysimage. so there is nothing wrong with the raid and nothing wrong with the lvm partitions.

it's just that for some reason, the fedora 11 boot process can't seem to recognize the logical volumes on the raid. seems like it can't even find the partitions--"No such file or directory while trying to open...."

the issue described here sounds somewhat similar:

http://blog.inuus.com/vox/2009/11/fe...adm-oh-my.html

except he's talking about fake raid and dmraid, whereas my raid is linux software raid using mdadm.

this machine is a headless server which acts as my home file, mail, and web server. it also runs mythtv with four hd tuners. i connect remotely to the server using nx or vnc to run applications directly on the server. i also run an xp professional desktop in a qemu virtual machine on the server for times when i need to use windows. so needless to say, it's a major inconvenience to have the machine down.

anyone have any ideas?

[bbobbo]

well, i was right that the fc12 preupgrade attempt had nothing to with the errors.

it turns out that the reason i rebooted my server 250+ days ago was to add a new hard drive to the RAID array. after adding the new drive, i powered on the server, resized the RAID, grew the LVM physical volume, and happily went about my business for 250+ days until i had to reboot again.

the problem was that i never changed /etc/mdadm.conf to reflect the added drive:

# mdadm.conf written out by anaconda
DEVICE partitions
MAILADDR root

ARRAY /dev/md0 level=raid5 num-devices=5 metadata=0.90 spares=1 UUID=1703de53:3d
6ddaf9:11a515ed:4405a64e

this was causing the kernel to be unable to start the RAID during boot--hence the fsck errors (strangely enough, LVM reported that it was able to start the logical volumes on the RAID, even though it obviously couldn't). all i had to do was change the '5' to a '6', and everything worked fine.

idiotically, i had previously grown the RAID from 4 drives to 5, and i must have changed the num-devices argument in /etc/mdadm.conf that time because i don't remember having problems back then. i guess the process went so smoothly and easily that i overlooked that step when i grew it from 5 drives to 6.

some useful things i did to diagnose the problem:

• added 'boot_delay=10' to the kernel line in /boot/grub/grub.conf--that slowed down the text output enough so i could capture it on video with my digital camera
• after booting in rescue mode using a fedora 14 dvd, followed the excellent instructions here System Recovery Week - Fedora Daily Package to make sure that there was nothing with the RAID or the LVM volumes
• in rescue mode, edited /etc/fstab to comment out all of the partitions which were giving errors
• after booting normally (but without the RAID logical volume partitions being mounted), checked /var/log/dmesg to discover that the RAID was not being started. annoyingly, however, there were no error messages in the log (e.g., "incorrect number of devices specified in /etc/mdadm.conf")--just the lack of any messages at all about RAID being started
• and finally, what gave me the solution after playing around with mdadm:

  mdadm -v --assemble --scan

hopefully this helps someone else who might find themselves in a similar situation.