Software RAID5 nightmare

[arear]

Hi
Am new to linux and am having trouble setting up a software RAID5 file server.

I have converted an old system containing 1800+ athlon, 1GB RAM, 400W PSU, 120GB IDE HDD.
I then added a 4 port Silicon Image SLI 3114 SATA RAID controller (cant afford proper hardware RAID). Connected to the controller are 3*300GB drives.

--RAID--
I managed to install FC5, do all updates currently available through yum.
Linux is installed on the 120GB IDE HDD.
I then partitioned each of the sata drives into 10 identical partitions per drive at approx 30GB each. These partitions are set to 'Linux RAID Autodetect'. I am currently testing with just 2 arrays assembled consisting of /dev/sda1 /dev/sdb1 /dev/sdc1 and /dev/sda2 /dev/sdb2 /dev/sdc2 and are named /dev/md1 and /dev/md2 respectivley.

mdadm.conf file is correct but linux refuses to assemble these on boot. It also refuses to assemble these from shell when I use the command 'mdadm --assemble /dev/md1'. I managed to get each of the arrays to assemble using 'mdadm --assemble --update summaries /dev/md1'

I added the assemble commands to /etc/rc.d/rc.local so at that point every time i restart the system the raid devices are mounted and reported claen and syncd .

--LVM--
Next i created an LVM VG named /dev/LVGraid which consisted of /dev/md1 and /dev/md2. I then added a LV named /dev/LVGraid/lv0 which used all of the space available to the VG. This formatted as ext3 and seems happy.
Whenever I restarted this VG was not available, presumably because the RAID arrays were being started after the VG was started. To fix this i added another command to /etc/rc.d/rc.local after the RAID assembles 'vgchange -ay LVGraid'.
This worked in that at every boot lv0 became available and was reported correctly.

To be sure with it all I ran 'e2fsck -f /dev/LVGraid/lv0' which reported problems with the image of the file system, I fixed and ran again. Every time I run it similar errors are reported. Even after running with bad blocks a number of times the immage is not correct.

I decided to continue anyway since the filesystem was reported clean unless i forced the check. I mounted to /raid and everything seemed fine untill I attempted to copy data to it. When transfering anything to it, after about 250mb i get an error saying canot write to lv0 so I am forced to abort the transfer. When trying to resume the transfer I get errors saying I dont have permission to write to /raid. After a reboot I am allowed to write again but after 250MB the same error repeats.

Any suggestions on how to fix this would be appreciated as I am on the verge of giving up and resorting to windows which i would rather not do!

Thanks

[ccrvic]

Originally Posted by arear

I then partitioned each of the sata drives into 10 identical partitions per drive at approx 30GB each. These partitions are set to 'Linux RAID Autodetect'. I am currently testing with just 2 arrays assembled consisting of /dev/sda1 /dev/sdb1 /dev/sdc1 and /dev/sda2 /dev/sdb2 /dev/sdc2 and are named /dev/md1 and /dev/md2 respectivley.

Eeek!

Personally, I'd have built one stonking great RAID & used LVM to partition it...

Originally Posted by arear

mdadm.conf file is correct

I wouldn't worry too much about mdadm.conf; the current RAID system doesn't need it unless you're doing something a bit off-the-wall (I only needed it when I was building RAIDs out of LVM partitions). I'd move it somewhere where it can't get involved in things, then build your RAIDs with mdadm --create & let the superblock magic do its stuff.

Originally Posted by arear

I added the assemble commands to /etc/rc.d/rc.local so at that point every time i restart the system the raid devices are mounted and reported claen and syncd .

Try the superblock method I've mentioned above - I think you'll be able to remove those assemble commands.

Originally Posted by arear

Whenever I restarted this VG was not available, presumably because the RAID arrays were being started after the VG was started.

Sounds like it. In my odd scenario above, I was using the mdadm.conf to force the RAID to be created after LVM had been set up. I suspect this is happening to you.

Originally Posted by arear

To fix this i added another command to /etc/rc.d/rc.local after the RAID assembles 'vgchange -ay LVGraid'.
This worked in that at every boot lv0 became available and was reported correctly.

I suspect you can get rid of this too.

Originally Posted by arear

To be sure with it all I ran 'e2fsck -f /dev/LVGraid/lv0' which reported problems with the image of the file system, I fixed and ran again. Every time I run it similar errors are reported. Even after running with bad blocks a number of times the immage is not correct.

Were you running fsck on a mounted file system? That's not a good thing to do...

Originally Posted by arear

I decided to continue anyway since the filesystem was reported clean unless i forced the check. I mounted to /raid and everything seemed fine untill I attempted to copy data to it. When transfering anything to it, after about 250mb i get an error saying canot write to lv0 so I am forced to abort the transfer. When trying to resume the transfer I get errors saying I dont have permission to write to /raid. After a reboot I am allowed to write again but after 250MB the same error repeats.

Hmmm. That sounds like filesystem corruption.

I'd recommend running fsck agains that filesystem - but *unmount it first*. If that doesn't come up clean, then we have a problem...

You might want to get your RAID/LVM things sorted out first, though.

Vic.

[arear]

I always did the e2fsck with /root unmounted.
After copying data and gettint the stop error I ran the check again and it came up with filesystem error.

Am creating the RAID arrays now using 'mdadm --create /dev/md1 --level=5 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1'. (same with 2s)
Have deleted all lines in mdadm.conf, will see if arrays start on boot once they are synced.

[ccrvic]

Originally Posted by arear

I always did the e2fsck with /root unmounted.

Is that where you were mounting /dev/LVGraid/lv0 ? That's an unusual place to put an independent RAID...

Originally Posted by arear

Have deleted all lines in mdadm.conf, will see if arrays start on boot once they are synced.

I've no idea what effect it will have if /etc/mdadm.conf exists but has no contents; personally, I'd either move the original file elsewhereville, or delete it entirely.

Vic.

[arear]

Array was assembled, synceed and persistent superblock. mdadm.conf deleted. On restert the arrays do not auto assemble. Assembly using following argument doesn't work either:

[root@localhost ~]# mdadm --assemble /dev/md1 /dev/sda1 /dev/sdb1 /dev/sdc1
mdadm: failed to add /dev/sdb1 to /dev/md1: Invalid argument
mdadm: failed to add /dev/sdc1 to /dev/md1: Invalid argument
mdadm: failed to add /dev/sda1 to /dev/md1: Invalid argument
mdadm: /dev/md1 assembled from 0 drives - not enough to start the array.

will assemble if I use the '--update summaries' argument.

Output from fdisk if thats any use:

[root@localhost ~]# fdisk -l

Disk /dev/hda: 123.5 GB, 123522416640 bytes
255 heads, 63 sectors/track, 15017 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 1 13 104391 83 Linux
/dev/hda2 14 15017 120519630 8e Linux LVM

Disk /dev/sda: 300.0 GB, 300090728448 bytes
255 heads, 63 sectors/track, 36483 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 3648 29302528+ fd Linux raid autodetect
/dev/sda2 3649 7296 29302560 fd Linux raid autodetect

Disk /dev/sdb: 300.0 GB, 300090728448 bytes
255 heads, 63 sectors/track, 36483 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 3648 29302528+ fd Linux raid autodetect
/dev/sdb2 3649 7296 29302560 fd Linux raid autodetect

Disk /dev/sdc: 300.0 GB, 300090728448 bytes
255 heads, 63 sectors/track, 36483 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdc1 * 1 3648 29302528+ fd Linux raid autodetect
/dev/sdc2 3649 7296 29302560 fd Linux raid autodetect

Disk /dev/dm-0: 121.2 GB, 121265717248 bytes
255 heads, 63 sectors/track, 14743 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-0 doesn't contain a valid partition table

Disk /dev/dm-1: 2080 MB, 2080374784 bytes
255 heads, 63 sectors/track, 252 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/dm-1 doesn't contain a valid partition table

[arear]

Looking better...
Had to create array, then reboot and it wasnt there. Then I assembled updating summaries and restarted again.
Its picking up the arrays and starting them automatically.

Am currently formatting exch array individually as ext3 so i can run checks on the system to make sure its not the arrays that are corrupting data.

Assuming everything is ok with them, do you advise I put a single partition on each array then LVM them or should I leave the arrays without partitions then put LVM on top?

[arear]

ARGH!
its stopped picking them up again and performing assemble without updating summaries results in invalid argument errors.

[arear]

Seems its sorted.
The PCI to SATA controller card was the problem. Have replaced it and all is well.