Some background. We do lots of coverage builds of the Wind River Linux products and we have a blade cluster attached to various SAN devices which hold the temporary build data. The builds are very CPU and disk intensive and push the limits of the SAN devices. The default configuration for a MD3000i SAN is one large RAID5 group, but this results in one unused RAID controller and unnecessary redundancy for our case of temporary build files.
So I reconfigured the MD3000i to have 2 RAID0 disk groups, one for each controller. This keeps both RAID controllers busy. Within each RAID group I made a disk for each host, i.e. 2 disks for each host. Each host then spread its builds evenly across the disks. Each builder runs 4 simultaneous builds, 2 on each disk.
Because MD3000i has redundant controllers, etc the multipath driver is necessary. I also found that I needed to make an alias for the wwid of each iSCSI disk to avoid naming problems when /dev/mapper/mpath0 became mpath3 for unpredictable reasons.
The only problem with RAID0 is that when a physical disk dies, the whole disk group dies too. No data is lost, but I thought I would capture the rebuild process here for future reference.
First log into dell storage manager.
Go to Modify > Delete Disk Groups and delete failed RAID 0 virtual disks and group.
Now create another RAID0 Disk group and the virtual disks. Make sure the names of the virtual disks and the host are the same as the previous virtual disk. Make sure the preferred owner for each disk used by the same host is different.
Log into machine. Unmount failed drive and remove from multipath. I use the device ba2 (buildarea2) in my examples.
puppet agent --disable
umount /ba2
multipath -f ba2
Run multipath -d (dryrun) to get the wwid of the new disk.
Edit /etc/multipath.conf to replace now invalid wwid with new wwid. Also change wwid in puppet class for this host. I am using extdata in puppet to manage the contents of the /etc/multipath.conf file.
Run multipath -d to verify that multipath alias is working. Run multipath (no args) to actually create disk.
Now create partitions. Ensure that partition is RAID stripe aligned. Use gpt because of large partitions (and it is new standard)
parted /dev/mapper/ba2 mklabel gpt
parted -s /dev/mapper/ba2 mkpart ba2 ext2 1M 100%
Format the drive. On CentOS 5 I have no choice but to use ext3. Use stride width of 32 which corresponds with 4k block x 32 = 128K raid block size.
mkfs.ext3 -m 0 -E stride=32 /dev/mapper/ba2p1
For some reason the format blazes through blocks 0 to 9000 and then slows to a crawl.
Final step, turn off fsck checks so reboots don’t hang. I don’t like it, but keeping the builders offline for hours to run fsck is not acceptable. It is easier just to reformat the drive regularly. Again the data on these drives is temporary and easily replaced.
tune2fs -i 0 /dev/mapper/ba2p1
With the disk recreated, I can run puppet to rebuild all infrastructure necessary for the coverage builders.
puppet agent --enable
puppet agent --test
All done. Take a couple hours. If lots of disks fail, then this takes too long. I never did a test with RAID5 in this configuration to see if the performance is acceptable. The builds are very sensitive to I/O bandwidth and are running well with RAID0 so I may not have time to run a comparison.