Rebuild Software RAID (Linux)

Here we show you how to rebuild a software RAID after replacing the defective hard disk.

Please note: After the hard disk has been replaced, it may be detected as sdc. This always happens during an exchange via hot-swap. Here only a reboot helps, so that the hard disk is recognized as sda or sdb again.

Example Scenario

The following configuration is assumed in this manual:

# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sda1[0] sdb1[1]
4194240 blocks [2/2] [UU]

md3 : active raid1 sda3[0] sdb3[1]
1458846016 blocks [2/2] [UU]

There are 2 arrays:

/dev/md1 as /

/dev/md3 for the log. Partitions /var /usr /home

Typically, with sda2 and sdb2 there are two more swap partitions that are not part of the RAID.

RAID Restore

The further procedure depends on whether hard disk 1 (sda) or hard disk 2 (sdb) has been replaced:

Case 1: Hard disk 2 has been replaced

If the second hard disk (sdb) has been replaced and a reboot has already been performed, it may be that the disk is directly recognized and mounted correctly. If this is the case, no further steps are necessary except activation of the swap partition, since a rebuild is already running.

You can check whether the rebuild is already running by using mdstat:

cat /proc/mdstat

Personalities : [raid1]
read_ahead 1024 sectors
md3 : active raid1 sdb1[1] sda1[0]
102208 blocks [2/2] [UU]

md1 : active raid1 sdb3[1] sda3[0]
119684160 blocks [2/1] [U_]
[>....................] recovery = 0.2% (250108/119684160) finish=198.8min speed=10004K/sec
unused devices: <none>

In this example, the rebuild is already running. By[U_] or[_U] you can see that a hard disk is not (yet) in sync. If the RAID array is intact, it says[UU].

If no rebuild can be seen here, you have to mount the hard disk manually and start the rebuild.

In the first step, copy the partition tables manually from the first to the second hard disk. This is done with this command:

sfdisk -d /dev/sda | sfdisk /dev/sdb

You may need to use the --force option:

sfdisk -d /dev/sda | sfdisk --force /dev/sdb

After that you can check with fdisk -l if the second hard disk is now as divided as the first one.

After restoring the partitioning, you can reinsert each part of the disk into the RAID:

mdadm /dev/md1 -a /dev/sdb1

mdadm /dev/md3 -a /dev/sdb3

You can ignore the output so far. It is only important that the rebuild runs under /proc/mdstat.

Once the rebuild is complete, you can activate the swap partition using the following commands:

mkswap /dev/sdb2

swapon -a

Case 2: Hard disk 1 has been replaced

If hard disk 1 (sda) has been replaced, you must check whether the hard disk was recognized correctly and reboot if necessary.

Then boot the server into the rescue system and follow these steps:

First copy the partition tables to the new (empty) hard disk:

sfdisk -d /dev/sdb | sfdisk /dev/sda

(You may need to use the --force option)

Now add the partitions to the RAID:

mdadm /dev/md1 -a /dev/sda1

mdadm /dev/md3 -a /dev/sda3

You can now use cat /proc/mdstat to track the rebuild of the RAID.

Then mount the partitions var, usr and home:

mount /dev/md1 /mnt
mount /dev/mapper/vg00-var /mnt/var
mount /dev/mapper/vg00-usr /mnt/usr
mount /dev/mapper/vg00-home /mnt/home

So that Grub can be installed error-free later, mount proc, sys and dev:

mount -o bind /proc /mnt/proc
mount -o bind /sys /mnt/sys
mount -o bind /dev /mnt/dev

After mounting the partitions, jump into the chroot environment and install the grub bootloader:

chroot /mnt

grub-install /dev/sda 

Exit Chroot with Exit and unmount all disks:

umount -a     

Wait until the rebuild process is complete and then boot the server back into the normal system.

Finally, you must activate the swap partition using the following commands:

mkswap /dev/sda2
swapon -a