Back to home

Disks: How is a defective hard disk replaced?

Briefly, the procedure looks something like this:

  1. Remove defective hard disk from RAID
  2. Replace with hard disk from nine.ch
  3. Prepare new hard disk for RAID
  4. Add hard disk to RAID
  5. Write bootloader to disk

Instructions

These instructions refer to dedicated servers equipped with at least two hard disks and software RAID 1 (with nine.ch preinstalled Ubuntu/Debian). To see whether your system uses a software RAID, check with cat /proc/mdstat. If you see more than just unused devices:, your system is equipped with a software RAID.

Check for software RAID:

root@server:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sda2[1] sdb2[0]
      972443840 blocks super 1.2 [2/2] [UU]

md0 : active raid1 sda1[1] sdb1[0]
      208640 blocks super 1.2 [2/2] [UU]

unused devices:

Uploaded Image

This picture shows the output from cat /proc/mdstat in a functioning, healthy RAID.

Partitions identified as failed are labelled with (F).

Uploaded Image

Defect partition in RAID

1. Removing a hard disk from RAID

A RAID partition is flagged as failed with the command mdadm --manage /dev/mdX -f /dev/sdY. X stands for the specific RAID device; Y stands for the specific true partition. When this has been done for all partitions on the hard disk, you can remove it from RAID (with the same numbers for each partition) using the command mdadm --manage /dev/mdX -r /dev/sdY.

For example, to remove the hard disk /dev/sdb from RAID, use the following commands:

root@server:~# mdadm --manage /dev/md1 -f /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md1

root@server:~# mdadm --manage /dev/md1 -r /dev/sdb2
mdadm: hot removed /dev/sdb2 from /dev/md1

root@server:~# mdadm --manage /dev/md0 -f /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md0

root@server:~# mdadm --manage /dev/md0 -r /dev/sdb1
mdadm: hot removed /dev/sdb1 from /dev/md0

2. Switching hard disk

Send a request to our support address support@nine.ch. For security purposes, we need the IP address of your server and the serial number of the functioning hard disk.

Show the serial number of the functioning hard disk, e.g. /dev/sda:

root@server:~# hdparm -I /dev/sda | grep -i serial
Model=WDC WD1003FBYX-01Y7B1, FwRev=01.01V02, SerialNo=WD-WCAW35284076

3. Preparing hard disk for RAID

The new hard disk must now be integrated into RAID. It may have a different hard disk letter than the one being replaced. You can find the hard disk “numbering” under the partition information in /proc/partitions by comparing the disk letters with the existing RAID partitions. For example, /dev/sdc was identified as the new hard disk and this now needs to be added to RAID:

root@server:~# cat /proc/partitions
major minor  #blocks  name

   8        0  976762584 sda
   8        1     208813 sda1
   8        2  972575100 sda2
   9        0     208640 md0
   9        1  972443840 md1
252        0   10485760 dm-0
252        1    8388608 dm-1
252        2  485490688 dm-2
252        3  104857600 dm-3
252        4   10485760 dm-4
   8       16  976762584 sdc

First, you must apply the partitioning of an existing hard disk to the new hard disk. You can do this for hard disks with MBR partition tables by entering the command sfdisk. The sfdisk command combined with the following command copies the partition table from /dev/sda to /dev/sdc:

root@server:~# sfdisk -d /dev/sda | sfdisk /dev/sdc

Disk /dev/sdc: 121601 cylinders, 255 heads, 63 sectors/track
Old situation:
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/sdc1          0       -       0          0    0  Empty
/dev/sdc2          0       -       0          0    0  Empty
/dev/sdc3          0       -       0          0    0  Empty
/dev/sdc4          0       -       0          0    0  Empty
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdc1   *        63    417689     417627  fd  Linux raid autodetect
/dev/sdc2        417690 1945567889 1945150200  fd  Linux raid autodetect
/dev/sdc3             0         -          0   0  Empty
/dev/sdc4             0         -          0   0  Empty
Successfully wrote the new partition table

Re-reading the partition table ...

Using a GPT partition table, you can perform the copy operation as follows: sgdisk -R=/dev/sdc /dev/sda (note the order!). With GPT, a random value must be entered for the ‘GUID’ so that the hard disk (in the same server) can be used: sgdisk -G /dev/sdc.

You now have two identically partitioned hard disks and can add the new hard disk to the existing RAID.

Uploaded Image

4. Adding the hard disk to RAID

The output from cat /proc/mdstat in turn provides a reference for determining the correct partition for the correct RAID device. In this example, the partition /dev/sdc1 would be added to the RAID device /dev/md0, and /dev/sdc2 to /dev/md1:

root@server:~# mdadm --manage /dev/md0 -a /dev/sdc1
mdadm: added /dev/sdc1
root@server:~# mdadm --manage /dev/md1 -a /dev/sdc2
mdadm: added /dev/sdc2
The RAID will now start synchronising data to the new hard disk. Depending on the hard disk size and server load, this can take some time.
root@server:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md1 : active raid1 sdc2[2] sda2[1]
      972443840 blocks super 1.2 [2/1] [_U]
      [>....................]  recovery =  0.0% (659456/972443840) finish=860.5min speed=18819K/sec

md0 : active raid1 sdc1[2] sda1[1]
      208640 blocks super 1.2 [2/1] [_U]
       resync=DELAYED

unused devices: 

5. Installing the bootloader

The last step is to install the bootloader on the new hard disk so that the server will boot correctly the next time it is restarted. To do this, enter the following command:

root@server:~# grub-install /dev/sdc
Installation finished. No error reported.

Tips

Increasing the speed

The speed of the recovery is limited, in order for the server to continue with active operations. To speed up the recovery, you can manually increase the limit.

Determining the current limit:

root@server:~# sysctl dev.raid.speed_limit_min
dev.raid.speed_limit_min = 40000

To set a new limit:

root@server:~# sysctl -w dev.raid.speed_limit_min=400000
dev.raid.speed_limit_min = 400000

Hard disk not recognised

In some cases, the hard disk is not recognised. Rescanning the SCSI adapter will resolve this. To do this, enter the following command (numbering of the ‘host’ may vary; e.g. 0, 1, 2, 3, etc.):

root@server:~# echo "- - -" > /sys/class/scsi_host/host0/scan

Didn't find what you were looking for?

Contact our support:

+41 44 637 40 40 support@nine.ch