Diagnosis and replacement of a defective hard drive (Linux Dedicated Servers and Bare Metal Servers with hardware RAID)
Please use the “Print” function at the bottom of the page to create a PDF.
If you receive a notification of a hard drive failure or notice any irregularities in the system, you must act quickly to restore redundancy to your RAID array. In this article, you will learn how to identify a defective hard drive and prepare the server for the replacement of the drive.
Notes
This article assumes a basic understanding of Linux server administration. If you have any questions or require assistance regarding the replacement of a defective hard drive, please contact IONOS Customer Support. You can find the contact details on the following page: IONOS Customer Support
This article primarily describes the diagnosis and preparation for replacing SATA/SAS drives in hardware RAID systems. Commands, device names, SMART/health values, and the identification of defective drives may differ for NVMe drives.
To ensure the highest possible level of reliability, you must monitor the hardware RAID on your Dedicated Server or Bare Metal Server. If you receive an email notification about a faulty hard drive or detect a fault in a hard drive, you must identify the faulty hard drive and prepare the server for the replacement of the faulty drive. You should then contact the IONOS Customer Support to arrange for the hard drive to be replaced.
Please Note
RAID systems offer greater reliability and/or higher speeds. However, they are no substitute for regular backups. To prevent data loss, we recommend that you create a backup regularly. Please also ensure that you create a backup before carrying out the following steps to safeguard your data.
For further information on creating backups, please refer to the following category in the IONOS Help Centre: Backup Solutions
Hardware RAID controllers: General information
A hardware RAID controller is a physical controller that is installed in the server as a hardware component. This controller has its own processor for performing RAID operations. This processor organises and manages the storage space. Consequently, the server’s CPU is not burdened by RAID calculations. With hardware RAID controllers, the RAID functionality is independent of the operating system. They are managed via specialised command-line interface (CLI) applications, which may vary depending on the manufacturer and model.
Diagnosis of hard drive faults
To detect hard drive faults, we recommend that you use the smartctl application. smartctl is a command-line program for monitoring storage devices using SMART (Self-Monitoring, Analysis, and Reporting Technology). You can use this program to check whether a hard drive is failing. It is part of the Smartmontools suite. Smartmontools are available as packages for many Linux distributions. Further information can be found on the following page: Smartmontools Packages
Note
In some cases, a hard drive fault may not be detected using the SMART values. We therefore recommend that you also analyse the system logs.
Installing smartctl
To install smartctl, log in to the server as an administrator.
Install the required packages depending on your distribution:
AlmaLinux 9 and 10, and Rocky Linux 9 and 10:dnf install smartmontools
Debian 12 and 13, and Ubuntu 24.04 and 26.04:
sudo apt install smartmontools
Determining the hardware controller type
To check which hardware controller is installed in your server, you can use the lshw program. This application provides detailed information about hardware components.
To install the program, enter the following command:
AlmaLinux 9 and 10, and Rocky Linux 9 and 10:
dnf install lshw
Debian 12 and 13, and Ubuntu 24.04 and 26.04:
sudo apt-get install lshw
Viewing hardware information
To display a summary of the hardware information, enter the following command:
lshw –short
To output the hardware information as a text file, enter the following command:
lshw > lshw_output.txt
In the following example, a PERC H330 hardware controller is installed in the server:
root@829F6DF:~# lshw -short
H/W path Device Class Description
==========================================================
system PowerEdge R230 (SKU=NotProvided;ModelName=PowerEdge R230)
/0 bus 0DWX9P
/0/0 memory 64KiB BIOS
/0/400 processor Intel(R) Xeon(R) CPU E3-1270 v6 @ 3.80GHz
/0/400/700 memory 256KiB L1 cache
/0/400/701 memory 1MiB L2 cache
/0/400/702 memory 8MiB L3 cache
/0/1000 memory 32GiB system memory
/0/1000/0 memory 16GiB DIMM DDR4 synchronous unbuffered (unregistered) 2133 MHz (0.5 ns)
/0/1000/1 memory 16 GiB DIMM DDR4 Synchronous Unbuffered (Unregistered) 2133 MHz (0.5 ns)
/0/1000/2 memory [empty]
/0/1000/3 memory [empty]
/0/100 bridge Intel Corporation
/0/100/1 bridge Skylake PCIe Controller (x16)
/0/100/1/0 scsi0 storage MegaRAID SAS-3 3008 [Fury]
/0/100/1/0/2.0.0 /dev/sda disk 799GB PERC H330 Adp
/0/100/1/0/2.0.0/1 /dev/sda1 volume 2047KiB BIOS boot partition
/0/100/1/0/2.0.0/2 /dev/sda2 volume 27GiB EXT3 volume
/0/100/1/0/2.0.0/3 /dev/sda3 volume 9536MiB Linux swap volume
/0/100/1/0/2.0.0/4 /dev/sda4 volume 707GiB LVM physical volume
/0/100/1.1 bridge Skylake PCIe Controller (x8)
/0/100/14 bus Sunrise Point-H USB 3.0 xHCI Controller
/0/100/14/0 usb1 bus xHCI Host Controller
/0/100/14/0/3 bus Gadget USB HUB
/0/100/14/1 usb2 bus xHCI Host Controller
/0/100/14.2 generic Sunrise Point-H Thermal subsystem
/0/100/16 communication Sunrise Point-H CSME HECI #1
/0/100/16.1 communication Sunrise Point-H CSME HECI #2
/0/100/17 storage Sunrise Point-H SATA controller [AHCI mode]
/0/100/1d bridge Sunrise Point-H PCI Express Root Port #9
/0/100/1d/0 eth0 network NetXtreme BCM5720 Gigabit Ethernet PCIe
/0/100/1d/0.1 eth1 network NetXtreme BCM5720 Gigabit Ethernet PCIe
/0/100/1d.2 bridge Sunrise Point-H PCI Express Root Port #11
/0/100/1d.2/0 bridge SH7758 PCIe Switch [PS]
/0/100/1d.2/0/0 bridge SH7758 PCIe Switch [PS]
/0/100/1d.2/0/0/0 bridge SH7758 PCIe-PCI Bridge [PPB]
/0/100/1d.2/0/0/0/0 display G200eR2
/0/100/1f bridge Sunrise Point-H LPC Controller
/0/100/1f.2 memory Memory controller
/0/100/1f.4 bus Sunrise Point-H SMBus
RAID controller management programs
The software required to manage the respective RAID controllers is installed by default. If the software is not installed on your server, you will find links in the following article where you can download it: Monitoring and rebuilding hardware RAID (Linux)
Use the CLI tool to view the status of the RAID array
Use the appropriate CLI tool to check the status of the RAID array (RAID 1, 5 or 6) and the physical drives.
The Broadcom RAID controller is managed and controlled using the StorCLI command-line tool. Depending on the controller generation and operating system, use:
storcli64: For common Broadcom/LSI controllers (e.g. MegaRAID, AVAGO)
storcli2: For the latest generation of Tri-Mode controllers (NVMe/SAS/SATA)
The Dell RAID controller is managed and controlled using the PERCCLI command-line tool. Depending on the controller generation and operating system, use:
perccli64: For older generations of Dell RAID controllers
perccli2: For the latest generation of Dell RAID controllers (e.g. GraniteRapids/Blackwell systems)
Notes
The following example uses storcli64. If you are using a Dell controller or a newer system, replace the command accordingly with perccli64, storcli2, or perccli2. The syntax remains virtually identical.
If you are unsure which command-line tool is intended for your RAID controller, try the commands listed below with the show option. Running the wrong tool will simply result in an error message; it will not cause any damage.
Identifying a defective hard drive
You can identify the defective hard drive using the following command: storcli64 /call show
The following information, amongst other things, will then be displayed:
Controller = 0
Status = Success
...
Drive Groups = 1
TOPOLOGY :
========
-----------------------------------------------------------------------------
DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace TR
-----------------------------------------------------------------------------
0 - - - - RAID1 Optl N 931,000 GB dflt N N dflt N N
0 0 - - - RAID1 Optl N 931,000 GB deflt N N deflt N N
0 0 0 16:4 0 DRIVE Onln N 931,000 GB default N N default - N
0 0 1 16:6 1 DRIVE Online N 931,000 GB default N N default - N
-----------------------------------------------------------------------------
DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID
DID=Device ID|Type=Drive or RAID Type|Onln=Online|Rbld=Rebuild|Optl=Optimal
Dgrd=Degraded|Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active PDC=PD Cache|PI=Protection Info|SED=Self-Encrypting Drive|Frgn=Foreign DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present TR=Transport Ready
Virtual Drives = 1
Check the controller number. Normally, the entry Controller = 0 is displayed. If a different number is displayed, you must take this into account when using the commands for the hardware RAID controller. Next, look in the PD LIST for the Slt (Slot) and State. A fault-free hard drive has the status Onln (Online).
A defective hard drive is often displayed as Failed, Missing, Offline, or Degraded. Note down the EID:Slt (Enclosure ID and Slot number), e.g., 252:1.
Viewing information about the hard drive
To retrieve information about the hard drive using smartctl, you must always specify the relevant command in combination with an option and a target device. The target device depends on the manufacturer of the controller.
Use the commands listed below to retrieve the information about the hard drive required for diagnostics:
ARECA:
Hard drive 1: smartctl -iHAl error /dev/sg1 -d areca,1
Hard drive 2: smartctl -iHAl error /dev/sg1 -d areca,2
LSI / 3Ware:
Hard drive 1: smartctl -iHAl error /dev/twe0 -d 3ware,0
Hard drive 2: smartctl -iHAl error /dev/twe0 -d 3ware,1
Adaptec:
Hard drive 1: smartctl -iHAl error /dev/sg2 -d sat
Hard drive 2: smartctl -iHAl error /dev/sg3 -d sat
Hard drive 3: smartctl -iHAl error /dev/sg4 -d sat
Hard drive 4: smartctl -iHAl error /dev/sg5 -d sat
Dell:
Hard drive 1: smartctl -iHAl error -d sat+megaraid,0 /dev/sda
Hard drive 2: smartctl -iHAl error -d sat+megaraid,1 /dev/sda
Broadcom:
Hard drive 1: smartctl -iHAl error -d sat+megaraid,0 /dev/sda
Hard drive 2: smartctl -iHAl error -d sat+megaraid,1 /dev/sda
Additional commands for supported hardware controllers can be found on the following page:
https://www.smartmontools.org/wiki/Supported_RAID-Controllers
Example:
[root@localhost ~]# smartctl -iHAl error /dev/sg1 -d areca,1
smartctl 7.0 30 December 2018 r4883 [x86_64-w64-mingw32-2016] (sf-7.0-1)
Copyright (C) 2002–18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K2
Device Model: HGST HUS722T1TALA604
Serial Number: WMC6M0JAUEV8
LU WWN Device Id: 5 0014ee 00482c2ec
Firmware Version: RAGNWA07
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA version: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local time: Thu 17 Jan 06:17:05 2019 CAST
SMART support is: Available – the device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART Attributes Data Structure revision number: 16
Vendor-specific SMART attributes with thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 141 140 021 Pre-fail Always - 3933
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 15
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 34
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 10
16 Gas_Gauge 0x0022 000 200 000 Old_age Always - 1822115874
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 6
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 9
194 Temperature_Celsius 0x0022 113 109 000 Old_age Always - 30
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
SMART Error Log Version: 1
No errors logged
Interpretation of the parameters
Analyse the detailed information you have retrieved. The first section contains information that will help you identify the hard drive. This section displays, for example, the device model, the serial number, and the capacity of the hard drive being tested.
Note
The SMART attributes listed are examples for ATA/SATA/SAS drives. NVMe drives use different health parameters.
=== START OF INFORMATION SECTION ===
Model Family: Hitachi/HGST Ultrastar 7K2
Device Model: HGST HUS722T1TALA604
Serial Number: WMC6M0JAUEV8
LU WWN Device ID: 5 0014ee 00482c2ec
Firmware Version: RAGNWA07
User Capacity: 1,000,204,886,016 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In the smartctl database [for details, use: -P show]
ATA version is: ACS-3 T13/2161-D revision 5
SATA version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local time is: Thu 17 Jan 06:17:05 2019 CAST
SMART support is: Available – the device has SMART capability.
SMART support is: Enabled
In the second section, smartctl assesses the current condition of the hard drive. If the value displayed is not PASSED but, for example, Failed or UNKNOWN, you should arrange for the hard drive in question to be replaced as soon as possible.
=== START OF READ SMART DATA SECTION ===
SMART overall health self-assessment test result: PASSED
The third section lists the calculated SMART values in detail. Alongside each current percentage value (VALUE), the worst value ever recorded (WORST) and the respective threshold (THRESH) are shown. If the current percentage value (VALUE) or the worst value ever recorded (WORST) exceeds the threshold (THRESH), a SMART warning is displayed in theWHEN_FAILED column (e.g. FAILING_NOW).
SMART Attributes Data Structure revision number: 16
Vendor-specific SMART attributes with thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 141 140 021 Pre-fail Always - 3933
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 15
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 34
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 10
16 Gas_Gauge 0x0022 000 200 000 Old_age Always - 1822115874
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 6
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 9
194 Temperature_Celsius 0x0022 113 109 000 Old_age Always - 30
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
The following parameters may indicate an impending hard drive failure before a SMART warning is displayed:
Reallocated_Sector_Ct: This parameter indicates the number of sectors that have already been reallocated due to read errors. If a sector can no longer be read, written to or checked correctly, the controller automatically assigns it a replacement sector from the hard drive’s reserve area. The original, faulty sector is permanently marked as defective and is no longer used.
Note
A value greater than zero is not necessarily a cause for concern, provided it remains stable over a long period of time. However, a steadily increasing number of reallocated sectors is a critical indicator of an impending hard drive failure. To detect a fault at an early stage, you should therefore log this value regularly and, if it continues to rise, arrange for the hard drive to be replaced immediately.
Current_Pending_Sector_Ct: Indicates the number of unstable sectors awaiting remapping. If a sector cannot be read or written to correctly, it is initially assigned the status ‘Current Pending Sector’. The sector is not remapped whilst in this state, as the data stored on it is unknown. Only after several unsuccessful read or write attempts is a replacement sector allocated, and the faulty sector is permanently marked as unreadable. The Current_Pending_Sector_Ct value is an important indicator that a hard drive needs replacing. If this value is not zero, a hard drive failure is often imminent.
Offline_Uncorrectable: Indicates the number of uncorrectable errors during read and write access to sectors.
The final section deals with the hard drive's internal log. Errors are recorded here when the server’s job requests have not been processed correctly by the hard drive. If this section displays an error count of at least two digits, you should arrange for the hard drive to be replaced as soon as possible.
SMART Error Log Version: 1
No errors logged
Alternatives to smartctl
If you are unable to retrieve any information using smartctl, modify the following command and then enter it:
Broadcom:
storcli64 /c[controller number]/e[enclosure ID]/s[slot ID] show smart
Example:
storcli64 /c0/e16/s4 show smart
Dell:
perccli64 /c[controller number]/e[enclosure ID]/s[slot ID] show smart
Example:
perccli64 /c0/e16/s4 show smart
Viewing log files
For information on how to access the log files, please refer to the documentation provided by the respective manufacturer.
Areca
https://www.starline.de/en/support/areca-service-blog
3ware (now Broadcom)
https://www.broadcom.com/products/storage/raid-controllers/
Adaptec (now Microchip)
https://storage.microsemi.com/en-us/support/
Dell
https://www.dell.com/support/home/en-us
Broadcom
https://www.broadcom.com/products/storage/raid-controllers/
Preparing to replace the hard drive
View detailed information on hard drive replacement
The following information is required so that the replacement of the defective hard drive can be arranged:
Designation of the hard drive in the RAID
Serial number
Model
Log file (optional)
Create a SMART log
Use the commands listed below to generate a complete SMART log:
ARECA systems:
Hard drive 1: smartctl -x /dev/sg1 -d areca,1
Hard drive 2: smartctl -x /dev/sg1 -d areca,2
LSI / 3Ware systems:
Hard drive 1: smartctl -x /dev/twe0 -d 3ware,0
Hard drive 2: smartctl -x /dev/twe0 -d 3ware,1
Adaptec systems:
Hard drive 1: smartctl -x /dev/sg2 -d sat
Hard drive 2: smartctl -x /dev/sg3 -d sat
Hard drive 3: smartctl -x /dev/sg4 -d sat
Hard drive 4: smartctl -x /dev/sg5 -d sat
Dell systems:
Hard drive 1: smartctl -x -d sat+megaraid,0 /dev/sda
Hard drive 2: smartctl -x -d sat+megaraid,1 /dev/sda
Broadcom systems:
Hard drive 1: smartctl -x -d sat+megaraid,0 /dev/sda
Hard drive 2: smartctl -x -d sat+megaraid,1 /dev/sda
Notes
If the SMART log has been generated as described above, this information is sufficient. You can then arrange for the faulty hard drive to be replaced. To do so, please contact IONOS Customer Support.
If you are unable to retrieve the serial number of the faulty hard drive using smartctl, you can alternatively provide customer service with the serial number of the functioning hard drive(s).
If you are unable to obtain the information required for the replacement and wish to have the hard drive replaced, the hardware must be checked before the replacement takes place. During this check, the server is usually temporarily unavailable. If a fault with the hard drive is detected during this check, it will be replaced.
Arrange for a hard drive replacement
You can then arrange for the defective hard drive to be replaced. To do so, please contact IONOS Customer Support. You will find the contact details on the following page: IONOS Customer Support
Steps to follow after replacing the hard drive
Once the faulty hard drive has been replaced, the RAID system rebuild will usually start automatically. Please check whether the RAID system rebuild has started and is proceeding successfully. To do this, monitor the progress using the controller tool:
Broadcom:
storcli64 /c0 /eall /sall show rebuild
Dell:
perccli64 /c0 /eall /sall show rebuild
Additional information on monitoring hardware RAIDs can be found in the following article: Monitoring and rebuilding hardware RAID (Linux)