RAID 5 build for home media storage

[MARSTG]

I decided to share with you my experience building a storage solution for my media files.
I'll also reiterate some good practices when it comes to storage/backup.
The whole thing started when I lost my 4TB 2.5" external drive and lost all my data. The critical documents you should keep in multiple locations such as USB drives, cloud storage, on a CD/DVD if you still use that.
A few years ago I got ahold of a 2TB Toshiba, enterprise-grade SATA HDD, and I started to slowly accumulate movies and tv shows. I was obsessed with the "what if" this drive will die as well. Prior to losing the external drive, my policy was to replace the HDD with newer ones when the old ones would approach the expiration date of the warranty provided by the manufacturer of the HDD. But I knew that enterprise-grade drives hold better in time than those intended for home use scenarios.
So I started to dream of a RAID 5 matrix using used enterprise-grade drives.
So I read up a lot on what RAID is and the alternatives. I wanted a hardware-based solution, so that meant either an Intel motherboard with RAID 5 logic or a dedicated hardware controller.
For those that don't already know, what RAID 5 does, is spreading information on 3 drives minimum, together with the so-called parity information, so if a drive fails, the data is still available, but the matrix will function with slower performance. If you replace the defective drive with a healthy one, the parity information from the remaining two drives will be reconstructed on the new one, thus re-establishing the full performance.
I was already accustomed to the process but never had the occasion before to get myself a raid 5 build. The menu in the Intel RST utility is very simple and intuitive but the parity information will be calculated by the host CPU taking quite a lot of cycles from it. Nobody really knows exactly how much as I haven't found specific reviews benchmarking the performance of a raid 5 setup on the onboard Intel raid controller, so I decided to go with a full hardware RAID 5 controller.
Modern raid 5 controllers don't have SATA connectors but something called either SAS or mini-SAS requires specific cables that open from that mini-SAS to 4 regular SATA ports. The cable in itself is called SFF 8087 mini-SAS to 4 SATA cables. The regular SAS connector is looking like 4 SATA data ports put together in a single piece of plastic while the mini-SAS one is like a larger SATA. Like I said I was already familiar with some controllers previously so I was thinking of getting myself an LSI-based controller and finally I decided on an LSI 9260-8i. I wanted something as close to the original design as I have found some problems online with semi-custom designs like 9267-8i so I decided to get something as close as possible to the original design. These days they are very cheap to buy from China, the disadvantage being the relatively long waiting time for the transit. I also wanted a package that would include cables too, as the cables can get quite expensive, around 35$ each, so a big drag for something costing just 50 CAD shipped. So, when I found a package with the controller and the cables, that also mentioned that shipping was done with SPEED PACK, the Express shipping method from China, I jumped on the occasion. In the meantime, I have transitioned my data from the old Toshiba to a brand new 4TB Seagate Ironwolf HDD that had a 3-year warranty and data rescue services included in the price for 3 years, for your peace of mind. This drive is designed for 24/7 operation, always on, so it never parks the heads like the infamous WD Green in order to save energy. This feature is very important, as many users discovered that this feature is very understandable in always-on environments, as if the HDD parks the heads and there is a read or write request to the raid matrix, by the time the drive comes online it will be too late and the controller will mark the HDD as failing, compromising the integrity and performance of the raid.
The initial estimated delivery time was at least 4 weeks, but I was very happy when I got home in 3.5 weeks. For RAID 5 you need at least 3 drives to build it, but you will have available just the space of double the capacity of the smallest drive, if there are drives of different sizes, or if they are identical, just double the capacity of a single drive. The capacity of the 3rd drive is used to write that parity information used to reconstruct the data in case a drive fails, and is not written on a single drive but distributed equally on each drive, so if any drive of the 3 fails your data is still safe. ( RAID 6 just adds a level of redundancy spreading the parity twice on every drive, so you will need a minimum of 4 drives while enjoying the space of just 2 drives, but you could lose up to two drives at once while still keeping your data safe ).
By the time the controller arrived I got myself a used 2 TB WD Black drive, the only consumer/desktop-grade drive I have, due to its known reliability, and also a 4 TB WD Ultrastar, which is something used in data centers, it is mentioned on the drive cover. So 2 x 2 TB + 1 x 4 TB, so the resulting capacity would have been 4 TB max with the 3 drives in raid 5 mode.

So I started to dream of a RAID 5 matrix using used enterprise grade drives.
So I read up a lot on what RAID is and the alternatives. I wanted a hardware based solution, so that meant either an Intel motherboard with RAID 5 logic or a dedicated hardware controller.
For those that don't already know, what RAID 5 does, is spreading information on 3 drives minimum, together with the so called parity information, so if a drive fails, the data is still available, but the matrix will function with slower performance. If you replace the defective drive with a healthy one, the parity information from the remaining two drives will be reconstructed on the new one, thus re-establishing the full performance.
I was already accustomed with the process but never had the occasion before to get myself a raid 5 build. The menu in the Intel RST utility is very simple and intuitive but the parity information will be calculated by the host CPU taking quite a lot of cycles from it. Nobody really knows exactly how much as I haven't found specific reviews benchmarking the performance of a raid 5 setup on the onboard Intel raid controller, so I decided to go with a full hardware RAID 5 controller.

Modern raid 5 controllers don't have SATA connectors but something called either SAS or mini-SAS requiring specific cables that open from that mini-SAS to 4 regular SATA ports. The cable in itself is called SFF 8087 mini-SAS to 4 SATA cables. The regular SAS connector is looking like 4 SATA data ports put together in a single piece of plastic while the mini-SAS one is like a larger SATA. Like I said I was already familiar with some controllers previously so I was thinking of getting myself a LSI based controller and finally I decided for a LSI 9260-8i. I wanted something as close to the original design as I have found some problems online with semi custom designs like 9267-8i so I decided to get something as close as possible to the original design. These days they are very cheap to buy from China, the disadvantage being the relatively long waiting time for the transit. I also wanted a package that would include cables too, as the cables can get quite expensive, around 35$ each, so a big drag for something costing just 50 CAD shipped. So, when I found a package with the controller and the cables, that also mentioned that shipping was done with SPEEDPACK, the Express shipping method from China, I jumped on the occasion. In the meantime I have transitioned my data from the old Toshiba to a brand new 4TB Seagate Ironwolf HDD that had 3 year warranty and data rescue services included in the price for 3 years, for your peace of mind. This drive is designed for 24/7 operation, always on, so it never parks the heads like the infamous WD Green in order to save energy. This feature is very important, as many users discovered that this feature is very understandable in always on environments, as if the HDD parks the heads and there is a read or write request to the raid matrix, by the time the drive comes online it will be too late and the controller will mark the HDD as failing, compromising the integrity and performance of the raid.
The initial estimated delivery time was at least 4 weeks, but I was very happy when I got home in 3.5 weeks. For RAID 5 you need at least 3 drives to build it, but you will have available just the space of double the capacity of the smallest drive, if there are drives of different sizes, or if they are identical, just double the capacity of a single drive. The capacity of the 3rd drive is used to write that parity information used to reconstruct the data in case a drive fails, and is not written on a single drive but distributed equally on each drive, so if any drive of the 3 fails your data is still safe. ( RAID 6 just adds a level of redundancy spreading the parity twice on every drive, so you will need a minimum of 4 drives while enjoying the space of just 2 drives, but you could loose up to two drive at once while still keeping your data safe ).
By the time the controller arrived I got myself a used 2 TB WD Black drive, the only consumer/desktop grade drive I have, due to it's known reliability, and also a 4 TB WD Ultrastar, which is something used in datacenters, it is mentioned on the drive cover. So 2 x 2 TB + 1 x 4 TB, so the resulting capacity would have been 4 TB max with the 3 drives in raid 5 mode.

 

[MARSTG]

Modern raid 5 controllers don't have SATA connectors but something called either SAS or mini-SAS requiring specific cables that open from that mini-SAS to 4 regular SATA ports. The cable in itself is called SFF 8087 mini-SAS to 4 SATA cables. The regular SAS connector is looking like 4 SATA data ports put together in a single piece of plastic while the mini-SAS one is like a larger SATA. Like I said I was already familiar with some controllers previously so I was thinking of getting myself a LSI based controller and finally I decided for a LSI 9260-8i. I wanted something as close to the original design as I have found some problems online with semi custom designs like 9267-8i so I decided to get something as close as possible to the original design. These days they are very cheap to buy from China, the disadvantage being the relatively long waiting time for the transit. I also wanted a package that would include cables too, as the cables can get quite expensive, around 35$ each, so a big drag for something costing just 50 CAD shipped. So, when I found a package with the controller and the cables, that also mentioned that shipping was done with SPEEDPACK, the Express shipping method from China, I jumped on the occasion. In the meantime I have transitioned my data from the old Toshiba to a brand new 4TB Seagate Ironwolf HDD that had 3 year warranty and data rescue services included in the price for 3 years, for your peace of mind. This drive is designed for 24/7 operation, always on, so it never parks the heads like the infamous WD Green in order to save energy. This feature is very important, as many users discovered that this feature is very understandable in always on environments, as if the HDD parks the heads and there is a read or write request to the raid matrix, by the time the drive comes online it will be too late and the controller will mark the HDD as failing, compromising the integrity and performance of the raid.
The initial estimated delivery time was at least 4 weeks, but I was very happy when I got home in 3.5 weeks. For RAID 5 you need at least 3 drives to build it, but you will have available just the space of double the capacity of the smallest drive, if there are drives of different sizes, or if they are identical, just double the capacity of a single drive. The capacity of the 3rd drive is used to write that parity information used to reconstruct the data in case a drive fails, and is not written on a single drive but distributed equally on each drive, so if any drive of the 3 fails your data is still safe. ( RAID 6 just adds a level of redundancy spreading the parity twice on every drive, so you will need a minimum of 4 drives while enjoying the space of just 2 drives, but you could loose up to two drive at once while still keeping your data safe )

I would have gone with some dedicated OS for the data integrity on the drives but I decided that I want this second machine to hold my data and be ready to be used as a regular PC in the other room or to be used as a guest machine.
The system is a Dell Optiplex 790, and as the motherboard doesn't have RAID logic, it is a Q65 express chipset I had two options : change the motherboard and case or just change the case and get a dedicated raid controller. If I would have changed the motherboard I would have lost the Windows licence as well so I decided against that. So I got myself an old style case, the type that can hold many 3.5" drives and swapped the motherboard in the new case. The process was a pain, as Dell corporate PC lines have proprietary connectors and would report all the missing hardware, like front panel for example, or the temperature sensor for the HDD which is mounted on the faceplate, the exhaust fan and even the power button presence. If I would have known all this before I started the process I would have gone with a z68/H67/P67 mobo and just swap in the CPU and the RAM, as I have 16 GB and a i7-2600. In hindsight, a Z68/P67/H67 mobo would have also included the needed RAID 5 chip already so you wouldn't need the additional controller, but it is hard to estimate the performance loss of a dedicated processor to do the parity calculations which might impact the write performance of the matrix. Read performance would not be impacted, just the writing one.

Prior to the arrival of the controller, out of curiosity, I benchmarked the HDDs to have a rough idea of what they were capable of in terms of read/write speeds, and found out that the Black was the slowest drive, peaking around at 125 MB/s at writing, while the Toshiba DTACA was topping at around 150 MB/s with the Ironwolf being the fastest at 180 MB/s despite spinning at only 5900 RPM compared to 7200 RPM for the other Black and the Toshiba. I didn't test the 4TB Ultrastar, I just made sure there were no errors on it reported by the checkdisk utility, and actually I inspected all the drives with the chkdsk command to make sure they passed the mustard. And they did, no errors.
So, the big day arrived, the controller is here, in my hands, I am popping it in the second x16 physical PCIe port of the Dell ( just x4 electrical ) and ... panic. The controller does not appear anywhere in the device manager . I made sure that I had the drivers, the software utility and a newer firmware already available on my boot drive to continue the installation process. I was so bummed that I gave up on all the troubleshooting. My first x16 slot was occupied with the GPU, an old busted Radeon HD 6570 and, also I had a 1x slot with a USB 3.0 card, as the system had only USB 2.0 and the available x4 2.0 slot in x16 format. The mobo has a Display Port and I had a DP to HDMI adapter to use, in case I had to ditch the GPU for troubleshooting purposes, but as it turns out, when I swapped the motherboard in the new case, I reused the I/O shield as well but the DP port was not exactly fitting perfectly so the I/O shield is blocking the connector by 1-2 mm making impossible to insert the DP to HDMI adapter. Bummer. Free advice : if you plan to use all options on a corporate mobo and you swap it in a regular PC case, don't use the I/O shield, it will be a PITA later. I had a x1 to x16 plastic riser, so I got the USB 3.0 card out and for the controller on the riser and stuck it on the x1 port. Still no change. Frustration was settling in. I was telling myself : I should have gotten that Z68 I saw locally on FB Marketplace. I was so bummed that I was ready to contact the seller to let him know I have a DOA card on my hands and try to solve it amicably. I even got a second GPU in the x4 slot and it was detected fine by Windows so the port was not the issue. I got so bummed that I even got the card out and put it back in it's plastic clamshell that arrived into from China. I even went to the recycling bag to get the return address, and was preparing to go back at the post office to return it. A Mississauga adresse was indicated for parcel return. I just told myself : I am finding it difficult to believe that LSI or IBM would have a DOA card. So in a last attempt to troubleshoot I took the GPU out and installed the controller in the first x16 slot. And, a miracle happened, the card was detected from the first boot, it's firmware was indicating the combination I had to input to enter in the bios of the raid card. The bios was having a 2011 build, so already 10 years old and it has a terrible old AMI BIOS like interface called Web bios. Initially these cards were made by 3ware, then they were bought by LSI then by Avago and then by Broadcom and the Broadcom web page with the drivers was useless for me, as it was a ftp server and Google and Mozilla have deactivated FTP browsing by default on their recent updates, so I found the drivers, RAID utility called MegaRaid Manager on a German website.

 

[MARSTG]

So, I got the drives installed, the utility installed, and got inside the firmware of the card. In that old bios the raid 5 stripe ( the basic data block ) was only 64 KB. I hastily build the raid 5 matrix and I had 3.6 TB of space available. I started to test the speeds by transferring a data folder with large video files, around 9 GB and found out that after the first few seconds of hundreds of MB/s, the transfer speed was dropping to 35-40 MB, and I was like WTF ? Only 40 MB/s ? That doesn't sound like server grade speeds. There are some important parameters when you choose to build the initial matrix like stripe size and disk policy. There are only 3 policies but they affect dramatically the performance of the setup or the integrity of the data during transfer. The disk policies are : 1) write-through 2) write-back and 3) always write-back and I will explain them succinctly.
The advantage of the hardware raid controller is that the parity calculations are done by a dedicated processor on the card itself, and, like any realtime processor it needs some local RAM to write the temporary values of the calculations locally. While the data is transferred the parity calculations are written to this local memory, which in the case of this controller is 512 MB of onboard cache. The write-through process means that the data copied to the matrix is writing all the information in real time to the drives, so if there is a power outage in the process the data doesn't get corrupted. In the write-back mode, this parity data is stored into the local cache and written to the drives after the process ends. However if a power outage happens now the data will be corrupted. This process can be mitigated by adding a battery to the raid controller, called BBU ( battery backup unit ) which, in case of a sudden power outage maintains the parity calculations in the local cache memory of the card until power to the system is resumed. I don't have a BBU now so I chose the write-through mode and the performance was what it was. So I destroyed the partition created and got in with the included software utility to play with that. First thing I did was to upgrade the firmware which allowed me to build volumes where the stripe would be up to 1MB in size. So I went with that and also enabled disk policy number 3, always write-back. And WOW : what a difference it made ! Writing to the new volume was peaking at 270 MB/a now, transferring 9 GB of data was a breeze. I installed HD Tune Pro which allows you not only to benchmark the read speeds for the whole volume, but, if you remove the partition you can also test the writing speeds ! So I did exactly that! I will attach the screenshots.