The Big WHS: May Update 60TB Edition

The Big WHS was originally supposed to house approximately 30TB of storage when the plans were first detailed on an Excel spreadsheet BOM in December 2009. This was a big upgrade to my first DIY Windows Home Server box that had well under 20TB. About five months later, the storage capacity has crested 60TB, with further room to expand. The Big WHS now spans two 4U Norco cases (using a total of 8U of rackspace and another 4U chassis is in the works) has over 60TB of storage, and requires well over a dozen ports on the gigabit switch.nvidia quadro m2000

The current hardware list is as follows:

1. CPU: Intel Core i7 920
2. Motherboard:  Supermicro X8ST3-F
3. Memory: Patriot Viper 12GB DDR3 1600
4. Case (1): Norco RPC-4020
5. Case (2): Norco RPC-4220
6. Drives: 12x Seagate 7200rpm 1.5TB, 2x 7200.11 1TB, 12x Hitachi 7200rpm 2TB and 2x 1TB, 8x Western Digital Green 1.5TB EADS, 2x Western Digital Green 2TB EARS.
7. SSD: 2x Intel X25-V 40GB
8. Controller: Areca ARC-1680LP
9. SAS Expanders: 2x HP SAS Expander (one in each enclosure)
10. NIC (additional): 2x Intel Pro/1000 PT Quad , Intel Pro/1000 GT (PCI)
11. Host OS: Windows Server 2008 R2 with Hyper-V installed
12. Fan Controllers: Various
13. PCMIG board to power the HP SAS Expander in the Norco RPC-4220
14. Main switch – Dell PowerConnect 2724

Changes and Future Plan

The configuration changes have had a few drivers:

1. I have been using many more Hyper-V virtual machines on the server lately. 8-10 VM’s run simultaneously on the box these days.
2. Storage growth is approximately 1.2TB/month due to additional witness preparation footage being shot, all in HD. The acceleration of storage growth has required space for additional drives.
3. The two Norco cases are LOUD with the stock fans with the RPC-4220 being significantly louder than a RPC-4020. I found that running the stock fans at 75% provided enough cooling while significantly reducing noise.

   

4. Since I generally like to give each Hyper-V VM its own gigabit NIC, I added the second Intel Pro/1000 PT Quad and an Intel Pro/1000 GT. The GT is a PCI card so it works fine in the Supermicro motherboard.
5. The second enclosure required SAS connectivity, so I am using the Areca 1680LP’s external port to connect to the HP SAS Expander’s external port. In the second enclosure, I am using a PCMIG board to power the HP SAS Expander so I can house a second server based on an Intel Xeon X3440 and Supermicro X8SIL-F.

   

6. The second 4U houses a Supermicro X8SIL-F with an Intel Xeon X3430. The majority of the drives in the chassis, and the HP SAS Expander are on one power supply while this secondary server is powered using a second PSU. This second mini server gives me a test bed for Hyper-V VM migration, VMWare ESXi appliances, and bare metal installs for things like WHS V2 codename VAIL. Having this test server allows me to test configurations and changes before running them on the Big WHS. Since it is in the secondary DAS Enclosure of the Big WHS, it allows me to re-allocate a row of disks by changing the termination point of one cable.
7. SSD’s were added to be used as basic cache drives, either for higher I/O VM’s (my SQL Server) or just to use to benchmark other VM’s. I can imagine adding additional SSD’s in the future and they may be added to the third 4U that is in progress.

So those are the basic changes at this point. Nothing fancy, but I am probably done buying drives until the 3TB hard drives come to market later in 2010.

I am not done yet though. There are a few things I have been meaning to add:

1. Supermicro X8SIL-F plus a Core i3 530 or Xeon X3440 in the second 4U (I did order a X3460 today also). Will need to have a separate power supply, but I would like to house a second CPU and more RAM in the second Norco enclosure. This will let me migrate some VM’s. I also have the opportunity to utilize the few unused hot swap slots in the Norco RPC-4220. Finally, I want a platform to test WHS V2 codename VAIL on a bare metal installation, and want a VMWare ESXi machine. Another use is that I want something that can host client Virtual Machines before putting them on the Big WHS.
2. 10BaseT Ethernet… at some point I want to upgrade the network. A big factor here is that the switch costs are very high as are the costs for the cards. Even with 2x SSD’s in Raid 0 on SATA II read speeds are below 600MB/s. This means that there is very little opportunity to use full 10BaseT 10 gigabit bandwidth between single clients and the server using a direct CAT 6 run.
3. I am thinking of utilizing the Norco RPC-470 from the first WHS to be a third 4U enclosure for holding optical drives, SSDs, and another virtual machine farm. I am more sold on the idea of it being an optical media expansion/ SSD case than I am of it being yet another virtual machine box. On the other hand, I am biased at the moment because I just lost an entire physical stack of installation DVD’s two weeks ago. The time involved in validating the non-operational status of those disks has been time consuming. I’ve started burning in the new 4U and am actually connecting it to a Silicon Image based eSATA 4-port card (I just wanted to see what this would be like).

   

4. Brace for the impending drive failures. With almost 40 drives spinning, I know that failures in the next few months are inevitable. Therefore, I need to start planning today for the replacement of disks in the near future. That is actually the reason that I am running more spare capacity than normal.

Finally, the first WHS was dismantled this weekend. Its Norco RPC-470 is being used as a test server (see above) and will most likely become a third 4U for the Big WHS. The Core 2 Duo 6420, Gigabyte motherboard, DDR2 memory, and Adaptec controllers will most likely be sold or given away.

That wraps up this edition of the Big WHS update. Stay tuned to see what happens next! It should be exciting as WHS V2 codename VAIL gets closer to release.