Since many of us are on Unraid and have limited drive slots and/or PCIe slots, I wanted to look into the idea of getting cheap(er) flash storage into our NAS. Unraid’s 6.9 will enable cache drives/pools to be used for individual shares which really has broader, positive implications.
One of the biggest downsides to a spinning HDD is that random IO isn’t great; well, if you have a share for Movies, for example, that generally receives one movie at a time (write) and some nominal amount of read-time on the new items added, you could consider isolating a single spinning HDD to be the cache drive of the Movies share. Same with TV, Videos, Photos, etc. That’s just one example where previously you’d have to use your cache drive with multiple shares which increases the likelihood of random IO bottlenecks.
Enter Flash. Many of us have gone down the path of using an SSD (regardless of its general kind like SAS, SATA, NVMe, etc). Some of us have gone the route of 1x 128GB SATA SSD for AppData, 2TB SATA SSD for a general cache across all shares. Some of us have gone the path of insanely high IOPS + a PCIe adapter card to throw everything onto a single 1.6/3.2TB NVMe. Flash opens up these options. However, not all flash storage is equal in price and most flash storage still takes up a SAS/SATA slot. It got me thinking about bifurcation of x8/x16 PCIe slots and the falling prices of NVMe drives. At times, you can find a 1TB NVMe drive for $70-80 USD. These drives often have better endurance than your similarly priced SATA SSD and don’t require SAS and don’t take up a hard drive slot (depending on your chassis and configuration). Note: I know some folks like to double-sided-tape their drives to their chassis. That does work in some cases but the more enterprise grade chassis like SuperMicro’s 836 don’t offer a ton of flat space to do this.
So this brings me to the interesting price-per-TB of flash storage coupled with dual or quad NVMe PCIe add-on cards. This unique combination results in the ability to pack TBs of NVMe storage into one PCIe slot, sufficient for the bulk of our use-cases in a home NAS and offers the ability to price hunt and expand over time. As of writing this, the 1TB NVMe are between $90-125 for a reasonable drive. But the inspiration for this came during Black Friday in 2020 when 1TB NVMe drives that are normally in that aforementioned range were dipping into $70s. At $75/TB, going NVMe is almost entirely upside compared to a similarly priced SATA SSD. Compared to a U2 NVMe, which can often be found in a similar price point, it’s still better in that you can get multiple NVMe drives on one PCIe slot. That’s really the magic here; higher density, better price hunting, and easier upgrading. If you start with 4x512GB NVMe and want to boost up the 4th to a 1TB due a great sale, you can. Each drive will show up as its own drive in the system which for our use-cases is another good thing. 1 for AppData, 1 for Movies, 1 for TV, 1 for everything else or VMs, etc.
On the topic of speed and bifurcation
Bandwidth on PCIe slots is sufficient. You will limit your NVMe speeds depending on the number of lanes. For example, PCIe v3 has a per-lane bandwidth of 985MB/s; x8 is 8 times that. So if you bifurcate and x8 into 2 x4’s, each drive will get a max of 4*985MB/s. Bifurcating an x8 into x2 gives you just under 2GB/s. While some NVMe could hit speeds of 3.4GB/s, I felt that anywhere from 1GB/s+ is more that sufficient for our use-cases. And you get the great IOPS. You can definitely bifurcate an x16 slot, if you have one, but I’ve found that often on the kind of server motherboards we’re all using we run into one of two issues: (1) lack of x16 slots or (2) x16 being used for a GPU.
Motherboard support for bifurcation
Not all motherboards support bifurcation. If your motherboard does support it, you can get dual and quad NVMe cards for cheap ($50). If your motherboard does not support it, or you just want to rely on a secondary chipset for whatever reason, the cards hit the $150-200 range.
Links from my research
- One of our commonly used boards, users report bios 3.3 bringing bifurcation
- Some of the products I have yet to dig deeper into
- QNAP’s 4x NVMe Built-in bifurcation chip
- SuperMicro’s AOC SHG3 4x NVMe Built-in bifurcation chip