QSAN XN8012R Storage Review: Virtualization + ZFS Onboard!

Today, many technical experts believe that ZFS is something that works only under FreeBSD, in clusters, with configuration from the command line, and it is better for mere mortals not to meddle there, because this is the lot of the elite. In fact, no, and even the ZFS On Linux port has already reached the level of stability when it is used in scalable production-grade hardware solutions. And many sysadmins of small organizations quite clearly understand the advantages of ZFS and can justify to the management the savings from software LZ4 compression and deduplication when buying a new storage system, but there were no inexpensive solutions for this file system. But a holy place is never empty, and QSAN decided to combine everything that we love in Linux and everything that we appreciate in ZFS in one device.

Why do we love ZFS? For snapshots, deduplication and absolute reliability, given to us by Oracle. Why don't we love you? For FreeBSD, which can do one thing, but got stuck in its development somewhere in the first decade of the 21st century, which means it is completely unsuitable for NAS.

Why?

Yes, because a modern NAS is not only a file storage, but also an application server that runs containers, virtual machines, databases and proprietary programs. Few NAS manufacturers are ready to develop software for FreeBSD, including because of licenses, and for Linux - please: here you have a debugged hypervisor, an excellent situation with drivers and a regularly updated kernel.

This is probably what the developers at QSAN, the oldest manufacturer of entry-level storage systems, thought. They decided that modern processors are already ready to handle all the delights of ZFS: memory is getting cheaper, SSD drives too, which means you can make a NAS, which has no analogue on the market. Previously, we have not tested QSAN products, and the more interesting it will be to get acquainted with the idea of ​​its authors: here everything is different - a different ideology, a different design ... well, let's not drag it out.

This is a single-controller NAS based on 3.3 GHz Quad-core Intel Xeon processor, The head unit has 12 bays for 3.5-inch SATA drives, plus 4 separate bays for SATA SSDs, plus 2 bays for NVME SSD 2.5, plus 2 bays for boards expansion, and this is only with regards to the `` iron ''. As for the software, the manufacturer promises tiering, SSD caching support, built-in virtualization, backup to the cloud, and all the delights of ZFS, including compression, deduplication and renowned reliability.

Design Features

In order to fit 12 large-capacity and 6 high-speed drives into one chassis, the developers of QSAN XN8012R placed SSD bays on the back of the chassis. This is a very daring solution to make the most of space in a 2U case, but it has some peculiarities. The fact is that drives installed at the outlet of the air flow tend to overheat, and to prevent this from happening, QSAN installed a separate fan on their basket.

Moreover, if cooling may not be necessary for 2.5-inch 7-mm SATA SSDs, then for 15-mm NVMe SSDs, of which there may be two, this is already a prerequisite for operation.

Two types of SSDs are needed to fine tune Tiering: Regular HDDs represent the lowest performance tier, SATA SSDs are medium, and NVMe SSDs are the highest. For each layer, its own RAID array can be used, and it is the RAID 5/6 that is understandable to everyone, and not the mysterious RAID Z1/Z2/Z3. The use of standard terminology can only be rejoiced, but more on that later.

I don't know who came up with such a serious NAS to ship with 1 Gigabit network interfaces. And look - a USB 3.0 port is installed under each of the 4 RJ45 ports. To say that this is inconvenient is like saying nothing, fortunately, the storage system has two low-profile expansion slots: PCI-E Gen3 x8 and Gen3 x4, in which we will put two 10-gigabit network cards, Intel X550-T2 for copper and Intel X520-DA2 for DAC. Although QSAN recommends using its own adapters (judging by the photo - re-labeled Intel), we had no problems with the native Intel, but God forbid you install a board with RJ45 ports in the left slot! Since the expansion bays are recessed into the case, your fingers won't get through the latches of the RJ45 chips, and removing the network cables without tweezers will be oh, how difficult.

Fortunately, this is the only design flaw found. Otherwise, everything is standard: a fault-tolerant power supply from Delta, an N + 1 cooling system, convenient disk trays and ... even an HDMI port. For Apple users, the company offers a Thunderbolt 3 controller, and a SAS-12Gb/s controller for connecting disk shelves. By default, the XN8012R comes with 8GB of RAM, but the memory can be expanded up to 64GB. The Linux-based operating system is not memory intensive, so adding RAM is only worth adding if you use deduplication and the built-in hypervisor. Well, while we're on the subject of software, let's start and configure the NAS.

Software and virtualization

To put it mildly, QSAN was inspired by Synology products when developing the NAS operating system. In the world of Linux software, there is nothing shameful about this: companies use OpenSource libraries and borrow ideas from each other, the only question is who will do something new or something better than others, and QSAN turned out... cool: HTML5, control panel, widgets, taskbar, monitoring the current load... Work,  just use Google Chrome, otherwise, under Firefox, not everything is displayed as needed.

If your company used FreeNAS, XigmaNAS or just one of the FreeBSD clones for the sake of ZFS goodies, then you probably lacked the built-in file manager: it is convenient to copy any amount of data inside the NAS without using a network interface. The FreeNAS developers reacted with hostility to the community's requests for a file manager, and the FreeNAS community administrator wrote that the Web file manager is the "dumbest idea"? Actually, QSAN QSM is an operating system with ZFS support and a normal web-based file manager with preview images, with basic functions (copy/delete), but this is not the only thing that distinguishes a commercial Linux product from the free FreeBSD.

Not so long ago, the Bhyde hypervisor was screwed under FreeNAS, but it has compatibility problems with any guest operating systems, including FreeBSD itself, and under Linux you have QEMU, the de facto industrial standard with proven VNC libraries and drivers which supports Linux and Windows equally well as guest systems. Of course, under Linux there is the Proxmox hypervisor, beloved by many system administrators, but there is no storage management and virtualization from a single window, as in NAS.

Of course, the presence of built-in virtualization makes it pointless to integrate many other packages that are not directly related to the NAS activity of the device, so QSAN does not bother with this direction. And although there is also a SQL server (MariaDB, and a VPN server), from my point of view, perhaps the antivirus that scans the contents of the NAS on a schedule deserves attention...

When it comes to backup, there are some very interesting features. First, as mentioned above, you can use Snapshot copies as an alternative to local backups. In a ZFS file system, Snapshot copies contribute to the total amount of data deduplicated, so in terms of copy storage costs, ZFS Snapshot copies are the best choice today. For large infrastructure, it makes sense to set up mirroring of the target folder on the NAS to a remote NAS (Qsan Xmirror Technology) or backup to the cloud. If you want to back up the contents of your workers' Windows computers, Qsan suggests using the Xreplicator program, which Acronis developed based on its True Image with a free license for 100 machines, and for Linux hosts it uses Rsync backup.

However, there are drawbacks: for example, there is no mechanism for backing up virtual machines for VMware ESX; as a cloud storage for backups, which we strongly recommend that you do, only three API providers are supported: Amazon S3, Hicloud S3 and Alibaba Cloud SS. There are also complaints about working with browsers that are not on the Chrome core (in particular, some buttons are not pressed in the interface under Firefox), but basically everything works flawlessly.

Disk Pools

As mentioned at the beginning of this article, QSAN XN8012R does not use ZFS terminology such as Zvol/Zdev, which even experienced sysadmins are confused about. Everything is traditional here: RAID, disk pools, shared folders, but with a ZFS kernel. For example, if you have an array for a large number of disks, the RAIDZ3 array will come in handy, which allows simultaneous failure of 3 hard drives. It is now considered good practice to constantly test hard drives for early detection of failures, and QSM has an S.M.A.R.T. tests, scanning the entire array (Scrub), as well as support for Seagate Ironwolf Health Management.

When creating a disk array, you have to choose what kind of load you will have: streaming or transactional, or maybe mixed? You also need to select the level of this array in a multilayer Tier-structure. For hard drives, this is the lowest level, and for SSDs of different speeds, it is medium or highest, as well as a data migration mechanism: on a schedule or manually. Take this stage responsibly, as further changes will be impossible.

Tiering

The Tiering function works as follows: the file system constantly records the frequency of block access, and then transfers data in layers, what is rarely used is stored on the HDD, and the most popular data is stored on an SSD or NVMe SSD. At the same time, the SSD layer is the same array as your disk RAID, on which you can create folders and iSCSI volumes. Since you can use it like a regular array, its size can be safely added to the amount of HDD storage, and this is a key difference from an SSD cache. Using Tiering, you specify a storage policy for each disk array when you create it. For example, what is higher in terms of their level can be transferred (from SATA SSD to NVMe SSD), but below it is impossible. The transfer of data from HDD to SSD itself is carried out on a schedule, for example, at night or on weekends.

When Auto Tiering is enabled the operating system independently determines the storage policy for the disk array, as well as the time for transferring data between levels. But it should be borne in mind that data usage statistics can accumulate for a very long time, and what you would like to see on an SSD can be dead weight on the HDD, so do not rely only on Tiering, but actively use the SSD pool for data that is important for low delay. In general, in an Enterprise environment, it is customary to pay extra for the Tiering function, and we are especially pleased that QSAN has multilayer storage included in the cost of storage.

SSD cache

ZFS has native support for SSD caching (L2ARC), and you can even use one drive for read/write caching. The cache warms up quickly enough, but the SSD does not reach its peak performance, so perform high-intensity operations with data on SSD arrays. For SSD cache, you can use any number of pooled drives. If you have only the read cache involved, then keep in mind that it will be reset to zero when the machine is rebooted. In general, the SSD cache is filled in blocks, and you can estimate the efficiency in the diagrams below. Please note: the access speed does not exceed 2500 IOPS, so for high-intensity workloads, an SSD cache over HDD will not replace a regular SSD pool, even in read mode.

The presence of an SSD read cache is a prerequisite for using the block deduplication function.

Block deduplication

This excellent ZFS feature works on the fly, saving you tens of terabytes of data. When writing data to disk, the storage processor finds duplicate extents and replaces them with links pointing to the block address, due to which duplicate data does not take up disk space, and is not rewritten at all. As a result, you not only gain in data volume, but also increase the speed of random reads and writes. If your storage system serves an average organization and works as a repository for backups, virtual machines, ordinary document archives, then here's how much you can save:

• Backups without software compression (usually - Rsync): from 85% to 90%
• Virtual machine disks: 50% to 90%
• Database dumps: 45% to 70%
• User home directories: 20% to 50%
• Mail archives: from 20% to 40%

The more data your organization has, the greater the benefit will be. You can enable shared deduplication for folders and iSCSI volumes that use a file as storage. In general, deduplication should greatly reduce the write speed to disk, and on the graph we can see that the speed of simple synthetics is noticeably reduced.

But in the patterns of real tasks, deduplication, on the contrary, gives a noticeable increase in speed.

When testing the access speed on 64-KB blocks, a surprise awaited us: the performance was higher than that of 4-KB blocks. I had to double-check the tests, but the results were repeated, and then I turned to the storage manufacturer for clarification. The engineers reported that these storage systems are written in 64-KB blocks to improve performance in most tasks. In general, such speed is just a gift for those who buy storage systems for use under Microsoft SQL Server, because it is for 64-kilobyte access recommends configuring storage itself by Microsoft .

So without a doubt, for transactional access in a small organization, the QSAN XN8012R is a great choice. What's more interesting is support for WORM (Write Once Read Many) technology, which in simple language means write protection for some resources. This protection can be installed for a limited time, or forever. It is very convenient for backups: you have saved a backup copy to the storage system, and you can already be sure that neither ransomware nor a hacker can delete or change your backups.

To test the speed of serial access, we used 2 ports of 10 Gb/s, but the indicators did not exceed the capabilities of one. In our test configuration for 3 HDDs and 3 SSDs, you will not go too far, but for large installations it is possible to connect up to 8 disk shelves.

Here you need to keep in mind that you have no fault tolerance: judging by the diagram, the shelves are connected in series, without forming a loop, so let's say if your 5th shelf fails, then both the 6th and 7th will turn off -th, and 8th. Well, in general, when you connect 8 QSAN XD5324 shelves, each of which has 24 bays for 3.5 HDDs, you can fill the entire storage system with 14 terabyte hard drives to get 2.83 with 210 hard drives with a capacity of 2886 terabytes of space for your needs.

Cost and licensing

The cost of the QSAN XN8012R head unit without hard drives is $ 6250 . Please note that by default the head unit is shipped even without a rack rail, with a minimum amount of memory and 1 Gigabit interfaces. The deduplication feature in ZFS heavily uses RAM to store extent tables, so if you want to save on the number of hard drives, please configure the storage system with at least 32GB of RAM. All software functionality such as Tiering, virtualization and snapshots does not require license activation.

Warranty

QSAN XN8012R comes with a 3-year limited warranty by default, which the customer can extend for another 2 years by purchasing the appropriate package at any time during the main warranty period. At the time of writing, QSAN did not have service packages with limited response times such as next business day replacement or onsite visits.

Conclusions

Today, every storage system supports SSDs, but not every one has 6 Hot-Swap SSD bays that you can use without compromising the HDD. Today, every storage system has support for Snapshot copies and SSD cache, but not all support multi-layer storage on NVMe SSDs with hot swap. Features such as Inline deduplication are the main criteria for choosing a storage vendor, and practically determine the choice of one or another solution. And if you choose QSAN, then this storage system will meet you with a friendly HTML5 interface, made according to the canons of traditional data storage systems. Everything here is simple and straightforward, so even a child can figure out how to set up a server, run virtualization and configure user access rights. The storage system has built-in security systems, including antivirus and Firewall, so that it can protect itself from any misfortunes.

Yes, of course, everything that goes beyond the ordinary has both its pros and cons: an unusual design causes difficulties when connecting cables, there are problems with the web interface under the Firefox browser, and 8 GB of RAM, and even typed in two 4GB modules look frivolous.

But it is not the first year that we have been talking about the fact that NAS-s with block access due to their versatility will push SAN solutions out of the initial price range. And here you are, the cost of the top-end QSAN XN8012R with virtualization and deduplication is comparable to the price of 2-controller SANs, which can do nothing but be 2-controller SANs, which means you can add memory, and run the missing software under a virtual machine ... In general, with a competent approach, this storage system will significantly save on expanding capacity as the volume of data in your organization grows, and it is this feature that makes you look closely at the XCubeNAS series today.

Mikhail Degtyarev (aka LIKE OFF)
08/28/2019