Storage and backup of virtual machines on QNAP NAS

In 2019, the world market for backup and data recovery software, according to estimates by Allied Market Research, amounted to $ 9.3 billion. An article of backups is a huge budget for any modern IT company, and every year the development of technology sets new requirements for IT directors, increasing deductions for data backup. Today we see two new directions of the IT industry: development of edge computing and the transition of staff to work from home due to the fact that after the epidemic, many companies realized that employees do not have to come to the office.

In general, we just have to realize how these two areas are being integrated: on the one hand, put servers right on the rig for processing Big Data, and on the other, decentralize all personnel by providing them with VPN access to the corporate network, or better - VM with RDP access and configured security policies. It remains only to understand what solution will take over all the work with data at once: in this case, hyper-converged infrastructure is unreasonably expensive both in purchase and in content, unconsolidated data storages are unreliable, and NAS, depending on the "caliber", can be not only that the place where the raw data is merged telemetry with a myriad of IoT sensors, but it itself can become a platform for virtualization. You can run powerful virtual machines on these devices using a convenient web interface. If you need to install compute servers - QNAP can work for them as iSCSI/NFS storage. If the site uses video surveillance, QNAP can act as a video recorder, both with built-in software and with any third-party (Macroscop, ZoneMinder) running in a virtual machine.

Using 16-terabyte hard disks, in the head unit alone, you can get 256 TB of RAW-capacity (which is 240 TB in RAID 5 or 214 TB in RAID 6), which is more than enough for all the needs of a remote office or entirely small enterprise. And even in the context of disputes about which is better - consolidated storage or hyperconverged infrastructure with software storage, QNAP does not argue with anyone, but offers its own view of HCI: "while you are arguing how to organize data storage in your infrastructure, think about it Is it really better to move the software infrastructure itself to the data warehouse?" Of course, someone will laugh at the idea of ​​running large Oracle databases on NAS-ah, and someone will calculate the real power consumption of virtual machines, the price per terabyte of data and understand that for small and medium-sized companies such an ideology of "use NAS for everything" ; translates into huge savings. And those who yesterday threw out servers, moving resources to the NAS, today cancel their subscription to the software, whose functions are performed by QNAP, as they say, out of the box. And now the time has come when QNAP aimed at sacred: virtual machine reservation.

QNAP recently introduced its proprietary software tool for saving and restoring virtual machine backups from VMware ESXi and Microsoft Hyper-V servers. Without stopping, with the ability to track changed blocks on the virtual disk, with the ability to coordinate the snapshot with the application and a modern Web interface. This solution is intended not only to save the company money, but also to consolidate the creation and storage of backups in one device, and we will now check how it works.

Reservation without stopping

At the heart of all modern means of backing up virtual machines is the mechanism for creating snapshots (snapshots) of the state of a VM, followed by saving the image of a virtual disk to third-party media. From a practical point of view, the hypervisor does everything itself through the API: it creates a snapshot and sends data to the backup device, so the latter's task is to send commands on a schedule and organize everything related to storing, backing up and compacting backups.

The process of creating snapshots itself interrupts the virtual machine for 1-2 seconds, and is native to virtualization. Since it is completely controlled by the hypervisor, there is no way to optimize or improve it. All software solutions that copy virtual machines on the fly, whether paid or free, use the same API.

VMware has a technology for tracking changed blocks on hard drives, and this is how it works: a regular virtual disk is a file whose size can be 100 and 200 GB, and the disk can change 1-2 GB per day. The backup system cannot access the contents of the virtual hard disk and make an incremental file-by-file backup, so the hypervisor takes over this task using Changed Block Tracking (CBT) technology. After creating a snapshot, a xxx-ctk.vmd file of a fixed size appears in the virtual machine storage for each vmdk virtual disk (there is about 500 KB of a ctk file per 10 GB disk): this file contains a list of blocks that are monitored for vmdk with their actual state from a certain point in time, fixed in ChangedId (changed or not). The next time you back up, QNAP Hyper Data Protector will copy only the changed data, drastically shortening the backup time frame (see our article 5 metrics efficiency of corporate backup ).

Please note that CBT is not enabled by default: you can manually enable it by following the VMware instructions and Microsoft , or if you give QNAP appropriate access rights, it will do it for you.

The second step in maintaining consistency on the part of QNAP is the use of technologies for matching redundancy with applications. It is needed so that QNAP can copy the files that the application is currently working with in the state in which they are used by the application. During the creation of a snapshot of the virtual machine, a short-term "freeze" of the guest OS file system occurs with the reset of caches, thereby ensuring data consistency. Support for these technologies requires that guest operating systems have the latest versions of VM Tools from VMWare or Microsoft installed. We recommend that you always enable this option.

But what is missing is the direct interaction between QNAP Hyper Data Protector and programs within the guest operating system. It would be very helpful if QNAP could issue a Flush Tables command to the database before the backup process, or save a file that is open in the editor to disk. There are such solutions, and I hope that they will appear here too.

AMD NAS

QNAP uses the EXT4 file system in its main line of NAS-s, which has established itself as the most reliable (on a par with NTFS), but at the same time does not have fashionable functions like Copy-On-Write. At the same time, the company began to use AMD processors in modern corporate NAS-s, which I fully welcome. And do not think that the developer is installing desktop CPUs in the Enterprise sector: our test QNAP TS-2477XU-RP has a powerful 8-core AMD Ryzen 7 Pro 2700 processor with a frequency of 3.2 to 4.1 GHz. This processor has hardly been seen in retail, and since it is not susceptible to speculative instruction execution vulnerabilities...:

• ... you will have to reboot your NAS less often to install security patches that are released on Intel platforms with sad regularity.
• ... your NAS has no software limitations that will kill random access performance.
• ... you are not afraid of attacks to increase the access level of virtual machines, so you, as a sysadmin, can just sleep peacefully.
• ... and with such a great processor, deduplication works on the fly during the transfer of data to the NAS.

Ryzen Pro processors also have a mind-blowing ability to encrypt RAM on the fly. For those users who prioritize security concerns, this means that it is useless for an attacker to try to carry out a Cold Boot attack in order to gain access to the cache and snapshot of the device's memory. This technology is discussed in great detail in our article Studying memory encryption in AMD processors  (it refers to AMD EPYC processors, but the Ryzen Pro is the same). By the way, since we are talking about encryption, QNAP has the most useful ability to save copies of virtual machines in encrypted folders, and at the time of writing this review, competitors did not have such a function, and its importance cannot be overestimated.

So that your backups can be protected from prying eyes in case, for example, you have to go to warranty repair and return the device along with the disks. Speaking of disks, the QNAP TS-2477XU-RP we are reviewing has 24 bays for 3.5"/2.5"; HDD/SSD, but no dedicated slots for NVMe. In this case, this is not a problem, because QNAP has very interesting adapters that combine a 10-Gigabit 10GBase-T port and two slots for SATA M.2 cards on one board, and there are also dedicated adapters for NVME SSD, but without network port. We talked about these expansion cards in our reviews QNAP TS-2483XU-RP .

SSDs can be used not only for write/read cache, but also for organizing tiered storage (Tiering), and this feature is basic in all QNAP devices, and is available without purchasing additional licenses. If you buy a NAS solely for backup, then SSD cache or tiered storage will hardly give you anything, but if you also use consolidated storage for your services and applications, or run a VM on the NAS itself (and the fast Ryzen processor 5 Pro has this), then of course it is better to install a board with M.2 SSD (read how virtualization and tier-ing works in QNAP NAS ).

Breaking free from Intel’s yoke, QNAP was given complete freedom to choose network interfaces for 10 Gigabit networks, and of course chose the best that is available on the 10 Gigabit motherboard market today: the Mellanox Connect-X 4 LX controller with RDMA support , iSER, RoCeV1/V2 and hardware offloading of the largest number of protocols available on the market today. This card is able to offload the traffic of virtual machines, encapsulated in VXLAN and NVGRE. Recently, the QNAP assortment has added dual-port Fiber Channel interface cards for 16 and 32 Gbps, respectively, which you can purchase yourself.

In general, a powerful processor is a necessity here, since resource-intensive functions such as compression and deduplication are available to reduce the size of the stored virtual machine backups. If your virtual machines use their own disk encryption at the operating system level (for example, Bitlocker or Veracrypt), you do not need these backup compaction functions, and you can turn them off. And what impresses QNAP is the ability to select these options individually for each folder for saving backups, which is convenient: we copy monotonous unencrypted virtual machines into one folder, and those that use Bitlocker or Veracrypt into another, thus reducing the CPU load, you can save reservation time frame within the limits of your organization.

Connecting to VMWare vSphere

QNAP itself recommends connecting the NAS not to the hypervisor itself, but to the vCenter management environment: this is understandable, because hosts can be added or disappeared from the infrastructure, and virtual machines can endlessly roam between servers and storage resources. Once you connect the NAS to VCenter, you will avoid the hassle of reconfiguring servers in the future: all virtual machines become available to you immediately. By the way, when starting the backup, it becomes clear that the process of creating snapshots is controlled by the vCenter itself, and QNAP Hyper Data Backup simply communicates with this platform through the API, and does not directly interfere with the work of virtual machines.

Let's see how redundancy works in different scenarios for using virtual machines. We used the following infrastructure for testing:

Restoring a virtual machine is only allowed to an existing host to existing storage. That is, despite the fact that QNAP TS-2477XU itself is a hypervisor with a powerful virtualization system, it cannot restore a VM on itself, but what is even sadder is that it cannot quickly connect a new volume via NFS as a "storage", so as not to transfer the entire volume of the virtual machine over the network. And there is a reason for this: the recovery process goes through the import of the OVF template. Such a solution has a big plus - compatibility with different versions of hypervisors, and minus one - the slowness of the process, so it can take an hour or even more to restore a 40-gigabyte VM.

Basically, if the backup window allows, you can restore to test hypervisors to check the backup performance, and we recommend using this option periodically, because there is no other method of checking the integrity of backups in nature.

Hyper-V Redundancy

When backing up Microsoft Windows Server, it is enough to connect to any machine on which Hyper-V virtualization is selected as the role. In our case, when using the Windows 10 guest OS on virtual machines, the backup was completed only when the virtual machine was disabled. That is, there was no such transparent and invisible redundancy as under VMware ESXi, but the rest of the process is completely identical.

Recovery to Hyper-V server takes a little less time, which is obviously related to the format of the stored backups. In general, the processes are essentially the same, and if you back up ESXi and Microsoft virtual machines, then deduplication works for both types of virtual machines.

Pricing Advantages and Conclusions

When choosing a backup solution, IT professionals usually ask Veeam and Nakivo. The pricing policy of the former depends on the number of virtual machines in your network, while the latter pay for the processor socket, but is limited by the number of sockets per organization. On average, when protecting 52 virtual machines on 2 sockets, you will pay $ 3600 a year when using Veeam and $ 400 for Nakivo, and of course you will have to purchase a dedicated data storage, but at least the same QNAP TS-2477XU-RP that we used for testing.

In general, QNAP developers have created various backup solutions over the years. Today NAS works with both virtual machines and clouds, but each such task is taken out in a separate application. I think it would be logical for developers to combine everything related to backups into a single application in order to use common deduplication for all protected objects, control the window for backups and test existing backups.

As for the equipment, as you can see, even the basic configuration of modern corporate QNAP NAS is a powerful 8-core AMD processor plus two 10 Gigabit network ports on Mellanox network controllers, which means you can take out redundant interfaces at no additional cost. copying to a separate network, without fear that during periods of daily activity, the backup process will slow down the network or the NAS processor, which means you can expand the backup window, and spend the free time on other scheduled activities, thus reducing the cost of maintaining your infrastructure .

By using QNAP Hyper Data Protector instead of third-party software, you can recoup your NAS investment over a 3-year horizon if you weren't planning to use it, or lower your digital infrastructure costs.

Mikhail Degtyarev (aka LIKE OFF)
28/09.2020