Big test Synology Active Backup for business - recovering from backup in 11 seconds!

In 2010, the founder of Veeam Software, Ratmir Timashev, said in one of his interviews that the main thing for backups is the recovery process, how fast, easy and reliable it is. Today Veeam software solutions have become the de facto standard in large businesses, and the only thing that can limit their distribution for small companies is the high price of paid versions.

In 2018, Synology says that quick recovery from backup is a matter of course, and if you've been making storage systems for 18 years, you should have a solution that works equally well with virtual and physical machines, backs up other NAS-s and SDS, determines when, where and what to back up, uses global deduplication, taking into account the repeated extents for all backups made by this package. As for recovery, why not give the user access to their copies so that he can pull the file out of the backup without tinkering with the sysadmin? I wonder if Synology Active Backup for Business can compete with all the Acronis and Veeams? To answer this question, we created a small test setup in which we first integrate the Synology RS4017xs +, and then we will break and restore everything.

Testbed configuration

Basically, if you understand the scheme below, then you don't have to read the article, and for those who need a pretty picture, here are some comments: our test setup is divided into two parts. On the left - everything related to Windows, and on the right - Linux/Unix. To test the performance of some functions, we used a virtual copy of our site (as of September 2018) under 1C-Bitrix with disabled CMS caching to increase the load on the database, processor and disk systems.

Test environment topology:

• 1 Windows 10 x64 Enterprise Workstation + Disk Spaces (2 x SAS HDD Mirror)
• 1 Windows 10 x64 Home Thin Client
• 1 VMware ESXi 6.7 server (as storage - virtual FreeNAS, NFS)
• 1 VMware ESXi 6.5 server (as storage - virtual XigmaNAS, NFS + iSCSI) (not shown in the diagram)
• 1 Windows Server 2016 + Hyper-V
• +1 Synology RS4017xs +, 3x Seagate IronWolf 10Tb RAID 5, BTRFS

Backup ESXi 6.7 VM under load

We will not consider the configuration and installation of the Synology RS4017xs +, because if you use virtualization, then you can easily figure out Synology DSM 6.2.

We launch the Active Backup for Business package - and we have a single backup system interface for all types of resources: physical, virtual and file. We immediately go with our trump cards and configure the reservation of virtual machines, for which, first, in the vSphere web interface, we need to enable SSH access and the TCM service by configuring the policy of these services to autostart along with the host.

Now, in the list of servers, select the virtual machines that will be backed up and the scheduling policy. In order to speed up the process of repeated backup, in the settings of virtual machines, you need to enable the VMware CBT (Changed Block Tracking) function, after which the VMware hypervisor will track the data blocks that have changed on the disks of virtual machines and it will be easier for our NAS to understand what exactly to save in an incremental form. This allows reducing the time for repeated copies of virtual machines to 2-3 minutes. By the way, due to my own laziness, I thought that it was useless to manually set some parameter for two dozen virtual machines and I was right - after the first backup, this parameter appeared on its own, but it turned out to be unpleasant - an incremental backup does not work with virtual machines into which physical disks (full backup only), so the test FreeNAS will wait for its fate for now.

Testbed:

• VMware ESXi 6.7 Server (Xeon E5-2603 v4, 32GB RAM)
• Virtual - a copy of the server HWP (Debian 9, LAMP, Bitrix, 120 GB data, 4 vCPU, 16Gb RAM).
• Storage - Virtual FreeNAS 11.2 NFS 3 (2vCPU, 16Gb RAM, 3xHDD Seagate Savvio 10K6 600Gb RAIDZ1)
• Load: Netpeak Spider, 100 threads @ AMD Ryzen 5 1600, 16 GB RAM, Gigabit connection.

We placed a virtual copy of the site on a server running ESXi 6.7 and turned on the load from the workstation - crawling pages by the SEO spider Netpeak Spider 3 in 100 simultaneous threads. On the ESXi 6.7 server during this test, the processor load was 100%, the typical response speed of the virtual machine was 112 URLs per second, including images, styles and fonts. By simulating a backup under load, we can see how the speed of our web server will change at the time of backup.

HWP virtual server scanning speed by Netpeak Spider

I expected the site to completely "lay down", but as you can see, the performance hit was about 20%. From my point of view, this is a small price to pay for the ability to silently copy data without stopping the service.

Now that we have copied the site, it's time to see what we have in our reserves. They are accessed through a separate Web interface called the `` Active Backup for Business Portal '', which is accessed according to the account policy of Synology DSM. Each user can be given the ability to access only certain backups, so as not to climb through other people's archives. From the web interface of Active Backup for Business Portal, we have access to the structure of the virtual disk of the machine with a search function by name, which makes it easy to restore a file. In our case, for a copy of the site, it makes sense to restore some of the JPEGs that were accidentally deleted while working with articles - you have the right to return it to its original location on the disk or download it to your work computer. Of course, you can rewind the history of file changes using a temporary tape, or you can immediately go to any date and restore the picture as of the specified day.

In principle, for virtual desktops (VDI), you can even restore some letter or document without touching the operation of the entire virtual machine, and note that at this stage, no software needs to be installed anywhere, so if you provide cloud services to clients, you do not have to prove to them that some kind of program for saving data is absolutely safe.

All the beauty of the restoration portal is in simplicity of forms and minimum functions. Three Find, Repair & Download Commands it performs perfectly, and most likely in real life it will be more in demand than other features of Active Backup for Business, but they are much more interesting, so let's move on.

Instant and full recovery

It is well known that data warehouses are a weak point of vSphere servers, especially if they are remote and connected via NFS, therefore, if with a "datastore" ESXi is in trouble, Synology will help out due to the technology of instant recovery of virtual machines. The Active Backup for Business package will take a backup stored on a NAS and present it as a remote storage from which it will launch a virtual machine, both on its own vSphere server, and on any other, including its own Synology VMM, which will be discussed later. You do not need to transfer the entire virtual machine over the network, spending time and nerves on this - use the backup where it is stored, especially since the virtual machine can work faster with the BTRFS file system than with EXT4 or ZFS virtualized storage systems. Let's test the fast recovery.

Recovery speed - 43 seconds for each virtual machine! However, there is one unpleasant moment, which is that at the stage of connecting your backup storage to ESXi, Synology software does not allow you to choose a network interface that will be used for NFS traffic. In our case, RS4017xs + had 6 network ports: ETH1-ETH4 at 1 Gbps and ETH5-ETH6 at 10 Gbps. The connection between the NAS and the hypervisor was established over ETH5, in the 192.168.2.x range, while vSphere and Synology were managed over ETH1, in the 192.168.1.x range. So, during a quick recovery, the NFS volume was connected only via ETH1 with an IP address of 192.168.1.x and a speed of 1 Gbit/s. Hopefully in the next update the developer will fix this issue.

Full restoration of the virtual machine is an extreme case, for which you need to be prepared. Regardless of the incremental step with which the copies were made, recovery affects the volume of all virtual machine disks, and there is something to complain about here too.

recovery of a virtual machine with a volume of 120 GB (time)

According to our tests, the speed of full recovery is about 30-40 Mb/s per virtual machine, that is, even the simplest machine with 200-300 gigabytes will come back to life for several hours. It is interesting that during the mass recovery of virtual machines, the speed is summed up and for five machines it is 120-150 Mb/s, reaching 200 Mb/s in peaks. As you noticed, a full restore uses the same network port as the backup (in our case, 10 Gb/s), but the best way out of the situation would be instant recovery, described two paragraphs above, followed by migration of the virtual machine using VCenter.

Sooner or later the server itself under vSphere may break down, and if you have a rare motherboard, the repair may take 3-4 weeks. Previously, such a host failure was akin to a catastrophe, but today NAS themselves are ready to work as computing nodes (see our articles on Synology hypervisor and Synology failover cluster), so of course knows how to run your backup on itself, deploying any of its incremental copies as a separate machine inside the Virtual Machine Manager. You can restore several states of one backup at the same time, if it is necessary for the work of programmers or for testing the application. Of course, frankly, the memory of most Synology NAS is not enough, and if your compute node is more powerful than the NAS, then it is better to test your application in advance for work in the Synology Virtual Machine Manager hypervisor, as we will now test our site.

The test showed that when restoring a virtual machine from Debian 9 from under vSphere 6.7 to Synology VMM, the only thing that needs to be done by hand is to change the number of the network controller in the operating system via the console (ens3 to ens33). If everything suddenly tightens itself and no action is required - recovery takes 11 seconds from pressing the button to the moment the operating system starts. There were no problems with virtual machines for Windows 7, Windows 10 and Windows Server 2016. Let's compare the site speed under load with different recovery options, for which we will make three launches of Netpeak Spider to warm up the database cache, and take readings on the fourth launch.

HWP virtual server scanning speed by Netpeak Spider (100 threads)

As for the site speed, we have a living example of how even on a 1-gigabit port external storage under BTRFS gives a 40% speed increase compared to the internal virtual FreeNAS. This is due to the fact that the FreeBSD operating system on which FreeNAS is built does not like working inside a virtual machine very much, as even the developers of this distribution say. As for working in Synology VMM Pro, here we ran into a lack of standard memory on the RS4017xs +, and starting from the 3rd run of the test, the machine went to Swap, which is why the diagram shows the data of the second run. Interestingly, the 8-core Xeon D-1541 in the RS4017xs + with one memory stick outperforms our test 6-core Xeon E5-2603 v4 with three DIMMs. Research has shown that when our site is running on Synology VMM, the NAS's disk pool and memory are up to 98% utilized, and it feels like give us more storage, give us more disks, and we'll go over 180 URL/s.

A backup of a virtual machine can be restored to another ESXi server, and this is the easiest way to transfer a machine from one host to another, without buying a Vmware vShere and vCenter volume license costing more than 10K $. For small companies this can be very useful, but you need to remember about the incompatibility of different versions of hypervisors: VMs from ESXi 6.7 will not be able to recover from ESXi 6.5.

In our server, the FreeNAS virtual machine consumes 16 GB of RAM for the LARC cache, running slower than a full-fledged Synology NAS, so we have no reason to leave the "All-In-One" design. live on, and our next task is to back up CIFS network file shares on Synology and remove FreeNAS permanently.

NAS Backup

The task of copying NAS-s is the simplest and most boring event in our testing. Synology Active Backup For Business supports only two protocols: SMB (including Volume Shadow Copy, which allows you to back up files opened in other programs if Windows Server 2016+ is running as a NAS) and Rsync, but that's why there is no NFS support is a big question, especially since we have a business-optimized environment, and today NFS is used even by some media players.

To access backups, the same recovery portal is used, which was mentioned above.

Someone thinks that it will be cool if Synology stores everyday copies of your network shares from the last year in case you deleted something or the software itself glitched. I do not agree with this approach, and I believe that as soon as a Synology NAS appears in your business park, you will delete the virtual SDS, as we removed FreeNAS, and give the freed up resources to some useful application. Of course, for a complete migration of network shares to Synology, it is not enough to restore network paths with user rights, but for the second, you can use a domain, and the first can be configured manually.

In our case, when FreeNAS just presented NFS- [storage to the ESXi host, so without regret we press the button "delete virtual machine" and do not forget to check the box `` delete all contents from disks ''. Let's go further, because we just need to configure the copying of our Windows devices.

Reserving Windows machines: computer, server, disk spaces and nested virtualization

Copying Windows hosts will require installing a small agent program on them that supports all modern Microsoft operating systems: desktop Windows 7 and above and server Windows 2008 and above with NTFS file system.

Usually, the process of backing up a Windows computer is a boring and routine task, and for Synology: you can select only one or several partitions, or even the entire computer, not worrying about what it has with the disks. It should be noted that backup is performed at the file (not block) level, so the use of Bitlocker and other encryption programs does not affect the backup.

By default, the Windows machine is defined in Synology Active Backup for Business as a "desktop PC", offering two types of recovery: file-by-file through the same portal and full through boot image.

To create a bootable disk, you should use the Synology Active Backup Recovery Media Creator utility, with which you can create a bootable media-flash drive, booting from which you will restore your computer to the state in which it existed at the time last backup. And if the computer is no longer amenable to resuscitation, disposed of or decommissioned, you can restore the entire system to a new PC or laptop, as long as the new drive is no less than the old one.

After booting from the installation flash drive, launch Synology Active Backup Recovery Manager, enter the data to connect to the NAS, select the backup and restore point and .... in general, that's it. The disk is restored, and after a reboot, we see our computer again in the form in which it was before the crash. This will work even if you change the motherboard or the entire laptop, as long as the new disk is not smaller than the previous one. In large companies, the administrator can create one such bootable USB flash drive once and, if something happens, restore the backup to bare-metal configurations in 20-30 minutes.

In the Active Backup for Business manager for a connected computer, you can change its role by clicking on the "update client" button, after which the PC will move to the "Physical server" section, and all the same methods will appear for it recovery as for virtual machines: to the VMware vSphere server (fully migrated or from local storage) or to the Synology VMM virtual environment. This is a completely different twist, because even if the motherboard burned out on the PC, nothing prevents you from deploying its image in the virtual machine of the NAS itself and connecting via RDP from any other device. From my point of view, this is the best way to restore from a backup ever.

But there is also a fly in the ointment - recovery to the hypervisor, both in ESXi and in Synology VMM, is not compatible with "disk spaces" Windows, and this problem is more serious than it seems at first glance, because as soon as you changed the role of a Windows computer from `` PC '' to the Physical Server, you do not have access to the Recovery Portal, and the only way to get to the data stored in the Windows software RAID is to restore a copy of it to the virtual machine. You do it, and here again - and `` unrecognized configuration '' hard drives included in Windows disk spaces. You are trying to bring everything back and change the state of the computer back to `` PC '', but there is no turning back, and everything that was stored in `` Windows Disk Spaces '' you can wave your hand.

Or such a trifle as the lack of autorun for an agent under Windows is perceived with surprise in 2018: well, how is it possible - I start Windows, but Synology Backup Agent needs to be started manually? But if the NAS administrator removes your PC from Active Backup for Business clients, then you can reconnect to the backup server only after a complete reinstallation of the agent.

But let's not talk about sad things, and let's move on. What do we do with Hyper-V virtual machines? If they use Windows, then you can install on each agent for backup to NAS, and if the guest systems are Linux, then back up the entire host and deploy it along with all the Hyper-V content, but already on an ESXi server or Synology VMM. Let's test this nested virtualization by backing up a physical host running Windows Server 2016 with the Hyper-V role enabled, running a virtual machine running Win10 x64 (1 vCPU + 1Gb vRAM).

Even after installing all updates, replacing drivers and installing guest tools, I was unable to get nested virtualization to work under Synology VMM, although this recovery worked with VMware ESXi. In principle, nothing prevents us from converting machines from Hyper-V to OVA, and then deploying in whatever hypervisor we want. The main thing is that the host is up and running again and the data has survived.

What's more interesting when backing up Windows?

First, Active Backup for Business is a very powerful SaaS solution for backing up and restoring files that is faster and easier than local Nortons and Acronis. Even from the deadliest computer, finding the oldest file and extracting it by attaching it to a letter or returning it to disk is a matter of five minutes. That is, if we are building a cloud for tenants with an additional service a la Google Drive, then we have an excellent local tool for this with autoloading all drives that does not clog the traffic of Internet channels. The cloud backup provider has a ready-made friendly storage shell for itself, centralized management for the administrator and recovery for the user, and this is very, very cool. And if you consider that you can distribute access and virtualize the Synology DSM itself, then the possibilities for multitantant services are endless, well, except that billing is not enough, but it can be configured through third-party tools and authorization domains deployed inside Synology VMM.

Secondly, if we are talking about the most modern workstations, then Active Backup for Business gives a very interesting opportunity to work with ramdisks, on which, when Windows starts, you can simply restore files from a NAS, with a complete history of saving data without fear of freezing the computer or turning off the UPS.

Third, you get a very powerful tool for migrating physical machines to virtual environments. For example, the disk2vhd program offered by Microsoft will require 2-3 hours per seat for backup only, plus migration to the hypervisor, plus compatibility issues. And with Active Backup for Business, you can go to the virtual world from anywhere in the past in seconds. If you don't like it in the virtual world, go back to hardware.

Deduplication efficiency

At the beginning of the article, we said that Active Backup for Business uses global deduplication, which takes into account all data that passes through the backup service - PCs, virtual machines, physical servers and NAS.

You can write a lot about the deduplication level, but the screenshot taken during the tests will say more than a thousand words. The process of assembling extents is carried out by the NAS processor at the time of backup, it is not configurable in any way, and the block size remains a mystery, but I rate the speed when writing over 200 MB/s as excellent. This algorithm practically does not consume memory, at least I did not notice any noticeable increase in the load on RAM.

Each connected BTRFS storage has its own deduplication, so it would be more correct to use one large volume than several small ones. Interestingly, Active Backup for Business creates a directory on the disk pool that is accessible only to itself, and even the administrator cannot copy the backup files, which is correct from a security point of view.

More memory!

The beauty of a converged NAS is that it can fix any problem by itself by running someone else's virtual machine, physical server or desktop in Virtual Machine Manager. In all these cases, you can not only pull out the data, but also continue to work until the broken piece of hardware is completely restored.

But virtualization requires large amounts of RAM, and even our test RS4017xs + with a very good Xeon D-1541 processor comes with only 8 GB of RAM, of which 5.5 GB is available for its own hypervisor. Of course, the memory in RS4017xs + can be expanded to 64 GB, because in the basic configuration it will not be enough for anything more serious than Windows 10 with an office, so since you are saving on disks due to deduplication, put the maximum amount of memory in the NAS, and instant recovery to your own hypervisor will be a joy!

Conclusions

Synology Active Backup for Business is the most significant step for the company in 2018. The release of this package can be compared in importance with the announcement of its own hypervisor, of which it is a logical continuation. As of today, the package has some shortcomings, which are mentioned in the text, but I hope that they will be fixed.

Active Backup for Business puts Synology on a par with companies like Veeam to reduce reliance on third-party software as the package itself is completely free. The concept of a Web portal for recovering individual files is modern and secure so that access can be given to any employee. The recovery rate of virtual machines exceeds the typical downtime in high availability clusters.

I am very pleased that Synology has started to use deduplication in BTRFS in production conditions, and doubly pleased that the backup speed remains high even on NAS with a middling Xeon D-1541 processor. In an environment where hundreds of terabytes of backups are stored on disks, incremental deduplication backups can save space tens of times.

Concluding the review on a positive note, I want to emphasize this figure again: 11 seconds - from the moment you decided to restore a physical or virtual machine from a backup until it starts inside Synology Virtual Machine Manager. Eleven (that's 10 + 1) seconds of downtime under load. I wonder if there is something faster?

Mikhail Degtyarev (aka LIKE OFF)
03/12.2018