Exploring SSD and RAM caching in Synology servers: how to get the most speed

Toshiba, a major manufacturer of flash drives and hard drives, forecasts only 10% of corporate data will be stored on SSDs in the year, although it seems that all storage manufacturers have forgotten about hard drives and focused on solid state drives. However, no matter how beautiful IOPS PR people promise us, no matter what synthetic tests show, the real speed of a typical virtual machine depends on how your server is configured and how the application itself works. Most server programs, from databases to the front-end with a web interface, use RAM caching, which means they may not depend on the disk system at all under any load.

A modern corporate NAS is not just a file storage, but also a computational node on which container (Docker) and host virtualization (hypervisor) are deployed, and since data in such a system is stored on the same host where it is processed, running which -or an application on the Synology VMM hypervisor, we can count on certain bonuses from the RAM side.

• First, it caches all file read operations of the virtual machine in the Diskstation/Rackstation RAM with the standard Linux cache.
• Secondly, it is the acceleration of write operations, which is achieved due to the absence of a layer between the hypervisor and the operating system. For example, when an ESXi 6.x host is connected to the NAS via NFS, all writes are necessarily synchronized by the VMware hypervisor, which leads to a decrease in speed.
• Thirdly, we can install a Redis caching server on Synology, both in the virtual machine and in the DSM itself via community packages or docker. We will have a non-volatile RAM caching server whose database is on a RAID array, isn't it lovely?

It is probably worth starting from the third point.

1. The most powerful way is to use Redis

If you already have a ready-made infrastructure with VMware vSphere servers, and you purchase a NAS only for storing backups or as a primary storage system, look at the memory: Synology RackStation RS18017xs + has 16 GB of RAM in the base, which can be expanded up to 128 GB. The entire DSM (DiskStation Manager) operating system rarely requires more than 2GB of RAM, so unused memory can be donated to Redis. It is a NoSQL server that stores data in memory, periodically flushing its database to disk. On reboot, Redis restores the database from disk, loading data into RAM, and even in a power outage, after a reboot, you have access to all data since the last synchronization. You can push inside Redis not only strings, but also files, and if your application constantly accesses a directory with tens of thousands of small files (for example, with machine learning), then you probably know that in this case any modern file system slows down, and Redis - no.

Redis can be installed through the Synology DSM package by linking the Synocommunity repository, but there is an old version 3.0.5-5 , so it's better to use Docker or a virtual machine running on a NAS. Install the package Synology Virtual Machine Manager and internally deploy the 5th version of Redis on the operating system .

Let's test the access speed using the built-in Redis benchmark. One and a half million transactions per second in pipelined mode and five hundred thousand with default settings. Redis itself is single-threaded, so you can clone a virtual machine to use more than 1 NAS processor core.

The example with Redis clearly demonstrates that today you can consider storage only as a file storage in two cases: when we are talking about a home 2-disk NAS, or vice versa, when we are talking about a powerful infrastructure of banks or airlines. For the rest, please, here's a centralized caching system: connect to it and save the resources of your hosts. You don't even need a 10 Gigabit network connection: in typical use cases, a 1 Gigabit network is sufficient for fast connected clients.

Don't forget that Synology DSM can use Snapshot copy protection of Redis data at the Btrfs file system level. Of course, using Redis will require you to rework the application a bit, which is not always possible, so let's take a look at how Synology DSM's built-in caching works.

2. NAS caching in RAM

Even if a virtual machine for Windows or Linux occupies hundreds of gigabytes on disk, units or tens of gigabytes of disk space are actively used: logs and database files, frequently requested files, in general, everything that is not cached in the memory of the guest operating system itself, or applications. Frequently requested data blocks are stored in the RAM of the Synology DSM itself, as we have seen many times in synthetic direct file access tests. The RAM caching mechanism is best observed on disk random reads.

This diagram shows the ideal access to a 16GB test area. Almost all of it can fit into the RAM of the NAS, which happens during the test. Note: Swinging The NAS is long enough - about 10 minutes, after which it reaches maximum performance.

When the cache fills up, the read speed increases by 3 times, but still remains small by the standards of what can be squeezed out of RAM. Does it make sense to add an SSD for operations that use a small active area of ​​the partition that can fit in the storage memory?

3. SSD cache in Synology implementation

SSD cache can operate in two modes: read-only and read/write. In the first case, you only need 1 solid-state drive, and in the second case, you will need at least a pair to combine into a `` mirror ''. Caching SSDs can be combined into more complex arrays, including RAID 5, the main thing is that fault tolerance is supported for write caching.

In the current version of DiskStation Manager, the contents of the SSD read cache are not retained after rebooting the NAS. That is, after the reboot, you will have a certain warm-up period, although DSM begins to push data to the SSD literally from the first minutes after launch. There is no such problem for the read/write cache.

Let's repeat our test, for which we will first use 1 SSD in read cache mode, and then 2 SSDs in read/write cache mode, combining them into mirrored RAID 1.

We see that SSDs, to put it mildly, work faster, and the mirrored array additionally increases performance by reading from two drives simultaneously.

It turns out that the NAS does not need to be asked to save data in the cache: SSDs reach maximum speed in 3-4 minutes, and RAM in 10-15 minutes. In addition, the SSD cache is more active in freeing data and rebuilding between workloads, although this is not shown in the diagrams. But, as they say, you will not be alive only by reading, and it is very interesting how the caches will behave in the patterns of VDI and SQL tasks. We will use 2 sizes of the test area: 16 GB, comparable to the amount of RAM, and 96 GB, three times more than the memory in the NAS.

Where a record is added, it is already necessary to more competently approach the choice of the SSDs themselves, given that most likely they will be constantly filled with data, and their speed will differ from the maximum. Let's enlarge the test area 6 times:

By the way, Synology DSM constantly monitors the health of SSDs and warns when it is best to replace the drive. For Seagate HDDs, there is advanced diagnostics through the IronWolf Health Management system.

Interestingly, in the SQL load, when the cache is full, the amplitude of performance fluctuations decreases. Let's compare the averages in different patterns.

Attentive readers have noticed that for the SQL pattern, we do not show a diagram for a 16 GB section area. Of course, you could give up and say: `` So everything is clear - you need to install at least 2 SSDs '', but we will not rush to conclusions, but we will run the OLTP test in a real application in a virtual machine.

Sysbench OLTP Test

Let's take a MariaDB database in a virtual machine with a small amount of memory, for example, 8 GB. Let's create a table of 50 million records so that its 11.2 GB size is more RAM available for the guest system, but less RAM of the NAS (16 GB) and force the machine to actively use the disk in transactional load mode using random read requests / Let's run this test three times: first, the virtual machine runs on a host running VMware ESXi 6.7 connected via iSCSI, then the same thing, but with NFS, and then we transfer the virtual machine to Synology Virtual Machine Manager using the package Synology Active Backup for Business.

Tests show that network traffic is 500-600 Mbps, but disk activity only appears in write operations, which means Synology DSM caches block access and file access operations equally well, which is not surprising since iSCSI LUNs are s are stored as files. Let me remind you that our test RackStation RS18017xs + has a base 16 GB of RAM.

In this case, SSD caching does not offer any special advantages due to the fact that the part of the virtual disk on which the database file is located can easily fit into the NAS's RAM. Let's create a situation in which the database is much larger than the free memory of the Rackstation RS18017xs +. It is impossible to increase the number of rows in the test table to billions: the database itself starts to slow down greatly, making the results not representative. It is much easier to take away the excess memory from the Synology DSM, for which we will launch a virtual machine with 12 GB of RAM in the Synology VMM hypervisor, as a result, only about 2.5 GB will be left for the cache.

And here the SSD cache smooths out the negative effect of the lack of memory, although the read performance is still worse than in the previous test. We need to make sure that the speed is affected by the lack of extra memory, and not by the Synology VMM hypervisor, for which we must run the same test on the NAS itself.

After moving the database to the Synology VMM hypervisor, we had to add another 16 GB of memory to RS18017xs + in order to keep the caching capability in the NAS's RAM. The benchmarks show the same performance, which is not surprising since a common pool is used for all file operations in the storage system. That is, for practical use of the database, you can completely get by with Synology VMM tools, reducing the number of servers in your company.

Going deeper into the buffering settings at the application level and experimenting with the InnoDB Buffer Pool Size parameter, I noticed that with values ​​from 1 GB to 6 GB, performance does not change significantly, so it is more profitable to give this amount of memory to the NAS. This is what hosting providers do when they lease virtual machines with a small amount of memory: the database actively works with a disk subsystem, which is a storage system with an SSD and a large amount of memory.

Moreover, it is worth noting that not every SAN array has the function of caching in RAM: LUN buffering at the block level is a rare feature, but Synology now even stores iSCSI LUNs as files, so in addition to scheduled snapshots , DSM can easily navigate what needs to be kept in mind and what not. Here's another plus of NAS over SANs.

Which SSD should you choose?

Try to choose MLC or SLC based drives for SSD caching over 3D NAND TLC. Whenever possible, choose a business-class SSD, and in reviews, pay attention to the distribution of IOPS over time, as in NVME SSD. Keep in mind that your SSD should definitely be on the Synology Compatibility List so you don't have to worry about the disk array.

Conclusions

What conclusions can be drawn from our testing? First of all, take a look at how aggressively Synology DSM writes data to SSD. In just a matter of minutes under load, they are copied to SSD drives, speeding up both NFS connections and iSCSI LUNs. According to synthetic tests, SSDs work even faster than RAM, but in reality it turns out quite differently: the more hot data in your infrastructure, the more memory you need to install in the NAS, no matter if it uses hard drives, SSDs or hybrid arrays.

Well, our example with Redis shows that if you have embarked on the path of good and decided to install a modern smart NAS with virtualization instead of the old SAN storage system, then use its capabilities to the maximum: it is not at all necessary to try to fill up all the storage bays solid-state drives - you can simply add support for NoSQL databases to your software and on the simplest model of Synology, the Rackstation series, get a miracle speed that no SSD will give for many years.

Mikhail Degtyarev (aka LIKE OFF)
07/03.2019