Huawei Oceanstor 2800 review - converged 2-controller storage for video surveillance

Video surveillance is one of the main driving forces in the storage market. The decline in the cost of IP cameras has led to the fact that today they are installed in hundreds, often next to each other, aimed at the same point, but with different focal lengths in order to shoot a wide shot and capture faces, use intelligent frame analysis and view from different angles. In the near future, we will see the spread of technologies based on Big Data, where the surveillance system will become the main data provider for analyzing the behavior of shoppers in stores, passengers at train stations and employees of business centers. The simple principle of 'write and save' already today it is supplemented with the word "analyzed", so the heart of video surveillance systems is not just a large shelf with disks where everything is recorded, but a powerful computing node capable of working with these records.

For these scenarios, Huawei has released a dedicated converged storage system, the OceanStor 2800, which operates on a storage + virtualization host in a single device. It is assumed that in projects where hundreds of IP cameras are used with a single computing center, where motion analysis functions are processed at the server level, the customer will not need to install a separate physical server, set up a fault-tolerant configuration, spend money on fast network interfaces: all this can be done on at the level of one storage system, on the controllers of which hypervisors are installed for the operation of virtual machines. Our readers are already familiar with similar solutions on industrial NAS, but we are testing virtualization on SAN devices for the first time, so first a little about the OceanStor 2800 architecture. We have a 2800 V3 model on the test, but the older 2800 V5 is built on the same principle, therefore much of this article applies to the newer modification.

The OceanStor 2800 V3 is an Active-Active dual-controller storage system with no single point of failure. At the heart of each controller is an Intel Xeon E5 2620v2 6-core processor with 48GB of RAM used for guest operating systems and write caching. For virtualization, the main advantage of a dual-controller solution is the built-in Failover mechanism, thanks to which you get two compute nodes with double protection: if one controller hangs or fails, within a few minutes the virtual machine will start on the second controller and the video surveillance will continue to work.

Initial setup

I would like to say right away that despite the very high-quality Web interface, setting up the Huawei OceanStor 2800 is difficult: everything related to virtualization is set through the command line using an SSH connection. The hypervisor itself (the development of Red Hat is used) is not the main function of the storage system, therefore, even to create virtual machines, you will have to use all possible means: the terminal, the web interface, and even the SFTP client to download the installer image, since this is done in video surveillance systems once. We start by initializing storage and setting up disk pools.

Huawei OceanStor 2800 is available only in 2U version for 3.5-inch hard drives, which, in general, is quite logical for a video surveillance system. By default, 4 bays are already occupied by system hard disks (600GB, 10K RPM, SAS, HGST body), which are combined into a single RAID 1 pool for storage software and do not participate in the shared storage space.

Like all Huawei storage systems, the OceanStor 2800 is hardwired to hard drives from the same manufacturer. As a rule, Huawei uses HGST drives, and only SATA drives are available for video surveillance, in our case, the He10 series, equipped with an adapter to SAS, which is necessary for connecting to two controllers. For large projects, the manufacturer suggests using 8 system hard drives with SAS interface instead of four. But it's hard for me to say how justified this step is, because these disks contain software that, after launch, practically does not use the system disk, and the entire archive is written to a separate pool, which we will now configure.

We combine 4 SATA disks into an analogue of RAID 5, which, thanks to block virtualization (Huawei RAID 2.0+ technology), can withstand the failure of 4 or more hard drives , after which we create LUNs to bind to virtual machines. They will be used for the video surveillance archive, and the connection between LUNs and virtual machines will be carried out via the vHBA virtual bus, so we do not need high-speed Front-end interfaces for storing video recordings, and it does not matter on which controller the virtual machine is running - disk E : will always be available under Windows.

Create a virtual machine from the terminal and allocate hardware resources to it. It should be borne in mind here that there is no dynamic memory allocation in the hypervisor, so for fault tolerance it is necessary to calculate that the total volume of all virtual machines (main and operating in the fault tolerance mode) does not exceed the amount of available memory on one controller. Up to 3 virtual machines per controller are supported and we will allocate 8 GB of RAM for the test machine. Using sFTP, connect to the controller and load the .iso image of the test Windows 7 onto it, mount it and start the installation. After installing Windows, we need to install the VMTools package to tell the operating system to use the VirtIO disk driver, which will enable us to configure fault tolerance. As soon as this step is completed, again from the terminal we indicate to our virtual machine to work in Failover mode. That's all, it remains only to test the transfer of the virtual machine between controllers during failover.

Virtual Machine Fault Tolerance and Power Protection

Let's check the operation of a failover cluster created on the basis of a single storage system. To do this, on running Windows 7, create a text file, save the current timestamp to it, and hot disable the controller on which the virtual machine is running.

Seeing that the connection with the computational node was lost, Huawei OceanStor 2800 starts the virtual machine from the backup controller, and after a couple of minutes we see our desktop with the saved timestamp. It should be borne in mind that there is no live migration of machines between compute nodes, so as soon as we connect the main controller back, our Windows 7 will be restarted on its controller. Thus, the hypervisors operate in the Active-Passive mode with the permanent role of the main and backup, and the downtime period of the guest operating system when switching the computational node is 1 minute 26 seconds.

Protection of memory from sudden power outages is assigned to massive lithium-ion batteries with a capacity like modern laptops - 4300 mAh. Two batteries are located in the power supply modules and can be replaced with a hot one without stopping the storage system. The use of such large batteries is due to the fact that in order to write memory data to a backup 64 GB MSATA disk, you need to support the operation of all storage systems: the chipset, processor and coolers. The batteries are not electrically connected directly to the power supply - only through a common board. So if the right power supply breaks and shuts down, the left one can charge the battery. The batteries are constantly under the flow of air from 16-speed fans, so they are not afraid of overheating, but for greater confidence, the manufacturer enclosed the batteries in steel cases, nevertheless, lithium-ion batteries, due to the activities of Samsung and Apple, are notoriously fire hazardous, so with powerful cooling and in a steel box it is somehow calmer for them.

Let's test how the virtual machine will behave when the power is turned off - open Notepad and write the current time in it without saving the file to disk, and then power off the storage system.

After power is restored, the Huawei OceanStor 2800 automatically starts the virtual machine, but we have an empty desktop: a snapshot of memory has not been saved. That is, if storage controllers operate in Active-Active mode, then the Active-Passive scheme is applied to the virtual machine. On average, Windows 7 startup time is about 40 seconds at 16 vCPU/16 Gb RAM settings and starting from system hard drives. It's pretty fast, so let's see what hardware the OceanStor 2800V3 hypervisor runs on.

Controllers

Each controller has one Intel Xeon E5-2620 v2 processor, which is outdated by modern standards. If CPU performance is critical to you, take a look at the newer OceanStor 2800 V5, although as we will see later in testing, even what is available is enough for video surveillance with hundreds of threads. The memory is recruited by six DDR3-1600 ECC modules with a capacity of 8 GB each, two more free slots are equipped with caps so as not to create a free corridor for air flow. Controllers, like interfaces, can be changed to hot without stopping the storage system.

For communication between controllers, the PCI Express 3.0 bus is used (up to 8 Gb/s), and Huawei emphasizes that for the OceanStor 2800, the main software layer is just SAN, so there is no way to load processors does not affect the data transfer rate, but we will now check this statement!

Test SAN speed vs. vCPU load

Up to 32 vCPUs are available for each virtual machine on the OceanStor 2800 V3, and we know that no application or database will put such a load on the processor as Burn -In OCCT test 4.5.1. Let's load all 32 logical cores for them from under Windows 7 and see how the time of access to disk resources via iSCSI changes, and for the sake of completeness, let's duplicate the experiment with one disabled controller.

Our tests show that the access time does not depend on the processor load, which means that the storage and the compute node are not connected at the hardware level, and there is no need to be afraid that video surveillance will affect the needs of others applications and servers: here it is, pure convergence.

Port Resiliency

By default, each controller already has 4 1Gbps BASET interfaces, which in total gives a speed of about 1 gigabyte per second on storage, plus the fifth control port can be used for a virtual machine (iSCSI connection is not available through it). Is it a lot or a little? If we take a Chinese IP-camera with a resolution of 1080p, then usually with the H.264 codec it creates a stream of about 5 Mbps, and the built-in network ports are enough to connect 1,500 such cameras, that is, more than officially announced by the manufacturer. So just for the needs of video surveillance, additional network interfaces may not be needed at all - it would be possible to divide traffic by ports, and here we are in for a little trouble: only those interfaces that are connected to the controller on which it works can be forwarded to a virtual machine , that is, 4 ETH Front-End ports and the fifth control port are available to us. When calculated in Chinese IP cameras, this is approximately 900 channels with a resolution of 1080p.

There are no such restrictions for iSCSI access, and you can connect to any controller through any of the Front-end interfaces, so for VMware vSphere it is enough to configure two connections to two IP addresses of the controllers , and get fault-tolerant access, and since we're talking about iSCSI, let's measure the speed of switching between controllers in Active-Active mode for the Random 4K Read test.

As we can see from the graph, switching between controllers takes about 37 seconds, and when the storage system is restored to normal Active-Active mode, the reverse transfer of the disk pool occurs in 5 seconds, while the time access is the same on the primary and backup controllers.

From the point of view of interfaces, convergence in Huawei OceanStor 2800 is not complete: you cannot send both iSCSI traffic and a stream from CCTV cameras to a virtual machine via the same ETH port. This is a feature of the SAN architecture, and in general, the solution is simple - each controller can accommodate up to two 4-port hot-swappable interface cards of the following type:

• 4 ports RJ45 1G BaseT
• 4 RJ45 10G BaseT ports
• 4 ports SFP + 10G ETH (SmartIO series adapters)

In our article on SFP+ vs. BASE-T we have shown that the latency for iSCSI access between copper and optical interfaces can be up to 20% (on flash arrays), but for the OceanStor 2800 it makes sense to install an additional 4 1G BaseT ports for video surveillance and 4 SFP + ports for SAN networks .

Software setup

Well, now we come to the final part of the setup - installing the video surveillance software. As you already understood, there is no built-in software solution on the OceanStor 2800 V3, so everything that science has created for Linux or Windows is available to us. It is not easy to figure it out among dozens of distributions, and friends who supply Zavvio IP cameras for integrators advised to pay attention to Axxon Next and Macroscop . The latter is being developed by our compatriots from Perm, this software package has additional modules for image analysis, such as "helmet detector" for construction projects, queue detector for cash registers, creation of heat maps of customer movement, state recognizer numbers and others. Plus, Macroscop uses disk resources for video archives more flexibly and is somehow easier to configure, so Axxon Next was postponed until better times, and the Perm product went into action.

Download the distribution kit from the Macroscop website and install a complete demo-set of video surveillance software under virtual Windows 7 (about 1 GB). In client + server mode, displaying heat maps and video streams results in approximately 30-40% CPU utilization for 32 vCPUs. This, of course, is not the case - we will leave only the server part of the DVR on the storage system, and move the client part to a separate host, because Macroscop supports accelerating the output to the GPU, which is very useful to us.

To set up fault-tolerant operation and distribute the load across controllers, we need a second virtual machine that will run on a second controller in order to use both processors for video surveillance needs. There is no cloning or migration functionality in the OceanStor 2800 V3 hypervisor, so we convert the virtual machine to a template by executing a simple command from the terminal, and then recreate a new virtual machine based on the saved image, thus reducing the installation time from several hours to several minutes.

After starting two Windows 7, make sure that Macroscop is running on both controllers, go to the program settings and install the second server. I transferred the client part to Windows 10 x64, running under VMware Workstation 14.1.3 from a desktop computer, so that all three components are virtualized - this will facilitate backup and backup recovery in case of a disaster. Yes, we are getting used to the fact that keeping a professional application on a physical host is possible only as a last resort.

Test 400 HD streams

In order to test the Huawei OceanStor 2800 V3's ability to work with video streams, we will use an IP camera emulator launched on the test bench, which can be downloaded from the software developer's website along with the distribution kit BY. Here you need to make one remark: the emulator operates in 1-channel mode, and multi-channel is configured by cloning the camera in the Macroscop server. At the same time, each channel at the same IP address requests a video stream from the emulator, so we get a load on both the network and the storage disk subsystem, everything as in combat conditions. On the Huawei OceanStor 2800 V3, 16 vCPUs were allocated for the virtual machine, and only the server part of the software was launched there, since the client is not very optimized - it requires acceleration through the GPU for normal operation, and running it on a gaming computer, I thought about another upgrade.

Test configuration characteristics

Huawei OceanStor 2800 V3

• 4 x 10Tb HDD (SATA to SAS, HGST He10 body) RAID 5
• 2 TB LUN vHBA
• Virtual Machine:
• 16 vCPU
• 16 Gb RAM
• Windows 7 SP1 Guest OS
• Macroscop 2.4 - writing the archive to the E: \ (2Tb LUN NTFS) drive.

Testbed

• Intel Xeon E5-2603 v4 (6C 1.7 GHz, 15M L3)
• 32 GB DDR4 ECC RDIMM
• Asrock Racka EPC612D4
• VMware vSphere 6.7
• FreeNAS ZFS (NFS storage lArc = 8Gb), 3xSavvio 10k.6 RAIDZ1
• Virtual Machine:
• 6 vCPU
• 16 Gb RAM
• Windows 10 x64 18.07
• Macroscop Virtual IP Camera

Interconnect: 1Gbps ETH direct connection

First video stream characteristics:

• 1280x720
• 28 FPS
• 1.9 Mbps
• H.264

Second video stream characteristics:

• 1920x1080
• 19-25 FPS
• 5.9 - 7.9 Mbps
• H.264
• Test recording from the Chinese camera Pinetron PNC-IV2E2

In this test, we faced two main difficulties: since the recorded stream from the FullHD camera was encoded with a variable bitrate, the load created on the storage system looked like this: "sometimes thick, sometimes empty" ;, monitoring showed a sharp increase in disk writes from 30 to 250 Mb/s, and something similar was observed on the test bench. Smoothly changing the number of threads, we reached 120, but stepped over the capabilities of the 1-gigabit interface, which immediately caused a high load on the test bench. A smooth decrease in flows to 100 allowed us to achieve a level when we are still within the gigabit range, but we are already running into the SAN disk subsystem, where 4 hard drives cannot cope with random write operations in any way. The software cursed that the disk subsystem was slowing down, and the hand did not rise to rebuild the excellent RAID 5 with block virtualization on RAID 0. In fact, in this configuration, we recorded stable operation of 95 Full HD streams, in which we did not run into either the test bench, the storage disk subsystem, or the gigabit interface.



The vCPU load shows that the Huawei OceanStor 2800 V3 still has a huge computing power reserve, which means that when using more hard drives, it is quite acceptable to reach the declared 400 threads.

Scaling and configuration

No vertical scaling: the only thing you can do with the OceanStor 2800V3 is to increase its performance by installing 8 SAS disks under the system files.

To expand disk space, 2 types of 4U disk shelves are used: one for 24 disks with front HDD connection and the other is a very interesting shelf for 75 disks. In total, up to 750 hard drives can be connected to the head unit, which will give 7.5 PB of net capacity. Is it a lot or a little? If we count in our favorite Chinese HD cameras, which produce a 7-megabit stream with continuous recording 24 hours a day, then this capacity will be enough for about 8 months of round-the-clock recording from 400 of these cameras (when using RAID 5) .

Interestingly, not only SSD drives, but also licenses for file access (NFS, CIFS), deduplication and Tiering are completely absent in the device configuration features. This should be taken into account if your goal is full convergence (NAS + SAN + VMS in one box), although in the Huawei OceanStor 2800V3 Web interface, you constantly find links not only to these technologies, but also to embedded applications, such as antivirus.

Warranty & Support

The standard device warranty is 3 years, this period can be extended by purchasing the appropriate service packages until December 31, 2023 (End of Support date for this model). You can also use packages of extended warranty service:

• 9х5 Next Business Day with a visit to the installation site
• 24x7x4H onsite
• 24х7х2H with on-site visit

Service is carried out by the authorized service centers and branches of the company.

Cost Effective

The main economic effect of buying Huawei OceanStor 2800V3 is that you save on servers allocated for VMS and software virtualization support. Plus, when using free software like ZoneMinder, you don't pay for licenses for the number of cameras. In fact, you get by with only one storage system with disks, which you will need for video archives anyway, and you can configure a failover cluster of 2 nodes with one command from the terminal.



RAID 2.0+ technology will allow the array to easily survive the failure of more than 1 disk in RAID 5 and quickly rebuild the pool in minutes. That is, the very planning of disk resources, again, allows you to save money compared to traditional hardware and software RAID.



The tested configuration used the price of Windows 10 x64 Pro, although we used Windows 7 "from old stock", but as you can see, this does not greatly affect the cost of even such a small project ..

Disadvantages

We have said a lot about the advantages of Huawei OceanStor 2800 V3, but the disadvantages should also be mentioned.

• From my point of view, the main disadvantage of this machine is the overly complicated process of installing virtual machines. Of course, from the manufacturer's point of view, this system is configured once by a certified integrator and is no longer touched until the end of the device's lifespan, but functions such as import/export of virtual machines and snapshots should be available at the click of a mouse.
• The lack of SSD drives can be understood and forgiven in a cheap school DVR, but if we have a converged SAN + VMS, then the fast layer should be here.
• Protection against sudden power outages covers only the cache of the disk system, although it directly suggests saving snapshots of virtual memory to disk and putting them into suspend mode.

Conclusions

If such a solution did not exist, it would be worth coming up with: a failover cluster of 2 nodes in one box, a hardware independent SAN architecture, RAID 2.0+ block virtualization and the ability to cram 7.5 petabytes of hard drives into one cabinet with a height of 44U. This is a huge savings in a single large-scale video surveillance project, and an opportunity to reduce the cost of software, licenses and maintenance.

The open platform allows you to choose any of the VMS software products, both free and commercial. If software is available, one of the controllers can be allocated for recording a video archive, and the other for analysis, not forgetting that even such a configuration will withstand the loss of one of the controllers. However, it should be borne in mind that Huawei OceanStor 2800 is primarily a dual video surveillance server, and secondly a storage system, so when planning a project, consider the possibility of using only block access (iSCSI).

Mikhail Degtyarev (aka LIKE OFF)



Tags surveillance Storage Array SAN Huawei OceanStor Macroscop