Server review and FAQ on Kunpeng 920, Huawei Taishan 2280 V2

To take for testing an ARM-based server from Huawei, created on the basis of a Huawei processor with a Huawei chipset and even a network card based on Huawei NIC, I have been in line since the end of August. And only in December, due to the proximity of the New Year holidays, I managed to snatch a test slot for 2 weeks: the Huawei Taishan test park is very tiny, and the interest in them is huge. Everyone wants to check what kind of new beast it is that came to compete with Intel Xeon and AMD EPYC, and first of all, all sorts of departments, structures and committees crowded next to me in the queue for testing.

1. Why?

Somewhere there, in other countries, they talk about saving electricity and CO2 emissions, someone believes that the Internet has outlived itself in the modern cloud world, and in other countries they look at the political situation. Anyway, Intel and AMD are corporations of USA, and the notorious “Uncle Sam” at any moment can not only open all the worms that are in processors and chipsets, but also simply limit the supply of technology to any country.

This is all very serious, no matter how strange it may sound, there is a huge demand for “non-American” protected from hacking and malware IT over the world. Here is only proposals almost there is no. And given that Huawei recently mentioned a desire to invest about $ millions to train specialists in the field of ARM and Cloud, it is already clear that all roads should be open for this company.

2. Are there many Taishans?

Yes, Huawei decided to cover the entire spectrum of the server market, except for blades and quite a budget. The considered Taishan 2280 V2 belongs to the class of universal servers in the opposed HP DL380, but there is also a Taishan 5280 in a 4U case for 40 hard drives, Taishan X6000 in a Hi-Density case for 4 nodes, an Edge server has been announced (read what is Edge computing ) , and in addition to this, the company will transfer its data storage systems to KunPeng (read Huawei OceanStor 2200 V3 review ), which now use Xeon processors.

That is, Taishan is not some one-time, experimental product, but an ARM-based world already ready for implementation, to which you can switch without looking at the x86 architecture, but we will still turn to compatibility over and over again with Intel/AMD.

3.So what kind of KunPeng is this?

We know about ARM processors that they are installed in smartphones, and today no one doubts their performance and energy efficiency. I will not take the liberty of explaining the pros and cons of the ARM architecture over x86, believing that someone has already done it before me or will do it instead. It is much more important that x86 and ARM are not compatible, and roughly speaking, you will not install regular Windows on KunPeng, and you will not transfer your old Linux by simply copying files to a new server. For some, the lack of live migration of VMs from Intel to AMD is already a disaster (why only for some - read our article ), but here the whole world is different: you have to use the ARM versions of your operating systems and their programs, and some of them must be compiled again. Inveterate IT specialists They will say that with such compatibility the server has no chance on the market, and this buried Itanium in due time and threw IBM Power into the backyard of the galaxy, but no: the ARM community is growing. Why?

Because ARM in hardware scales easily. Today, Kunpeng has up to 64 cores on one processor manufactured using 7 nm technology, an 8-channel controller of advanced DDR4-2933 memory, PCI Express 4.0, an integrated 100-Gigabit network controller with RDMA support, 64 MB L3 cache, and tomorrow it can be will take it and double it, because the chipset design on ARM architecture allows for almost infinite scalability of computing power. I almost forgot - one server can have up to 4 KunPeng processors.

But do not rush to drive horses, because today there are at least 7 modifications of Kunpengs, and you need to look for information about them on the Huawei website. By the way, all the chips have the Hisilicon logo on whose site there is even less information. And even the fact that it is a subsidiary of Huawei only issue copyrights at the bottom of the web pages. Obviously, this is another political affair, the best way not to get bogged down in which is to look at the iron with your own, well, or our eyes.

4. What is the quality of Taishan 2280 v2 in hardware

I mentioned that Huawei Taishan should be cheaper than analogues from HPE/Dell, but this does not mean at all that the manufacturer will save on matches. And if you are the sysadmin who opens every new server that comes into operation, then the first thing I draw your attention to is attention to detail. The fans of the Taiwanese company AVC are: a) hot-swappable, b) anti-vibration suspension, c) a curtain that prevents reverse air circulation when stopped. You are looking for something to find fault with, but in general there is nothing to complain about, so instead of the usual description of the insides, let's answer frequently asked questions.

Well, for example, expansion boards in risers are fastened with soft rotary latches, and the risers themselves are installed in the case, sliding along the rails of powder aluminum. Instead of risers for PCI-E boards, you can order I/O modules for installation in the back of four 2.5" SSD if 6 PCIe slots is a lot for you. And if you are interested in PCI Express 4.0 support, then I hasten to please you: all slots in Taishan 2280 v2 are PCIe 4.0 16x (well, except for the slot occupied by the mezzanine of the RAID controller).

5. Can I install my own expansion cards?

It seems to me that the server was specially created to install expansion cards in it.

By default, the server has 5 PCIe 4.0 16x full height and length slots. For the sake of experiment, we installed a network card Intel Xa520-DA it worked.

6. Does Taishan 2280 V2 support GPU installation?

Only boards without external power supply, such as Tesla T4. For this, additional power is supplied to the two PCI slots of the riser. In general, the power of small 2-kilowatt power supplies of 80 Plus Platinum standard should be enough for at least 4 of the most powerful motherboards, but the problem is that there is nowhere to connect additional power cables, and they are not included in the kit, and this question is not raised in the manual. Huawei has its own artificial intelligence accelerators, and this may be the reason, but in any case, this server is not claimed to be GPU-compatible.

7. What built-in network interfaces does the server have?

Huawei Taishan 2280 V2 has two slots for network cards in mezzanine format, and one of them can be changed without access to the server (cold swap). In Huawei terminology, these mezzanines are called FlexIO, and today there are two types of such cards: 4-port Gigabit copper (model TM210) and 4-port 25 Gigabit optics (model TM280).

As you can see, we have a simple TM210 installed, and it does not represent anything interesting: there is no hot swap, the dull Realtek 8211 is used as chips.

8. Whose chips are used for high-speed interfaces?

A server with an additional network card with 4 ports of 25 Gbps came to us for testing. It will probably be superfluous to say that there is no information on this controller named BC51ETHB, and in order to shed light, we had to check the processor marking. Imagine my surprise when I saw again the brainchild of Huawei, the HiSilicon I1822 FPGA processor. This is the same chip that is used in the Huawei IN200 controller, which is positioned by the company as an Intelligent NIC, and not just a network card (read how SmartNIC differs from a network card ).

The controller supports RoCE, iWARP, SR-IOV and has the following hardware offloads:

• VLAN offload
• Stateless offload (Checksum/TSO/LRO/RSS)
• VxLAN tunnel encapsulation/decapsulation offload
• OVS TC Flower offload

For heavy network traffic, such as security gateways, this card can reduce CPU usage by 15-20% and lower latency by 30%. The power consumption of the board is up to 15 W, and by the way, the card itself is supported by all operating systems, including VMware ESXi, Windows Server, RHEL, Ubuntu and others.

9. Is Huawei also responsible for data security?

The holy of holies of any server, the disk subsystem is at the mercy of LSI/Avago/Broadcom products: an active backplane and a SAS3508 mezzanine controller, supporting RAID up to level 60 with an immortal super-capacitor for cache protection -memory and the ability to configure via the WEB interface. The controller can switch to JBOD mode for use in software storage, especially with a ZFS file system. The Backplane also uses an LSI processor, so as elsewhere in the server world, this is nothing new.

Media has Vendor Lock, and as an OEM core they mainly use HGST for HDD and Samsung for SSD, but not necessarily.

10. What about memory?

In general, everything is interesting with memory: there are 32 slots for DDR4 DIMM ECC Registered modules with a frequency of 2933 MHz. You can install dies from 16 GB to 128 GB, typing a crazy 4 Terabytes on one host, but keep in mind: all memory modules must be the same in terms of size, frequency and others parameters.

11. Can I replace the processor, or buy a 2-processor server with 1 CPU?

No, because Kunpeng 920 processors are soldered on the motherboard in the amount of 2 pieces. From the Hi1620 marking, you can guess that we have a 48-core Kunpeng 920-4826. It has 48 physical cores, 46 MB of L3 cache, a frequency of 2.6 GHz and a TDP of around 158 W. In these parameters, the Chinese is very similar to AMD EPYC, but even more.

Processors do not support HyperThreading, so 96 cores means 96 threads. The CPUs are interconnected by two Hydra Link buses, each of which has a bandwidth of about 30 Gbit/s. If you are worried that this will not be enough for cases when, for example, a virtual machine screwed to the core of the left processor works with memory served by the right one, then you can enable the `` One NUMA Per Socket '' policy in the BIOS, then the hypervisor will be able to consider the memory and the core as a single the object to which the VM is assigned. By the way, it's very cool that this was done in the server settings, and not at the software level: VMWare has a separate case dedicated to how they treated this problem on AMD EPYC Napples processors.

12. Does the ARM server have a BIOS?

If you were worried about whether the ARM server has a BIOS, then breathe out with relief: there is. Not AMI or Award, but our own, and by server standards - very technological. You can configure PCI Express slots parameters, memory polling rates, configure the RAID controller and other basic parameters from the common interface.

Remember to change your remote control password.

13. Do you have remote control?

Admit it is unusual to see a server without an ASpeed ​​AST1500/2500 chip, which is responsible for BMC, and all because it uses a Hi1710 controller from the same Huawei. From the point of view of all sorts of "anti-sanctions" - this is ideal, because ASpeed ​​is a Taiwanese company, and Hisilicon is the same Huawei, and it is unsafe and not prestigious to trust a third-party server to manage the server.

In addition, we have a full-fledged HTML5 interface, which (let's be honest with ourselves) is still far from ASpeed: there is no support for mobile browsers, you cannot connect an image via CIFS/NFS, but the truth is, there are some very good things, for example, simultaneous work of several operators with the console, automatic screenshots and video recording of errors and the choice of the BMC user role, as well as integration into the domain.

In general, the design shows that Huawei has extensive experience in the production of x86 servers, and here the company simply realized its potential, which is read literally in every detail. And before we revive this monster by pressing the virtual Power button, let's repeat which components are from which country. Considering that everything is made in China, I am interested in what jurisdiction the headquarters of the developer of this or that device belongs to.

Of course, for many years we have been writing in reviews that it is time to switch to SSD, but the reason has never been political.

14. Which Linux is installed on the server?

Almost all server distributions have an ARM64 version called "aarch64". Almost all of the same packages that are available for the AMD64 version, compiled for ARM64, are installed by the same commands from the repositories. For testing, I used Debian 10, the installation of which did not differ at all from the installation on other hardware: all network interfaces are picked up without problems, so you can use the Netinstall distribution.

15. What software is available?

I love Debian because it has there is a list of available packages with an indication of the architectures for which they are compiled, so you can see in advance before buying a server - is there any software you need in the repository, or can it be compiled from source? Immediately after installing the OS, I installed: Webmin and Cockpit for remote monitoring, MariaDB, QEMU-KVM for virtualization with GUI control and Jupyter to run Python notepads and Git to access the world's largest source code library.

That is, it is important to understand: such basic components of Enterprise distributions as DBMS, remote control, Web server and virtualization are debugged and work in such a way that you will not notice the difference with x86: the same commands, the same configs.

16. What software are you having problems with?

With scientific software developed by the non-community. Well, for example, we start with Tensorflow: yes, it is not for ARM64, but the developers site has an instruction how to compile it from source: it's only 3 commands and 5 minutes in the console, but as soon as you start installing libraries via pip, then here you will have problems: you need to install something from Github, look for something in the source and compile if you find it. An excellent environment manager like Anaconda under ARM64 there is no at all, and as a result, the Keras libraries, including matplotlib, pandas and others, are full of errors during installation.

Of course, all the software within the framework of the review cannot be checked, and you cannot tell about everything, but you must understand that for the operation of some software that is not part of the linux distribution, it is better to add an abstraction level and use third-party assemblies if there is. And here, for the first time in my entire acquaintance with the Taishan server, I meet a serious drawback of this whole project: Huawei does not have any service or mini-market like Bitnami, from which one could download applications packed in a container or virtual machine.

17. What about virtualization?

VMware this October showed ESXi work on ARM architecture, but so far this is a long-term affair, and today virtualization works only under Linux, and without x86/AMD64 emulation. This means that inside the VM you can only install ARM64 operating systems, and you will not have mos compatibility with the Intel/AMD based stack.

Let's just say: not very good, because the hypervisor does not have x86 emulation, which means you can virtualize, but all your VMs will have the ARM64 architecture. Simply put, you won't install Windows through a hypervisor, you won't get compatibility with the Intel/AMD-based stack, so either you have to build a closed cluster on the aarch64/ARM architecture, or use container virtualization, and run docker pods under KVM.

19. What about containers?

Everything is fine: Docker is installed the same under x86 and ARM64. Using Portainer , I have combined in one interface a server on AMD EPYC and our test Huawei Taishan 2280 v2 to visually show that you can have a common interface for managing your containers under any platform. True, you cannot transfer image files between AMD64 and ARM64, but you can independently package your application into a docker image and convert it to any architecture.

If you google deep, you can find that people manage to build ( example 1 , example 2 ) shared cluster with hosts on x86 and ARM64, but mostly Raspberry Pi and TV boxes. If you don't set yourself the goal of making friends with architecture from different eras, then you can not bother and leave everything as it is.

I think it would be superfluous to say that Nginx, Redis, MySQL/Mariadb, and even ready-made Wordpress/Drupal/Joomla images are installed in one click.

20. What's up with speed?

To one degree or another, the performance of all modern CPUs in typical tasks is comparable, and in its class the Kunpeng 920 focuses on the niche of top-end Intel and AMD processors with 48 cores and more. We do not have a 2-processor server at hand to put ARM architecture and x86 face to face, we only have a 32-core AMD EPYC 7551p, which we can use just to understand if it is possible to compare AMD64 and ARM64 in terms of speed?

And of course, we apologize in advance to those who were waiting for a war like `` Intel vs Huawei '', but it has been proven many times that in the server segment the software version has a greater impact on performance than the architecture and processor model. But since there is a complete misunderstanding among my colleagues about the speed prospects of ARM64, our tests will show you what you can count on when purchasing these servers.

Let's take two databases: for 100 thousand and 10 million records and compare the behavior of machines under the same conditions, which will allow us to extrapolate the results a little for your needs.

With the increase in the number of streaming connections to the base, Taishan 2280 v2 begins to break away, and I would not say that it is a matter of the number of physical cores (96 for Huawei versus 32 for EPYC), because even on 16 threads the difference is almost twofold. In 1-stream mode, we will not consider the data as not having any meaning.

In terms of the number of transactions per second, the difference in speed becomes greater with an increase in the number of threads and an increase in the dataset.

In a non-relational database, the difference is less noticeable, which once again confirms the thesis that “there is power not in processors, but in software code!” But in summary, we can say that the performance of ARM servers is quite comparable to similar x86 machines with the same number of cores. In some specific scenarios, ARM can give a breakthrough, and in some - x86.

21. What's the money?

Literally in every announcement regarding servers on ARM, a paragraph skips about how beneficial they are in terms of CAPEX and OPEX, so let's use the retail GPL and calculate how much Huawei Taishan 2800 V2 is more profitable than its competitors on Intel and AMD? Today, Intel only has processors up to Xeon Platinum 8180 on the open market, but to make the comparison more honest in terms of operating frequency, we will use Xeon Platinum 8180 (2.5 GHz, 28 cores), that is, we keep in mind that in the case of Intel there are almost half of us physical nuclei. With AMD, the same situation: HPE currently does not have models with EPYC Rome, so we are limited to the first series of processors, having 64 cores on board. In addition, we will ditch the US products and take a couple of SSDs, no hard drives.

And now it is clear that the only competitor in terms of price is a mythical server with two AMD EPYC 7542s, which HPE did not have at the time of writing.

Recommendations for IT specialists when ordering

Today, demand for ARM processors in the data center segment is showing double-digit growth y/y, and it is mainly supported by hyperscalers purchasing servers in the hundreds and thousands. Seeing a ready-made working server on ARM64 in front of you, you understand that the future has already arrived, and you can smoothly begin to build data center segments on an alternative architecture, using Taishan 2280 v2 as bricks, and leave the functions of storage systems, SD-WAN and security gateways at the mercy of software.

Things to remember before buying the Taishan 2280 V2:

• This is the same high-tech hyperconverged server like some DL380 G10 or PowerEdge R840. But unlike them, this is a real New Age, really something new over the past 30 years.
• Only Linux is here now! No VMware, FreeBSD, or holy-holy-holy, Windows Server is not here, and whether or not it will appear is not clear. Commercial distributions are friends with Kunpeng processors and provide technical support for them.
• Everything is ready to implement the import substitution plan
• Standard simple tasks such as hosting, working with databases, traffic balancing, VPNs work here right out of the box, installed and configured with the same commands as for x86/AMD64.
• The more applications you have moved into containers, the easier the deployment will be.
• Problems may arise with commercial software that does not distribute source codes and with some libraries, especially scientific ones.

Considering that this server has the highest degree of independence from American companies, it is an ideal solution for building a scalable infrastructure based on Kubernetes or Docker Swarm, with a performance of at least x86/AMD64, but at the same time resistant to trade wars and sacks.

Mikhail Degtyarev (aka LIKE OFF)
15/04.2020