Review of ASRock Rack EP2C621D12 WS motherboard for double Xeon LGA3647 processors

Intel is actively promoting its LGA-3647 platform not only for the server market, but also for top-end computers designed for enthusiasts, as well as for workstations optimized for complex parallel computing. Figuratively speaking, today this platform allows you to assemble a working machine with 56 physical cores (112 CPU threads) in one desktop case. Usually, this power is required for any scientific calculations with high accuracy, performed on the processor. However, both Intel and motherboard manufacturers understand that the customer is likely to use GPUs or discrete accelerators for machine learning functions, so Xeon Scalable processors in this case are considered the only Intel solution with a 48-lane PCI Exprees 3.0 controller. Want to connect more than three PCI-Express 16x GPUs? Opt for a dual-processor motherboard and go for even the simplest 6-core Xeon Bronze 3104, you have 96 PCI-E lanes.

ASRock Rack's EP2C621D12 WS motherboard is just that universal case when you choose a platform for a machine for LGA-3647, want to have a large number of PCI-Express slots and today's basic storage options. The main advantage of the motherboard is as many as 7 PCI Express 16x slots. As usual, clarification is required here: PCIE 3 and PCIE 4 slots share one x16 bus and work either in x16 + 0 format or in x8 + x8 format. The PCIE 2 slot operates in x8 mode, that is, there are actually 5 slots left for x16 speed, three of which you can use with 2-slot GPU cards without additional extension cords.

But besides the traditional PCI Express slots for expansion cards, the EP2C621D12 WS motherboard also has two M.2 slots supporting up to PCIE (x4) speeds. The first one is connected to CPU1, and the second one is connected to the Intel C621 PCH south bridge. Both slots support SATA3 drives in sizes from Key 2230 to Key 2280. Using PCI Express M.2 drives does not affect the speed of the expansion card slots.

Of course, the main storage system in this case is placed on a SATA array. To boot the hypervisor, the board provides two ports for SATA DOM drives (red) with additional power, but for the main array there are 10 SATA 600 ports, of which 8 are connected to the first SATA controller, and two more - to the second. In total, 14 SATA ports are available to you, which will be enough, without exaggeration, for any storage needs. The motherboard supports RAID levels 0, 1, 10 and 5 through Intel drivers.

But since the ASRock Rack EP2C621D12 WS motherboard was created not for servers, but for workstations, there is no SAS controller or PCI Express ports for connecting 2.5-inch SSDs. How important is it? From our point of view, two M2 slots are enough to hold a GUI operating system and cache data from SATA drives. As a last resort, it is always possible to install an accelerator board for 4 M.2 drives.

Peripheral ports

But the situation with peripheral ports is, to put it mildly, strange. In general, the board in question was created for workstations, but this is indicated only by the built-in audio path, and even that is not particularly high-quality: the Realtek ALC892 codec goes back to 2011, on the board the analog path is located in the lower corner, from which without additional of shielding, the signal simply goes through the entire very long board to 3.5 mm jacks, which in their appearance come from the same place, from the early 2000s.

The choice of network connections is even stranger: 4 interfaces of 1 Gbps, for which the rare Marvell 88E1543 controller is selected, a separate BMC port for remote management, and only 2 USB 3.0 ports. It's good at least that this network controller is supported by ESXi 6.x +, and the board has connectors for connecting two front USB 3.0 ports and two USB 2.0 ports: not enough, comrades.

In general, I believe that there is enough space on an EEB motherboard to install a USB 3.1 controller, and I absolutely do not understand the point of installing 4 network ports in a device that clearly does not want to be a network gateway. You need a fast network - put 10/25G, BMC management for a workstation is not relevant at all. In this regard, the Asus C621E Sage is more comprehensible and balanced.

Perhaps there is nothing to play with - ASRock Rack just undeservedly placed this board in the "for workstations" section. If you look at everything the same from the point of view of the server, then everything is clear, except for the Marvell network chip.

Power Patterns

The motherboard supports processors with a TDP of up to 205 W, that is, all the existing Xeons of the Skylake series, including the top-end Xeon Platinum 8180. Power is supplied to each processor via an 8-pin ATX 12V connector, but ATX 24-pin There is only one connector, so one power supply for the machine will be enough.

The memory modules use a 2-phase power supply circuit with solid-state polymer capacitors E88E with a temperature rating of 105 degrees Celsius.

Each processor has a 6 + 1-phase power system with a Renesas ISL69127 controller.

On the reverse side of the board are solid polymer capacitors, as on most modern professional motherboards.

For fans here, frankly, heaven: 7 double 6-pin connectors, each of which has its own LED indicator of failure. The board has automatic speed control for processor, front and case fans.

IPMI control

BMC-control is carried out using the modern Aspeed AST2500 chip, the most modern solution with a normal HTML5 interface, which is also compatible with mobile devices. You do not need to install any client software on the operator's computer, you do not need to bother with compatibility with browsers - everything works under PC, and under MacOS, and under Linux and Android.

From the KVM interface, you can connect virtual media from a local computer, but this is the last century. It is much more convenient to mount a folder on a NAS with OS installers and choose which image and how to connect it. The machine scans the CIFS/NFS folder for .ISOs and offers you a connection not only in the form of a floppy disk or CD-ROM, but also in the form of an HDD so that you can boot over the network. True, the installation showed that Virtual Media works at a speed of about 1.5 Mb/s, which is frustrating: there was no breakthrough, but in vain, because this is not just a method of installing an OS remotely, but also a practical replacement for PXE boot.

Of course, the Java application also remains, but now it is only needed to launch the on-screen keyboard, if, for example, you find it difficult to press the F11-F12 keys under MacOS.

Power management is highlighted as a separate item, sensors and logs are separate. It's a pity that there are no graphs in the interface, I really liked to monitor temperature changes, but this is partially offset by a very clear interface for viewing logs.

BIOS

The motherboard has a standard AMI (Aptio) BIOS with a standard set of settings for server boards. I would like to draw your attention to the fact that the motherboard has the Watch Dog function, which for some reason is not installed in the workstation model (it has two options: NMI and Reset).

Basically, the BIOS has everything a server motherboard needs, including boot and error logs.

Testing

• Processor: 2x Intel Xeon Gold 6146
• Memory: 32GB (2x16GB) DDR PC2400
• Vmware ESXi 6.7
• Debian 9

MySQL OLTP latency, lower = better

Testing the speed of Virtual Media:

• AMI Virtual CD (image is stored on NAS) - 1512 Kb/s
• AMI Virtual CD (the image is connected from a computer) - 1680 Kb/s

At times the speed jumps up to 4 Mb/s, but this is more like a cache. So, for the needs of BMC monitoring, a regular 100-Mbps switch will be more than enough.

Conclusions

ASRock Rack makes excellent server motherboards, and if you consider EP2C621D12 WS as an option for a machine for scientific computing and machine learning, then this is the best option. There are 7 PCI Express slots for expansion cards, and a large number of SATA ports, well, in general, everything that should be in a modern highly integrated motherboard.

What we liked:

• Rackmount board design with airflow perpendicular to the ports on the motherboard.
• Two M.2 slots that do not share performance with other PCI Express ports
• Two SATA DOM ports
• Very good IPMI/BMC interface
• Legacy BIOS
• 6-pin cooling headers for powerful fans

What I didn't like:

• Only 2 USB ports on the board panel
• Lack of 10G Ethernet ports
• Audio interface is only suitable for working through RDP

Mikhail Degtyarev (aka LIKE OFF)
25/03.2019