Review of ASRock Rack EPYCD8-2T server motherboard for AMD EPYC 7000

The good thing about the AMD EPYC 7000 platform is that a large number of PCI Express lanes are available to you in 1-socket format. This is a serious advantage for building a server with several GPUs, because the 1-processor layout allows you to assemble compact machines in the ATX form factor.

This motherboard from ASRock Rack is designed for just such an application, and unlike its counterparts, this model already has a pair of 10 Gigabit network ports per bot, that is, you do not need to spend money on interconnects.

Memory and expansion slots

The platform has 8 DIMM slots so you can enjoy the performance of an 8-channel processor memory controller, but this is not the main plus of the EPYCD8-2T. Here you have 4 PCI Express 3.0 16x slots available, plus 3 PCI Express 3.0 8x slots, and 2 more NVME M.2 slots. The standard ATX layout allows you to use any number of expansion slots without the fear of a video card or any expansion card running into memory slots or cables.

Of course, the expansion slots are not quite standard: the topmost slot is located closer to the memory slot than is usually allowed, so in some cases you may not be comfortable opening the latch if the nearest DIMM is busy. Moreover, this is not the only complexity of the assembly.

For example, opposite the fifth PCI Express slot, there are three slots for connecting system fans, and a long video card will easily block them.

Cooling system

No, of course this motherboard has no shortage of connectors for fans: there are 7 of them (1 CPU/4 Front/2 Rear), of which 4 are located near the VRM processor, next to OCULink ports. And the SATA ports are deployed parallel to the board so as not to interfere with your video cards.

Each fan connector has a breakdown LED to make it easier for you to see which connector to disconnect if the impeller stops. For fans, automatic speed control is available depending on the processor temperature. In total, the motherboard monitors three temperature indicators: CPU, Motherboard, MB Card Side and TR1 remote sensor (not included). Of course, the monitoring system can read temperature readings from the controllers of the RDIMM modules.

Storage Engine

ASRock Rack EPYCD8-2T has an abundance of storage in terms of storage. First, 8 SATA ports are implemented here via 2 Mini-SAS connectors for storing cold data and data-at-rest data. There is also a ninth SATA port for a SATA-DOM flash drive, on which you can install an operating system. Please note that RAID functions are not declared, so you will have to programmatically merge disks into arrays.

Secondly, for PCIE SSD you are given two OCULink ports, which are likely to be out of work, because there are two M.2 ports supporting media up to 22110. Please note that the use of PCI Express SSD has no effect on the bus width of the PCI Express expansion slots: EPYC 7000 processors have 128 PCI Express 3.0 lanes, and you can not save them here.

Network & Peripherals

We are considering a top-end version of this motherboard with 2 10G Base-T ports based on the Intel X550 network controller. Yes, ASRock Rack EPYCD8-2T is the very case when, meeting on the same board, Intel and AMD do not conflict, but complement each other.

The network controller is cooled by some prohibitively large heatsink, which is clearly designed for the fact that in ordinary life it will be closed on top by several hot video cards. The presence of a fast network interface is a huge plus for this motherboard, because you simply do not spend money and a PCI slot for an additional controller.

However, EPYCD8-2T is a server board, without a hint of use in workstations, so there are only two USB 3.0 ports (plus one more block for connecting two front USB 3.0 ports), no audio path and no frills kind of bluetooth. Oh, what a pity, because the company does not have a similar card for graphics stations.

Power circuit and components

The EPYCD8-2T motherboard features an 8-phase processor power supply with solid capacitor filters. Here, power filtering is installed even on PCI Express slots, although not every vendor takes such excessive care of power supply. But in general, only 2 complex chips are soldered on the motherboard: a network controller and a remote control processor. In the absence of the southern one, the board turns out to be less loaded in terms of temperature and power supply, and simply put, there is nothing to break here, it is eternal.

Let's calculate how PCI Express 3.0 lanes are allocated in this card:

• PCI Express x16 Slots: 64 Lanes
• PCI Express x8 Slots: 24 Lanes
• M.2 PCIe 3.0 x4 Slots: 8 Lanes
• Intel X550 Network Card: 8 Channels
• 8 SATA ports: 8 lanes
• 2 Oculink ports x4: 2 channels
• Aspeed AST2500 chip: 1 channel

Total: 115

As you can see, there is still a margin for PCI Express lanes, because we counted only 115 out of 128 possible lanes, so that's why people choose AMD EPYC: you can use all PCI Express devices at full speed.

Remote control

For remote management, an ASPeed AST2500 chip is used with a dedicated 1 Gigabit network port, which is used only for BMC functions and nothing else. I would like to say that AST2500 is the best solution for today, having a modern HTML-5 interface, optimized for tablets and smartphones. You have access to fast and convenient console management, connecting a remote disk for installing the operating system both from a computer and through a NAS. There is a very convenient temperature monitoring and the ability to record POST tests.

Compared to Dell iDRAC, HPE iLO and Lenovo XClarity management systems, here you do not pay for remote management, but get almost the same capabilities, not to mention the fact that you can simply forget about IP-KVM like a bad dream ... Yes, and one more thing: if you do not use a dedicated gigabit network port, you can access the BMC through a 10-gigabit port: the board implements a kind of network switch, so that we can access the hypervisor web interface via one network wire, and in the BIOS via the console. Cool!

The motherboard has an AMI BIOS with a standard set of settings. Among the sensors, the temperature of the Intel X550 network card should be noted, and among the peripheral settings - the bandwidth limitation of PCI Express slots.

Testing

First of all, let's check at what speed the Virtual Media image is mounted, because this is a very important moment when deploying a software environment in a data center.

The performance here is on par with other motherboards with the AST2500 chip on the Intel platform. Let's look at the speed of the network connection:

In general, Intel X550 is a very interesting network card, which should be said about the following. Firstly, unlike Intel X540 and Intel X557 (used in Xeon-D), this chip has no compatibility problems with old 1G equipment, especially from Mikrotik and Ubiquiti, that is, old switches may not work with the above two chips. and with Intel X550 they always work. Of course, you can say that there is no point in plugging a 10 Gigabit card into 1 Gigabit ports, but I disagree with you, because anything happens at work. Secondly, Intel X550 will inherently support intermediate speeds of 2.5 Gb/s and 5 Gb/s when using a standard Category 5E cable, but the implementation of these standards depends on the device that uses this chip. For the server market, where switching is carried out between racks, intermediate speeds are of no value, because inside the rack, 10G works quite well over 5E cable, and it is easy to stretch CAT6/7 between the racks. In our test, for fun, I tried to get intermediate speeds, but it came out either 1 Gb/s or 10 Gb/s. Well, as they say, no one else promised: the main thing is that 10 Gbit/s works as well as on the discrete Intel X550-T2 (read our AQuantia AQC107 vs Intel X550 comparison)

How to buy this board via tender

To get the ASRock Rack EPYCD8-2T at the tender, indicate the 10-gigabit network with 10GBase-T ports, the presence of Oculink and M.2 ports in a single-socket platform for Epyc 7000. It is such one, and no substitutes will be brought to you .

Conclusions

If you want to build a server with 4 GPUs in an ATX case, then here's a great solution. You don't need to buy a high-speed network interface or a controller for SATA drives - ASRock Rack EPYCD8-2T already has everything you need on board, plus a gorgeous remote control system with iKVM that you get for free. Comparing with analogs, you are surprised that ASRock Rack engineers did not limit the flight of their imagination and even fit two Oculink ports and two more popular M.2 ports on the board.

As such, the motherboard has no shortcomings, this is an excellent combine that lacks only the audio path for use in workstations, but if your goal is to build a server for GPU computing, then EPYCD8-2T is a platform that you can feel free to put into the project, that rare case when we have a board without flaws.

Mikhail Degtyarev (aka LIKE OFF)
06/06.2019