Review of Asrock Rack EPC612D4U-2T8R motherboard for cloud storage

For keeping the 'cold' data, where, first of all, the volume in terabytes per one unit of the server rack is important, there are compact 1U enclosures that can accommodate 12 3.5-inch hard drives. Integrators and assemblers can use for these needs both ready-made server kits and assemble storage systems on their own in special cases, such as Chenbro RM14500 .

In such cases, the motherboard is given a minimum of space, and not so much is required from it - to have a maximum of SATA/SAS ports and be as compact as possible in order to give as much space as possible for hard drives. It is for such purposes that ASRock Rack, the server division of the familiar ASRock company, releases the EPC612D4U-2T8R motherboard for Intel Xeon LGA2011 v3 processors.

This board is interesting in that the manufacturer has integrated an 8-port SAS-12Gbps controller (LSI3008 chip) into a compact MicroATX form factor, two 10-Gigabit BASE-T network ports on an Intel X540 controller, two PCI Express 3.0 16x and one PCI Express 3.0 8x slots. In total, you can connect as many as 16 hard drives and 14 network ports to one Micro-ATX board, which is impressive!

To ensure high-speed communication with other nodes, the board is equipped with a dual-port Intel X540-T2 network controller with RJ45 interfaces for copper cabling. Today, almost all Asrock Rack 10 Gigabit motherboards are BASE-T, as the general trend in storage systems is to use conventional twisted pair for network connectivity, while the optical media remains for faster interfaces. The Intel X540 chip is not the newest in the 10 Gigabit controller family, it does not have the power saving features of Intel Ethernet Power Management, but like all Intel 10 Gigabit chips, it supports iSCSI, FCoE and NFS hardware offloading and has SR-IOV functionality. that allows you to share board resources between virtual machines. At a maximum 10 gigabyte speed, the cable length of the 6th category can be up to 55 meters, and when using the cable category 6A - up to 100 meters. Network ports can be grouped according to the IEEE-802.3ad standard for increased throughput and resiliency.

The next important element of the Asrock Rack EPC612D4U-2T8R is the 12Gb/s SAS controller Broadcom (LSI) 3008. This is the simplest chip in the SAS-12 family of Broadcom chips, having a PowerPC 476 processor with a frequency of 1.2 GHz. It supports RAID 0,1,1E and 10, has a PCI Express 3.0 8x bus, supports the T-10 data protection type and DataBolt technology, which allows operating at SAS-12Gb/s speeds when using SAS-6 Gb/s drives and backplanes ... Overall controller performance is claimed to be 1 million IOPS , so the headroom is huge.

By the way, continuing the topic of storage, 8 more SATA-600 ports operate on two SATA controllers built into the Intel C612 chip, and devices can be combined into RAID 0, 1, 5 and 10 arrays.

Due to such a tight layout, there is no space left on the motherboard for the Ultra M2 drive, although this is where it would be very useful for caching I/O operations, or for the operating system if you plan to install Windows. For Linux, VMware and FreeBSD there is an internal USB 3.0 Type A port, into which you can install a low USB flash drive with a bootloader and an OS kernel.

The board has a large number of connectors, connectors and a whole block of jumpers. Here are just a few:

• Pins for connecting the NMI button (softer alternative to reset)
• TPM connection block
• Thermocouple probe (TR1)
• Front VGA out
• Dual USB 3.0 ports
• Dual USB 2.0 output

Reboot via NMI is more "gentle" an alternative to reset, as it helps to avoid data loss during intensive disk writes.

One LAN1 network port is dedicated for BMC remote management. Remote monitoring is assigned to the Aspeed AST2400 chip (network controller - Realtek RTL8211E), the most common server solution to date. In fact, although the board is announced as 3-port, there are two active network ports, but their total speed of 20 Gbps is more than enough for the next 5-6 years.

It should be noted that although the motherboard has 3 PCI Express 3.0 slots, they share the data bus with the following restrictions:

• Upper PCI Express 3.0 16x (PCIE5 slot) when the SAS controller LSI3008 is enabled, it switches from 16x mode to 8x mode.
• The bottom PCI Express 3.0 16x (PCIE7 slot) shares 8 lanes with the middle PCI Express 8x (PCIE6) slot, therefore, when an expansion card is connected there, it goes into 8x mode.

The form factor of the middle PCIE6 slot allows you to install long expansion cards designed for 16x into it, but when installing the motherboard for its intended purpose, in cloud storage systems, most likely all three slots will remain empty, since installing there is simply nothing in them: everything you need is already integrated on the motherboard.

The processor socket has Narrow ILM format, which is important to consider when choosing a cooler. For floor-standing servers, you can find on sale cooling systems Noctua NH D9DX i4 3U high. In general, there are no problems with low-profile coolers - for 2U cases you can use Supermicro SNK-P0048AP4, and for 1U cases - Dynatron T318 or Dynatron R31.

The motherboard has 4 DDR4 RDIMM/LRDIMM slots with frequencies 1600/1866/2133 MHz and 2400 MHz for Xeon E5-1600/2600 V4 processors. The maximum supported capacity for a single RDIMM is 32GB and an LRDIMM is 64GB. If we assume that for 1 TB of disk space it is desirable to have 1 GB of RAM for cache and deduplication tables, then this is enough for a 1U case crammed with 12 hard drives of 21 TB each.

I would like to note the support of 6 fans, each of which is connected to a 4-pin PWM socket with LED indication. If the cooler stops, a red light near the PWM connector will turn on. Automatic CPU cooler adjustments tend to keep the fan speed to a minimum.

There is only a minimal set of outputs on the rear panel: two USB 2.0, two USB 3.0, three RJ45 ports, VGA, RS232 and a UID indicator.

Remote control

BMC supports a virtual desktop function with the ability to mount ISO files as a CD-ROM, flash drive or floppy drive, the so-called Virtual Media function. Any of the system sensors, for example, the processor temperature, can be opened as a widget and constantly track its change on the graph. At the same time, surfing between windows of the web interface does not affect open widgets. KVM software console allows you to record videos, take screenshots and dump Blue Screen of Death settings to facilitate diagnosis. True, if you plan to use a GPU to display an image on the screen, then the virtual KVM console will become unavailable, and for such cases it is better to purchase a separate IP-KVM, such as ATEN KN2124VA .

We tested Virtual Media speed and got about 3.7 Mb/s , so a gigabit connection is clearly redundant here, and by connecting remote monitoring to a 100 Mbps network switch, you are nothing you will lose, on the contrary - it is very convenient when installing updates supplied as ISO images for hypervisors on FreeBSD.

After testing the BMC for two weeks, let's just say - there are no serious complaints about the device's operation: the mouse cursor is synchronized almost perfectly, keystrokes are transmitted without delay, but sometimes, with intensive work of the main network adapter Intel X540, the BMC interface becomes unavailable, and it was not possible to understand the pattern of this behavior. I would also like to draw your attention to the fact that power management is carried out in the java-widget window responsible for KVM, so to turn the server on and off, you will have to enable pop-up windows and JAVA in the browser.

The hardware monitoring section is very happy - the board displays readings of all temperature sensors, including the motherboard, integrated network card, south bridge, memory modules and processor, as well as readings of all voltage sensors and all cooler speeds, which is in the system.

Total, for the functionality of the remote control system, you can safely put the top five: fast, convenient and logical.

Operating system compatibility

We have tested the compatibility of ASRock Rack EPC612D4U-2T8R with the latest operating systems. In all cases, the installation was carried out via a remote connection to the BMC using Virtual Media.

• Windows 10 x64 build 1807 - requires network drivers to be installed after installation
• Windows server 2016 build 1705 - everything works out of the box
• VMWare ESXi 6.7 - everything works out of the box, including hardware monitoring
• Debian 9 Stretch initial release - everything works out of the box

UEFI (BIOS)

ASRock Rack EPC612D4U-2T8R uses AMI BIOS with traditional text interface. It is in the general BIOS that the built-in SAS controller LSI3008 is configured, which, by the way, by default operates in "IT Mode", which is important for modern file systems such as Btrfs and Zfs.

Generally, most of the settings are related to various CPU functions, serial ports, and control system. The function of waking the motherboard from sleep mode is supported through a signal from PCI Express, as well as over the network. But the possibility of turning on by signal from USB is not provided.

The motherboard has a built-in Watch Dog timer that operates in two modes - a simple cold reset and an NMI interrupt call. The latter is recommended just when using smart filesystems with extensive RAM caching. The Watchdog timer requires support from the operating system, and from the tested operating systems, it was easiest to achieve its work only in Windows Server 2016.

Testing

For testing, we used a computer with the following configuration:

• Intel Xeon E5-2603 v4 (6C 15M L3, 1.7 GHz)
• 16 Gb DDR4 ECC RDIMM
• 8 x SSD Samsung MZ-7KM480E, 480 Gb, SATA-600

Below are the AIDA64 benchmarks to evaluate the board's performance.

Let's test the speed via iSCSI and compare the response times on the built-in and discrete network controller.

Integrated Intel X540-T2 vs Intel X550-T2 vs Intel X520-DA2

During testing, we noticed that the only very hot element on the board was the PCH, whose temperature exceeded 60 degrees Celsius without a fan installed, so if you want to use ASRock Rack EPC612D4U-2T8R for a tower server in a non-industrial case or for a NAS (ideal, by the way, option) - be sure to provide good airflow to the radiators.

Tender protection

ASRock Rack EPC612D4U-2T8R is a unique motherboard, of which neither Tyan nor Asus makes analogues . A very similar X10SRM-TF board is manufactured by Supermicro, but it does not have an integrated SAS controller. Therefore, the simultaneous requirement to support 10GB LAN, SAS-12G and 3 PCI Express slots in MicroATX format will not allow entering the project with products from other vendors.

Conclusions

ASRock Rack is a motherboard specialist, and the best part is that the company's lineup includes non-standard highly integrated solutions in a compact form factor. These cards are versatile and can be used for virtualization compute nodes, SDS solutions, and high-density storage systems, including Flash arrays in 1U and larger enclosures.

The versatile MicroATX form factor allows the board to be installed not only in high-density full-depth enclosures, but also in shortened half-depth models designed for installation in wall-mounted telecom cabinets. This flexibility enables a server manufacturer to offer a customer a unique solution that will have no competitors in retail and will be able to pass through tenders without the risk of supplying analogues from cunning competitors . It is this feature that makes ASRock Rack EPC612D4U-2T8R a great buy on the market.

Mikhail Degtyarev (aka LIKE OFF)
21/08.2018