Details of Facebook Open Compute cloud servers

By and large, Facebook's basic architecture is not very different from other high-performance web services. However, Facebook & ndash; advertising brainchild of new generation cloud applications. Its high popularity and high degree of interactivity in itself requires much more scalability and accessibility than a conventional site providing mainly information or a simple service.

Generation of 'cloud apps' does not use the classic high-level principles of redundancy with massive relational database management systems. The combination of x86 scale-out clusters, open source software and non-SQL is what Facebook, Twitter, Google, and other systems are based on.

However, Facebook has put together some pieces of the puzzle in open source software to make it more suitable for extreme scalability. The Facebook developers chose PHP as their proxy language because it is easy to learn, easy to write, and easy to read. Despite this, PHP is very CPU and memory intensive.

According to Facebook's own data, the PHP code is almost 39 times slower than C ++. So it was obvious that Facebook had to solve this problem first. The traditional approach is to rewrite the highest performing critical pieces of code in C ++ as PHP extensions, but Facebook decided to take a different approach: the developers improved HipHop, a source code converter. HipHop converts PHP source code to fast C ++ code and compiles it with g ++.

The next piece of the Facebook puzzle is Memcached. Memcached is an in-memory data caching program with some very interesting features. Memcached is a distributed caching system; this means that memory caching with Memcached can span multiple servers. In this case, the term "cache" should be understood as a collection of smaller caches. Basically, the cache is recovered at the expense of unused RAM, which is likely to be wasted by the operating system for less efficient file system caching. These "cache nodes" are not synchronized or forwarded, and as a result, the cache memory scales well.

Facebook quickly became the largest user of memcached technology in the world and improved it significantly. The developers have ported the technology for support in 64-bit systems, reduced memory usage in TCP, enabled multi-core support (previously only one core was used), etc. Today, Facebook mainly uses memcached technology to ease the load on databases.

Facebook Server

In the basement floor of Facebook's Palo Alto, California headquarters, three custom engineers from the company designed servers, power supplies, server racks, and backup power systems that any hp server would envy. The Facebook server was supposed to be much cheaper than the average server and also more energy efficient. They called their project the Open Compute Project, bringing open source ideas to server hardware.

The first thing they changed was the chassis height. The 1.5U format was chosen as a compromise between the occupied space and the possibility of good cooling of the server. The 1.5U format allows the use of taller radiators, larger (60x60 mm) slow-speed fans, rather than the shrill “forties” energy-eaters used in the 1U chassis. As a result, fans consume only 2% to 4% of the total electrical power, which is quite surprising since it is known that 1U fans can require up to one third of the total system power consumption. It seems that the air cooling in the Open Compute 1.5U server is as efficient as in the 3U servers, previously considered the best in terms of energy efficiency.

At the same time, Facebook's R&D department uses very simple chassis with no plastic parts. At the same time, the air movement inside the server becomes more uniform, and the weight is reduced by reducing the number of parts in the case. The bottom wall of one server serves as the top one for the server below it. I saw how in one of the data centers there was not enough space in the racks and several hp proliant dl360 g6 servers were installed one on top of the other. To be honest, it looked strange.

Facebook has developed motherboards for AMD and Intel processors; motherboards are produced at Quanta factories. Much attention was paid to the efficiency of volt mods (efficiency = 94%). Another idea was to remove something that could be done without. These motherboards lack BMC monitoring, they have two USB and RJ45 ports, one expansion slot, and they lack integrated graphics.

The only thing that is available to the system administrator remotely is a reboot over LAN ("reboot over LAN"). The idea is that if this does not help, then the problem in 99% of cases is so serious that you have to send the system administrator to the server anyway. So why have extra components that make the product more expensive?

Machines based on AMD processors are mainly used as Memcached servers, since AMD Magny-Tours Opteron 6100 processors have 4 HyperTransport bus channels and can use 12 DIMM slots per processor - in total, this is 24 DIMM slots per system board. Thus, up to 384 GB of cached memory can be used, and this is the cheapest platform to install this amount of memory on a single server.

In comparison, Facebook Open Compute servers that use Xeon processors have only 6 DIMM slots and are used for high-CPU tasks: they are used as PHP "assembled" data servers.

Dual Bus Power Supplies

The power supply has two input connectors, one for 277 VAC and the other for 48 VDC. The PSU can run for about 10 minutes on the 48 V bus before overheating and will shutdown, so it is not designed to operate continuously on 48V DC. The idea behind the 48V DC bus is to get away from the traditional UPS system: the batteries must keep the servers running for a few minutes until power is restored AC bus 277 V.

Where does 277 volts come from? In the United States, power standards are somewhat different, in commercial buildings and enterprises. And 277 Volts are generated when input is connected to the output according to the 3-phase + neutral scheme at 480 Volts at the input (3-phase, star).

The power supply efficiency is extremely high - up to 94.5%.

By using 277V, Facebook can save about 3-4% in power consumption compared to 208V - a result of lower transmission current loss.

Conclusions

Traditional servers such as the HP DL380 G7 can be considered very efficient compared to previous generations. An integrated graphics adapter allows console management with IP KVM, and for remote management and monitoring of a fleet of servers, Hewlett Packard has created an excellent Integrated Lights Out management system.

Facebook Open Compute servers are a very interesting round of evolution. Cloud services built on servers are now building their own servers for cloud services. Interesting, right?

By itself, the type of server developed by Facebook will be interesting as a Cloud-only solution available for data centers offering various cloud products. It is unlikely to appear on the open market, but it is quite possible that the solutions used in this project will inspire the largest server manufacturers, Dell, HP, IBM and Supermicro to create their own servers of this type.

03/02/2011