What is Remote Direct Memory Access (RDMA)?

Remote Direct Memory Access, RDMA for short, makes it possible to send data quickly and easily from a computer’s main storage to the main storage of another computer. This is es­pe­cially helpful when dealing with large data sets or complex processes, which tend to be required, for example, for machine learning. But what exactly is RDMA and how does it work?

As is the case with the related direct memory access (DMA), one can use remote direct memory access (RDMA) to transfer data directly from a computer’s main memory or working memory to the storage of another computer. In this case, operating systems, CPUs, and cache are com­pletely ignored so that fewer hardware resources are burdened. For this purpose, the storage transfer with RDMA runs via a private buffer in which the user transfers their data. In doing so, the tech­no­logy accesses a system’s network cards, where the transfer is processed via Ethernet or In­fin­i­Band.

Many products (hardware and software) already support RDMA. Among these solutions are:

• Apache Hadoop
• Spark
• Baidu Paddle
• Dell EMC PowerEdge Server
• Intel Xeon Scalable Pro­cessors
• Microsoft Windows Server (2012 and more recent versions)

For the data exchange via RDMA to work, a suitable set of protocols in the network cards is required. Typically, TCP/IP is used as a basis. Only through a suitable transport protocol can the tech­no­logy support so-called zero copy net­work­ing, for example. Zero copy net­work­ing does not utilise the computer processor. Where both systems enable the use of Remote Direct Memory Access, the data transfer between them is con­sid­er­ably faster than that between systems without RDMA support.

For a smooth transfer with RDMA, the following network tech­no­lo­gies and in­ter­faces offer the best re­quire­ments:

• RDMA over Converged Ethernet (RoCE): RoCE enables RDMA via an Ethernet con­nec­tion.
• Internet Wide Area RDMA Protocol (IWARP): For the data transfer, IWARP relies on the transport protocol TCP, or al­tern­at­ively, on streaming TCP. IWARP was initiated by the Internet En­gin­eer­ing Task Force (IETF) in order to write tasks and processes directly onto ap­plic­a­tions in another system.
• In­fin­i­Band: In­fin­i­Band is a com­mu­nic­a­tion standard for high-per­form­ance computers in order to transfer files with lower latency. It is fre­quently used in computer centres in order to connect computer clusters with one another. RDMA via In­fin­i­Band is one of the most popular methods for quick data exchange.

However, Remote Direct Memory Access can also be used in con­junc­tion with Flash or SSD storage mediums and NVDIMMs (non-volatile dual in-line memory modules).

The evolution of RDMA is still in full swing. As with RDMA over Fabrics, its next potential ap­plic­a­tion is already in the starting blocks. In­fra­struc­tures between several servers and computers are referred to as a fabric. They support the data transfer via fibre channel networks (storage area networks) and PCI Express (standard for high-speed con­nec­tions).

One of the main ad­vant­ages of RDMA is its out­stand­ing speed when compared with other tech­no­lo­gies and protocols for data transfer such as iSCSI (SCSI protocol via TCP), fibre channel (FC) or fibre channel over Ethernet (FCoE). However, the final speed of the data exchange also depends on the RDMA variant. Ethernet and In­fin­i­Band are es­pe­cially popular, as these enable transfer speeds of 10 to 100 Gigabit per second. This is es­pe­cially suitable for areas of ap­plic­a­tion that require high computing power, such as dis­trib­uted databases, big data analyses or ap­plic­a­tions in data centres.

However, RDMA also has dis­ad­vant­ages when compared with fibre channels, which are still used by many companies. In order to introduce RDMA, companies must arrange high in­vest­ments, as the tech­no­logy requires the ac­quis­i­tion of new hardware and protocol com­pon­ents. The costs of Remote Direct Memory Access are thus con­sid­er­ably higher than those for FC or FCoE. In addition, the speedy data transfer via RDMA only works if all systems are supported by the tech­no­logy.