A Comprehensive Analysis of InfiniBand and RoCE Networking: Focusing on GPU Applications

In the fields of artificial intelligence training, high-performance computing (HPC), and scientific computing, the performance of GPU computing clusters is paramount, and networking technology is a critical factor affecting overall performance. As model sizes grow and computational demands increase, choosing the right network technology is crucial for unlocking the full potential of graphics cards. This article provides an in-depth analysis of two mainstream high-performance network technologies—InfiniBand (IB) and RoCE—examining their application effectiveness with different GPUs to help you make optimal choices when building computing clusters.

1 IB and RoCE: Core Technologies for High-Performance Computing Networks

In an era where traditional Ethernet struggles to meet the demands of high-performance computing, InfiniBand and RoCE have emerged as leading solutions supporting Remote Direct Memory Access (RDMA) technology, addressing the two major bottlenecks of network latency and CPU overhead.

1.1 InfiniBand: Network Technology Born for High Performance

InfiniBand is an open-standard network interconnect technology designed specifically for high-performance computing. It differs fundamentally from traditional Ethernet in its architecture, creating an independent, complete end-to-end RDMA network system. IB natively supports RDMA, allowing data to be transferred directly between application memory spaces without passing through the remote CPU or operating system kernel, significantly reducing latency and CPU overhead. IB networks have their own physical, link, network, and transport layer protocols, along with dedicated network cards, switches, and cables, independent of the TCP/IP protocol stack.

IB employs a Queue Pair (QP)-based communication model. Applications submit Work Requests to QPs, which are processed directly by the hardware on the network card, enabling a "zero-copy" communication path from software to hardware. Furthermore, the IB protocol includes comprehensive, hardware-implemented, credit-based end-to-end flow control and hop-by-hop congestion control mechanisms. These mechanisms react rapidly to network congestion, ensuring high throughput and fairness. Leveraging its streamlined protocol stack and hardware offloading capabilities, IB networks can provide end-to-end latency below 1 microsecond and bandwidths up to 400Gbps (with 200G/400G being mainstream currently).

1.2 RoCE: RDMA over Ethernet

The core idea of RoCE (RDMA over Converged Ethernet) is to "run RDMA over Ethernet." It attempts to combine the ubiquity and cost advantages of Ethernet with the high performance of RDMA. RoCE has two main versions: RoCE v1, based on the Ethernet link layer (L2), operates only within a single broadcast domain (the same VLAN); RoCE v2, based on the UDP/IP protocol (L3), can be routed across IP subnets, greatly enhancing deployment flexibility and making it the current mainstream.

RoCE encapsulates the IB transport layer protocol packets inside Ethernet frames (v1) or UDP/IP packets (v2), enabling standard RDMA operations to run on common Ethernet infrastructure. The key prerequisite for RoCE's success is the construction of a lossless Ethernet environment. Because Ethernet itself is "best-effort," packet loss can catastrophically impact RoCE performance. Therefore, RoCE networks must be configured as lossless Ethernet, utilizing technologies like Priority-based Flow Control (PFC) and Explicit Congestion Notification (ECN) to pause transmission for specific priorities (i.e., RoCE traffic) during link congestion, preventing packet loss.

Similar to IB, RoCE also requires specialized network cards that support RoCE to offload the RDMA protocol stack, avoiding kernel involvement. Its greatest advantage lies in leveraging existing data center Ethernet switches (which must support PFC/ECN) for deployment, protecting existing investments.

2 Matching GPUs with Networking Technologies

Different GPUs, with varying performance levels and positioning, yield different results when paired with network technologies. Selecting the appropriate networking solution is crucial for optimizing cost and performance.

2.1 The Golden Pairing: High-End GPUs and IB Networking

For high-end data center GPUs like the NVIDIA A100, H100, and H200, IB networking is considered the gold standard. These GPUs are designed for large-scale AI training and HPC applications, featuring powerful computing capabilities and high-speed NVLink interconnect technology. When these GPUs handle large models with hundreds of billions of parameters, frequent exchange of massive gradient parameters between nodes makes network bandwidth and latency critical bottlenecks for overall performance.

Taking cloud servers equipped with NVIDIA A100 GPUs as an example, the g2na100_ib model specifically includes IB network cards, supporting internal network bandwidth from 12.5Gbps to 100Gbps, targeting "AI, deep learning, and HPC business scenarios that require high-speed data transmission." In such high-performance environments, IB networks can fully unleash the computational potential of high-end GPUs, ensuring the network does not become a performance bottleneck during distributed training.

For NVIDIA's latest Blackwell architecture GPUs (like B200, GB200), due to their even larger model sizes and interconnection needs, the advantages of IB networks become even more pronounced. In these high-end application scenarios, the superior performance and stability of IB networks fully justify their higher initial investment cost, making them the preferred choice for large-scale clusters.

2.2 The Cost-Effective Choice: Mid-to-High-End GPUs and RoCE

For mid-to-high-end graphics cards such as the NVIDIA A100 (in some scenarios), A800, H20, and L40S, as well as previous-generation flagship GPUs like the NVIDIA V100, RoCE networking offers excellent cost-effectiveness. These GPUs are typically used for medium-scale AI training, inference tasks, and departmental HPC clusters where network demands remain high, but budget becomes a significant consideration.

Practice has shown that in 64-GPU AI computing environments, RoCE networks using H3C S9827 series switches and IB networks using NVIDIA QM9700 switches demonstrated "comparable performance and consistent user experience" in tests with the NCCL collective communication library and the Llama3 large model. Similarly, RoCE networks using H3C S9825-G and S9855-G series switches also achieved performance levels comparable to IB networks.

For domestic GPUs (such as products from Moore Threads, Biren Technology, Enflame, etc.), RoCE networks also show strong adaptability. BaiRui Data's hybrid networking solution using 32 domestic GPU servers (256 cards) demonstrated excellent performance with a RoCE solution, breaking foreign technology monopolies and providing strong support for domestic alternatives. Cost-wise, BaiRui Data's RoCE networking solution saved about 36% compared to the InfiniBand solution at 100Gbps rates, and achieved about 14.3% cost savings at 400Gbps rates.

2.3 Networking Considerations for Consumer-Grade GPUs

For small-scale research teams or startups using consumer-grade flagship GPUs like the NVIDIA RTX 4090 and 4090D, or development and testing environments, network choices are often more complex. These graphics cards typically lack official multi-node interconnection support but can still be deployed in clusters through third-party solutions.

In such scenarios, the economic advantages of RoCE networks are even more evident. By installing RoCE-capable Ethernet network cards (such as the NVIDIA ConnectX-6 or ConnectX-7 series) and configuring PFC-supported Ethernet switches, a low-cost, high-performance training cluster can be built. It is important to note that such configurations often require more tuning and testing to ensure network performance meets requirements.

Table: Recommendations for Pairing Different GPUs with Networking Technologies

3 Analysis of Key Factors Affecting Networking Performance

When selecting a networking technology, it is essential to comprehensively consider various performance-influencing factors to ensure the full performance potential of the GPUs is realized.

3.1 PCIe Bottleneck: A Non-Negligible Limiting Factor

Regardless of whether you choose IB or RoCE networking, attention must be paid to the potential bottleneck of PCIe bandwidth. Taking a single server with 8 A100 GPUs as an example, there are significant differences in the bandwidth of key links:

• Between GPUs on the same host: Using NVLink, bidirectional bandwidth is 600 GB/s, unidirectional bandwidth is 300 GB/s.

• Between a GPU and its network interface card (NIC) on the same host: Utilizing PCIe Gen4 switch chips, bidirectional bandwidth is 64 GB/s, unidirectional bandwidth is 32 GB/s.

• Between GPUs across different hosts: Data transfer relies on NICs, bandwidth depends on the specific NIC used. Currently, the mainstream unidirectional bandwidth for commonly used A100/A800 model NICs in China is 100 Gbps (12.5 GB/s).

This indicates that inter-host communication performance significantly drops compared to intra-host communication. 200 Gbps (25 GB/s) is already close to the unidirectional bandwidth limit of PCIe Gen4. Therefore, using a 400 Gbps NIC in a PCIe Gen4 configuration does not yield significant benefits; PCIe Gen5 performance is required to fully utilize 400 Gbps bandwidth. When selecting a network solution, it is crucial to ensure that network bandwidth matches PCIe bandwidth to avoid resource waste.

3.2 Network Architecture and Congestion Control

Network architecture design and congestion control mechanisms significantly impact performance. IB networks employ native congestion control mechanisms implemented in hardware, allowing rapid response to changes in network state. In contrast, RoCE networks rely on mechanisms like PFC (Priority-based Flow Control) and ECN (Explicit Congestion Notification) to build a lossless network. Improper configuration can lead to issues like "head-of-line blocking" and "PFC deadlock."

New network architectures like DDC (Distributed Disaggregated Chassis) technology, utilizing advanced hardware technologies such as VOQ (Virtual Output Queue)/CELL switching, improve link utilization and throughput, fully meeting the low forwarding latency and low packet loss rate requirements of HPC and AI services on transmission networks. Test results show that in RoCE networks, DDC offers significant advantages in total bus bandwidth compared to traditional ECMP hash methods.

Network architecture choices also vary depending on the cluster scale. For small-scale networking, a single-box or single-chassis solution is recommended, where all parameter network cards of the GPU servers are connected to the switch. Network cards with the same sequence number from different servers are configured in the same VLAN to reduce communication hops and latency. For medium to large-scale networking, domestic GPU servers can adopt a multi-rail access method, achieving efficient data transmission through a Leaf-Spine network architecture.

3.3 Software Ecosystem and Operational Considerations

The software ecosystem and operational complexity are also key factors in the selection. IB networking is technologically more mature, requires less complex parameter tuning, and allows for faster deployment, but its ecosystem is relatively monolithic, dominated by NVIDIA. RoCE, based on Ethernet, has a more open ecosystem with multiple suppliers (including Huawei, H3C, ZTE, Ruijie Networks, etc.), but requires more complex parameter tuning.

Regarding operations and maintenance, IB networks, due to their specialized nature, require dedicated teams for management. RoCE networks can leverage existing Ethernet operational experience and toolchains, but configuring and troubleshooting lossless networks requires deep learning. In terms of networking scale, IB networks can support clusters with tens of thousands of GPUs, while RoCEv2 performs well on clusters with thousands of GPUs, with its networking performance continuously being optimized.

4 Practical Application Scenarios and Selection Guide

Based on the above analysis, we can provide specific networking recommendations for different application scenarios.

4.1 Large-Scale Clusters and Cutting-Edge Research

For national supercomputing centers, top-tier AI research institutions, and large cloud service providers, where every microsecond of latency and every watt of efficiency are critical, InfiniBand is the undeniable choice. This is especially true when using top-tier hardware like NVIDIA H100, H200, or Blackwell architecture GPUs, where IB networks ensure the network does not become a bottleneck for overall performance.

Such scenarios typically have the following characteristics:

• Large computing cluster scale (thousands of GPUs or more)

• Sufficient budget, pursuit of ultimate performance

• Possession of professional IB network operation teams

• Processing ultra-large models with hundreds of billions of parameters

In these environments, the ultimate performance and mature stability advantages of IB networks are most apparent, fully justifying their higher initial investment.

4.2 Medium Scale and Enterprise Applications

For enterprise data centers, medium-sized AI labs, and departmental HPC clusters, RoCE networking provides an ideal balance of performance and cost. These scenarios typically:

• Have cluster sizes ranging from hundreds to a few thousand GPUs

• Need to balance performance with TCO (Total Cost of Ownership)

• Have operations teams more familiar with Ethernet technology

• Use mid-to-high-end graphics cards like A100, A800, H20, etc.

Such applications can choose Ethernet switches like the H3C S9825/9855-G series that support high-density 400GE/100GE ports, combined with RoCE network cards to build high-performance networks. In practice, such configurations can provide performance experiences close to IB networks while controlling costs.

4.3 Small Scale and Experimental Environments

For startups, academic research teams, and development and testing environments building clusters using consumer-grade graphics cards or a small number of data center GPUs, RoCE or even high-speed TCP/IP networks are reasonable choices. These scenarios typically:

• Have cluster sizes within a few dozen GPUs

• Have strictly limited budgets

• Require rapid deployment and easy maintenance

• May include domestic GPUs or consumer-grade GPUs

In such environments, priority can be given to leveraging existing Ethernet infrastructure, paired with RoCE-capable network cards, to build cost-effective high-performance computing clusters. BaiRui Data's small-scale networking solutions offer single-box or single-chassis options, simplifying deployment and management processes.

5 Future Technology Trends

As computational demands continue to evolve, both IB and RoCE technologies are also developing, showing several clear trends.

On the one hand, IB technology continues to lead performance peaks; next-generation IB products will support higher bandwidth and lower latency, meeting the needs of future trillion-parameter model training. On the other hand, RoCE technology is maturing rapidly, with more optimization technologies (such as DDC, SmartNICs, etc.) narrowing the performance gap with IB at the extreme end.

Regarding standardization and interoperability, RoCE, with its Ethernet foundation, is promoting the development of a more open ecosystem, breaking the traditional closed nature of the IB field. Experts from Wuzhen Institute note: "RoCE network is a purely distributed network that applies RDMA technology to traditional Ethernet, essentially a network card encapsulation technology." This characteristic makes RoCE easier to gain broad industry support.

Furthermore, the trend towards localization is also driving the development of RoCE technology. With the advancement of domestic GPUs and switch technologies, RoCE, as a more open technical standard, naturally becomes the preferred choice for domestic solutions. Domestic data center switch manufacturers (including Huawei, H3C, ZTE, Ruijie Networks, etc.) are actively deploying RoCE-related products.

6 Conclusion

In the fields of AI and HPC, choosing between InfiniBand and RoCE networking is not merely a simple technical selection but a multi-dimensional decision requiring comprehensive consideration of GPU type, application scenario, cluster scale, and total cost of ownership.

For large-scale clusters using high-end data center GPUs (like H100, A100) and pursuing ultimate performance, InfiniBand remains the preferred solution. Its excellent latency performance and mature ecosystem provide reliable assurance for mission-critical tasks.

For medium-scale clusters using mid-to-high-end GPUs (like A800, L40S) or domestic GPUs, RoCE offers an attractive cost-performance ratio, capable of meeting the performance requirements of most application scenarios while controlling costs.

Regardless of the chosen technology, attention must be paid to key factors such as PCIe bandwidth, network architecture, and software tuning to ensure the network does not become the bottleneck of the entire computing cluster. As technologies continue to develop, the competitive landscape between IB and RoCE may change, but the goal remains consistent: to fully leverage the computational potential of every graphics card and accelerate scientific discovery and technological innovation.