For the RDMA/GPUDirect Storage technology to take effect, the following requirements must be met:
- All the cluster servers support RDMA networking
- For a client:
- GPUDirect Storage - the IB interfaces added to the Nvidia GPUDirect configuration should support RDMA
- RDMA - all the Infiniband Host Channel Adapters (HCA) used by Content Software for File support RDMA networking
- Encrypted filesystems: The framework is not used for encrypted filesystems and falls back to work without RDMA/GPUDirect for IOs to encrypted filesystems.
- An HCA is considered to support RDMA Networking if the following requirements are met:
- For GPUDirect Storage only: InfiniBand network
- The NIC supports RDMA. See Supported network adapters.
- OFED 4.6-1.0.1.1 or higher
- For GPUDirect Storage: install with --upstream-libs and --dpdk.
Note: GPUDirect Storage completely bypasses the kernel and does not utilize the page cache. Standard RDMA clients still utilize the page cache.
Note: RDMA/GPUDirect Storage technology is unsupported when working with a mixed IB and Ethernet networking cluster.
Running weka cluster processes indicates if the RDMA is used.
Example:
# weka cluster processes PROCESS ID HOSTNAME CONTAINER IPS STATUS ROLES NETWORK CPU MEMORY UPTIME 0 weka146 default 10.0.1.146 UP MANAGEMENT UDP 16d 20:07:42h 1 weka146 default 10.0.1.146 UP FRONTEND DPDK / RDMA 1 1.47 GB 16d 23:29:00h 2 weka146 default 10.0.3.146 UP COMPUTE DPDK / RDMA 12 6.45 GB 16d 23:29:00h 3 weka146 default 10.0.1.146 UP COMPUTE DPDK / RDMA 2 6.45 GB 16d 23:29:00h 4 weka146 default 10.0.3.146 UP COMPUTE DPDK / RDMA 13 6.45 GB 16d 23:29:00h 5 weka146 default 10.0.1.146 UP COMPUTE DPDK / RDMA 3 6.45 GB 16d 22:28:58h 6 weka146 default 10.0.3.146 UP COMPUTE DPDK / RDMA 14 6.45 GB 16d 23:29:00h 7 weka146 default 10.0.3.146 UP DRIVES DPDK / RDMA 18 1.49 GB 16d 23:29:00h 8 weka146 default 10.0.1.146 UP DRIVES DPDK / RDMA 8 1.49 GB 16d 23:29:00h 9 weka146 default 10.0.3.146 UP DRIVES DPDK / RDMA 19 1.49 GB 16d 23:29:00h 10 weka146 default 10.0.1.146 UP DRIVES DPDK / RDMA 9 1.49 GB 16d 23:29:00h 11 weka146 default 10.0.3.146 UP DRIVES DPDK / RDMA 20 1.49 GB 16d 23:29:07h 12 weka147 default 10.0.1.147 UP MANAGEMENT UDP 16d 22:29:02h 13 weka147 default 10.0.1.147 UP FRONTEND DPDK / RDMA 1 1.47 GB 16d 23:29:00h 14 weka147 default 10.0.3.147 UP COMPUTE DPDK / RDMA 12 6.45 GB 16d 23:29:00h 15 weka147 default 10.0.1.147 UP COMPUTE DPDK / RDMA 2 6.45 GB 16d 23:29:00h 16 weka147 default 10.0.3.147 UP COMPUTE DPDK / RDMA 13 6.45 GB 16d 23:29:00h 17 weka147 default 10.0.1.147 UP COMPUTE DPDK / RDMA 3 6.45 GB 16d 23:29:00h 18 weka147 default 10.0.3.147 UP COMPUTE DPDK / RDMA 14 6.45 GB 16d 23:29:00h 19 weka147 default 10.0.3.147 UP DRIVES DPDK / RDMA 18 1.49 GB 16d 23:29:00h 20 weka147 default 10.0.1.147 UP DRIVES DPDK / RDMA 8 1.49 GB 16d 23:29:00h 21 weka147 default 10.0.3.147 UP DRIVES DPDK / RDMA 19 1.49 GB 16d 23:29:07h 22 weka147 default 10.0.1.147 UP DRIVES DPDK / RDMA 9 1.49 GB 16d 23:29:00h 23 weka147 default 10.0.3.147 UP DRIVES DPDK / RDMA 20 1.49 GB 16d 23:29:07h . . .
Note: GPUDirect Storage is auto-enabled and detected by the system. To enable/disable RDMA networking altogether on the cluster or a specific client, contact customer support.