Build InfiniBand fabrics that stay fast, stable, and observable for AI and HPC workloads.
Running InfiniBand at production scale is hard. Physical plant, routing policy, partitions, congestion control, and day two operations all interact. Teams need a practical reference that connects design choices to the counters, commands, and workflows used in real clusters.
Practical InfiniBand Administration provides that reference. It moves from RDMA fundamentals to fabric design and bring up, then into monitoring, troubleshooting, and workload integration with UCX, NCCL, SHARP, Kubernetes, and OpenShift, so you can keep throughput high and variance low.
- clear rdma object model, pd qp cq mr, address handles, with a clean qp lifecycle using rdmacm
- throughput math operators use, sdr hdr ndr, lanes, mtu, payload efficiency that matches real links
- cabling and optics choices, qsfp56 qsfp112 osfp twin port, aoc dac acc, mpo polarity and bend radius
- opensm configuration that sticks, routing engines minhop ftree updn torus 2 qos and predictable sweeps
- partitions and qos in practice, p keys, sl to vl mapping, arbitration that protects storage and control traffic
- nvidia congestion control deployed safely, policy files, keys, staged rollout, and validation on live clusters
- adaptive routing and shield, enablement steps with acceptance tests that prove path dispersion
- ib routers for multi subnet scale out, practical design patterns and known limits
- fabric design and bring up, spine leaf fat tree planning, oversubscription math, baseline and burn in gates
- day 0 to day 2 operations, ibdiagnet ibnetdiscover perfquery ibtracert used as an actionable toolkit
- firmware and driver lifecycle, repeatable workflows with mlxfwmanager and mstflint
- troubleshooting playbooks, physical faults with symbolerrors and linkdowned, xmitwait analysis, path checks and sweep timing
- ai and hpc integration, ucx and hpc x transport selection, parameters, and a measurement strategy that holds up
- nccl over infiniband, essential environment variables, topology awareness, and stable channel plans
- sharp collectives offload, setup, orchestration, and verification on both mpi and nccl stacks
- platform features, gpudirect rdma with dma buf checks, ipoib connected mode tuning, and practical sysctls
- virtualization with sr iov on connectx, host and hypervisor configuration, and guest level validation
- kubernetes and openshift patterns, nvidia network operator and doca ofed with a nicclusterpolicy that matches the fabric
- rdma shared device plugin and cdi, device exposure inside pods, and sriov infiniband cni with multus attachments
This is a code heavy guide with working Bash, C, YAML, and JSON snippets that map directly to bring up, validation, and on call workflows.
Add a reliable InfiniBand playbook to your toolkit, get your copy today.
