Whether coincidental or intentional, AMD engineers tend to drop interesting, feature patches for the open-source/Linux space on Friday afternoons. Hitting the kernel mailing list minutes ago were two patch series sent out by AMDGPU maintainer Alex Deucher for introducing UALink infrastructure.

Ultra Accelerator Link "UALink" is the open standard and high speed networking protocol for connecting GPUs and other AI accelerators within scale-up pods in the data center. UALink is designed to allow linking up to 1,024 accelerators in a single domain.

Sent out was the first patch series that comes in at 95 patches, about 7k lines of code. That adds in the core infrastructure and AMDGPU kernel graphics driver integration for UALink.

"Connected GPUs in a pod can directly access the [remote] memory on another GPU over UALink. Unlike RMDA, there is no copy involved; it is direct loads/stores over the fabric. Shared memory can only be accessed by a remote GPU if the memory was exported and the importer has been authorized. For the memory to be shared, it must be part of a unified physical address space shared between nodes. This address space is called NPA (Network Physical Address) space. This address space is partitioned between
the GPUs so that each GPU has it's own segment of the address space in which to export its memory. Each GPU maintains a dedicated set of page tables for their NPA space similar to GPUVM.

Exported memory is not pinned. The exporter may need to move memory or make it unavailable to the importers. This is handled via remote TLB shootdowns on the exporter in the case of TTM evictions or MMU notifiers and remote [presence] checks with retry on the importer.

To access remote memory, the driver can map NPA addresses into its per process GPUVM page tables just like local memory. Applications use opaque handles to represent remote memory. GPUs in a pod communicate with eachother directly to exchange NPA addresses between importers and exporters. If a node goes offline or is reset, their peers will clean up any remaining [references] that are lost when that happens."

The second patch series is just three patches and exposes the new IOCTL interface and documentation around the UALink integration for memory import and export.

For readers the likely most interesting bit is the documentation patch.

Great seeing this UALink Linux kernel integration work finally surfacing by AMD for their AMDGPU driver.