Cross-Device Communication Design

Cross-Device Communication Design#

To support collaboration across multiple devices, communication needs to be done among them. This page briefly introduce the design and technology used for the communication.

What To Transfer for Places#

For Places, each device is responsible for processing one block of the complete Places grid. For APIs like the getNeighborsPtr to work correctly, we add additional Place instances around each block holding read-only local copies of the instances at neighboring devices. These regions containing such read-only instances are called Ghost Regions. Each Ghost Region reflects a region of Place instances with the same shape at the boundary of the block on a neighboring device. These regions reflected by Ghost Regions on neighboring devices are called Boundary Regions.

The attribute values of the Place instances within the Boundary Regions need to be copied to the corresponding Ghost Regions on the neighboring devices.

Consider a 2D Places, the relationship and data flow between the Boundary Regions and the Ghost Regions are presented in the following figure.

boundary_and_ghost_regions

What To Transfer for Agents#

For Agents, there are two types of data to transfer:

  1. If an Agent instance is migrating to a neighboring device, its attribute values need to be transferred to the destination device.

  2. If an Agent instance requests spawning child instances on a neighboring device, its Agent Spawning Request needs to be collected and transferred to the neighboring device.

The Agent Spawning Request in the second case is a pair containing the address of the Place instance that the child Agent instances should be spawned on, and the number of children to spawn there.

Collaborative Communication Process#

The actual communication is a collaborative process between the RootDispatcher and the DeviceDispatcher using the Communication Buffers as a bridge between them.

The Communication Buffers is a set of contiguous buffers temporarily holding the data to be transferred and the data received. The DeivceDispatcher is responsible for collecting the data to transfer and copy them to the corresponding Communication Buffers and also copy the received data from the Communication Buffers to the targeted destination. On the other hand, the RootDispatcher is responsible for handling the actual data transfer by copying the data from one Communication Buffers to another.

The following sequence diagram presents the high-level process of how the Collaborative Communication works to transfer data on GPU 0 to GPU 1

collaborative_communication

Technology for Data Transfer between Devices#

We use Nvidia Collective Communication Library (NCCL) to handle the actual data transfer between devices. This is a powerful yet easy-to-use library providing high-level communication primitives. It supports devices connected over different mechanisms, including NVLink, PCIe, and even network connections. Users don’t have to specify how each pair of devices are connected, the NCCL can detect that automatically and pick the optimal one for communication.

For GPUs connected over network on different hosts, NCCL works better when the Remote Direct Memory Access (RDMA) feature is available. So it is recommended to use GPUs with RDMA support when trying to run large scale simulations on multiple GPUs distributed on multiple hosts.