# Cross-Device Communication Design

To support collaboration across multiple devices, communication needs to be done among them. This page briefly introduce the design and technology used for the communication. 


## What To Transfer for `Places`

For `Places`, each device is responsible for processing one block of the complete `Places` grid. For APIs like the `getNeighborsPtr` to work correctly, we add additional `Place` instances around each block holding read-only local copies of the instances at neighboring devices. These regions containing such read-only instances are called Ghost Regions. Each Ghost Region reflects a region of Place instances with the same shape at the boundary of the block on a neighboring device. These regions reflected by Ghost Regions on neighboring devices are called Boundary Regions. 

The attribute values of the `Place` instances within the Boundary Regions need to be copied to the corresponding Ghost Regions on the neighboring devices. 

Consider a 2D `Places`, the relationship and data flow between the Boundary Regions and the Ghost Regions are presented in the following figure.

![boundary_and_ghost_regions](../assets/boundary_ghost_regions.png)


## What To Transfer for `Agents`

For `Agents`, there are two types of data to transfer: 

1. If an `Agent` instance is migrating to a neighboring device, its attribute values need to be transferred to the destination device. 
2. If an `Agent` instance requests spawning child instances on a neighboring device, its Agent Spawning Request needs to be collected and transferred to the neighboring device. 

The Agent Spawning Request in the second case is a pair containing the address of the `Place` instance that the child `Agent` instances should be spawned on, and the number of children to spawn there. 


## Collaborative Communication Process

The actual communication is a collaborative process between the `RootDispatcher` and the `DeviceDispatcher` using the `Communication Buffers` as a bridge between them. 

The `Communication Buffers` is a set of contiguous buffers temporarily holding the data to be transferred and the data received. The `DeivceDispatcher` is responsible for collecting the data to transfer and copy them to the corresponding `Communication Buffers` and also copy the received data from the `Communication Buffers` to the targeted destination. On the other hand, the `RootDispatcher` is responsible for handling the actual data transfer by copying the data from one `Communication Buffers` to another. 

The following sequence diagram presents the high-level process of how the Collaborative Communication works to transfer data on GPU 0 to GPU 1

![collaborative_communication](../assets/collaborative_communication.png)


## Technology for Data Transfer between Devices 

We use Nvidia Collective Communication Library (NCCL) to handle the actual data transfer between devices. This is a powerful yet easy-to-use library providing high-level communication primitives. It supports devices connected over different mechanisms, including NVLink, PCIe, and even network connections. Users don't have to specify how each pair of devices are connected, the NCCL can detect that automatically and pick the optimal one for communication. 

For GPUs connected over network on different hosts, NCCL works better when the Remote Direct Memory Access (RDMA) feature is available. So it is recommended to use GPUs with RDMA support when trying to run large scale simulations on multiple GPUs distributed on multiple hosts.