Cross-Host Parallelism Design

Contents

Cross-Host Parallelism Design#

When GPUs participating in the simulation are distributed on different hosts, we need to spawn different processes running in parallel on each host to dispatch jobs to the local GPUs. This requires us to have a pattern for the coordinating such cross-host parallelism.

MASS CUDA adoptes a different pattern compared to MASS Java and MASS C++: we use hybrid pattern that combines Master-Slave pattern and Peer-to-Peer (P2P) pattern.

  • During library initialization (i.e.: the MASS::init() method), Master-Slave is used so that a master node is responsible for launching processes on all the other hosts and coordinate the initialization process.

  • During the simulation execution process, switch to P2P pattern where each process executes the same program independently.

In the P2P stage, when communication related APIs are not invoked (e.g.: syncPlaceBorder, manageAll), all the processes focus only on their own assigned data and run in full parallelism. When any communication related APIs are invoked, due to the fact that they are running the same program, all the processes will eventually reach the same API call and start to communicate with their neighboring processes.

To make this design possible, it is important to make sure that all the hosts share the same executable binary of the simulation program and related resources. The only difference should be the rank of host specified in the CLI launch arguments.

Important Notes#

To make this design possible, it is important to make sure that all the hosts share the same executable binary of the simulation program and related resources. The only difference should be the rank of host specified in the CLI launch arguments.

In addition, the implementation of the simulation should avoid any branching conditions depending on host-specific data (e.g.: host name, rank of host, current time). Such branching conditions might cause the processes on different hosts to diverge and invoke different MASS CUDA APIs, which can leads to undefined behaviors.

Currently, the only reasonable situation for such divergence is collecting the downloaded attribute values from the devices on current hosts and transferring them to the master node for further processing (e.g.: visualization, analysis, etc.). In this case, the code should determine whether the current host is the master node, if so, it should receive data from all the other hosts, otherwise, it should send the attribute values to the master node.

Note

In the future, we should provide more sophisticated APIs for easier collection of attribute values from devices across all the hosts to the master node to avoid manual implementation of such pattern.

Contents