Place Neighbors#

Each Place instance can have a set of other instances as its neighbors. It can access these neighbors and read / modify their attribute values.

By default, all the Place instances do not have any neighbors and require users to explicitly define that. This is done using the updateNeighbours method of Places, which takes in a set of mass::NDOffset instances indicating the N-dimensional offsets of each neighbor.

For example, for a 2D Places, if we want to define the neighbors of each Place instance to be 4 instances in the north, east, south and west direction, we can do that as follows:

places->updateNeighbours({
    {  0,  1 },   // north
    {  1,  0 },   // east
    {  0, -1 },   // south
    { -1,  0 }    // west
});

Then for each Place instance, the library will calculate the addresses of the neighboring instances base on these offsets and store these addresses in a pre-defined attribute. Some instances at the border of the grid may not have all the specified neighbors since there is no more instances there.

Note that the updateNeighbors function accept any type of container as long as it can be an argument to std::data() and std::size().

To access the addresses of the neighboring instances for each Place instance, use the getNeighborsPtr method provided in Place class, whose return value is a pointer-like type where each pointed value is the pointer to one neighboring Place instance. If the pointed value is nullptr, it means that there is no neighbor at that direction.

A common pattern is to access all the neighboring instances and gather some attribute values from all of them. Here is an example where we sum the attribute values from all the neighbors (assume 4 neighbors):

class MyPlace final : public mass::Place {
public:
    __universal__ explicit MyPlace(const mass::Index localIndex) : Place(localIndex) {}

    enum Attribute { VALUE, SUM };
    enum Function { SUM };

    __device__ void callMethod(const mass::FunctionId functionId, void*const args) override {
        switch (functionId) {
        case Function::SUM: sum(); break;
        default:            break;
        }
    }

    __device__ void sum() {
        const auto neighborPtrs = getNeighborsPtr();
        auto sum = 0;
        for (auto i = 0; i < 4; i++) {
            if (neighborPtrs[i] != nullptr) {   // Only gather value from neighbors that exist
                sum += *(neighborPtrs[i]->getAttribute<int>(Attribute::VALUE));
            }
        }
        *getAttribute<int>(Attribute::SUM) = sum;
    }
};

Accessing Neighbors on Another Device & Ghost Region Update#

Due to the multi-GPUs context, Place instances at the boundary may have some of their neighboring instances staying on another device. The library provides the Ghost Regions to allow accessing such neighbors with the same getNeighborsPtr method. However, users still need to manually update the Ghost Regions before invoking any callAll that access the neighbors.

Updating the Ghost Regions is done by the syncPlaceBorder method, which will trigger the process of copying the latest attribute values of the Place instances at the boundary of each block to the corresponding Ghost Regions on the neighboring devices. If we forgot to call this method before invoking any callAll method that access the neighbors, the attribute values retrieved from neighboring instances on neighboring device may not be the latest ones, leading to incorrect simulation results.

places->syncPlaceBorder();
places->callAll(MyPlace::Function::SUM);

The simplest strategy of using the syncPlaceBorder method is to always call it before any callAll invocation that will read attribute values of neighboring instances through getNeighborsPtr. However, there might be more efficient strategies based on different simulation implementation, and the users are responsible for finding them.

Selective Ghost Region Update#

By default, the syncPlaceBorder method will include all the attributes excluding those marked as “permanently excluded” during attribute creation (i.e.: the permernentlyExcludedFromSyncBorder parameter set to true). This strategy is simple and safe, but may lead to unnecessary data transfer since not all the attributes need to be updated for the Ghost Regions.

To further optimize the execution of syncPlaceBorder, we can only include the attributes that are actually modified and required to be updated for the instances within the Ghost Regions. For example, consider a simulation process with the following codes:

places->syncPlaceBorder();  // update all the attributes
places->callAll(MyPlace::Function::MODIFY_A);    // only modify attribute A of all instances

places->syncPlaceBorder();  // update all the attributes
places->callAll(MyPlace::Function::READ_A_B);    // access both A and B from neighbors

Here, although the last callAll access both attribute A and B from the neighbors, attribute B is not modified since the first syncPlaceBorder invocation, which means that the values of B of the Ghost Regions are already up-to-date. As a result, we can safely exclude attribute B from the second syncPlaceBorder invocation to avoid unnecessary data transfer.

To do that, the MASS CUDA provides 2 methods. The first one is passing a list of attribute tags to the syncPlaceBorder method indicating the attributes included. The configuration set by this method will not be memorized for future calls.

// assume that we have attribute A, B and C
places->syncPlaceBorder({ MyPlace::Attribute::A, MyPlace::Attribute::C });  // only include A and C
places->syncPlaceBorder({ MyPlace::Attribute::B, MyPlace::Attribute::C });  // only include B and C

Another method is configuring the included attributes for all subsequence syncPlaceBorder invocations. This can be achieved with the following 4 methods on Places:

Method	Description
`removeAttrsFromSyncBorder`	Exclude a list of attributes for subsequence invocations
`addAttrsForSyncBorder`	Include a list of attributes for subsequence invocations
`setAttrsForSyncBorder`	Set the included attributes for subsequence invocations, overwriting any previous configurations
`resetIncludedAttrForSyncBorder`	Reset the included attributes to the default setting for subsequence invocations

// Assume that we have attribute A, B, C and D.

// currently included: [A, B, C, D]
places->removeAttrsFromSyncBorder({ MyPlace::Attribute::B });
// currently included: [A, C, D]
places->syncPlaceBorder();
// currently included: [A, C, D]
places->setAttrsForSyncBorder({ MyPlace::Attribute::B, MyPlace::Attribute::D });
// currently included: [B, D]
places->addAttrsForSyncBorder({ MyPlace::Attribute::A });
// currently included: [A, B, D]
places->resetIncludedAttrForSyncBorder();
// currently included: [A, B, C, D]

Place Neighbors

Contents

Place Neighbors#

Accessing Neighbors on Another Device & Ghost Region Update#

Selective Ghost Region Update#