# Place Neighbors

Each `Place` instance can have a set of other instances as its neighbors. It can access these neighbors and read / modify their attribute values. 

By default, all the `Place` instances do not have any neighbors and require users to explicitly define that. This is done using the `updateNeighbours` method of `Places`, which takes in a set of `mass::NDOffset` instances indicating the N-dimensional offsets of each neighbor. 

For example, for a 2D `Places`, if we want to define the neighbors of each `Place` instance to be 4 instances in the north, east, south and west direction, we can do that as follows: 

```cpp
places->updateNeighbours({
    {  0,  1 },   // north
    {  1,  0 },   // east
    {  0, -1 },   // south
    { -1,  0 }    // west
});
```

Then for each `Place` instance, the library will calculate the addresses of the neighboring instances base on these offsets and store these addresses in a pre-defined attribute. Some instances at the border of the grid may not have all the specified neighbors since there is no more instances there. 

Note that the `updateNeighbors` function accept any type of container as long as it can be an argument to `std::data()` and `std::size()`. 

To access the addresses of the neighboring instances for each `Place` instance, use the `getNeighborsPtr` method provided in `Place` class, whose return value is a pointer-like type where each pointed value is the pointer to one neighboring `Place` instance. If the pointed value is `nullptr`, it means that there is no neighbor at that direction. 

A common pattern is to access all the neighboring instances and gather some attribute values from all of them. Here is an example where we sum the attribute values from all the neighbors (assume 4 neighbors): 

```cpp
class MyPlace final : public mass::Place {
public:
    __universal__ explicit MyPlace(const mass::Index localIndex) : Place(localIndex) {}

    enum Attribute { VALUE, SUM };
    enum Function { SUM };

    __device__ void callMethod(const mass::FunctionId functionId, void*const args) override {
        switch (functionId) {
        case Function::SUM: sum(); break;
        default:            break;
        }
    }

    __device__ void sum() {
        const auto neighborPtrs = getNeighborsPtr();
        auto sum = 0;
        for (auto i = 0; i < 4; i++) {
            if (neighborPtrs[i] != nullptr) {   // Only gather value from neighbors that exist
                sum += *(neighborPtrs[i]->getAttribute<int>(Attribute::VALUE));
            }
        }
        *getAttribute<int>(Attribute::SUM) = sum;
    }
};
```


## Accessing Neighbors on Another Device & Ghost Region Update

Due to the multi-GPUs context, `Place` instances at the boundary may have some of their neighboring instances staying on another device. The library provides the Ghost Regions to allow accessing such neighbors with the same `getNeighborsPtr` method. However, users still need to manually update the Ghost Regions before invoking any `callAll` that access the neighbors. 

Updating the Ghost Regions is done by the `syncPlaceBorder` method, which will trigger the process of copying the latest attribute values of the `Place` instances at the boundary of each block to the corresponding Ghost Regions on the neighboring devices. If we forgot to call this method before invoking any `callAll` method that access the neighbors, the attribute values retrieved from neighboring instances on neighboring device may not be the latest ones, leading to incorrect simulation results.

```cpp
places->syncPlaceBorder();
places->callAll(MyPlace::Function::SUM);
```

The simplest strategy of using the `syncPlaceBorder` method is to always call it before any `callAll` invocation that will read attribute values of neighboring instances through `getNeighborsPtr`. However, there might be more efficient strategies based on different simulation implementation, and the users are responsible for finding them. 

### Selective Ghost Region Update

By default, the `syncPlaceBorder` method will include all the attributes excluding those marked as "permanently excluded" during attribute creation (i.e.: the `permernentlyExcludedFromSyncBorder` parameter set to true). This strategy is simple and safe, but may lead to unnecessary data transfer since not all the attributes need to be updated for the Ghost Regions. 

To further optimize the execution of `syncPlaceBorder`, we can only include the attributes that are actually modified and required to be updated for the instances within the Ghost Regions. For example, consider a simulation process with the following codes: 

```cpp
places->syncPlaceBorder();  // update all the attributes
places->callAll(MyPlace::Function::MODIFY_A);    // only modify attribute A of all instances

places->syncPlaceBorder();  // update all the attributes
places->callAll(MyPlace::Function::READ_A_B);    // access both A and B from neighbors
```

Here, although the last `callAll` access both attribute `A` and `B` from the neighbors, attribute `B` is not modified since the first `syncPlaceBorder` invocation, which means that the values of `B` of the Ghost Regions are already up-to-date. As a result, we can safely exclude attribute `B` from the second `syncPlaceBorder` invocation to avoid unnecessary data transfer.

To do that, the MASS CUDA provides 2 methods. The first one is passing a list of attribute tags to the `syncPlaceBorder` method indicating the attributes included. The configuration set by this method will not be memorized for future calls. 

```cpp
// assume that we have attribute A, B and C
places->syncPlaceBorder({ MyPlace::Attribute::A, MyPlace::Attribute::C });  // only include A and C
places->syncPlaceBorder({ MyPlace::Attribute::B, MyPlace::Attribute::C });  // only include B and C
```

Another method is configuring the included attributes for all subsequence `syncPlaceBorder` invocations. This can be achieved with the following 4 methods on `Places`: 

| Method | Description | 
| --- | --- |
| `removeAttrsFromSyncBorder` | Exclude a list of attributes for subsequence invocations |
| `addAttrsForSyncBorder` | Include a list of attributes for subsequence invocations |
| `setAttrsForSyncBorder` | Set the included attributes for subsequence invocations, overwriting any previous configurations |
| `resetIncludedAttrForSyncBorder` | Reset the included attributes to the default setting for subsequence invocations |

```cpp
// Assume that we have attribute A, B, C and D.

// currently included: [A, B, C, D]
places->removeAttrsFromSyncBorder({ MyPlace::Attribute::B });
// currently included: [A, C, D]
places->syncPlaceBorder();
// currently included: [A, C, D]
places->setAttrsForSyncBorder({ MyPlace::Attribute::B, MyPlace::Attribute::D });
// currently included: [B, D]
places->addAttrsForSyncBorder({ MyPlace::Attribute::A });
// currently included: [A, B, D]
places->resetIncludedAttrForSyncBorder();
// currently included: [A, B, C, D]
```