DPEcho

Follow this procedure to run DPEcho, a general relativistic magneto hydrodynamics simulator used to model instabilities, turbulence, propagation of waves, stellar winds and magnetospheres, and astrophysical processes around black holes.

  • Operating system: Ubuntu* 22.04

  • Hardware: Intel® Data Center Max GPUs

  • Software: Intel® oneAPI Base toolkit, Intel® oneAPI HPC toolkit

  • Time to complete: 15 minutes

For more information on DPEcho, see DPEcho description and DPEcho repository.

  1. Check whether the driver stack is installed.

    $ xpu-smi discovery

    The command should return at least one Intel® Data Center GPU Max device.

  2. Check whether the oneAPI toolkit is installed.

    $ apt list intel-basekit intel-hpckit

    Expected output:

    Listing... Done
intel-basekit/all,now 2023.2.0-49384 amd64 [installed]
intel-hpckit/all,now 2023.2.0-49438 amd64 [installed]

  3. If you previously have not configured your environment, install the Ubuntu 22.04 graphics driver. See dgpu-docs for details.

    Note

    Access to Ubuntu repositories, such as https://repositories.intel.com and https://apt.repos.intel.com, is required for installation. If proxy settings involve changes to environment variables such as http_proxy or https_proxy, small modifications are required in the following steps, such as adding -E (preserve environment) to sudo commands.

  4. If you previously have not configured your environment, enable access to the Intel repo serving the oneAPI packages and install the oneAPI Base toolkit and HPC toolkit for Ubuntu 22.04.

    wget -O- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB \ | gpg --dearmor | sudo tee /usr/share/keyrings/oneapi-archive-keyring.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main" | sudo  tee /etc/apt/sources.list.d/oneAPI.list
sudo apt update

    sudo apt install -y intel-basekit intel-hpckit

  5. Build DPEcho.

    git clone https://github.com/LRZ-BADW/DPEcho.git
cd DPEcho
mkdir build && cd build
source /opt/intel/oneapi/setvars.sh
cmake .. -DCMAKE_BUILD_TYPE=Release -DENABLE_MPI=OFF -DFD=6 -DSYCL=oneAPI -DFILE_IO=DISABLED -DSYCL_DEVICE=GPU -DNRK=3 -DPHYSICS=GRMHD -DREC_ORDER=5 -DSINGLE_PRECISION=OFF
make -j

  6. Run DPEcho.

     ./echo_gpu_gr ../examples/alfven.par

The following example presents a result snapshot from Intel® Data Center GPU Max 1550:

I:     Raising Hx to allowed min 3
I:     Raising Hy to allowed min 3
I:     Raising Hz to allowed min 3
[0][DeviceConfig::deviceWith] Looking at DPEcho default device.
[0][DeviceConfig::printTargetInfo]
        Hardware Intel(R) Data Center GPU Max 1550GPU
        Max Compute Units  : 896
[0]
        Max Work Group Size: 1024
        Global Memory / GB : 128
        Local  Memory / kB : 128

[0][Domain::cartInfo] CartCoords:    (1919378037 1936749935 1869116255) CartPeriodic:  (1 1 1)
[0][Domain::locInfo] Local size (1 1 1) From (1.91938e+09 1.93675e+09 1.86912e+09) To (1.91938e+09 1.93675e+09 1.86912e+09)
[0][Grid::print]  halos_(3 3 3)  cellsWH(70 70 70)  cellsNH(64 64 64)  Dxyz   (0.015625 0.015625 0.015625)
[0][main] Step 1 out 0 characteristic: 52.049 in 1.97405 s
[0][main] Step 2 out 0 characteristic: 52.0485 in 0.00511193 s
[0][main] Step 3 out 0 characteristic: 52.0486 in 0.00496292 s
[0][main] Step 4 out 0 characteristic: 52.049 in 0.0049932 s
[0][main] Step 5 out 0 characteristic: 52.0483 in 0.00497198 s
[0][main] Step 6 out 0 characteristic: 52.0488 in 0.00499296 s
[0][main] Step 7 out 0 characteristic: 52.049 in 0.0049839 s
[0][main] Step 8 out 0 characteristic: 52.0481 in 0.005023 s
[0][main] Step 9 out 0 characteristic: 52.0489 in 0.00497198 s
[0][main] Step 10 out 0 characteristic: 52.049 in 0.00496912 s
[0][main] Step 11 out 0 characteristic: 52.0479 in 0.00497603 s
[0][main] Step 12 out 0 characteristic: 52.049 in 0.00498796 s
[0][main] Step 13 out 0 characteristic: 52.0489 in 0.0049789 s
[0][main] Step 14 out 0 characteristic: 52.0481 in 0.00497699 s
[0][main] Step 15 out 0 characteristic: 52.0491 in 0.00499701 s
[0][main] Step 16 out 0 characteristic: 52.0488 in 0.00499701 s
[0][main] Step 17 out 0 characteristic: 52.0483 in 0.00499701 s
[0][main] Step 18 out 0 characteristic: 52.0491 in 0.00496602 s
[0][main] Step 19 out 0 characteristic: 52.0487 in 0.00500202 s
[0][main] Step 20 out 0 characteristic: 52.0485 in 0.00496411 s
[0][main] Step 21 out 0 characteristic: 52.0491 in 0.00498676 s