System Administration
Usage of MPS daemon
In case when multi-process application like PostgreSQL uses GPU device, it is a well known solution to use MPS (Multi-Process Service) to reduce context switch on GPU side and resource consumption for device management.
https://docs.nvidia.com/deploy/mps/index.html
It is also recommended for PG-Strom to apply MPS, however, you need to pay attention for several limitations below.
One MPS daemon can provide its service for up to 48 clients (16 clients if Pascal or older). So, DB administration must ensure number of PostgreSQL processes using GPU (including background workers in CPU parallelism) is less than 48 (or 16 if Pascal).
MPS does not support dynamic parallelism, and load GPU programs using the feature. GPU programs automatically generated from SQL will never use dynamic parallelism, however, PL/CUDA user defined function may use dynamic parallelism if it links CUDA device runtime to invoke sub-kernels. So, we don't use MPS for invocation of PL/CUDA functions.
MPS document recommends to set compute-mode EXCLUSIVE_PROCESS, however, PG-Strom requires DEFAULT mode.
Several operations, including PL/CUDA above, call CUDA APIs with MPS disabled explicitly, so other processes than MPS daemon must be able to use GPU devices.
The following commands start MPS daemon. Replace <UID> with user-id of PostgreSQL process.
$ nvidia-cuda-mps-control -d
$ echo start_server -uid <UID> | nvidia-cuda-mps-control
nvidia-smi command shows MPS daemon is using GPU device.
$ nvidia-smi
Sat Nov 3 12:22:26 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.48 Driver Version: 410.48 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla V100-PCIE... Off | 00000000:02:00.0 Off | 0 |
| N/A 45C P0 38W / 250W | 40MiB / 16130MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 11080 C nvidia-cuda-mps-server 29MiB |
+-----------------------------------------------------------------------------+
Knowledge base
We publish several articles, just called "notes", on the project wiki-site of PG-Strom.