System Administration

Usage of MPS daemon

In case when multi-process application like PostgreSQL uses GPU device, it is a well known solution to use MPS (Multi-Process Service) to reduce context switch on GPU side and resource consumption for device management.

https://docs.nvidia.com/deploy/mps/index.html

It is also recommended for PG-Strom to apply MPS, however, you need to pay attention for several limitations below.

One MPS daemon can provide its service for up to 48 clients (16 clients if Pascal or older). So, DB administration must ensure number of PostgreSQL processes using GPU (including background workers in CPU parallelism) is less than 48 (or 16 if Pascal).

MPS does not support dynamic parallelism, and load GPU programs using the feature. GPU programs automatically generated from SQL will never use dynamic parallelism, however, PL/CUDA user defined function may use dynamic parallelism if it links CUDA device runtime to invoke sub-kernels. So, we don't use MPS for invocation of PL/CUDA functions.

MPS document recommends to set compute-mode EXCLUSIVE_PROCESS, however, PG-Strom requires DEFAULT mode. Several operations, including PL/CUDA above, call CUDA APIs with MPS disabled explicitly, so other processes than MPS daemon must be able to use GPU devices.

The following commands start MPS daemon. Replace <UID> with user-id of PostgreSQL process.

$ nvidia-cuda-mps-control -d
$ echo start_server -uid <UID> | nvidia-cuda-mps-control

nvidia-smi command shows MPS daemon is using GPU device.

$ nvidia-smi
Sat Nov  3 12:22:26 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.48                 Driver Version: 410.48                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-PCIE...  Off  | 00000000:02:00.0 Off |                    0 |
| N/A   45C    P0    38W / 250W |     40MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     11080      C   nvidia-cuda-mps-server                        29MiB |
+-----------------------------------------------------------------------------+

Knowledge base

We publish several articles, just called "notes", on the project wiki-site of PG-Strom.

https://github.com/heterodb/pg-strom/wiki