Installation

This chapter introduces the steps to install PG-Strom.

Checklist

Server Hardware
- It requires generic x86_64 hardware that can run Linux operating system supported by CUDA Toolkit. We have no special requirement for CPU, storage and network devices.
- note002:HW Validation List may help you to choose the hardware.
- GPU Direct SQL Execution needs NVME-SSD devices, or fast network card with RoCE support, and to be installed under the same PCIe Root Complex where GPU is located on.
GPU Device
- PG-Strom requires at least one GPU device on the system, which is supported by CUDA Toolkit, has computing capability 7.5 (Turing generation) or later;
- Please check at 002: HW Validation List - List of supported GPU models for GPU selection.
Operating System
- PG-Strom requires Linux operating system for x86_64 architecture, and its distribution supported by CUDA Toolkit. Our recommendation is Red Hat Enterprise Linux or Rocky Linux version 10.x, 9.x or 8.x series.
- GPU Direct SQL (with cuFile driver) needs the nvidia-fs driver distributed with CUDA Toolkit, and Mellanox OFED (OpenFabrics Enterprise Distribution) driver.
PostgreSQL
- PG-Strom v6.1 requires PostgreSQL v15 or later.
- Some of PostgreSQL APIs used by PG-Strom internally are not included in the former versions.
CUDA Toolkit
- PG-Strom requires CUDA Toolkit version 12.2update1 or later.
- Some of CUDA Driver APIs used by PG-Strom internally are not included in the former versions.

Steps to Install

The overall steps to install are below:

Hardware Configuration
OS Installation
MOFED Driver installation
CUDA Toolkit installation
HeteroDB Extra Module installation
PostgreSQL installation
PG-Strom installation
PostgreSQL Extensions installation
- PostGIS
- contrib/cube

OS Installation

Choose a Linux distribution which is supported by CUDA Toolkit, then install the system according to the installation process of the distribution. NVIDIA DEVELOPER ZONE introduces the list of Linux distributions which are supported by CUDA Toolkit.

In case of Red Hat Enterprise Linux 8.x series (including Rocky Linux 8.x series), choose "Minimal installation" as base environment, and also check the "Development Tools" add-ons for the software selection

In case of Red Hat Enterprise Linux 9.x series (including Rocky Linux 9.x series), choose "Minimal installation" as base environment, and also check the "Standard" and "Development Tools" add-ons for the software selection

In case of Red Hat Enterprise Linux 10.x series (including Rocky Linux 10.x series), choose "Minimal installation" as base environment, and also check the "Standard" and "Development Tools" add-ons for the software selection

RHEL10 Software Selection

Next to the OS installation on the server, go on the package repository configuration to install the third-party packages.

If you didn't check the "Development Tools" at the installer, we can additionally install the software using the command below after the operating system installation.

# dnf groupinstall 'Development Tools'

Tip

If GPU devices installed on the server are too new, it may cause system crash during system boot. In this case, you may avoid the problem by adding nouveau.modeset=0 onto the kernel boot option, to disable the inbox graphic driver.

Disables IOMMU

GPU-Direct SQL uses GPUDirect Storage (cuFile) API of CUDA.

Prior to using GPUDirect Storage, it needs to disable the IOMMU configuration on the OS side.

Configure the kernel boot option according to the NVIDIA GPUDirect Storage Installation and Troubleshooting Guide description.

To disable IOMMU, add amd_iommu=off (for AMD CPU) or intel_iommu=off (for Intel CPU) to the kernel boot options.

Configuration at RHEL9/RHEL10

The command below adds the kernel boot option.

# grubby --update-kernel=ALL --args="amd_iommu=off"

Configuration at RHEL8

Open /etc/default/grub with an editor and add the above option to the GRUB_CMDLINE_LINUX_DEFAULT= line.

For example, the settings should look like this:

  :
GRUB_CMDLINE_LINUX="rhgb quiet amd_iommu=off"
  :

Run the following commands to apply the configuration to the kernel bool options.

-- for BIOS based system
# grub2-mkconfig -o /boot/grub2/grub.cfg
# shutdown -r now

-- for UEFI based system
# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
# shutdown -r now

Disables nouveau driver

When the nouveau driver, that is an open source compatible driver for NVIDIA GPUs, is loaded, it prevent to load the nvidia driver. In this case, reboot the operating system after a configuration to disable the nouveau driver.

To disable the nouveau driver, put the following configuration onto /etc/modprobe.d/disable-nouveau.conf, and run dracut command to apply them on the boot image of Linux kernel. Then, restart the system once.

# cat > /etc/modprobe.d/disable-nouveau.conf <<EOF
blacklist nouveau
options nouveau modeset=0
EOF
# dracut -f
# shutdown -r now

Enables extra repositories

EPEL(Extra Packages for Enterprise Linux)

Several software modules required by PG-Strom are distributed as a part of EPEL (Extra Packages for Enterprise Linux). You need to add a repository definition of EPEL packages for yum system to obtain these software.

One of the package we will get from EPEL repository is DKMS (Dynamic Kernel Module Support). It is a framework to build Linux kernel module for the running Linux kernel on demand; used for NVIDIA's GPU driver and related. Linux kernel module must be rebuilt according to version-up of Linux kernel, so we don't recommend to operate the system without DKMS.

epel-release package provides the repository definition of EPEL. You can obtain the package from the Fedora Project website.

-- For RHEL10
# dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-10.noarch.rpm

-- For RHEL9
# dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm

-- For RHEL8
# dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm

-- For Rocky8/Rocky9
# dnf install epel-release

Red Hat CodeReady Linux Builder

Installation of MOFED (Mellanox OpenFabrics Enterprise Distribution) driver requires the Red Hat CodeReady Linux Builder repository which is disabled in the default configuration of Red Hat Enterprise Linux 8.x installation. In Rocky Linux, it is called PowerTools

To enable this repository, run the command below:

-- For RHEL10
# subscription-manager repos --enable codeready-builder-for-rhel-10-x86_64-rpms

-- For RHEL9
# subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms

-- For Rocky9/Rocky10
# dnf config-manager --set-enabled crb

-- For RHEL8
# subscription-manager repos --enable codeready-builder-for-rhel-8-x86_64-rpms

-- For Rocky8
# dnf config-manager --set-enabled powertools

DOCA(OFED) Driver Installation

To use direct data from NVME-SSD to GPU (GPU-Direct SQL), you need to replace the operating system default inbox driver with the NVME drive distributed by DOCA (OFED) module. This section introduces the steps to install the DOCA (OFED) module.

Info

Note that the Linux kernel driver portion of the DOCA (OFED) module was formerly called MOFED (Mellanox Open Fabric Enterprise Driver), and can still be downloaded from here. However, it has already entered Long Term Support mode, and there is no prospect of the driver being provided for newer environments such as RHEL10. So, we recommend using the DOCA (OFED) package for the PG-Strom installation procedure.

The official installation instructions for the DOCA (OFED) module can be found here.

Add repository definition

Add the DOCA repository definition as follows. For RHEL 8/9 series, the ${releasever_major} and ${releasever_minor} macros are not defined, so write the OS version directly.

Example configuration for RHEL10

# cat > /etc/yum.repos.d/doca.repo <<EOF
[doca]
name=DOCA Online Repo
baseurl=https://linux.mellanox.com/public/repo/doca/latest/rhel\${releasever_major}.\${releasever_minor}/\${basearch}/
enabled=1
gpgcheck=0
EOF

Example configuration for RHEL9

# cat > /etc/yum.repos.d/doca.repo <<EOF
[doca]
name=DOCA Online Repo
baseurl=https://linux.mellanox.com/public/repo/doca/latest/rhel9.6/\${basearch}/
enabled=1
gpgcheck=0
EOF

Installation of DOCA packages

Install the necessary components from the DOCA package distributed by NVIDIA. As shown in the example below, please install kernel-module-extra-<KERNEL VERSION> (which should already be installed), doca-ofed, and doca-extra.

# dnf install -y kernel-modules-extra-$(uname -r)
# dnf install -y doca-ofed doca-extra

Building the NVME driver

Although the DOCA (OFED) package distributed by NVIDIA includes the NVMe driver, this driver is not built with the --with-gds option, which is necessary to enable GPU-Direct Storage.

So, we need to recompiler the driver to turn on GPU-Direct SQL. The procedure is fully scripted and can be executed as follows: /opt/mellanox/doca/tools/doca-kernel-support.

# /opt/mellanox/doca/tools/doca-kernel-support
doca-kernel-support: Building under /tmp/DOCA.EhUCIyOzbU
           :
doca-kernel-support: Rebuilding kernel modules
doca-kernel-support: Building mlnx-ofa_kernel under /tmp/DOCA.EhUCIyOzbU/build/mlnx-ofa_kernel with log /tmp/DOCA.EhUCIyOzbU/logs/mlnx-ofa_kernel.log
doca-kernel-support: Building iser under /tmp/DOCA.EhUCIyOzbU/build/iser with log /tmp/DOCA.EhUCIyOzbU/logs/iser.log
doca-kernel-support: Building isert under /tmp/DOCA.EhUCIyOzbU/build/isert with log /tmp/DOCA.EhUCIyOzbU/logs/isert.log
doca-kernel-support: Building srp under /tmp/DOCA.EhUCIyOzbU/build/srp with log /tmp/DOCA.EhUCIyOzbU/logs/srp.log
doca-kernel-support: Building mlnx-nfsrdma under /tmp/DOCA.EhUCIyOzbU/build/mlnx-nfsrdma with log /tmp/DOCA.EhUCIyOzbU/logs/mlnx-nfsrdma.log
doca-kernel-support: Building mlnx-nvme under /tmp/DOCA.EhUCIyOzbU/build/mlnx-nvme with log /tmp/DOCA.EhUCIyOzbU/logs/mlnx-nvme.log
doca-kernel-support: Building virtiofs under /tmp/DOCA.EhUCIyOzbU/build/virtiofs with log /tmp/DOCA.EhUCIyOzbU/logs/virtiofs.log
doca-kernel-support: Building knem under /tmp/DOCA.EhUCIyOzbU/build/knem with log /tmp/DOCA.EhUCIyOzbU/logs/knem.log
doca-kernel-support: Building xpmem under /tmp/DOCA.EhUCIyOzbU/build/xpmem with log /tmp/DOCA.EhUCIyOzbU/logs/xpmem.log
doca-kernel-support: Building kernel-mft under /tmp/DOCA.EhUCIyOzbU/build/kernel-mft with log /tmp/DOCA.EhUCIyOzbU/logs/kernel-mft.log
doca-kernel-support: Creating a rpm meta package:
doca-kernel-support: Creating a package repository in /tmp/DOCA.EhUCIyOzbU/repo/usr/share/doca-host-25.07-0.9.7.0/Modules/6.12.0-55.41.1.el10_0.x86_64
doca-kernel-support: Built single package: /tmp/DOCA.EhUCIyOzbU/doca-kernel-repo-25.07.0.9.7.0-1.kver.6.12.0.55.41.1.el10.0.x86.64.x86_64.rpm
doca-kernel-support: Done
Now you should install the generated single repository package to make its
files available:

  rpm -Uvh /tmp/DOCA.EhUCIyOzbU/doca-kernel-repo-25.07.0.9.7.0-1.kver.6.12.0.55.41.1.el10.0.x86.64.x86_64.rpm

After installing the package, doca-kernel-6.12.0.55.41.1.el10.0.x86.64 metapackage should be available.

Next steps could probably be:
  dnf makecache

Then, install ofed packages (e.g. doca-all, doca-ofed):
        dnf install doca-ofed-userspace
        dnf install --disablerepo=doca doca-kernel-6.12.0.55.41.1.el10.0.x86.64

Once the script finishes successfully, it will suggest the next steps you should take. First, install doca-kernel-<KERNEL_VERSION> to build a local repository containing the rebuilt kernel module.

[root@saba~]# rpm -Uvh /tmp/DOCA.EhUCIyOzbU/doca-kernel-repo-25.07.0.9.7.0-1.kver.6.12.0.55.41.1.el10.0.x86.64.x86_64.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:doca-kernel-repo-25.07.0.9.7.0-1.################################# [100%]

Next, install the doca-kernel-<KERNEL_VERSION>, kmod-mlnx-nvme and kmod-mlnx-nfsrdma packages according to the doca-kernel-support script output.

# dnf makecache
# dnf install doca-ofed-userspace
# dnf install --disablerepo=doca doca-kernel-6.12.0.55.41.1.el10.0.x86.64
# dnf install --disablerepo=doca kmod-mlnx-nvme kmod-mlnx-nfsrdma

Confirm location of the nvme kernel module using modinfo command. It was successfull installed if this kernel module is installed at extra/mlnx-nvme/host directory. (The OS default INBOX driver is stored at kernel/drivers/nvme/host.)

# modinfo nvme
filename:       /lib/modules/6.12.0-55.41.1.el10_0.x86_64/extra/mlnx-nvme/host/nvme.ko
description:    NVMe host PCIe transport driver
version:        1.0
license:        GPL
author:         Matthew Wilcox <willy@linuxintel.com>
rhelversion:    10.0
srcversion:     951B0D9C33E4E9A9D30FF50
alias:          pci:v*d*sv*sd*bc01sc08i02*
alias:          pci:v0000106Bd00002005sv*sd*bc*sc*i*
           :

Update the boot image using dracut command and restart the system to reflect the new nvme driver.

# dracut -f
# shutdown -r now

CUDA Toolkit Installation

This section introduces the installation of CUDA Toolkit. If you already installed the latest CUDA Toolkit, you can check whether your installation is identical with the configuration described in this section.

NVIDIA offers two approach to install CUDA Toolkit; one is by self-extracting archive (runfile), and the other is by RPM packages. We recommend the RPM installation for PG-Strom setup.

You can download the installation package for CUDA Toolkit from NVIDIA DEVELOPER ZONE. Choose your OS, architecture, distribution and version, then choose "rpm(network)" edition.

CUDA Toolkit download

Once you choose the "rpm(network)" option, it shows a few step-by-step shell commands to register the CUDA repository and install the packages. Run the installation according to the guidance. The example below shows the steps to install CUDA Toolkit 13 on RHEL10 environment.

# dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel10/x86_64/cuda-rhel10.repo
# dnf clean all
# dnf install cuda-toolkit-13-0
# dnf install nvidia-gds-13-0

Next to the installation of the CUDA Toolkit, two types of commands are introduced to install the nvidia driver.

Please use the open source version of nvidia-driver here. Only the open source version supports the GPUDirect Storage feature, and PG-Strom's GPU-Direct SQL utilizes this feature.

Next, install the driver module nvidia-gds for the GPU-Direct Storage (GDS). Please specify the same version name as the CUDA Toolkit version after the package name.

-- For RHEL9
# dnf module install nvidia-driver:open-dkms
-- For RHEL10
# dnf install nvidia-open

Once installation completed successfully, CUDA Toolkit is deployed at /usr/local/cuda.

# ls /usr/local/cuda/
bin                             extras   man               share
compute-sanitizer               gds      nsightee_plugins  src
CUDA_Toolkit_Release_Notes.txt  include  nvml              targets
DOCS                            lib64    nvvm              tools
EULA.txt                        LICENSE  README            version.json

Once installation gets completed, ensure the system recognizes the GPU devices correctly. nvidia-smi command shows GPU information installed on your system, as follows.

# nvidia-smi
Tue Oct 28 14:22:43 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.95.05              Driver Version: 580.95.05      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA H100 PCIe               Off |   00000000:42:00.0 Off |                    0 |
| N/A   37C    P0             48W /  350W |       0MiB /  81559MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Tips

Additional configurations for systems with NVSwitch

For systems with multiple GPUs that use NVSwitch for interconnect them, the nvidia-fabricmanager module must be installed. If this package is not installed, cuInit(), which initializes CUDA, will fail with the CUDA_ERROR_SYSTEM_NOT_READY error, and PG-Strom will not start. Run the following commands to install the nvidia-fabricmanager package. (source)

# dnf install nvidia-fabricmanager
# systemctl enable nvidia-fabricmanager.service
# systemctl start nvidia-fabricmanager.service

Check GPUDirect Storage status

After the installation of CUDA Toolkit according to the steps above, your system will become ready for the GPUDirect Storage. Run gdscheck tool to confirm the configuration for each storage devices, as follows. (Thie example loads not only nvme, but nvme-rdma and rpcrdma kernel modules also, therefore, it reports the related features as Supported)

# /usr/local/cuda/gds/tools/gdscheck -p
 GDS release version: 1.15.1.6
 nvidia_fs version:  2.26 libcufile version: 2.12
 Platform: x86_64
 ============
 ENVIRONMENT:
 ============
 =====================
 DRIVER CONFIGURATION:
 =====================
 NVMe P2PDMA        : Unsupported
 NVMe               : Supported
 NVMeOF             : Unsupported
 SCSI               : Unsupported
 ScaleFlux CSD      : Unsupported
 NVMesh             : Unsupported
 DDN EXAScaler      : Unsupported
 IBM Spectrum Scale : Unsupported
 NFS                : Supported
 BeeGFS             : Unsupported
 ScaTeFS            : Unsupported
 WekaFS             : Unsupported
 Userspace RDMA     : Unsupported
 --Mellanox PeerDirect : Disabled
 --rdma library        : Not Loaded (libcufile_rdma.so)
 --rdma devices        : Not configured
 --rdma_device_status  : Up: 0 Down: 0
 =====================
 CUFILE CONFIGURATION:
 =====================
 properties.use_pci_p2pdma : false
 properties.use_compat_mode : true
 properties.force_compat_mode : false
 properties.gds_rdma_write_support : true
 properties.use_poll_mode : false
 properties.poll_mode_max_size_kb : 4
 properties.max_batch_io_size : 128
 properties.max_batch_io_timeout_msecs : 5
 properties.max_direct_io_size_kb : 16384
 properties.max_device_cache_size_kb : 131072
 properties.per_buffer_cache_size_kb : 1024
 properties.max_device_pinned_mem_size_kb : 33554432
 properties.posix_pool_slab_size_kb : 4 1024 16384
 properties.posix_pool_slab_count : 128 64 64
 properties.rdma_peer_affinity_policy : RoundRobin
 properties.rdma_dynamic_routing : 0
 fs.generic.posix_unaligned_writes : false
 fs.lustre.posix_gds_min_kb: 0
 fs.beegfs.posix_gds_min_kb: 0
 fs.scatefs.posix_gds_min_kb: 0
 fs.weka.rdma_write_support: false
 fs.gpfs.gds_write_support: false
 fs.gpfs.gds_async_support: true
 profile.nvtx : false
 profile.cufile_stats : 0
 miscellaneous.api_check_aggressive : false
 execution.max_io_threads : 4
 execution.max_io_queue_depth : 128
 execution.parallel_io : true
 execution.min_io_threshold_size_kb : 8192
 execution.max_request_parallelism : 4
 properties.force_odirect_mode : false
 properties.prefer_iouring : false
 =========
 GPU INFO:
 =========
 GPU index 0 NVIDIA H100 PCIe bar:1 bar size (MiB):131072 supports GDS, IOMMU State: Disabled
 ==============
 PLATFORM INFO:
 ==============
 IOMMU: disabled
 Nvidia Driver Info Status: Supported(Nvidia Open Driver Installed)
 Cuda Driver Version Installed:  13000
 Platform: AS -2015CS-TNR, Arch: x86_64(Linux 6.12.0-55.41.1.el10_0.x86_64)
 Platform verification succeeded

Tips

Additional configuration for RAID volume

For data reading from software RAID (md-raid0) volumes by GPUDirect Storage, the following line must be added to the /lib/udev/rules.d/63-md-raid-arrays.rules configuration file.

IMPORT{program}="/usr/sbin/mdadm --detail --export $devnode"

Then reboot the system to ensure the new configuration. See NVIDIA GPUDirect Storage Installation and Troubleshooting Guide for the details.

PCI Bar1 Memory Configuration

GPU-Direct SQL maps GPU device memory to the PCI BAR1 region (physical address space) on the host system, and sends P2P-RDMA requests to NVME devices with that as the destination for the shortest data transfer.

To perform P2P-RDMA with sufficient multiplicity, the GPU must have enough PCI BAR1 space to map the device buffer. The size of the PCI BAR1 area is fixed for most GPUs, and PG-Strom recommends products whose size exceeds the GPU device memory size.

However, some GPU products allow to change the size of the PCI BAR1 area by switching the operation mode. If your GPU is either of the following, refer to the NVIDIA Display Mode Selector Tool and switch to the mode that maximizes the PCI BAR1 area size.

NVIDIA L40S
NVIDIA L40
NVIDIA A40
NVIDIA RTX 6000 Ada
NVIDIA RTX A6000
NVIDIA RTX A5500
NVIDIA RTX A5000

To check the GPU memory size and PCI BAR1 size installed in the system, use the nvidia-smi -q command. Memory-related status is displayed as shown below.

$ nvidia-smi -q
        :
    FB Memory Usage
        Total                             : 46068 MiB
        Reserved                          : 685 MiB
        Used                              : 4 MiB
        Free                              : 45377 MiB
    BAR1 Memory Usage
        Total                             : 65536 MiB
        Used                              : 1 MiB
        Free                              : 65535 MiB
        :

Edit `/etc/cufile.json`

Edit the configuration file /etc/cufile.json, which is used internally by GPU Direct SQL.

In the JSON-formatted configuration, change the value of "parallel_io" in the "execution" section to false, from the default value (true).

  "execution" : {
          :
      // enable support for parallel IO
      "parallel_io" : false,
          :
  },

Tips

cuFile splits I/O requests larger than a certain size (default 8MB) and uses an internal thread pool to execute multiple I/O requests in parallel. While this is an effective mechanism for single-threaded GPU applications to perform I/O efficiently, it actually increases overhead for applications like PG-Strom, which are multi-threaded and execute finely granular I/O requests concurrently. Therefore, PG-Strom recommends disabling cuFile's Parallel-I/O functionality.

HeteroDB extra modules

PG-Strom and related packages are distributed from HeteroDB Software Distribution Center. You need to add a repository definition of HeteroDB-SWDC for you system to obtain these software.

heterodb-swdc package provides the repository definition of HeteroDB-SWDC. Access to the HeteroDB Software Distribution Center using Web browser, download the heterodb-swdc package on top of the file list, then install this package. (Use heterodb-swdc-1.3-1.el10.noarch.rpm for RHEL10) Once heterodb-swdc package gets installed, yum system configuration is updated to get software from the HeteroDB-SWDC repository.

Install the heterodb-swdc package as follows.

# dnf install https://heterodb.github.io/swdc/yum/rhel10-noarch/heterodb-swdc-1.3-1.el10.noarch.rpm

heterodb-extra module enhances PG-Strom the following features.

multi-GPUs support
GPUDirect SQL
GiST index support on GPU
License management

If you don't use the above features, only open source modules, you don't need to install the heterodb-extra module here. Please skip this section.

Install the heterodb-extra package, downloaded from the SWDC, as follows.

# dnf install heterodb-extra

License activation

License activation is needed to use all the features of heterodb-extra, provided by HeteroDB,Inc. You can operate the system without license, but features below are restricted.

Multiple GPUs support
Striping of NVME-SSD drives (md-raid0) on GPUDirect SQL
Support of NVME-oF device on GPUDirect SQL
Support of GiST index on GPU-version of PostGIS workloads

You can obtain a license file, like as a plain text below, from HeteroDB,Inc.

IAgIVdKxhe+BSer3Y67jQW0+uTzYh00K6WOSH7xQ26Qcw8aeUNYqJB9YcKJTJb+QQhjmUeQpUnboNxVwLCd3HFuLXeBWMKp11/BgG0FSrkUWu/ZCtDtw0F1hEIUY7m767zAGV8y+i7BuNXGJFvRlAkxdVO3/K47ocIgoVkuzBfLvN/h9LffOydUnHPzrFHfLc0r3nNNgtyTrfvoZiXegkGM9GBTAKyq8uWu/OGonh9ybzVKOgofhDLk0rVbLohOXDhMlwDl2oMGIr83tIpCWG+BGE+TDwsJ4n71Sv6n4bi/ZBXBS498qShNHDGrbz6cNcDVBa+EuZc6HzZoF6UrljEcl=
----
VERSION:2
SERIAL_NR:HDB-TRIAL
ISSUED_AT:2019-05-09
EXPIRED_AT:2019-06-08
GPU_UUID:GPU-a137b1df-53c9-197f-2801-f2dccaf9d42f

Copy the license file to /etc/heterodb.license, then restart PostgreSQL.

The startup log messages of PostgreSQL dumps the license information, and it tells us the license activation is successfully done.

    :
 LOG:  HeteroDB Extra module loaded [api_version=20231105,cufile=on,nvme_strom=off,githash=9ca2fe4d2fbb795ad2d741dcfcb9f2fe499a5bdf]
 LOG:  HeteroDB License: { "version" : 2, "serial_nr" : "HDB-TRIAL", "issued_at" : "2022-11-19", "expired_at" : "2099-12-31", "nr_gpus" : 1, "gpus" : [ { "uuid" : "GPU-13943bfd-5b30-38f5-0473-78979c134606" } ]}
 LOG:  PG-Strom version 5.0.1 built for PostgreSQL 15 (githash: 972441dbafed6679af86af40bc8613be2d73c4fd)
    :

PostgreSQL Installation

This section introduces PostgreSQL installation with RPM. We don't introduce the installation steps from the source because there are many documents for this approach, and there are also various options for the ./configure script.

PostgreSQL is also distributed in the packages of Linux distributions, however, it is not the latest one, and often older than the version which supports PG-Strom. For example, Red Hat Enterprise Linux 7.x distributes PostgreSQL v9.2.x series. This version had been EOL by the PostgreSQL community.

PostgreSQL Global Development Group provides yum repository to distribute the latest PostgreSQL and related packages. Like the configuration of EPEL, you can install a small package to set up yum repository, then install PostgreSQL and related software.

Here is the list of yum repository definition: http://yum.postgresql.org/repopackages.php.

Repository definitions are per PostgreSQL major version and Linux distribution. You need to choose the one for your Linux distribution, and for PostgreSQL v15 or later.

You can install PostgreSQL as following steps:

Installation of yum repository definition.
Disables the distribution's default PostgreSQL module
Installation of PostgreSQL packages.

When PostgreSQL v18 is used for instance, installation of PG-Strom requires postgresql18-server and postgresql18-devel packages.

Below is the steps to install PostgreSQL v18 for RHEL10.

# dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-10-x86_64/pgdg-redhat-repo-latest.noarch.rpm
# dnf install -y postgresql18-devel postgresql18-server

Note

On the Red Hat Enterprise Linux 8.x or 9.x, the package name postgresql conflicts to the default one at the distribution, thus, unable to install the packages from PGDG. So, disable the postgresql module by the distribution, using dnf -y module disable postgresql. The AppStream feature was improved, Red Hat Enterprise Linux 10.x does not need this operation.

libarrow/libparquet Installation

PG-Strom v6.1 and later requires libarrow and libparquet for building and installation.

Since the packages provided by the Linux distribution may be outdated, please follow the developer community guidance and install the arrow-devel and parquet-devel packages.

The steps required for a minimal installation, excluding any overlaps with the installation steps above, are as follows.

$ sudo dnf install -y https://packages.apache.org/artifactory/arrow/almalinux/$(cut -d: -f5 /etc/system-release-cpe | cut -d. -f1)/apache-arrow-release-latest.rpm
$ sudo dnf install -y arrow-devel
$ sudo dnf install -y parquet-devel

PG-Strom Installation

RPM Installation

PG-Strom and related packages are distributed from HeteroDB Software Distribution Center. If you repository definition has been added, not many tasks are needed.

We provide individual RPM packages of PG-Strom for each PostgreSQL major version. pg_strom-PG17 package is built for PostgreSQL v17, and pg_strom-PG18 is also built for PostgreSQL v18.

It is a restriction due to binary compatibility of extension modules for PostgreSQL.

# dnf install -y pg_strom-PG18

That's all for package installation.

Installation from the source

For developers, we also introduces the steps to build and install PG-Strom from the source code.

Getting the source code

Like RPM packages, you can download tarball of the source code from HeteroDB Software Distribution Center. On the other hands, here is a certain time-lags to release the tarball, it may be preferable to checkout the master branch of PG-Strom on GitHub to use the latest development branch.

$ git clone https://github.com/heterodb/pg-strom.git
Cloning into 'pg-strom'...
remote: Counting objects: 13797, done.
remote: Compressing objects: 100% (215/215), done.
remote: Total 13797 (delta 208), reused 339 (delta 167), pack-reused 13400
Receiving objects: 100% (13797/13797), 11.81 MiB | 1.76 MiB/s, done.
Resolving deltas: 100% (10504/10504), done.

Building the PG-Strom

Configuration to build PG-Strom must match to the target PostgreSQL strictly. For example, if a particular strcut has inconsistent layout by the configuration at build, it may lead problematic bugs; not easy to find out. Thus, not to have inconsistency, PG-Strom does not have own configure script, but references the build configuration of PostgreSQL using pg_config command.

If PATH environment variable is set to the pg_config command of the target PostgreSQL, run make and make install. Elsewhere, give PG_CONFIG=... parameter on make command to tell the full path of the pg_config command.

$ cd pg-strom/src
$ make PG_CONFIG=/usr/pgsql-18/bin/pg_config
$ sudo make install PG_CONFIG=/usr/pgsql-18/bin/pg_config

Post Installation Setup

Creation of database cluster

Database cluster is not constructed yet, run initdb command to set up initial database of PostgreSQL.

The default path of the database cluster on RPM installation is /var/lib/pgsql/<version number>/data. If you install postgresql-alternatives package, this default path can be referenced by /var/lib/pgdata regardless of the PostgreSQL version.

# su - postgres
$ /usr/pgsql-18/bin/initdb -D /var/lib/pgsql/18/data
            :
Success. You can now start the database server using:

    /usr/pgsql-18/bin/pg_ctl -D /var/lib/pgsql/18/data -l logfile start

Setup postgresql.conf

Next, edit postgresql.conf which is a configuration file of PostgreSQL. The parameters below should be edited at least to work PG-Strom. Investigate other parameters according to usage of the system and expected workloads.

shared_preload_libraries
- PG-Strom module must be loaded on startup of the postmaster process by the shared_preload_libraries. Unable to load it on demand. Therefore, you must add the configuration below.
- shared_preload_libraries = '$libdir/pg_strom'
max_worker_processes
- PG-Strom internally uses several background workers, so the default configuration (= 8) is too small for other usage. So, we recommand to expand the variable for a certain margin.
- max_worker_processes = 100
shared_buffers
- Although it depends on the workloads, the initial configuration of shared_buffers is too small for the data size where PG-Strom tries to work, thus storage workloads restricts the entire performance, and may be unable to work GPU efficiently.
- So, we recommend to expand the variable for a certain margin.
- shared_buffers = 10GB
- Please consider to apply SSD-to-GPU Direct SQL Execution to process larger than system's physical RAM size.
work_mem
- Although it depends on the workloads, the initial configuration of work_mem is too small to choose the optimal query execution plan on analytic queries.
- An typical example is, disk-based merge sort may be chosen instead of the in-memory quick-sorting.
- So, we recommend to expand the variable for a certain margin.
- work_mem = 1GB

Expand OS resource limits

GPU Direct SQL especially tries to open many files simultaneously, so resource limit for number of file descriptors per process should be expanded.

Also, we recommend not to limit core file size to generate core dump of PostgreSQL certainly on system crash.

If PostgreSQL service is launched by systemd, you can put the configurations of resource limit at /etc/systemd/system/postgresql-XX.service.d/pg_strom.conf.

RPM installation setups the configuration below by the default.

It comments out configuration to the environment variable CUDA_ENABLE_COREDUMP_ON_EXCEPTION. This is a developer option that enables to generate GPU's core dump on any CUDA/GPU level errors, if enabled. See CUDA-GDB:GPU core dump support for more details.

[Service]
LimitNOFILE=65536
LimitCORE=infinity
#Environment=CUDA_ENABLE_COREDUMP_ON_EXCEPTION=1

Start PostgreSQL

Start PostgreSQL service.

If PG-Strom is set up appropriately, it writes out log message which shows PG-Strom recognized GPU devices. The example below recognized two NVIDIA A100 (PCIE; 40GB), and displays the closest GPU identifier foe each NVME-SSD drive.

# systemctl start postgresql-18
# journalctl -u postgresql-18
# journalctl -u postgresql-18
Oct 28 16:17:16 saba.heterodb.in systemd[1]: Starting postgresql-18.service - PostgreSQL 18 database server...
Oct 28 16:17:16 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:16.574 JST [4325] LOG:  HeteroDB Extra module loaded [api_version=20250316,cufile=on,nvme_strom=off,nvidia-fs=on,githash=d5044aceca80aecee04c7c1a662c066122986291]
Oct 28 16:17:16 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:16.574 JST [4325] LOG:  HeteroDB License: { "version" : 2, "serial_nr" : "HDB-TRIAL", "issued_at" : "2025-10-28", "expired_at" : "2099-12-31", "nr_gpus" : 1, "gpus" : [ { "uuid" : "GPU-156b86ad-864c-d97d-4729-5c3405d8...
Oct 28 16:17:16 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:16.574 JST [4325] LOG:  PG-Strom version 6.1.0.el9 built for PostgreSQL 18 (githash: )
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.011 JST [4325] LOG:  PG-Strom binary built for CUDA 12.9 (CUDA runtime 13.0)
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.011 JST [4325] LOG:  PG-Strom: GPU0 NVIDIA H100 PCIe (114 SMs; 1755MHz, L2 51200kB), RAM 79.18GB (5120bits, 1.52GHz), CC 9.0
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.012 JST [4325] LOG:  [0000:42:00:0] GPU0 (NVIDIA H100 PCIe; GPU-156b86ad-864c-d97d-4729-5c3405d88e76)
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.012 JST [4325] LOG:  [0000:09:00:0] nvme8 (SAMSUNG MZ1LB960HAJQ-00007) --> GPU0 [dist=130]
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.012 JST [4325] LOG:  [0000:c5:00:0] nvme2 (INTEL SSDPF2KX038TZ) --> GPU0 [dist=130]
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.012 JST [4325] LOG:  [0000:88:00:0] nvme4 (INTEL SSDPF2KX038TZ) --> GPU0 [dist=130]
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.012 JST [4325] LOG:  [0000:c3:00:0] nvme1 (INTEL SSDPF2KX038TZ) --> GPU0 [dist=130]
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.012 JST [4325] LOG:  [0000:c7:00:0] nvme6 (INTEL SSDPF2KX038TZ) --> GPU0 [dist=130]
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.012 JST [4325] LOG:  [0000:c1:00:0] nvme0 (INTEL SSDPF2KX038TZ) --> GPU0 [dist=130]
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.012 JST [4325] LOG:  [0000:41:00:0] mlx5_0 (MT4125) --> GPU0 [dist=130]
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.013 JST [4325] LOG:  [0000:82:00:0] nvme3 (INTEL SSDPF2KX038TZ) --> GPU0 [dist=130]
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.013 JST [4325] LOG:  [0000:01:00:0] nvme5 (INTEL SSDPF2KX038TZ) --> GPU0 [dist=130]
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.013 JST [4325] LOG:  [0000:41:00:1] mlx5_1 (MT4125) --> GPU0 [dist=130]
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.013 JST [4325] LOG:  [0000:84:00:0] nvme7 (INTEL SSDPF2KX038TZ) --> GPU0 [dist=130]
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.142 JST [4325] LOG:  redirecting log output to logging collector process
Oct 28 16:17:17 saba.heterodb.in postgres[4325]: 2025-10-28 16:17:17.142 JST [4325] HINT:  Future log output will appear in directory "log".
Oct 28 16:17:17 saba.heterodb.in systemd[1]: Started postgresql-18.service - PostgreSQL 18 database server.

Creation of PG-Strom Extension

At the last, create database objects related to PG-Strom, like SQL functions. This steps are packaged using EXTENSION feature of PostgreSQL. So, all you needs to run is CREATE EXTENSION on the SQL command line.

Please note that this step is needed for each new database. If you want PG-Strom is pre-configured on new database creation, you can create PG-Strom extension on the template1 database, its configuration will be copied to the new database on CREATE DATABASE command.

$ psql -U postgres
psql (16.3)
Type "help" for help.

postgres=# CREATE EXTENSION pg_strom ;
CREATE EXTENSION

That's all for the installation.

PostGIS Installation

PG-Strom supports execution of a part of PostGIS functions on GPU devices. This section introduces the steps to install PostGIS module. Skip it on your demand.

PostGIS module can be installed from the yum repository by PostgreSQL Global Development Group, like PostgreSQL itself. The example below shows the command to install PostGIS v3.6 built for PostgreSQL v18.

# dnf install postgis36_18

Start PostgreSQL server after the initial setup of database cluster, then run CREATE EXTENSION command from SQL client to define geometry data type and SQL functions for geoanalytics.

postgres=# CREATE EXTENSION postgis;
CREATE EXTENSION

Installation on Ubuntu Linux

Although PG-Strom packages are not available for Ubuntu Linux right now, you can build and run PG-Strom from the source code.

After the installation of Ubuntu Linux, install the MOFED driver, CUDA Toolkit, and PostgreSQL for Ubuntu Linux, respectively.

Next, install the heterodb-extra package. A .deb package for Ubuntu Linux is provided, so please obtain the latest version from the SWDC.

$ wget https://heterodb.github.io/swdc/deb/heterodb-extra_6.7-1_amd64.deb
$ sudo dpkg -i heterodb-extra_6.7-1_amd64.deb

Checkout the source code of PG-Strom, build and install as follows. At this time, do not forget to specify the target PostgreSQL by pg_config.

Post-installation configuration is almost same as for Red Hat Enterprise Linux or Rocky Linux.

$ git clone https://github.com/heterodb/pg-strom.git
$ cd pg-strom/src
$ make PG_CONFIG=/path/to/pgsql/bin/pg_config -j 8
$ sudo make PG_CONFIG=/path/to/pgsql/bin/pg_config install

However, if you use a packaged PostgreSQL and start it by systemctl command, the PATH environment variable will be cleared (probably for security reasons). As a result, the script launched to build the GPU binary on the first startup will not work properly. To avoid this, if you are using Ubuntu Linux, add the following line to /etc/postgresql/main/PGVERSION/environment.

/etc/postgresql/PGVERSION/main/environment:


PATH='/usr/local/cuda/bin:/usr/bin:/bin'

Installation

Checklist

Steps to Install

OS Installation

Disables IOMMU

Configuration at RHEL9/RHEL10

Configuration at RHEL8

Disables nouveau driver

Enables extra repositories

EPEL(Extra Packages for Enterprise Linux)

Red Hat CodeReady Linux Builder

DOCA(OFED) Driver Installation

CUDA Toolkit Installation

Check GPUDirect Storage status

PCI Bar1 Memory Configuration

Edit /etc/cufile.json

HeteroDB extra modules

License activation

PostgreSQL Installation

libarrow/libparquet Installation

PG-Strom Installation

RPM Installation

Installation from the source

Getting the source code

Building the PG-Strom

Post Installation Setup

Creation of database cluster

Setup postgresql.conf

Expand OS resource limits

Start PostgreSQL

Creation of PG-Strom Extension

PostGIS Installation

Installation on Ubuntu Linux

Edit `/etc/cufile.json`