Self-Hosting an AI Server: PVE GPU Passthrough, Drivers, CUDA and cuDNN for Running Local LLMs

With open source LLMs and the tools the community keeps shipping, AI is finally within everyone’s reach. Plenty of people can now run large models on a local laptop — but laptops are underpowered, and they have other work to do; you can’t keep a model running on them around the clock. I prefer PVE, so here’s my deployment walkthrough, a guide for my future re-deploys.

The hardware

• Motherboard: ASUS Z97-K R2.0
• CPU: Intel Core i7-4790 @ 3.6GHz
• RAM: 32 GB (4×8 GB) DDR3 1600MHz
• GPU: NVIDIA GeForce RTX 2080 Ti 22G

This hardware runs a 4-bit quantized Qwen2.5 32B Q4_K_M model quite smoothly — the whole model runs on the GPU and its VRAM, with almost no load on the CPU and RAM. Bigger models, say 72B, have to spill over onto the CPU and memory, and it gets very slow. It runs, but it’s not pleasant.

The other machine (not covered here)

• Motherboard: X99-F8D PLUS
• CPU: 2× Intel Xeon E5-2660 v3
• RAM: 512 GB (8×64 GB) 4DRX4 2400T
• GPU: 2× NVIDIA GeForce RTX 2080 Ti 22G

This one can run the full DeepSeek-R1 671B Q4_K_M at 4-bit quantization. The VRAM can’t hold it all, so part of it lands on the CPU and memory — very slow, two or three seconds per character. It runs; using it is another matter.

PVE initial setup

I’ll skip the PVE installation itself — assume PVE is already installed.

Switch the apt sources to the Tsinghua mirror

cp /etc/apt/sources.list /etc/apt/sources.list.bak
nano /etc/apt/sources.list

Replace with:

# 默认注释了源码镜像以提高 apt update 速度，如有需要可自行取消注释
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm main contrib non-free non-free-firmware
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm main contrib non-free non-free-firmware
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-updates main contrib non-free non-free-firmware
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-updates main contrib non-free non-free-firmware
deb https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-backports main contrib non-free non-free-firmware
# deb-src https://mirrors.tuna.tsinghua.edu.cn/debian/ bookworm-backports main contrib non-free non-free-firmware
# 以下安全更新软件源包含了官方源与镜像站配置，如有需要可自行修改注释切换
deb https://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware
# deb-src https://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware

Then edit /etc/apt/sources.list.d/pve-no-subscription.list:

nano /etc/apt/sources.list.d/pve-no-subscription.list
deb https://mirrors.tuna.tsinghua.edu.cn/proxmox/debian/pve bookworm pve-no-subscription

Remove the enterprise sources:

rm /etc/apt/sources.list.d/ceph.list
rm /etc/apt/sources.list.d/pve-enterprise.list

Update and upgrade:

apt update
apt upgrade

Install your usual tools

I’m a vim person; everything below edits with vim:

apt install vim

GPU passthrough

BIOS changes

• Enable VT-d / AMD-Vi
• Make sure the motherboard and CPU support IOMMU (Intel VT-d or AMD-Vi) and enable it in the BIOS
• Enable UEFI boot and disable CSM

Enable IOMMU

Edit /etc/default/grub and add the IOMMU parameters to GRUB_CMDLINE_LINUX_DEFAULT. Intel systems:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"

AMD systems:

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

Update GRUB:

update-grub

After a reboot, verify IOMMU is active:

dmesg | grep -e DMAR -e IOMMU

Configure PCI passthrough

Get the GPU’s PCI device IDs — this lists the GPU (01:00.0) and its audio device (01:00.1):

lspci -nn | grep -i nvidia

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] [10de:1e07] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1)
01:00.2 USB controller [0c03]: NVIDIA Corporation TU102 USB 3.1 Host Controller [10de:1ad6] (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C UCSI Controller [10de:1ad7] (rev a1)

Bind the devices to the VFIO driver — edit /etc/modprobe.d/vfio.conf:

options vfio-pci ids=10de:1e07,10de:10f7

Replace 10de:1e07 and 10de:10f7 with your actual GPU and audio device IDs.

Blacklist the default drivers — edit /etc/modprobe.d/blacklist.conf and add:

blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist rivafb

Regenerate the initramfs:

update-initramfs -u

Reboot and verify the devices are bound to vfio-pci:

lspci -nnv | grep -i vfio

Create the VM

Create a KVM virtual machine — not an LXC/CT container, they’re different. Things to watch:

• Machine type must be q35
• CPU type must be host
• Don’t use UEFI mode — Secure Boot’s kernel signature verification is a pain
• In Hardware, add a PCI device: raw device, pick your GPU, check All Functions and PCI-Express

Inside the VM

Boot the VM and SSH in. I installed Ubuntu 24.04.1; I’ll skip the OS install and the Tsinghua mirror switch. Everything below runs as root. Update and install what’s needed:

apt update
apt upgrade
apt install gcc make dkms qemu-guest-agent

Disable the open source nouveau driver — edit /etc/modprobe.d/blacklist-nouveau.conf and add:

blacklist nouveau
options nouveau modeset=0

Update the kernel modules: update-initramfs -u. Reboot: reboot. SSH back in.

Download the driver: https://www.nvidia.cn/drivers/lookup/ Download CUDA: https://developer.nvidia.com/cuda-downloads Download cuDNN: https://developer.nvidia.com/cudnn-downloads

Note that driver and CUDA versions have compatibility requirements — you can’t mix them freely. The versions I downloaded:

• NVIDIA-Linux-x86_64-565.77.run
• cuda_12.6.3_560.35.05_linux.run
• cudnn-local-repo-ubuntu2404-9.6.0_1.0-1_amd64.deb

Install the driver

Install the driver first:

sh ./NVIDIA-Linux-x86_64-565.77.run

The prompts are just: pick Nvidia, continue, agree. If you don’t click continue, it aborts. When it finishes, run nvidia-smi — if you see something like this, it worked:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.77                 Driver Version: 565.77         CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 2080 Ti     Off |   00000000:01:00.0 Off |                  N/A |
|  0%   50C    P0             33W /  260W |       1MiB /  22528MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

Install CUDA

When installing CUDA, disable OpenGL by adding --no-opengl-libs — driver files only, no OpenGL, to avoid conflicting with Ubuntu’s graphics stack (otherwise you get an infinite login loop):

sh ./cuda_12.6.3_560.35.05_linux.run --no-opengl-libs

Accept the license: type accept, Enter. Use the arrow keys to get to - [X] Driver, press space to uncheck it so it reads - [ ] Driver, then select Install and hit Enter. Usually this just works, but for whatever reason mine printed:

===========
= Summary =
===========

Driver:   Not Selected
Toolkit:  Installed in /usr/local/cuda-12.6/

Please make sure that
 -   PATH includes /usr/local/cuda-12.6/bin
 -   LD_LIBRARY_PATH includes /usr/local/cuda-12.6/lib64, or, add /usr/local/cuda-12.6/lib64 to /etc/ld.so.conf and run ldconfig as root

To uninstall the CUDA Toolkit, run cuda-uninstaller in /usr/local/cuda-12.6/bin
***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 560.00 is required for CUDA 12.6 functionality to work.
To install the driver using this installer, run the following command, replacing <CudaInstaller> with the name of this run file:
    sudo <CudaInstaller>.run --silent --driver

Logfile is /var/log/cuda-installer.log

“Driver: Not Selected” is expected — we installed the driver ourselves above, no need for CUDA to do it again. The toolkit installed fine, but the environment variables weren’t set automatically: /usr/local/cuda-12.6/bin needs to go into PATH, plus LD_LIBRARY_PATH and /etc/ld.so.conf. If you see the same output, set them manually like me; if not, skip ahead to verification.

Edit ~/.bashrc and add these two lines at the end:

export PATH=/usr/local/cuda-12.6/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-12.6/lib64:$LD_LIBRARY_PATH

Then source ~/.bashrc to apply immediately.

Edit /etc/ld.so.conf and add the CUDA library path:

/usr/local/cuda-12.6/lib64

Then run ldconfig. After installation, run nvcc -V — if you see version info, it worked:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Oct_29_23:50:19_PDT_2024
Cuda compilation tools, release 12.6, V12.6.85
Build cuda_12.6.r12.6/compiler.35059454_0

Install cuDNN

My package is cudnn-local-repo-ubuntu2404-9.6.0_1.0-1_amd64.deb — swap in your own filename:

dpkg -i cudnn-local-repo-ubuntu2404-9.6.0_1.0-1_amd64.deb
cp /var/cudnn-local-repo-ubuntu2404-9.6.0/cudnn-*-keyring.gpg /usr/share/keyrings/
apt update
apt install cudnn

This one practically never fails, so I didn’t bother verifying — properly verifying would mean running a neural network program, and I couldn’t be bothered.

Install Ollama

I use Ollama to run LLMs, so next up is the Ollama install. One-line script:

curl -fsSL https://ollama.com/install.sh | sh

The script downloads files from GitHub — if your network is restricted it may fail. Manual install docs: https://github.com/ollama/ollama/blob/main/docs/linux.md

My network wasn’t restricted (I use a proxy), so the automatic install just worked. If you install manually and need help, you can contact me for paid support.

ollama -v showing a version number means success.

I also needed to edit /etc/systemd/system/ollama.service so Ollama listens on 0.0.0.0. Add these environment variables to the service file:

# 监听的地址和端口
Environment="OLLAMA_HOST=0.0.0.0:11434"
# 保持模型不被自动卸载
Environment="OLLAMA_KEEP_ALIVE=-1"
# 并发数
Environment="OLLAMA_NUM_PARALLEL=4"
# 模型加载超时时间，大型模型好几百G的加载很慢，所以需要设置大
Environment="OLLAMA_LOAD_TIMEOUT=90m"
# 给GPU预留的显存空间，不能占满，要不推理没地方了
Environment="OLLAMA_GPU_OVERHEAD=536870912"
# 启用 CUDA 统一内存
Environment="GGML_CUDA_ENABLE_UNIFIED_MEMORY=1"

Then restart Ollama:

systemctl daemon-reload
systemctl restart ollama

Pull a model and run it — ollama run starts a chat:

ollama pull deepseek-r1:32b
ollama run deepseek-r1:32b

Install Docker

Dify (next time) depends on Docker, so let’s install that too:

apt install ca-certificates curl

install -m 0755 -d /etc/apt/keyrings

curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc

chmod a+r /etc/apt/keyrings/docker.asc

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

apt update

apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

If there’s time in the next post, shall we talk about Dify?