Utilizing PCI Passthrough (VT-d) on Proxmox VE

Print
You are here:

Overview

Proxmox Virtual Environment (Proxmox VE) is an open-source software server for virtualization management.

On Proxmox VE, hardware devices are seen as “virtual” hardware and can be used by all the Virtual Machines (VM) and the host in a system. Proxmox provides “PCI Passthrough” which enables direct access via the PCI Express bus to physical devices such as a network ports, WiFi cards, storage devices, GPU, and more. The advantage of PCI Passthrough is lower latency and higher performance. However, once a device is passed through to a VM, it can’t be used by other VMs or the host in the system.

PCI Passthrough will work with both AMI and coreboot BIOS, but certain devices like thermal sensors and the HD audio controller cannot be passed with coreboot BIOS.

Please see the chart at the bottom of this article for VT-d/PCI Passthrough compatibility. (FW2B and FW4B are not supported)

*Important Note: IOMMU must be enabled for PCI Passthrough to work. If you are using a fresh install of Proxmox VE version 7.2 or above, PCI passthrough will be enabled by default. If you update the OS, IOMMU will most likely become disabled, meaning you must configure and enable IOMMU via the command line. Instructions to configure PCI Passthrough/IOMMU can be found at: https://pve.proxmox.com/wiki/Pci_passthrough.

*Additional Note: All of the instructions regarding changing machine type and passing through PCI devices is under the assumption you have already properly installed a VM on Proxmox VE. If you need instructions on installing VMs on Proxmox VE, please check out our articles for installing a pfSense® CE VM or installing an OPNsense VM.

Enabling IOMMU

  • Access the Proxmox VE console via an external monitor or through the Shell on the web management interface
  • Type and enter: nano /etc/default/grub
  • Add intel_iommu=on to GRUB_CMDLINE_LINUX_DEFAULT=”quiet” (See the screenshot below)
  • Write Out the settings and Exit
  • Run the command update-grub to finalize changes
  • Reboot your Vault

Follow the official Proxmox instructions for more in-depth information (https://pve.proxmox.com/wiki/Pci_passthrough)

Make sure to save these settings!

 

Choosing q35 Machine Type for VMs

Selecting the q35 machine type allows for proper utilization of passed-through devices. Compared to the default machine type of i440fx, q35 is more closely related to how a real system’s PCIe topology looks like, and has better support for native PCIe devices.

The q35 machine type can be selected via the Hardware tab of any given VM you have created on Proxmox VE.

Selecting q35 Machine Type

OVMF (UEFI) vs SeaBIOS

Linux, Debian, and Windows

For Linux, Debian, and modern Windows Operating Systems, it is best to initially install the VM with the OVMF (UEFI) BIOS. Keep in mind there is not a single configuration that will work for absolutely every single PCI device, but selecting OVMF (UEFI) alongside the q35 machine type has the best success rate for PCI passthrough with Protectli Vaults.

Some OS’s may not support UEFI, like Sophos XG.

FreeBSD (OPNsense and pfSense® CE)

For FreeBSD based OS’s like OPNsense and pfSense® CE, using the default SeaBIOS with the q35 machine type will work with passing through network ports (NICs) and WiFi adapters.

Selecting and Passing Through PCI Devices

On the Hardware tab of any VM, click Add>PCI Device

Adding PCI Device to a VM

 

From here, you will see a list of all PCI devices found within the Vault. The following example showcases PCI devices on an FW6E with AMI BIOS.

Selecting PCI Device

 

Simply choose the device you wish to add, and press the Add button. From here, the VM should be able to utilize the device the next time you start it.

Potential Uses

There is a plethora of possibilities with how you can utilize PCI devices with different OS’s, so it’s impossible to cover everything. However, we will go over a few interesting/useful options that can get you started.

Ethernet Ports/NICs

You can passthrough individual NICs to improve performance and decrease latency opposed to the default Linux Bridges. This can be done with both OVMF(UEFI) and SeaBIOS.

You should not passthrough the same NIC that is being used as Proxmox VE’s management port, as this will cause a loss of connection to the management web interface, or will cause crashing.

Simply choose your desired interface connection as a PCI device, and it should immediately work for data throughput the next time you launch the VM.

If you are using OPNsense or pfSense® CE, you will need to assign these interfaces on the OS’s console.

In the following example you can see the VP2410’s available interfaces.

Adding NICs as PCI Device

USB Port Passthrough

By selecting the USB Controller as a PCI Device, you can connect USB storage devices, keyboards/mice, or whatever USB device you may want to use. In the following example we have a Protectli USB Drive plugged into the Vault, and it is accessible on an Ubuntu 22.04 VM.

USB Passthrough

Thermal Subsystem Passthrough

You can passthrough the thermal subsystem to a VM to detect internal running temperatures. This will only be possible if your Vault is flashed with the AMI BIOS. The VP2410 does not have the thermal subsystem available for passthrough.

In the following example, we have passed through the thermal subsystem to detect temperatures on a Fedora VM. We used the xsensors package to display the temperatures.

Temperatures on Fedora Using xsensors

GPU Passthrough

You can passthrough the integrated graphics controller to a VM, allowing for higher video performance and the ability to use an external monitor via the HDMI port.

This tends to work better with Linux distributions opposed to Windows.

When adding the graphics processor as a PCI device, you will need to check the Primary GPU and All Functions box.

Checking the Advanced box and choosing PCI-Express may help if you are experiencing any trouble with passing through the GPU.

Selecting Primary GPU and All Functions

 

In the following example we have passed through the Intel UHD Graphics 620 on an FW6E, allowing for a monitor to display the Ubuntu VM’s output via the HDMI port. Use this alongside USB passthrough so you can utilize a keyboard and mouse!

Using an External Monitor with an Ubuntu VM to play Half-Life

WiFi Modules

You can passthrough WiFi Modules to use with your virtual machines.

If your Vault has an mPCIe or M.2 WiFi kit, you can choose them as PCI devices to passthrough. If you have the WAP01K WiFi kit, you need to passthrough your USB controller to utilize it.

Further information regarding OS compatibility is outlined in our WiFi on the Vault knowledge base article.

When adding the mPCIe or M.2 WiFi kit as a PCI device, expand the Advanced options and enable the ROM-Bar and PCI-Express options

mPCIe and M.2 WiFi kit settings

 

The following example shows that the WAP01K WiFi kit can be added as a Wireless Interface on a pfSense® CE virtual machine. This can also be done on OPNsense.

Adding the WAP01k WiFi kit to pfSense® CE

 

Conclusion

There are many different possibilities with how you can utilize PCI Passthrough, so we encourage you to experiment with it. There will occasionally be some troubleshooting required when getting PCI devices to work with different OS’s. This typically revolves around changing the advanced settings when adding the PCI device, or sometimes installing drivers.

If you ever come across any issues, please reach out to us at support@protectli.com, or open up a support ticket.

Table of Contents