Enabling PCI Passthru with Proxmox 7.x
I am virtualizing my pfSense CE machine onto Proxmox 7.x (7.4 currently). Here are the steps I took to accomplish this. I am using the Proxmox installation and not installing proxmox over an existing Debian installation. This changes how the kernel parameters are updated. All screenshots are taken in dark mode.
Enable IOMMU
There are multiple steps to enable IOMMU. The first is to enable IOMMU in the kernel. It is disabled by default.
Add the proper MMU kernel parameters to the /etc/kernel/cmdline configuration file. The parameters that are needed are based on your CPU manufacturer. If you are passing through a GPU, it is highly recommended having hardware that can support the ‘pt’ pass-thru acceleration option.
For Intel CPUS:
If your hardware supports pass-thru acceleration,
intel_iommu=on iommu=pt
If your hardware doesn’t,
intel_iommu=on iommu=on
For AMD CPU
If your hardware supports pass-thru acceleration,
amd_iommu=on iommu=pt
If your hardware doesn’t support pass-thru acceleration,
amd_iommu=on iommu=on
Then, you need to rebuild the grub menu to incorporate your new kernel parameters. You can do this by running the command as root: proxmox-boot-tools refresh
Next, you need to make sure the proper kernel modules are loaded. Add the following modules to the /etc/modules configuration file.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqf
Now reboot the hypervisor
Verify successful IOMMU enabling by searching the boot logs with the following command: sudo dmesg|grep -e DMAR -e IOMMU
You should see a line like this for each CPU socket:
DMAR: IOMMU enabled
Assigning PCI device to the VM
If you have completed the Enable IOMMU section, now you need to assign the devices the PCI devices to the VM. If your CPU and devices do not have SR-IOV, you can only pass thru whole IOMMU groups. This means will have to share groups of PCI devices. Some PCI devices are sub-devices identified by 0000:0000:00.x. The .x of the PCI id is the sub-device. You have to assign all of the devices in 0000:0000:00 together.
Adding a PCI device to the VM
Next, you need to add the PCI device to the VM. You can add it via the WebGUIThe through the following menu:
Then, add select the whole device ID from the list. The device selector may add the PCI device with the subdevice ID. You will have to remove the subdevice from the ID.
Take note of the PCI id in the virtualized hardware list. You will need it in the next step. With this particular hardware, it is 0000:02:00
Blacklisting Pass-Thru PCI Ids
In order to allow non-SRV-IO hardware to pass through to the VM, you have to block the host OS from loading drivers and taking control of the hardware. If you don’t do with, the VM will fail to start. In the previous step, you should have taken note of the PCI id that you want to pass thru. Now you need to find the string to JUST blacklist the PCI id and not the whole driver in case other hardware requires it. To get the modalias to blacklist the PCI id, you need to use the /sys filesystem. The ‘file’ you need to query is /sys/bus/pci/devices/{pci_device_id.subdevice}/modalias. In our use case, it would be, cat /sys/bus/pci/devices/0000\:02\:00.0/modalias. The output should look something similar to this: pci:v000014E4d0000165Fsv00001028sd000005E5bc02sc00i00
You have to do this for all PCI devices and subdevices in the same IOMMU.
Then, you need to add the PCI device modalias id to the /etc/modprobe.d directory. I recommend creating a file that describes your use case. My blacklist file is /etc/modprobe.d/pci-passthru-blacklist.
In the text file, you’ll need to add the blacklist directive and then the PCI device modalias id. If there are multiple PCI device modalias ids, add each one per line. Here is an example of mine.
# Onboard NIC cards for pfsense VM
blacklist pci:v000014E4d0000165Fsv00001028sd000005E5bc02sc00i00
blacklist pci:v000014E4d0000165Fsv00001028sd000005E5bc02sc00i00
Now reboot to apply these settings.
This should complete all of the steps to enable PCI(e) pass-thru for Proxmox.
Leave a Reply