[{"content":"Proxmox Virtual Environment (PVE) offers powerful virtualization capabilities but tinkering with PCIe devices can lead to unexpected challenges. If like me, you didn\u0026rsquo;t realize that messing around with PCIe devices in your Proxmox PVE host would break your VMs, keep reading.\nIn my case, I added a new NVMe SSD, but this can also happen when removing PCIe devices.\nAccess the Proxmox PVE host with mouse and keyboard and log in as a privileged user.\nGathering info: PCIe bus ids It\u0026rsquo;s EXTREMELY likely that the device addresses change upon adding or removing any PCIe devices:\n[For each restart] the PCI BIOS must walk the base PCI bus (starting at bus 0), subsequent bridges, and bridged devices to search and identify other PCI buses as if it were the first time.\nEach time the PCI BIOS discovers another PCI bus after a physical configuration change is made, it increments the bus number and continues to walk the bus until all other buses are discovered.\nAs it discovers each bus and/or bridge, the PCI BIOS:\nRecords each unique bus number Associates the bus number to a bus or bridged PCI device ― Compaq, PCI Bus Numbering in a Microsoft Windows NT Environment In order to better understand how the PCIe bus ids have changed, lets run lspci to get a list of PCIe devices installed and their respective bus ids.\n1 2 3 4 5 6 7 8 $ lspci [...] 01:00.0 Non-Volatile memory controller: MAXIO Technology (Hangzhou) Ltd. NVMe SSD Controller MAP1202 (rev 01) 02:00.0 Ethernet controller: Intel Corporation Device 125c (rev 04) 03:00.0 Ethernet controller: Intel Corporation Device 125c (rev 04) 04:00.0 Ethernet controller: Intel Corporation Device 125c (rev 04) 05:00.0 Ethernet controller: Intel Corporation Device 125c (rev 04) The NVMe drive I just installed is 01:00.0. That bus id used to belong to the first Intel ethernet controller. So in this case, all NICs bus ids have increased by one.\n01:00.0 is now 02:00.0 02:00.0 is now 03:00.0 \u0026hellip;and so on Potential issues Issue 1: Network connectivity is gone If your NICs are PCIe (which is very likely!), they would be affected by this:\nThe Linux kernel assigns names to network interfaces by combining a fixed prefix and a number that increases as the kernel initializes the network devices. [\u0026hellip;] If you add another network interface card to the system, the assignment of the kernel device names is no longer fixed. Consequently, after a reboot, the kernel can name the device differently.\nWhen the consistent network device name feature is enabled, the udev device manager creates the names of devices based on different criteria.\nen for Ethernet\n[P\u0026lt;domain_number\u0026gt;]p\u0026lt;bus\u0026gt;s\u0026lt;slot\u0026gt;[f\u0026lt;function\u0026gt;][d\u0026lt;device_id\u0026gt;]\n― RedHat, RHEL 8 documentation. Chapter 1. Consistent network interface device naming Run ip link (with the ethernet cable connected) in order to get the interface name for the Proxmox Linux Bridge.\n1 2 3 4 5 6 7 8 $ ip link 1: lo: \u0026lt;LOOPBACK,UP,LOWER_UP\u0026gt; mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 5: enp5s0: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000 link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff 7: vmbr0: \u0026lt;BROADCAST,MULTICAST,UP,LOWER_UP\u0026gt; mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff Leaving the loopback (lo) and Linux Bridge (vmbr0) interfaces aside, the new interface name for the Linux Bridge is enp5s0.\nComparing against the naming criteria, we can understand that enp5s0:\nIs an ethernet connection. Its bus is 5. Its slot within the bus is 0. Can now edit /etc/network/interfaces to reflect the change, and restart the networking service.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 # /etc/network/interfaces auto lo iface lo inet loopback auto vmbr0 iface vmbr0 inet static address 192.168.1.120/24 gateway 192.168.1.1 - bridge-ports enp4s0 + bridge-ports enp5s0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 1 $ systemctl restart networking.service At this point, network connectivity should be restored. Check by pinging Google: ping -c3 google.com.\nIssue 2: Broken Passthrough PCI devices Now, one of Proxmox PVE features is the ability to do PCI Passthrough, allowing the guest VM to access host physical resources. As you have probably guessed by now, if any of your VMs use PCI passthrough, they will also be affected by the PCIe bus changes.\nThis is required for every VM that does PCI passthrough.\nRetrieve the VM id of the relevant machine, with the following command:\n1 2 3 $ qm list VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID 100 routervm running 8192 32.00 8705 Then export the VM id as an environment variable, to avoid having to remember it every time: export VM_ID=100.\nEnsure the VM is (gracefully) stopped, by running qm shutdown $VM_ID. Once stopped, check the VM config, with qm config $VM_ID.\n1 2 3 4 5 6 7 8 9 $ qm config $VM_ID boot: order=scsi0 cores: 4 cpu: kvm64,flags=+aes hostpci0: 0000:02:00 hostpci1: 0000:03:00 hostpci2: 0000:04:00 [...] hostpci0, hostpci1 and hostpci2 are all NICs that are PCI passthrough devices. As mentioned earlier, the bus ids for the NICs has increased by one, so just need to edit the VM config file located in /etc/pve/qemu-server/${VM_ID}.conf in my case:\nRemember to shutdown the machine first if you haven\u0026rsquo;t yet: qm shutdown $VM_ID\n1 2 3 4 5 6 7 8 9 10 11 12 13 # /etc/pve/qemu-server/$VM_ID.conf [...] cores: 4 cpu: kvm64,flags=+aes - hostpci0: 0000:02:00 + hostpci0: 0000:03:00 - hostpci1: 0000:03:00 + hostpci1: 0000:04:00 - hostpci2: 0000:04:00 + hostpci2: 0000:05:00 memory: 8192 [...] Now you should just be able to either start the VM from the Proxmox Web UI (remember, we just restored connectivity to it!). Or by running the following command: qm start $VM_ID.\n","date":"2023-08-03T22:10:52Z","image":"https://nullcheck.me/p/fixing-changes-in-proxmox-pcie-ids/pexels-pok-rie-1432794_hu3d03a01dcc18bc5be0e67db3d8d209a6_357243_120x120_fill_q75_box_smart1.jpg","permalink":"https://nullcheck.me/p/fixing-changes-in-proxmox-pcie-ids/","title":"Fixing Proxmox PVE issues when adding or removing PCIe devices"},{"content":"Welcome to nullcheck.me!\nThis blog is your go-to destination for all things software development. From DevOps and cloud computing to CI/CD and IaC. But that\u0026rsquo;s not all – we\u0026rsquo;ll also explore other hobbies like electronics, 3D printing, and simracing!\nI\u0026rsquo;m your host, a passionate software engineer, and together with you, we\u0026rsquo;ll embark on a journey of learning and exploration. I believe in keeping things simple and straightforward, so expect practical insights, how-to guides, and informative posts without any unnecessary fluff.\nSo, whether you\u0026rsquo;re a tech guru or just starting your coding journey, join us in this adventure of learning and exploration. Let\u0026rsquo;s dive into thisthe world of software development and discover the wonders it holds together!\nStay tuned for more, and happy coding!\nDan @ nullcheck.me\n","date":"2023-07-29T00:27:40Z","image":"https://nullcheck.me/p/hello-world/cover_hu3d03a01dcc18bc5be0e67db3d8d209a6_253548_120x120_fill_q75_box_smart1.jpg","permalink":"https://nullcheck.me/p/hello-world/","title":"Hello World!"}]