Force remove Proxmox Virtual Machine or Container
This was driving me crazy but ended up being ridiculously simple.
For background, here’s a high-level overview of my setup:
- 3 Proxmox nodes
- Ceph storage
- Each node is set as a Ceph Manager, Monitor and Metadata Server
The Problem
For a few weeks I’ve had issues destroying VM’s and containers (I mostly use containers, when I can).
I initiated the destruction process using the Proxmox UI: VMID > More > Remove
The process is logged using the typical percentage complete format, then produces the following error:
TASK ERROR: rbd error: rbd: error opening image vm-115-disk-0: (2) No such file or directory
The LXC remains present in the UI. Retrying the above does nothing except produce the same error.
I would then follow up with qm destroy [vmid] from the cli which produces the following:
Configuration file 'nodes/larry/qemu-server/100.conf' does not exist
…and still, the CT remains in the UI. The ID is never released back to Proxmox and Proxmox Backup Server continues to back up…nothing, essentially.
Given the text of the error, I figured it was an issue with Ceph. (RBD stands for RADOS which is a key component of Ceph.)
All my RBD’s checked out fine: the CT image was gone as were any secondary storage volumes.
Web searches for this problem turned up nothing and Perplexity queries seemed sketchy, at best.
I decided to just take a chance on an idea: the only artifact of the dead LXC was the configuration file found in /etc/pve/lxc . So I simply issued the following command to rename (but not delete, just in case) the offending LXC:
mv /etc/pve/lxc/100.conf /etc/pve/lxc/100.conf.old
…then restarted Proxmox cluster services:
systemctl restart pve-cluster
The LXC just disappeared from the Proxmox UI.
After that, I deleted the renamed configuration file and all was well.
Now, it should be noted that, in a cluster configuration when shared Ceph storage is used, the LXC should exist in:
nodes/NODE-NAME/lxc/LXCID.conf
But that’s a different day of troubleshooting.