How to Migrate a Large Proxmox Virtual Machine to another Host
TL;DR: This sounds simple but like anything else, you have to plan ahead.
This task took four days of trial and error (mostly waiting for backups to compress and lengthy network file transfers).
Had I planned ahead and not rushed into it (thinking that I knew enough), I could have completed this in a day.
Learn from my mistakes, Padawan.
As usual, this is based on my personal experience. I eventually achieved the desired result but I’m learning as I go, so be sure to consult other sources that more closely match your situation.
Background
I created a virtual machine on my three-node Proxmox VE cluster using Ceph storage that runs Open Media Vault, with the sole purpose of storing automated Kopia backups from four Macs.
Configuration
- 4 Cores
- 4 GB RAM
- 32 GB OS disk
- 2 TB Storage disk
- 2.5Gb Ethernet
I subsequently purchased a 2-Bay N100 mini PC and two 22 TB rust drives that I configured in a ZFS mirror. This machine runs Proxmox VE on the host and Proxmox Backup Server running in a VM and is deliberately not part of the cluster. I created a 4 TB storage volume for PBS. With a current deduplication factor of 32:1, this seems like overkill but it works great.
I wanted to free up some of my Ceph storage pool and isolate all my backups to the storage on the mini PC that I bought for this purpose so I thought, “I’ll just move the VM.” Simple, right?
Not so much.
Anyway, if you have two or more PVE hosts that do not share storage, read on.
First Challenge: Datastore and Encryption
The most obvious solution here is to restore a PBS backup of the VM onto the same PVE host used by the PBS guest.
In order to do this, the backup host needs to share the same PBS datastore as the original host. At first, I thought it had to use the same PBS instance but that was incorrect.
Lesson learned.
Now, if you’ve never restored from a PBS backup, the user experience feels a little strange. Even though you’re backing up to a PBS instance and you can see all of your incremental backups in the UI, the restore process happens in the PVE GUI.
When attempting to restore a backup from PBS to the new host, it wouldn’t work because the encryption key on the new host is different than the one used to encrypt the backup. I could find no way to pass the proper encryption key to the restore command in the GUI.
If you know of a way to do this, please let me know in the comments or via Mastodon.
What about the CLI?
That’s what I thought, too. But the CLI assumes that the backup archive is on a file system that it can access. I could find no way to mount the PBS datastore from the CLI on the host.
Again, if you know of a way to do this, please let me know in the comments or via Mastodon.
Fine. I’ll just copy a full backup over the network to the new host.
Second Challenge: Block Storage and Network Throughput
I have a 2.5Gb Ethernet link between the cluster nodes and the backup host. This speed is fine for my use case (and my current budget). Incremental backups happen in the middle of the night so I don’t have to worry about latency. Even migration between cluster nodes is pretty quick because they share the same Ceph file system.
But this was different.
Even though I’m only using a fraction of the virtual storage I created for the Open Media Vault VM, it’s block storage. That means that 2 TB of storage has been allocated and damn it, it’s going to transfer the full 2 TB over the network no matter how much is actually used.
Rclone
I figured rclone would do the job since it’s what I use for my offsite backups. But the size of the file and speed of the network would cause the rclone process to fail about halfway through and don’t you know it: rclone doesn’t have a block-level resume function. That means when an hours-long rclone process fails, you have to start over and attempt to transfer the entire file again, which is common for large files.
I tried this twice before I figured it out.
Rsync to the rescue. Sort of.
I switched to rsync. While rsync supports the resumption of failed transfers of large files, you have to specify that by including a flag in the command.
I tried this twice before I figured it out.
That command is:
rsync -avP --partial --inplace --no-whole-file /mnt/pve/cephfs/dump/vzdump-qemu-111-2025_06_24-09_41_02.vma.zst [email protected]:/zfs-mirror/
This assumes that you have an ssh key configured for the selected user on the new host. Obviously change file name, user, IP address, and destination path for your configuration.
For reference, here’s what the flags mean:
-a(archive mode) - preserves permissions, timestamps, etc.-v(verbose) - shows file being transferred-P- this is shorthand for--partial --progress--partialkeeps partially transferred files so rsync can resume from where it left off--progressshows transfer progress
--inplace- updates the destination file in-place rather than creating a temp file (crucial for resuming large files)--no-whole-file- forces rsync to use delta transfer even on local networks (better for resume capability)
If you’re using --no-whole-file make sure that you check the size of the file in the destination directory. Because the process isn’t using a temp file before writing to the final destination, the transfer may appear to be complete even if it failed.
ls -l is your friend.
Important Note: Shut down the source VM during this process.
Backups were configured on the client machines to write to the storage volume on the VM every hour and transfering the VM was taking forever. I had to pause the source VM along with the client backup processes so that the backups wouldn’t be out of sync once the VM was resumed on the new host.
I realized this halfway through an rsync process and had to start over.
Finally
After a successful transfer of the VM, I was able to restore it on the new host.
qmrestore vzdump-qemu-111-2025_06_24-09_41_02.vma.zst 200
Again, change the file name and VMID based on your situation.
Post Mortem
What would I do differently? Lots of things.
- Dig deeper into backup restoration of PBS archives
- Learn more about the tools I plan to use. Before using rclone, I would have read more about its capabilities. Ditto for rsync. Transferring a 2 TB file over the network would have taken me only a single day if I had read more about the resume function and didn’t just assume that it would “just work.”
- Test these tools on a smaller file. It would still need to be a large file (1TB?) but I would have probably learned faster if I had gotten feedback sooner.
- Upgrade my backhaul network to 10Gb. 😉 I’m joking. Sort of. That will happen once the price of 10Gb Thunderbolt network adapters come down.
- Keep a spare 5 TB - 10 TB rust drive around. This isn’t as crazy as it sounds. Sure, I would have missed out on the opportunity to learn something new, but used drives can be found cheap on eBay or Facebook Marketplace and direct transfer via a USB adapter would have been much faster - even faster than upgrading my backhaul network to 10Gb.
Update: If you have Ceph storage, Reddit user /u/HTTP_404_NotFound suggested another option:
As one note though, you can attach the ceph storage to non-clustered hosts, and attach the disk that way via copy-pasting from the vm’s config, and still use the disk migration feature.
Example from vm-config-
virtio0: ceph-block:vm-146-disk-0,cache=writeback,iothread=1,size=32G
The first segment identifies the storage name. Next segment is the name of the block in ceph.
To add the storage, datacenter -> storage -> add rbd
Uncheck the
use proxmox managed pool. feed in monitor IPs & keyring. Boom done. Shared hyperconverged storage across clusters.
Thanks for the suggestion - I wish I would have thought of this before I started the process.
Is there anything else you’d recommend?