Hello and I hope this finds you well. Today I wanted to make a guide to help with a problem I am guessing most have likely solved at this point. I, however, chose to wait to upgrade my Proxmox cluster until after the last minute because it felt a bit intimidating. I finally did it though using the official guide, a walkthrough that missed a couple things at the end for me, a pretty decent youtube video, and a touch of AI. In the end, over the course of about 5 hours I was able to upgrade my 4 node cluster to version 9 with no downtime, so I am calling this a success. Anyway this is to help anyone who might have put off their upgrade the way I did.
Upgrading a multi-node Proxmox Virtual Environment (PVE) cluster with an integrated Ceph storage backend requires a strict order of operations. This guide covers upgrading Ceph from Reef to Squid, transitioning package files to the modern DEB822 format, and mitigating the common aes256k cipher security errors.
Proxmox upgrade from 8 to 9 official guide
Proxmox VE 8 to 9 Upgrade Guide: Step-by-Step - Paolo Valsecchi web guide
⚠️ Prerequisites & Precautions
- Backup Everything: Ensure all VMs and containers have verified, external backups.
- Maintain Quorum: Upgrade nodes one at a time. Never reboot or stop services on multiple nodes simultaneously.
- No-Subscription Focus: This guide is tailored for standard environments utilizing free no-subscription repositories.
- Ensure you are on the latest version of proxmox 8.
- Before working on a node ensure to migrate the VMs off of that node to ensure uptime and no issues while updating.
🗒 Phase 1: Ceph Upgrade (Reef to Squid)
Before upgrading the underlying operating system to Proxmox VE 9 (Debian Trixie), you must first upgrade your Ceph storage layer to Ceph Squid while still running Proxmox VE 8.
Perform these steps on Node 1, then repeat sequentially for Node 2, Node 3, and Node 4.
Step 1: Set Storage Protection Flags
Prevent Ceph from attempting to rebalance or heal data pools while services are restarting:
ceph osd set nooutStep 2: Switch the Ceph Repository URL
Update the legacy repository file configuration to target the newer version release:
sed -i 's/reef/squid/' /etc/apt/sources.list.d/ceph.listStep 3: Fetch and Install Packages
Refresh your index database maps and pull down the modern binaries:
apt update && apt upgrade -yVerify the installation by validating the running binary metadata details:
ceph --versionStep 4: Restart Core Monitors & Managers
Gracefully reload your monitor cluster daemons to force them to run the newer release code:
systemctl restart ceph-mon.targetVerify the monitor health ring status before proceeding:
ceph -s
ceph mon dump | grep min_mon_releaseOnce the monitors register cleanly, restart the primary manager daemons:
systemctl restart ceph-mgr.target
ceph -s🗒 Phase 2: Proxmox VE 8 to 9 Distribution Upgrade
Once all nodes are happily running Ceph Squid, begin upgrading the operating system layer. Perform these steps one node at a time.
Step 1: Put the Node into Maintenance Mode
Gracefully clear running tasks and notify the cluster high-availability tracker:
ha-manager crm-command node-maintenance enable PVENX(Replace PVENX with your specific node hostname, e.g., PVEN1)
Step 2: Protect and Check Pre-flight Readiness
Set your storage protection flags again to guard your active disks during the OS transition:
ceph osd set nooutRun the official Proxmox configuration analyzer utility to catch layout issues:
pve8to9 --fullNote: Make sure to correct any errors, warnings were ok for me. If using standard UEFI boot layouts, clean out legacy boot blocks to avoid conflicts:
apt remove systemd-boot -yStep 3: Modernize Repositories to DEB822 Standard
Proxmox 9 drops support for legacy .list configuration rows in favor of structured .sources blocks. To prevent critical meta-package removal safety prompts (proxmox-ve), you must clean out old system configurations and generate fresh files targeting Debian Trixie.
First, scrub historical baseline rows out of your directory path maps:
# Clear out traditional file locations completely
rm -f /etc/apt/sources.list
rm -f /etc/apt/sources.list.d/ceph.listNext, deploy your clean, modern .sources components:
- Create the Main Proxmox No-Subscription Source:
cat > /etc/apt/sources.list.d/proxmox.sources << EOF
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF- Create the Free Ceph Squid Source:
cat > /etc/apt/sources.list.d/ceph.sources << EOF
Types: deb
URIs: http://download.proxmox.com/debian/ceph-squid
Suites: trixie
Components: no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF- Create the Core Debian Trixie Base OS Source:
cat > /etc/apt/sources.list.d/debian.sources << EOF
Types: deb
URIs: http://ftp.debian.org/debian
Suites: trixie trixie-updates
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb
URIs: http://security.debian.org/debian-security
Suites: trixie-security
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
EOF- Disable Ghost Enterprise Files:
If stray enterprise trackers are lingering on your file grid, rename them out of sight to block401 Unauthorizederrors during synchronization:
mv /etc/apt/sources.list.d/pve-enterprise.sources /etc/apt/sources.list.d/pve-enterprise.sources.bak 2>/dev/nullStep 4: Execute the Distribution Upgrade
Clear the internal cache storage folder completely and kick off your core upgrade procedure:
rm -rf /var/lib/apt/lists/*
apt update
apt dist-upgrade -yThere will be an apt-listchanges: News page that comes up, just hit q to quit, or read it, up to you. During configuration file prompts, press N (keep local version) unless you have highly specific manual custom system configuration parameters to re-inject.
Step 5: Reboot and Verify Cluster State
Bring the host back up on the modern Proxmox 9 system kernel:
rebootOnce the machine initializes and rejoins your network console, run your operational validation checks:
pveversion
pvecm statusStep 6: Bring the Node Back Online
Take the node out of standby and resume storage traffic:
ha-manager crm-command node-maintenance disable PVENX
ceph osd unset noout
ceph -s🗒 Phase 3: Mitigating Ceph Security Key Errors (aes256k)
After completing your upgrades, Ceph Squid will trigger critical dashboard security alerts (AUTH_INSECURE_CLIENT_KEY_TYPE). This is because it requires legacy authentication tokens to be rotated over to stronger aes256k ciphers.
Run these final repair routines from your primary cluster node shell (Node 1):
Step 1: Rotate Cluster Service Keys
Rotate your underlying daemon and system connection keyrings securely:
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-cluster-keys --applyStep 2: Stage the New Admin Key
Stage an upgraded secret for your global administrator account. This runs concurrently with the old key so active connections do not get terminated:
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --rotate-admin-key --applyStep 3: Refresh Active Virtual Machine Client Sessions
Running VMs hold onto the old session keys in memory. To move them to the modern cipher, you must force them to reconnect to storage.
- Zero Downtime Method: Go to your cluster GUI and live-migrate your running VMs across different hardware nodes. The live handoff drops old sessions and picks up the fresh key format seamlessly.
- Alternative Method: Perform a clean stop-and-start reboot cycle on your hosted guest machines.
Step 4: Confirm Key Migrations & Lock Down Ciphers
Once all active clients are holding the updated credentials, commit the tracking changes permanently:
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --confirm-all-clients-refreshed --applyFinally, restrict your system monitors from acknowledging weak encryption attempts moving forward:
/usr/share/pve-manager/migrations/pve-cephx-rotate-service-keys --apply --restrict-ciphersNote: The remaining transient tracking alerts (AUTH_INSECURE_ROTATING_SERVICE_KEY_TYPE) will clear off your main dashboard metrics view entirely within a few hours as remaining temporary cache tickets reach their expiration limits.