Ransomware-safe offsite backups with Proxmox Backup Server

We know that cyberactors not only exfiltrates and encrypt your data, they also target the backup. If they can kill your ability to restore your services then they will be in a better position to hold your money ransom and in the end game giving you no other option than to pay up. So basically, if your offsite copy can be reached with credentials from your own environment, it goes too.

This post shows a simple Proxmox Backup Server (PBS) setup where two independent sites keep copies of each other’s backups, and neither can delete anything on the other. It works for a company and a service provider, two companies, or two homelabbers swapping disk space.

• If the primary is compromised, the data is safe at the offsite.
• If the offsite is compromised, the data is safe at the primary.

Everything is done in the PBS GUI and utilizes only read tokens to get the data. All pruning and garbage collection is done locally on the data the local PBS has in its datastore.

Note; this article focuses on the how to copy and protect the data. Everything else you should have done like SSL, firewall, VPN, MFA, encryption etc I just assume you have control of.

Update 23.10.26, I added a new section for network below just to add another layer to the securitty, also protecting against API vulnerabilities. Thanks to @Johannes S from the Proxmox Communinty Forum suggesting this as an improvement.

1. Pull, don’t push

#

With push, the primary holds credentials that can write to the offsite, and usually delete as well. Anyone who owns the primary owns the offsite copy too.

So we use pull instead. Each side gets a read-only token for the other, and only ever fetches.

2. Keep the sites separate

#

The tokens only help if the sites are truly separate:

• Different admins, ideally different organisations.
• No shared passwords, anywhere.
• No shared password vaults, even if you have separate passwords, please dont save them in same place!
• Only use tokens, for all the automation, use only tokens.
• No AD, LDAP or SSO on the offsite. Local accounts only.
• 2FA on all interactive accounts.
• Separate hardware and network. Only port 8007 open between the sites, from known addresses.
• No SSH trust or shared automation accounts.

These things are very important. The offsite PBS needs to be in a jail that is not approachable execept if the wind is blowing south while the wolves are houling and its raining angels in hell.

3. Secure the network design

#

We need to start with a solid network design. The whole point is that the Proxmox VE (PVE) source side where the primary PBS is placed, needs to only communicate with the remote PBS on restricted ports. The design should be on of the following two states:

• Restore and testing, when doing testing and setup the PBS <-> PBS network communication should be only port TCP 8007 on each side. Nothing else. So the source can connect to the API on the offsite on port 8007, and the pulling PBS offsite can connect to the primary on port 8007. When the network is set up like this then the primary site can pull data from offsite to do restore testing.
• Normal production, as a security measure over and beyond just using pull tokens I propose to close the access from Primary PBS -> Offsite PBS on port 8007. This means only offsite can connect back to primary to pull the data but not the otherway. Why do this? Well its the scenario where there is a zero day or exploitable vulnerability not patched on the PBS API. This could in theory be used by an attacker to do privilege escalation to the offsite PBS. Closing this access down during normal operations kills this possiblity.

In both the above scenarioes we limit the communication between the PBS servers to only be allowed to speak to eachother. The primary PVEs can only speak to the local PBS, the primary PBS can only speak to the offsite PBS - this means, the PVE (and any other devices on the primary site) cannot speak directly with the offsite PBS, even on port 8007. Look at the following diagram:

I host PBS on a remote PVE cluster so I create a firewall rule using PVE firewall rules for this. Just go to the offsite PVE hosting PBS, enter the firewall for the VM and add a rule similar to the below. The example here blocks from the source, but you can of course set no source so you block from all. I have multiple sources syncing to my offsite PBS so I need to be more granular.

The network between primary and offsite locations could be done over the Internet with static IPs opening only the 8007 ports, or you could use a VPN (I suggest Wireguard, but IPsec and others could also be used). This part I will leave to you to find out, but make sure the network layer between the sites is set up in a secure way so you do not expose the API to the general internet.

4. Encrypt, and store the key elsewhere

#

Use client-side encryption in PVE, so the offsite only sees encrypted chunks. Export the key as a paper key and keep it outside both environments. This key is vital if you need to restore, so keep it safe!

How to set up PBS encryptionn

5. Naming and certificates

#

Every object below lives on one of the two servers. <site> is a short name for the site being protected, for example acme. Two rules of thumb make the names easy to read:

• A token is named after who holds it. sync-reader@pbs!offsite lives on the primary, but it is the offsite that holds and uses it.
• A remote is named after where it points. The remote offsite on the primary points to the offsite server.

On the primary:

On the offsite:

So each server holds exactly one token for the other one, and both tokens are read-only.

Certificates: optional, but recommended. Give both servers a valid certificate, for example via ACME. Then the remote’s Fingerprint field stays empty and you connect with the FQDN from the certificate. With a self-signed certificate, use the fingerprint from Dashboard → Show Fingerprint.

6. Read-only user and token

#

Repeat this process on both PBS servers, using the naming standard above (or adapt it to your liking). This will be used in in sections 6 and 8.

1. Configuration → Access Control → User Management → Add. Realm pbs, long random password that you don’t store - just max the 64 charactes with random data.
2. API Token → Add. Copy the secret, it is shown once.
3. Permissions → Add → User Permission. Path /datastore/<datastore>, role DatastoreReader.
4. Permissions → Add → API Token Permission. Same path and role.

Both user and token need the permission. Never grant it on /datastore as a whole. DatastoreReader can read backups, but not create, prune or delete them. Especially important if the remote PBS will store data for more than one “customer”.

My token name is the wrong naming standard… sorry about that!

7. Primary: read access for the offsite

#

1. Create sync-reader@pbs with token offsite and DatastoreReader on the primary datastore.
2. Send the secret (and fingerprint, if needed) to the offsite admin over a secure channel.

No need for a new image, it would look the same as 5, just the sync-reader user and token.

8. Offsite: datastore and pull job

#

Datastore: create <site>-offsite. Create sync-owner@pbs and give it DatastoreBackup on that datastore.

Configuration → Remotes → Add:

Datastore → <site>-offsite → Sync Jobs → Add Pull Sync Job:

Important: Never check Remove Vanished. Deletions on the primary must not reach the offsite. If this is on an atacker could just delete the local copies, and when the next pull sync job runs it will delete also the data on the offsite.

Prune & GC Jobs: set retention and GC here. Retention is set separately for the offiste copy, make this as long as you think you need. I suggest a minimum of 14 days of daily copies. A Verify Job is a good idea too.

9. Offsite: read access for the primary

#

This part is needed when the primary wants to do a restore. I suggest setting it up to test and verify. What good is a backup without the ability to restore?

1. Create <site>-restore@pbs with token primary and DatastoreReader on /datastore/<site>-offsite.
2. Send the secret to the primary admin over a secure channel.

10. Primary: pull back and test

#

By having a local namespace you can use the same datastore that you have to pull back to. This also ensures minimal data transfer - but you can of course also create a separate datastore locally if you want. If you just use a namespace only chunks missing locally are transferred.

Content → Add Namespace: from-offsite.

Configuration → Remotes → Add: Remote ID offsite, the offsite’s FQDN, Auth ID <site>-restore@pbs!primary, token secret as password.

Sync Jobs → Add Pull Sync Job:

Try to restore the VM in PVE. This also proves the encryption key is available. For a real recovery, change the group filter and run the job manually.

Now look back on section 3 that we will block this access during normal operations. So enable that block when the testing is done. If you should do a restore test or real restore in the future you need to unblock this firewall port to enable the restore. You do not have to do this, but think about it and make your own decisions.

11. What an attacker can do

#

Only both sites compromised at once beats this. That’s what section 2 is for.

12. Worth remembering

#

• Retention. Attackers can wait weeks. Keep enough history to reach a clean point.
• Monitor from the offsite. A failing pull job is an early warning, and a compromised primary can’t be trusted to alert. See my post on monitoring PBS with Uptime Kuma.
• Choose the other admin carefully. They are part of your security.

13. Potential next steps?

#

If this is no enough I have only two things to suggest.

First would be to create an immutable snapshot (WORM - Write Once, Read Many) of your datastore. Some NAS systems can do this, create a snapshot that is set immutable and basically makes it undeletable by the storage admin. This functionality needs to be used with caution though. Do this wrong and you fill up your disks without the ability to delete, it really is immutable! I know that NetApp, Dell, Synology, Pure and QNAP (and probably many more) can do this. Also you can get this functionality on public cloud providers if you choose to build your offsite solution there.

Second option is the obvious one, use tape. Saving a copy of your backup on tape is never wrong and will protect your data from any network based attack (as long as you take the tapes out of the drive!).

14. Postludium

#

What do you think? Is this a a good and secure approach to protect offsite copies of your backup data? Please suggest improvements if you have any, or just want to share your voice.

15. AI disclaimer

#

I have used AI to help me with drawings and also to read QA and generate some text. All the suggestions in this article I have set up and tested in my lab so this is actually a working design that I use in prodcution and not some AI slop solution.