BlogSystems
SystemsProxmox VE and ZFS: Cluster Notes from a Two-Node Lab
How a two-node Proxmox cluster can use local ZFS for guests, a TrueNAS backend for shared data, and replication that is honest about two-node limits.
- Published
- Reading time
- 9 min read
- Category
- Systems
- proxmox
- zfs
- clustering
- truenas
On this page
Two Proxmox nodes are enough to practice cluster operations. They are not enough to pretend you have automatic, lossless failover for every guest. This lab treats pve-01 as the primary hypervisor, pve-02 as a full peer that can take scheduled replicas, and TrueNAS as the shared capacity layer — not as a substitute for a third vote.
What the cluster is for
I use the cluster for a single inventory, shared API tokens, and live migration when both nodes are healthy. I do not use it as an excuse to skip backups. With only two nodes, a failed disk shelf or a split brain is still a human event.
The practical layout:
- Each node has a local ZFS mirror (
rpool) on a pair of 1.6 TB NVMe drives for guest disks that want low latency. - Shared datasets and backup targets live on TrueNAS, reached on VLAN 30 over 10 GbE.
- Corosync and management ride VLAN 20 / VLAN 10 respectively, never the storage path.
Creating the cluster and the extra vote
On the first node:
pvecm create lab-cluster
pvecm status
Join the second node with pvecm add 10.10.20.11. A two-node cluster still needs a reliable quorum device so a single reboot does not fence the remaining node. I run a qdevice on the TrueNAS management address:
# On both nodes, after installing corosync-qdevice
pvecm qdevice setup 10.10.10.10
pvecm status
If pvecm status does not show a qdevice and three votes, I do not start moving guests around.
Local ZFS for guests
The NVMe mirror is created at install time, but I still set a few properties so snapshots and replication behave:
zpool status rpool
zfs set compression=lz4 rpool
zfs set atime=off rpool
zfs set acltype=posixacl rpool/data
zfs list -o name,used,avail,refer,compression,compressratio rpool
Guest volumes live under rpool/data. I avoid putting large media or backup trees here; those belong on the NAS. A local rpool is for disks I might want to snapshot and replicate between nodes on a schedule.
Replication, not a fantasy of HA
Proxmox can replicate ZFS-backed guests from pve-01 to pve-02 on a schedule. That is a recovery point, not a heartbeat. I keep the job conservative — hourly for the few guests that matter, daily for the rest — and I test a failover during a maintenance window rather than discovering the replica is stale during an outage.
# On pve-01, after the replication job exists in the UI
pvesr status
pvesr list
When pve-02 is in standby, I still want replication to succeed. A node can be out of the guest scheduler and still accept ZFS receives. If a job fails, it is almost always a dataset rename, a full rpool, or a broken storage VLAN — not Corosync itself.
TrueNAS as capacity, not as the guest disk for everything
NFS and iSCSI from TrueNAS are reserved for shared data and for guests that are explicitly designed to boot or store data off-box. A typical NFS storage entry on both nodes:
pvesm add nfs tank-nfs \
--server 10.10.30.10 \
--export /mnt/tank/proxmox \
--content images,iso,backup \
--options vers=4.2
pvesm status
I keep iSCSI for the odd guest that wants a raw shared block device. I do not put the Proxmox root or Corosync on NFS.
A short recovery drill
Once a quarter I pick a non-critical guest, confirm the latest replica or backup, and start it on the other node:
qm list
# After confirming the replica date on pve-02:
qm start 121
journalctl -u pvedaemon -n 50 --no-pager
If that start is not boring, the cluster notes get updated the same day. The goal is a two-node lab that tells the truth: shared management, honest replication, and a NAS that holds data, not hope.