Building a Proxmox cluster with Ceph
What is Ceph on Proxmox VE, and what does it need at a minimum?
According to the Proxmox documentation, Ceph is a distributed object store and file system that provides block, file system and object storage. Proxmox VE can install and manage Ceph directly on the hypervisor nodes. Here, “hyper-converged” means that the same servers compute (VMs and containers) and store (replicated storage), and a separate SAN or NAS is not needed.
The lower limit is hard. A hyper-converged Proxmox and Ceph cluster needs at least three, preferably identical, servers; two nodes are not enough. The route described is for “small to medium-sized” deployments.
Among the advantages of Ceph on Proxmox VE, the documentation lists:
- thin provisioning and snapshots
- self healing
- replicated, fault-tolerant data
- running on commodity hardware, without RAID controllers
- open source
What requirements does the documentation name?
How many nodes and monitors do I need?
- For high availability, a Proxmox cluster needs at least three nodes for a reliable quorum.
- For Ceph, at least three monitors are also needed for quorum.
- For operation, the documentation recommends a Ceph cluster with at least three nodes and at least 12 OSDs, evenly distributed among the nodes.
What network do I need?
A bandwidth of at least 10 Gbps, used exclusively for Ceph traffic, is recommended. Otherwise recovery traffic can disturb the latency-sensitive Corosync and, in the worst case, cost the cluster its quorum.
For high-performance setups, if unsure, the documentation recommends three separate physical networks:
- very high bandwidth (25+ Gbps) for internal Ceph traffic
- high bandwidth (10+ Gbps) for public Ceph traffic
- 1 Gbps exclusively for Corosync
For Corosync alone, the cluster chapter says a dedicated 1 Gbit NIC is enough in most situations.
What hardware do I need?
- Rule of thumb: at least one CPU core (or thread) per Ceph service.
- For OSDs at least 8 GiB of memory; the OSD daemon needs 4 GiB by default.
- For small clusters SSDs rather than HDDs, because recovery then takes less time.
- An evenly sized and evenly distributed set of disks per node is better than a mixed one.
- No RAID controller: Ceph handles redundancy itself, so the documentation advises using a host bus adapter (HBA) instead.
All figures are recommendations, not guarantees. The documentation itself says to test the setup and monitor health and performance continuously.
Why are two servers not enough?
The cluster quorum: on a network partition the cluster needs a majority of votes, otherwise it switches to read-only mode. Each node has one vote by default. For small two-node clusters the documentation describes a QDevice: an external service on an independent server that provides a third vote. That solves only the cluster quorum; the three-server requirement for Ceph is stated elsewhere.
Ceph itself: here the documentation explicitly requires at least three servers. A pool defaults to three replicas (size 3) and requires at least two replicas for I/O (min_size 2), so that no data is lost if an OSD fails.
You should not set a min_size of 1: I/O is then allowed on an object with only one replica, which can lead to data loss. The Ceph documentation adds that for high availability more than two copies should be stored, so that the cluster can continue to run in a degraded state while maintaining data safety.
How do I build the cluster?
The commands come from the Proxmox documentation (as of Proxmox VE 9.2.13, edition of 24 September 2026) and were not run on a test cluster.
How do I create the cluster?
1. Create the cluster. Install Proxmox VE on all nodes with the final hostname and IP configuration (neither can be changed after the cluster is created, and neither can the cluster name). On the first node (web interface: Datacenter → Cluster → Create Cluster):
pvecm create CLUSTERNAME
pvecm status
2. Add nodes. On each further node, with the IP address of an existing node:
pvecm add IP-ADDRESS-CLUSTER
Joining overwrites the existing configuration under /etc/pve; a joining node therefore cannot hold any guests.
How do I install and configure Ceph?
3. Install Ceph. In the web interface, open the Ceph section of a node; a wizard offers the installation there. Alternatively, on each node:
pveceph install
4. Configure Ceph. This is needed once per cluster; Proxmox VE distributes the configuration itself. On the command line, on one node:
pveceph init --network 10.10.10.0/24
The network is the example from the documentation. The wizard asks for the public network (required) and optionally a separate cluster network for OSD replication and heartbeat, plus the first monitor node.
How do monitors and OSDs come about?
5. Monitors and manager. Three monitors are recommended, and small to medium-sized clusters need no more; install the manager, according to the documentation, best on the same nodes, and for high availability more than one (only one is active at any time). On each of these nodes:
pveceph mon create
pveceph mgr create
6. Create OSDs. One OSD per physical disk. A disk that was in use before must be cleaned first, which destroys all data on it; the command applies only to such a disk:
ceph-volume lvm zap /dev/sd[X] --destroy
Then the OSD itself:
pveceph osd create /dev/sd[X]
How does the pool come about?
7. Create a pool and use it as storage. Without further options, Proxmox VE sets 128 placement groups, size 3 and min_size 2. With --add_storages the pool is also set up as a storage:
pveceph pool create POOLNAME --add_storages
Which Ceph repository applies in production?
The preconfigured Ceph source is the enterprise repository. According to the documentation it is recommended for production use and accessible if the node has a valid subscription of any level. The no-subscription repository is meant for testing and non-production use. For the cost of a subscription, see Proxmox subscription cost.
For Proxmox VE 9 the documentation lists the Ceph releases Tentacle (recommended in the enterprise repository) and Squid (estimated end of life 2026-09).
How do I shut down a Ceph cluster?
The documentation gives a fixed order for shutting down the whole cluster:
First stop all Ceph clients, mainly VMs and containers (including CephFS or RADOS gateway clients).
Check that the cluster is healthy:
ceph -sSet the
nooutflag so that no recovery starts during shutdown and power-on:ceph osd set nooutShut down the nodes without a monitor first, then the nodes with monitors.
When powering on, start the nodes with monitors first. Once all nodes are up and all Ceph services are running, the only warning left is that noout is set. Remove the flag:
ceph osd unset noout
Then the guests can be started.
Sources
- Proxmox VE Administration Guide: Deploy Hyper-Converged Ceph Cluster, retrieved on 11 November 2026
- Proxmox VE Administration Guide: Cluster Manager, retrieved on 11 November 2026
- Proxmox VE Administration Guide: Host System Administration (Ceph Repositories), retrieved on 11 November 2026
- Ceph Documentation: Architecture, retrieved on 11 November 2026