Infrastructure/VM cluster

Open Food Facts uses a Proxmox based cluster to host different virtual machines (VM) on OVH provided servers.

The cluster is made of 4 physical machines ("nodes" or "hosts" in Proxmox jargon):

• ovh1 and ovh2 are computation oriented nodes: 24 cores, 256 GB RAM, 1TB nvme SSD
• ovh3 and ovh4 are storage oriented nodes: 32GB RAM, 6x12 TB HDD + 512GB NVMe cache

ovh1 and ovh3 are in Roubaix datacenter, ovh2 and ovh4 in Strasbourg.

At initial setup (january 2021), v6.3 of proxmox has been installed (based on Debian 10 "buster").

Proxmox GUI is available on any of the cluster nodes on port 8006.

At the networking level, a vRack links the cluster nodes with a 3Gbps private network used to access data on storage servers and replicate data between nodes.

All storage is managed using ZFS which provides:

• volume management (like lvm)
• redundancy (like mdadm)
• encryption (like luks)
• compression
• snapshots
• quota

Snapshots allow efficient synchronization between remote storage, and is used extensively by Proxmox to replicate data across the nodes. Snapshots simplify backups and allow rollbacks.

Proxmox allows full virtualization (VM, using QEMU) and containerization (CT using LXC). For performance reason, LXC CTs are prefered over QEMU VMs.

All resources are shared and dynamically allocated, thus can be reallocated at any time without reboot.

Usage guidelines (to be completed)

Here is a few guides to follow for all new virtual machines:

1. MUST: no direct root access on the nodes, even with SSH key.
2. MUST: sudoers (root access using sudo) limited to SSH key based authentication
3. SHOULD: use SSH keys published on Github: giving access to a server is then simple and secure:

   curl https://github.com/CharlesNepote.keys | tee -a ~/.ssh/authorized_keys

4. SHOULD: take care of production resources: use "nice" / "ionice" for scripts manually launched. Stéphane's tip: just use

   nice ./mycommand whatever arguments

   (nice default to lower the priority). CPU and I/O priorities can be set if needed at the virtualization level.