Micro XCP-ng cluster Thoughts and Suggestions. (2024)

Hello everyone
Im Building a XCP-ng Cluster with low end hardware to Run HA services.

1. I want to test out HA
2. It seems fun
3. I want to see how low i can go in terms of hardware and cost
4. Im in SA and we have rolling blackouts in need to conserve energy on my UPS

for those curious this is what im using

Celeron j1900
8GB DDR3L SODIMM
120gb patriot SSD for boot
Huntkey H702 Mini Itx Case with 60 watt psu

local-storage i want to use 3 SSDs or HDDs with XOSTOR
external-storage 2 NVME SSDs in a ZFS mirror in my NAS for NFS

Any thoughts or suggestions.

Ive always wanted to build this and it would be a great use For runX containers

I would like to know though because xostor is still in development when it comes to release will there be a path to going form the testing to the production release?

Thanks

@planedrop Hi thanks for the input. From what i can tell the j in the j1900 denotes that these are low end atom cores so im not expecting great performance.

pcie pass through is not that big a deal since its pcie 2.0 x1 and the case does not support additional pcie cards

My biggest problem is that dom0 normally uses 1.7 -2 GB of ram and these boards have a max of 8gb of ram

@brodiecyber Personally, I think that the setup is very limited in terms of performance with RAM being the greatest challenge from the beginning.

While I understand your initial idea of a low hardware and cost micro XCP-ng cluster to check out HA and learning runX/XOSTOR, you might be paying some good money for a low performance setup with little head room.

Probably not an option, but if it is about learning and testing, you could go for a used mini pc (i10500t, 64GB RAM, 1TB boot and ISO drive, 2TB VM drive) and setup everything as a nested virtualization XCP-ng cluster? That would allow for snapshots and backups to make testing and rollback a breeze. My HP ProDesk 600 G6 is at an 20W average with 16 VMs running. Going for a physical XCP-ng cluster is much more fun though .

@gskger I agree that the ram is the greatest challenge especially since dom0 takes 1.5 -2gb of ram in this setup.

I wanted to do ServeTheHomes Tiny,Mini,Micro nodes but Ive tried looking for them secondhand and the cost of a single dell micro i5-8th gen were I live. I was able to get 3-j1900 motherboards off eBay including shipping.

That's why when runX containers are more mature it would be a great platform for that. But apart form XCP-ng it works well as k3s nodes for kubernetes

@brodiecyber Ok, that makes sense.

I once run XCP-ng on a HP T620 PLUS Thin Client with some lightweight Debian VM for a test. The HPs AMD GX-420CA has a bit higher CPU Mark compared to the j1900 (1787 vs. 1148), but the single thread performance is more or less the same.

Nothing to run major workloads on, but should be ok for testing HA and general XCP-ng operations at low speed. No idea how XOSTOR will be performing on that platform, so please update us on your findings.

@gskger

Sure thing Ill update the post.

Just waiting on a the cases and boot SSDs should be here Thursday. Then need to figure out if i want to use some cheap HDDs or cheap SSDs for XOSTOR

@gskger But in the absence of runX containers

I used some Alpine Linux VMs with an overkill amount of ram at 512mb yes mb and 10gb of storage and because it supports docker and compose you can run any docker app you just need to make sure their is enough ram for the running container.

only thing is no VM icon

@gskger Thought id update you the cases are coming today

Apparently yesterday was a public holiday

Great for relaxation.
Bad for when your waiting on components.

But on another not i had an idea
I have an available PCIE 2.0 x1 slot that maxes out at the speed of SATA
So i had the idea for using an m.2 to PCIE adapter and some cheap nvme drives
so i can have a local SR on each node and an HA SR distributed on all nodes. not sure yet but this could be interesting

Hosts are online and updating before i cluster them

@brodiecyber After 2 hours of waiting its fianlly updates and cluster configured.

I will note out of 8GB of RAM. Dom0 is using 1.9GB leaving me with 6.1GB of RAM

Update the cluster is doing good i just need to get some SSDs for XOSTOR.

I have a 1cpu 1gb ram 25gb storage VM running over gigabit networking for ha storage. i don't recommend it its just for testing because disk io and bandwidth is limited.

VM runs and live migration takes about 17 Mississippi

@brodiecyber

Update Setting up cloud-init templates to provision VMs with Terraform and Ansible for application and patch management.

Maybe ill even setup a XO-ce dedicated to this cluster

@brodiecyber ok

new update

Redoing my cloud-init templates and preparing for the next version of Debian in June release 12.
Am using terraform to provision virtual machines and cloud-init and ansible to install software

I'm also using the boot disk as VM storage i don't recommend it as its a 120GB disk and i had to redo my templates as the VM they provision are to big for the little storage.

but Im running

• HA pihole with a keepalived

• WIreguard

• Terraform

• Dashboards. RSS, Archivebox

Will be adding

• dedicated XO-ce VM

• Unifi Controller

Ive still got 4GB of ram to spare on Xen- 2 and 3 so abit more room for 1 CPU/RAM vms to deploy

I still think that this a the perfect environment for RUNx containers just the efficiency especially with memory will help alot

New update

I got a asus n3150i-c motherboard and found a spare ssd

Im reusing a node for parts and am installing truenas scale as a shared storage target for NFS

Its only 100gb of SSD storage but before i redo my workloads im waiting on debian 12 to be released

so im down a compute node but am up on shared storage with dedicated node