At my workplace, we do a lot of software and SCADA type development which necessitates setting up virtual machines for testing projects. Then there’s the needs of production workloads. So a private cloud was needed.
At my home, I have an aging Intel Atom D525-based webserver that has served me well for the past 5½ years, but is now starting to run out of puff. In addition, it’s a base-load energy consumer, running 24×7. In addition to having something that can run VMs, it’d be great to have some redundancy and be semi-independent from the mains grid.
The aim of this project is to come up with a hardware and software stack that can be scaled up to meet the needs of small/medium business in a cost-effective manner and as an engineering challenge, be green at the same time.
- Storage and compute nodes:
- Motherboard: Supermicro A1SAi-2750F
- CPU: 8-core Intel Atom C2750 at 2.4GHz
- SSD: Samsung 850EVO 120GB
- OS: Gentoo Linux
- Storage nodes (3×):
- RAM: 16GB ECC DDR3
- HDD: HGST HTS541010A9 1TB
- Storage software: Ceph 10.2
- Compute nodes (2×):
- RAM: 32GB ECC DDR3
- Virtualisation software: KVM managed through OpenNebula
- Network fabric: Linksys LGS326-AU, modified to run off 12V. (I am looking to upgrade this to a Netgear GS748T as I have filled up the Linksys’ ports. I have the relevant parts to do this.)
- Solar input: 3×120W 12V panels
- Battery bank: 2×105Ah 12V AGM
- Monitoring node:
- Technologic Systems TS-7670v2 revision D
- CPU: Freescale^WNXP i.MX286 at 454MHz ARMv5 (single-core)
- RAM: 128MB
- SSD: On-board 2GB eMMC
- OS: Debian Stretch for now, but am compiling a port of Gentoo/musl which I’ll release in due course.
What is this cluster doing?
Since late June 2016, it has been in production running a number of websites:
- My blog
- Brisbane Area WICEN Group‘s OwnCloud site and other miscellaneous sites (e.g. their calendar and some of their mailing lists)
- My photo gallery
- #Hackaday.io Spambot Hunter Project (which is helping keep the spam down on this site)
…among lots of ad-hoc projects.
It also has been used on a few occasions to run test instances at my workplace, in one case, providing about 5 virtual machines to try out Kubernetes, and in another, spinning up a test instance of WideSky, because our usual hosting provider (Vultr) was full (specifically their Sydney data centre). For that reason, this equipment appears on my tax records.
In March 2018, I officially decommissioned the old Intel Atom D525 server that had been running much of my infrastructure to date, doing a physical-to-virtual migration of the old server onto a VM. The old box was re-configured to just power on at 9PM so that its cron jobs could do a back-up of the real instances, then shut down. This machine has since been reloaded, still performs the same function but now the OS is stripped down to the bare essentials. (Thank-you Gentoo/musl.)
I may yet convert it to run off 12V with the cluster too as the PSU fan is making noises, we’ll see.
Can it run entirely off solar?
In good weather, yes. If there’s good sunlight during the day. An extra battery and another panel would help here, and I’m considering doing exactly that.
For now though, it runs both mains and solar, which already has reduced our power bill.
If doing it again, what would I do different?
- The switch: the Linksys can’t do LACP with more than 4 LAGs, whereas the Netgear one can do the required number of LAGs.
- At the time that Supermicro board was one of the best options available, but buying DDR3 ECC SO-DIMMs is a pain. There are newer ones now that aside from having more cores (up to 16!), take full-size DDR4 ECC DIMMs which are easier to come by.
- The rack could be a bit taller (not a show stopper though)
- Getting ATX DC-DC PSUs that can tolerate up to 16V natively. (I think mini-box were out-of-stock of the other models, hence I took the ones I have and used LDOs to hack around the problem.)
See the Solar-powered Cloud Computing category.