Well, figured I’d document this project here in case anyone was interested in doing this for personal amusement or for their workplace.
The list I’ve just chucked up is not a complete list, nor is it a prescribed list of exactly what’s needed, but rather is what I’ve either acquired, or will acquire.
The basic architecture is as follows:
- The cluster is built up on discrete nodes which are based around a very similar hardware stack and are tweaked for their function.
- Persistent data storage is handled by the storage nodes using the Ceph object storage system. This requires that a majority quorum is maintained, and so a minimum of 3 storage nodes are required.
- Virtual machines run on the compute nodes.
- Management nodes oversee co-ordination of the compute nodes: this ideally should be a separate pair of machines, but for my use case, I intend to use a virtual machine or container managed using active/passive failover techniques.
- In order to reduce virtual disk latency, the compute nodes will implement a local disk cache using an SSD, backed by a Rados Block Device on Ceph.
I’ll be using KVM as the virtualisation technology with Gentoo Linux as the base OS for this experimental cluster. At my workplace, we evaluated a few different technologies including Proxmox VE, Ganeti, OpenStack and OpenNebula. For this project, I intend to build on OpenNebula as it’s the simplest to understand and the most suited to my workplace’s requirements.
Using Gentoo makes it very easy to splice in patches as I’ll be developing as I go along. If I come to implement this in the office, I’ll be porting everything across to Ubuntu. This will be building on some experimental work I’ve done in the past with OpenNebula.
For the base nodes themselves, I’ve based them around these components:
- Supermicro A1SAi-2750F: This is a mini-ITX form-factor server motherboard which features an 8-core Intel Atom C2750 CPU on-board, quad-ethernet and IPMI. Newegg sell these for about AU$500. Scorptec were able to procure some for me. (Thanks!)
- Kingston KVR16LSE11/8: 8GB ECC SO-DIMMs. Techbuy sell these for about AU$120. There are other options out there so look around. Supermicro seem to recommend pairs of sticks, but mine seem to be happy with one of these 8GB sticks in slot A1 (next to ATX PSU socket).
- Samsung 850EVO 120GB SSD: There are other SSD options out there, I got these through U-Mart at AU$95 each. We’re using the 1TB versions in a few of our servers at work and they do okay and have a good warranty period.
- Mini-Box M350: Nice minimal mini-ITX enclosure, which has DIN-rail mounting options.
- Mini-Box PicoPSU 160-XT: 12V ATX 160W power supply.
- A power connector for the PSU. I think this one from Jaycar should fit, I’ll know tomorrow.
For the storage nodes, add to the list:
- Hitachi TravelStar 5K1000 0J22413: Fairly run-of-the-mill 5400RPM 1TB laptop hard drive. U-Mart had them cheap at about AU$83.
- M350 HDD bracket: as the M350 normally only takes one drive.
- Molex?SATA power adapter: the PicoPSU provides one SATA and one Molex connector for power. U-Mart normally stock these but this week they’re out, so I’ll be visiting Jaycar tomorrow to pick some up there.
Other things you may want/need:
- A managed switch, I ended up choosing the Linksys LGS-326AU which U-Mart were selling at AU$294. If you’ve ever used Cisco’s small business offerings, this unit will make you feel right at home.
- DIN rail. Jaycar sell this in 1m lengths, and I’ll grab some tomorrow.
Most of the above bits I have, the nodes are all basically built as of this afternoon, minus the SATA adaptors for the three storage nodes. All units power on, and do what one would expect of a machine that’s trying to boot from a blank SSD.
I did put one of the compute nodes through its paces, network booting the machine via PXE/NFS root and installing Gentoo.
Power consumption was below 1.8A for a battery voltage of about 13.4V, even when building the Linux 4.4.6 kernel (using make -j8), which it did in about 10 minutes. Watching this thing tackle compile jobs is a thing of beauty, can’t wait to get distcc going and have 40 CPU cores tear into the bootstrap process. The initial boot also looks beautiful, with 8 penguins lined up representing the 8 cores — don’t turn up here in a tuxedo!
So hardware wise, things are more or less together, and it’ll mostly be software. I’ll throw up some notes on how it’s all wired, but basically the plan in the short term is a 240V mains charger (surplus from a caravan) will keep the battery floated until I get the solar panel and controller set up.
When that happens, I plan to wire a relay in series with the 240V charger controlled by a comparator to connect mains when the battery voltage drops below 12V.
The switch is a 240V device unfortunately (couldn’t find any 24-port 12V managed switches) so it’ll run from an inverter. Port space is tight, and I just got the one since they’re kinda pricey. Long term, I might look at a second for redundancy, although if a switch goes, I won’t lose existing data.
ADSL2+ will be managed by a small localised battery back-up and a small computer as router, possibly a Raspberry Pi as I have one spare (original B model), which can temporarily store incoming SMTP traffic if the cluster does go down (heaven forbid!) and act as a management endpoint. There are a few contenders here, including these industrial computers, for which I already maintain a modern Linux kernel port for my workplace.
Things are coming together, and I hope to bring more on this project as it moves ahead.