Jun 292017

So, there’s some work still to be done, for example making some extension leads for the run between the battery link harness, load power distribution and the charger… and to generally tidy things up, but it is now up and running.

On the floor, is the 240V-12V power supply and the charger, which right now is hard-wired in boost mode. In the bottom of the rack are the two 105Ah 12V AGM batteries, in boxes with fuses and isolation switches.

The nodes and switching is inside the rack, and resting on top is the load power distribution board, which I’ll have to rewire to make things a little neater. A prospect is to mount some of this on the back.

I had a few introductions to make, introducing the existing pair of SG-200 switches to the newcomer and its VLANs, but now at least, I’m able to SSH into the nodes, access the IPMI BMC and generally configure the whole box and dice.

With the exception of the later upgrade to solar, and the aforementioned wiring harness clean-ups, the hardware-side of this dual hardware/software project, is largely complete, and this project now transitions to being a software project.

The plan from here:

  • Update the OSes… as all will be a little dated. (I might even blow away and re-load.)
  • Get Ceph storage up and running. It actually should be configured already, just a matter of getting DNS hostnames sorted out so they can find eachother.
  • Investigating the block caching landscape: when I first started the project at work, it was a 3-horse race between Facebook’s FlashCache, bcache and dmcache. Well, FlashCache is no more, replaced by EnhancedIO, and I’m not sure about the rest of the market. So this needs researching.
  • Management interfaces: at my workplace I tried Ganeti, OpenNebula and OpenStack. This again, needs re-visiting. OpenNebula has moved a long way from where it was and I haven’t looked at the others in a while. OpenStack had me running away screaming, but maybe things have improved.
Jun 252017

So, having got the rack mostly together, it is time to figure out how to connect everything.

I was originally going to have just one battery and upgrade later… but when it was discovered that the battery chosen was rather sick, the decision was made that I’d purchase two new batteries. So rather than deferring the management of multiple batteries, I’d have to deal with it up-front.

Rule #1 with paralleling batteries: don’t do it unless you have to. In a perfect world, you can do it just fine, but reality doesn’t work that way. There’s always going to be an imbalance that upsets things. My saving grace is that my installation is fixed, not mobile.

I did look at alternatives, including diodes (too much forward voltage drop), MOSFET switching (complexity), relay switching (complexity again, plus contact wear), and DIY uniselectors. Since I’m on a tight deadline, I decided, stuff it, I’ll parallel them.

That brings me to rule #2 about paralleling batteries: keep everything as close to matched as possible. Both batteries were bought in the same order, and hopefully are from the same batch. Thus, characteristics should be very close. The key thing here, I want to keep cable lengths between the batteries, load and charger, all equal so that the resistances all balance out. That, and using short runs of thick cables to minimise resistance.

I came up with the following connection scheme:

You’ll have to forgive the poor image quality here. On reflection, photographing a whiteboard has always been challenging.

Both batteries are set up in an identical fashion: 40A fuse on the positive side, cable from the negative side, going to an Andersen SB50/10. (Or I might put the fuse on the negative side … haven’t decided fully yet, it’ll depend on how much of each colour wire I have.) The batteries themselves are Giant Power 105Ah 12V AGM batteries. These are about as heavy as I can safely manage, weighing about 30kg each.

The central harness is what I built this afternoon, as I don’t yet have the fuse holders for the two battery harnesses.

The idea being that the resistance between the charger and each battery should be about the same. Likewise, the resistance between the load and each battery should be about the same

The load uses a distribution box and a bus bar. You’ve seen it before, but here’s how it’s wired up… pretty standard:

You might be able to make out the host names there too (periodic table naming scheme, why, because they’re Intel Atoms) … the 5 nodes are on the left and the two switches to the right of the distribution box. I have 3 spare positions.

In heavy black is the 0V bus bar.

This is what I’ve been spending much of my pondering, doing. Part of this harness is already done as it was installed that way in the car, the bit that’s missing is the circuit to the left of the relay that actually drives it. Redarc intended that the ignition key switch would drive the relay, I’ll be exploiting this feature.

Some time this week, I hope to make up the wiring harnesses for the two batteries, and get some charge into them as they’ve sat around for the past two months in their boxes steadily discharging, so I’d be better to get a charger onto them sooner rather than later.

The switch-over circuit can wait for now: just hard-wire it to the mains DC feed for now since there’s no solar yet. The principle of operation is that the comparator (an LM311) compares the solar voltage to a reference (derived from a 5V regulator) and kicks in when the voltage is high enough. (How high? No idea, maybe ~18V?). When that happens, it outputs a logic high signal that turns off the MOSFET. When too low, it pulls the MOSFET gate low, turning it on.

The MOSFET (a P-channel) provides the “ignition key switch” signal to the BCDC1225, fooling it into thinking it is connected to vehicle power, and the charger will boost as needed. The key being that the BCDC1225 makes the decision as to whether the battery needs charging, and how much charge.

By bolting together off-the-shelf parts, we should have something that I can source replacements for should the smoke escape, and there’s no high voltages to deal with.

Jun 252017

Well, it’s been a while since I last updated this project. Lots have been due to general lethargy, real life and other pressures.

This equipment is being built amongst other things to host my websites, mail server, and as a learning tool for managing clustered computing resources. As such, yes, I’ll be putting it down as a work expense… and it was pointed out to me that it needed to be in operation before I could start claiming it on tax. So, with 30th June looming up soon, it was time I pulled my finger out and got it going.

At least running on mains. As for the solar bit, well we will be doing that too, my father recently sent me this email (line breaks for readability):

Subject: Why you're about to pay through the nose for power - ABC News
 (Australian Broadcasting Corporation)
To: Stuart Longland
From: David Longland

Hi Stuart,

This is why I am keen to see your cluster up and running.  Our power 
bill is about $300 every 3 months, a lift in price by 20% represents 
$240pa hike.


Umm, yeah… good point. Our current little server represents a small portion of our base-load power… refrigeration being the other major component.

I ordered the rack and batteries a few months back, and both have been sitting here, still in the boxes they were shipped in, waiting for me to get to and put them together. My father got fed up of waiting and attacked the rack, putting it together one evening… and last night, we worked together on putting a back on the rack using 12mm plywood.

We also fitted the two switches, mounting the smaller one to the lid of the main switch using multiple layers of double-sided tape.

I wasn’t sure at first where the DIN rail would mount. I had intended to screw it to a piece of 2×4″ or similar, and screw that to the back plane. We couldn’t screw the DIN rail directly to the back plane because the nodes need to be introduced to the DIN rail at an angle, then brought level to attach them.

Considering the above, we initially thought we’d bolt it to the inner run of holes, but two problems presented themselves:

  1. The side panels actually covered over those holes: this was solved with a metal nibbling tool, cutting a slot where the hole is positioned.
  2. The DIN rail, when just mounted at each end, lacked the stability.

I measured the gap between the back panel and the DIN rail location: 45mm. We didn’t have anything that was that width which we could use as a mounting. We considered fashioning a bracket out of some metal strip, but bending it right could be a challenge without the right tools. (and my metalwork skills were never great.)

45mm + 3mm is 48mm… or 4× plywood pieces. We had plenty of off-cut from the back panel.

Using 4 pieces of the plywood glued together and clamped overnight, I made a mounting to which I could mount the DIN rail for the nodes to sit on. This afternoon, I drilled the pilot holes and fitted the screws for mounting that block, and screwed the DIN rail to it.

At the far ends, I made spacers from 3mm aluminium metal strap. The result is not perfect, but is much better than what we had before.

I’ve wired up the network cables… checking the lengths of those in case I needed to get longer cables. (They just fit… phew! $20 saved.) and there is room down the bottom for the batteries to sit. I’ll make a small 10cm cable to link the management network up to the appropriate port on the main switch, then I just need to run cables to the upstairs and downstairs switches. (In fact, there’s one into the area already.)

On the power front… my earlier experiments had ascertained the suitability of the Xantrex charger that we had spare. The charger is a smart charger, and so does various equalisation and balancing cycles, thus gets mightily confused if you suddenly disconnect the battery from it by way of a MOSFET. A different solution presented itself though.

My father has a solar set-up in the back of his car… there’s a 12V 120W panel on the roof, and that provides power to a battery system which powers an amateur radio station and serves as an auxiliary battery. There’s a diode arrangement that allows charging from the vehicle battery system.

In an effort to try and upgrade it, he bought a Redarc BCDC1225 in-vehicle MPPT charger. This charger can accept power from either the 12V mains supply in a vehicle, or from a “12V” solar panel. The key here, is it relies on a changeover relay to switch between the two, and this is where it wasn’t quite suitable for my father’s needs: it assumed that if the vehicle ignition was on, you wanted to charge from the vehicle, not from solar.

He wanted it to switch to whichever source was more plentiful, and had thought the unit would drive the relay itself. Having read the manual, we now know the signal they tell you to connect to the relay coil is there to tell the charger which source it is plugged into, not for it to drive the relay.

The plan is therefore:

  • use a 240V→12V AC-DC switch-mode power supply to provide the “vehicle mains” DC input to the charger.
  • measure the voltage seen at the solar input with a comparator and switch over when it is above some pre-defined voltage (use hysteresis to ensure it doesn’t oscillate)
  • use the output to drive a P-channel MOSFET attached to the “vehicle mains”, which drives the relay.
Jun 112017

So… when laying out this board, I decided I’d swap the 74HC374 for the much nicer ‘574 for managing the MOSFETs. No problem, well, one minor snag, neither Jaycar nor Altronics carry the ‘574.

They carry the ‘374… Jaycar is where I got mine originally. They also carry the very similar ‘573. I had intended to order some ‘574s for when the boards arrived, but they sort of turned up unexpectedly… so didn’t get a chance.

The fundamental difference? Apart from the ‘574 being edge triggered, the ‘573 also is active high on the logic enable pin. I had wrongly assumed it was active low .

Naturally, I did try to hack around this fundamental difference:

Tried to get that leg in focus, but it’s difficult when the LCD of the camera is such low resolution (and the viewfinder is an even lower resolution LCD). Basically, I nibbled the leg of the IC with my sidecutters and bent it up. To the pad, I solder the gate of a 2N7000 MOSFET, bend the drain pin up to meet the now floating-in-air LE pin, and run a resistor (a 3k3) to Vcc.

That works somewhat… it might be possible to introduce some more state machine cycles to handle the kludge… although the real fix is to use the correct part in the first place.

Jun 102017

So, I haven’t yet tried out the MOSFET outputs, but nearly all the GPIO inputs work. I say nearly… there is one that doesn’t on channel 4.

I can test by shorting out the 0V and GPIO pins with a multimeter probe. Not sure why one channel isn’t working yet, that I’ll have to debug tomorrow.

It is worth noting that the MC14066s here are older than I am if I am reading the date codes correctly. Still, they’ve been kept in a tube all these years, so no reason why they’d suddenly pop. More probable, I’ve goofed somewhere in the wiring of this.

At worst, I should be able to de-solder the faulty chip (if that is the case) and put another one in its place… I have plenty lying around. The fact that its mate on the other side is working fine, is promising.

The up side is this board doesn’t look like a big mess of wires like the last prototype. Once I get the faulty channel sorted, I should be able to make the I/O modules that will provide the MOSFET outputs, ESD protection and switch debouncing.

One thing I’ll probably do in future: use bigger vias for my jumper wires, and actually mark on the silk screen component values and the jumper wire routes.

Did some probing… turns out my jumper wire wasn’t making good contact with the via … a dry joint. Definitely I’ll make the vias bigger next time!