Jun 292017

So, there’s some work still to be done, for example making some extension leads for the run between the battery link harness, load power distribution and the charger… and to generally tidy things up, but it is now up and running.

On the floor, is the 240V-12V power supply and the charger, which right now is hard-wired in boost mode. In the bottom of the rack are the two 105Ah 12V AGM batteries, in boxes with fuses and isolation switches.

The nodes and switching is inside the rack, and resting on top is the load power distribution board, which I’ll have to rewire to make things a little neater. A prospect is to mount some of this on the back.

I had a few introductions to make, introducing the existing pair of SG-200 switches to the newcomer and its VLANs, but now at least, I’m able to SSH into the nodes, access the IPMI BMC and generally configure the whole box and dice.

With the exception of the later upgrade to solar, and the aforementioned wiring harness clean-ups, the hardware-side of this dual hardware/software project, is largely complete, and this project now transitions to being a software project.

The plan from here:

  • Update the OSes… as all will be a little dated. (I might even blow away and re-load.)
  • Get Ceph storage up and running. It actually should be configured already, just a matter of getting DNS hostnames sorted out so they can find eachother.
  • Investigating the block caching landscape: when I first started the project at work, it was a 3-horse race between Facebook’s FlashCache, bcache and dmcache. Well, FlashCache is no more, replaced by EnhancedIO, and I’m not sure about the rest of the market. So this needs researching.
  • Management interfaces: at my workplace I tried Ganeti, OpenNebula and OpenStack. This again, needs re-visiting. OpenNebula has moved a long way from where it was and I haven’t looked at the others in a while. OpenStack had me running away screaming, but maybe things have improved.
Mar 192017

So, further progress on the charge controller.

Thinking about the problem … I realised that I really do not want to be testing for VBNVH when entering the CHARGE_CHECK state, as it’ll prematurely terminate the charge when the battery is being bulk-charged.

Better to wait until the charger decides to stop … which we’ll see due to the battery voltage ceasing to increase. We do want to check we’re not critically high however, so we can swap out VH for VCH.

Next, when we find that VBNVBL, meaning the battery is not charging, there we can check for VBNVH and stop charging at that point.

That change, worked pretty well, but it was still flapping between sources. A little state management helped this. If we declare another state variable, charger_warning, we can flip this to 1 upon first detecting that VBNVBL, then wait a little longer. If after a time-out this is still the case, then we can take action. Thus we define a new timer tCWARN, which delays acting on the not-charging case.

A bit of threshold tweaking, and things are behaving themselves. I’m using an el-cheapo 3-way camping fridge as the stand-in for the cluster. This is an Aldi special bought some years ago that draws about 5-6A… and when running on 12V power, features no thermostat.

We’re finding that its cooling capacity is no match for Brisbane’s early autumn weather anyway, so it’s pretty much would be a constant load even if the thermostat worked on 12V.

The battery we’re using is an old 105Ah AGM battery… which is one of two batteries from the caravan we have here. They were the original batteries, and this battery’s mate had failed when both were replaced. We’ve noted this battery getting warm whilst charging, so we think it might now be on the way out too.

What to replace it with? LiFePO₄ is AU$1000 for 100Ah, so too expensive. AGM is still the better bet. I can get 300Ah AGM batteries, but they weigh nearly as much as I do. I can just manage the 105Ah, so we’ll stick with those. I will need more than one long-term.

That brings the thorny issue of connecting them. I am not keen to hook batteries in parallel for various reasons. At least not permanently.

Now, the charger I’m using for mains is a 3-channel charger. I can make additional charge controllers (with the caveat that I need heatsinks for the MOSFETs…sigh!) and I can look for 3-channel solar chargers, or just get multiple chargers for the extra batteries. This can be done.

The load is the elephant in the room. I’d ideally like to manage it as a single load, although conceivably, I could put the switch and one storage node on one battery, a second storage node and a compute node on a second, and the final storage and compute nodes on the third. If the switch goes however, my cluster is toast.

I can put batteries in parallel, but this really does need to be done with care, using carefully matched batteries. So the better solution is to have a controller that chooses the battery with the highest voltage.

There might be an analogue means of implementing this, but a microcontroller is a single-chip solution. The ATTiny24As have up to 8 ADC/GPIO pins and three non-ADC GPIO pins. It’s what I’m already using for the charge controller, so is an easy choice.

The cluster will not tolerate a break-before-make switch-over. I thought about using a capacitor bank to keep the cluster alive during a brief (~1sec max) switch-over. Back-of-the-envelope calculations suggested I would need a 10F capacitor bank. I can get a 16V 470mF capacitor for AU$70 each… and would need 20 of these. Ouch!

A small battery is another option, maybe a 7Ah, but that has its own maintenance issues, and represents a single point of failure.

I can get Schottky diodes capable of 40A, they still present a 0.6V voltage drop. At 30A, that represents 16W! For comparison, a relay with a 225ohm coil resistance will draw ~60mA when the battery is at the maximum of 15V, representing a load of 1W.

Or I can use more MOSFETs like the ones I’m already using, which draw even less power; poor man’s solid-state relays. Latching relays also exist, but they can be rather expensive, more so than a solid-state relay.

I can probably get away with temporary parallel connections, so a make-before-break would let me switch sources. Or, I could place my switch across a Schottky, meaning I put up with that 16W load for a brief moment while I switch sources.

So more to think about, but we are getting close. I can defer this decision until I get a second battery, but I am getting close to the point where the cluster will be running full-time.