powertech-mp-3735

Solar Cluster: Re-wiring the rack

It’s been on my TO-DO list now for a long time to wire in some current shunts to monitor the solar input, replace the near useless Powertech solar controller with something better, and put in some more outlets.

Saturday, I finally got around to doing exactly that. I meant to also add a low-voltage disconnect to the rig … I’ve got the parts for this but haven’t yet built or tested it — I’d like to wait until I have done both,but I needed the power capacity. So I’m running a risk without the over-discharge protection, but I think I’ll take that gamble for now.

Right now:

  • The Powertech MP-3735 is permanently out, the Redarc BCDC-1225 is back in.
  • I have nearly a dozen spare 12V outlet points now.
  • There are current shunts on:
    • Raw solar input (50A)
    • Solar controller output (50A)
    • Battery (100A)
    • Load (100A)
  • The Meanwell HEP-600C-12 is mounted to the back of the server rack, freeing space from the top.
  • The janky spade lugs and undersized cable connecting the HEP-600C-12 to the battery has been replaced with a more substantial cable.

This is what it looks like now around the back:

Rear of the rack, after re-wiring

What difference has this made? I’ll let the graphs speak. This was the battery voltage this time last week:

Battery voltage for 2019-05-22

… and this was today…

Battery voltage 2019-05-29

Chalk-and-bloody-cheese! The weather has been quite consistent, and the solar output has greatly improved just replacing the controller. The panels actually got a bit overenthusiastic and overshot the 14.6V maximum… but not by much thankfully. I think once I get some more nodes on, it’ll come down a bit.

I’ve gone from about 8 hours off-grid to nearly 12! Expanding the battery capacity is an option, and could see the cluster possibly run overnight.

I need to get the two new nodes onto battery power (the two new NUCs) and the Netgear switch. Actually I’m waiting on a rack-mount kit for the Netgear as I have misplaced the one it came with, failing that I’ll hack one up out of aluminium angle — it doesn’t look hard!

A new motherboard is coming for the downed node, that will bring me back up to two compute nodes (one with 16 cores), and I have new 2TB HDDs to replace the aging 1TB drives. Once that’s done I’ll have:

  • 24 CPU cores and 64GB RAM in compute nodes
  • 28 CPU cores and 112GB RAM in storage nodes
  • 10TB of raw disk storage

I’ll have to pull my finger out on the power monitoring, there’s all the shunts in place now so I have no excuse but to make up those INA-219 boards and get everything going.

Solar Cluster: Considering options for over-discharge protection

Right now, the cluster is running happily with a Redarc BCDC-1225 solar controller, a Meanwell HEP-600C-12 acting as back-up supply, a small custom-made ATTiny24A-based power controller which manages the Meanwell charger.

The earlier purchased controller, a Powertech MP-3735 now is relegated to the function of over-discharge protection relay.  The device is many times the physical size of a VSR, and isn’t a particularly attractive device for that purpose.  I had tried it recently as a solar controller, but it’s fair to say, it’s rubbish at it.  On a good day, it struggles to keep the battery above “rock bottom” and by about 2PM, I’ll have Grafana pestering me about the battery slipping below the 12V minimum voltage threshold.

Actually, I’d dearly love t rip that Powertech controller apart and see what makes it tick (or not in this case).  It’d be an interesting study in what they did wrong to give such terrible results.

So, if I pull that out, the question is, what will prevent an over-discharge event from taking place?  First, I wish to set some criteria, namely:

  1. it must be able to sustain a continuous load of 30A
  2. it should not induce back-EMF into either the upstream supply or the downstream load when activated or activated
  3. it must disconnect before the battery reaches 10.5V (ideally it should cut off somewhere around 11-11.5V)
  4. it must not draw excessive power whilst in operation at the full load

With that in mind, I started looking at options.  One of the first places I looked was of course, Redarc.  They do have a VSR product, the VS12 which has a small relay in it, rated for 10A, so fails on (1).  I asked on their forums though, and it was suggested that for this task, a contactor, the SBI12, be used to do the actual load shedding.

Now, deep inside the heart of the SBI12 is a big electromechanical contactor.  Many moons ago, working on an electric harvester platform out at Laidley for Mulgowie Farming Company, I recall we were using these to switch the 48V supply to the traction motors in the harvester platform.  The contactors there could switch 400A and the coils were driven from a 12V 7Ah battery, which in the initial phases, were connected using spade lugs.

One day I was a little slow getting the spade lug on, so I was making-breaking-making-breaking contact.  *WHACK*… the contactor told me in no uncertain terms it was not happy with my hesitation and hit me with a nice big back-EMF spike!  I had a tingling arm for about 10 minutes.  Who knows how high that spike was… but it probably is higher than the 20V absolute maximum rating of the MIC29712s used for power regulation.  In fact, there’s a real risk they’ll happily let such a rapidly rising spike straight through to the motherboards, frying about $12000 worth of computers in the process!

Hence why I’m keen to avoid a high back-EMF.  Supposedly the SBI12 “neutralises” this … not sure how, maybe there’s a flywheel diode or MOV in there (like this), or maybe instead of just removing power in a step function, they ramp the current down over a few seconds so that the back-EMF is reduced.  So this isn’t an issue for the SBI12, but may be for other electromechanical contactors.

The other concern is the power consumption needed to keep such a beast activated.  The other factor was how much power these things need to stay actuated.  There’s an initial spike as the magnetic field ramps up and starts drawing the armature of the contactor closed, then it can drop down once contact has been made.  The figures on the SBI12 are ~600mA initially, then ~160mA when holding… give or take a bit.

I don’t expect this to be turned on frequently… my nodes currently have up-times around 172 days.  So while 600mA (7~8W at 12V nominal) is high, that’ll only be for a second at most.  Much of the current will be holding current at, let’s call it 200mA to be safe, so about 2~3W.

That 2-3W is going to be the same, whether my nodes collectively draw 10mA, 10A or 100A.

It seemed like a lot, but then I thought, what about a SSR?  You can buy a 100A DC SSR like this for a lot less money than the big contactors.  Whack a nice big heat-sink on it, and you’re set.  Well, why the heat-sink?  These things have a voltage drop and on resistance.  In the case of the Jaycar one, it’s about 350mV and the on resistance is about 7mΩ.

Suppose we were running flat chat at our predicted 30A maximum…

  • MOSFET switch voltage drop: 30A × 350mV = 10.5W
  • Ron resistance voltage drop: (30A)² × 7mΩ = 6.3W
  • Total power dissipation: 10.5W + 6.3W = 16.8W OUCH!

16.8W is basically the power of an idle compute node.  The 3W of the SBI12 isn’t looking so bad now!  But can we do better?

The function of a solid-state relay, amongst other things, is to provide electrical isolation between the control and switching components.  The two are usually galvanically isolated.  This is a feature I really don’t need, so I could reduce costs by just using a bare MOSFET.

The earlier issues I had with the body diode won’t be a problem here as there’s a definite “source” and “load”, there’ll be no current to flow out of the load back to the source to confuse some sensing circuit on the source side.  This same body diode might be an issue for dual-battery systems, as the auxiliary battery can effectively supply current to a starter motor via this body diode, but in my case, it’s strictly switching a load.

I also don’t have inductive loads on my system, so a P-channel MOSFET is an option.  One candidate for this is the Infineon AUIRFS3004-7P.  The Ron on these is supposedly in the realm of 900µΩ-1.25mΩ, and of course, being that it’s a bare MOSFET and not a SSR, there’s no voltage drop.  Thus my power dissipation at 30A is predicted to be a little over 1W.

There are others too with even smaller Ron values, but they are in teeny tiny 5mm square surface-mount packages.  The AUIRFS3004-7P looks dead-buggable, just bend up the gate pin so I can solder direct to it, and treat the others as single “pins”, then strap the sucker to a big heatsink (maybe an old PIII heatsink will do the trick).

I can either drive this MOSFET with something of my own creation, or with the aforementioned Redarc VS12.  The VS12 still does contain a (much smaller) electromechanical relay, but at 30mA (~400mW), it’s bugger all.

The question though was what else could be done?  @WIRING_SOLUTIONS suggested some units made by Victron Energy.  These do have a nice feature in that they also have over-voltage protection, and conveniently, it’s 16V, which is the maximum recommended for the MIC29712s I’m using.  They’re not badly priced, and are solid-state.

However, what’s the Ron, what’s the voltage drop?  Victron don’t know.  They tell me it’s “minimal”, but is that 100nV, 100mV, 1V?  At 30A, 100mV drop equates to 3W, on par with the SBI12.  A 500mV drop would equate to a whopping 15W!

I had a look at the suppliers for Victron Energy products, and via those, found a few other contenders such as this one by Baintech and the Projecta LVD30.  I haven’t asked about these, but again, like the Victron BatteryProtect, neither of these list a voltage drop or Ron.

There’s also this one from Jaycar, but given this is the same place that sold me the Powertech MP-3735, and sold me the original Powertech MP-3089, provided a replacement for that first one, then also replaced the replacement under RMA.  The Jaycar VSR also has practically no specs… yeah, I think I’ll pass!

Whitworths marine sell this, it might be worth looking at but the cut-out voltage is a little high, and they don’t actually give the holding current (330mA “engage” current sounds like it’s electromechanical), so no idea how much power this would dissipate either.

The power controller isn’t doing a job dissimilar to a VSR… in fact it could be repurposed as one, although I note its voltage readings seem to drift quite a lot.  I suspect this is due to the choice of 5% tolerance resistors on the voltage sensing circuit and my use of the ~1.1V internal voltage reference.  The resistors will drift a little bit, and the voltage reference can be anywhere from 1.0 to 1.2V.

Would a LM311N with good quality 1% resistors and a quality voltage reference be “better”?  Who knows?  Maybe I should try an experiment, see if I can get minimal drift out of a LM311N.  It’s either the resistors, the voltage reference, or a combination of the two that’s responsible for the power controller’s drift.

Perhaps I need to investigate which is causing the problem and see what can be done in the design to reduce it.  If I can get acceptable results, then maybe the VS12 can be dispensed with.  I may be able to do it with another ATTiny24A, or even just a simple LM311N.

Solar Cluster: Reverting back to the Powertech MP-3735

So, for the past few weeks I’ve been running a Redarc BCDC-1225 solar controller to keep the batteries charged.  I initially found I had to make my little power controller back off on the mains charger a bit, but was finally able to prove conclusively that the Redarc was able to operate in both boost and float modes.

In the interests of science, I have plugged the Powertech back in.  I have changed nothing else.  What I’m interested to see, is if the Powertech in fact behaves itself, or whether it will go back to its usual tricks.

The following is the last 6 hours.

Next week, particularly Thursday and Friday, are predicted to have similar weather patterns to today. Today’s not a good test, since the battery started at a much higher voltage, so I expect that the solar controller will be doing little more than keeping the battery voltage up to the float set-point.

For reference, the settings on the MP-3735 are: Boost voltage 14.6V, Float voltage 13.8V. These are the recommended settings according to Century’s datasheets for the batteries concerned.

Interestingly, no sooner do I wire this up, but the power controller reaches for the mains. The MP-3735 definitely likes to flip-flop. Here’s a video of its behaviour shortly after connecting up the solar (and after I turned off the mains charger at the wall).

Now looking, it’s producing about 10A, much better than the 2A it was doing whilst filming.  So it can charge properly, when it wants to, but it’s intermittent, and inside you can sometimes hear a quiet clicking noise as if it’s switching a relay.  At 2A it’s wasting time, as the cluster draws nearly 5× that.

The hesitation was so bad, the power controller kicked the mains charger in for about 30 minutes, after that, the MP-3735 seems to be behaving itself.  I guess the answer is, see what it does tomorrow, and later this week without me intervening.

If it behaves itself, I’m happy to leave it there, otherwise I’ll be ordering a VSR, pulling out the Powertech MP-3735 and re-instating the Redarc BCDC-1225 with the VSR to protect against over-discharge.


Update 2018-10-28… okay, overcast for a few hours this morning, but by 11AM it had fined up.  The solar performance however was abysmal.

Let’s see how it goes this week… but I think I might be ordering that VSR and installing the Redarc permanently now.


Today’s effort:

Each one of those vertical lines was accompanied by a warning email.

Solar Cluster: Making the BCDC1225 get up and boogy!

So, I’ve been running the Redarc controller for a little while now, and we’ve had some good days of sunshine to really test it out.

Recall in an earlier posting with the Powertech solar controller I was getting this in broad daylight:

Note the high amount of “noise”, this is the Powertech solar controller PWMing its output. I’m guessing output filtering is one of the corners they cut, I expect to see empty footprints for juicy big capacitors that would have been in the “gold” model sent for emissions testing. It’ll be interesting to tear that down some day.

I’ve had to do some further tweaks to the power controller firmware, so this isn’t an apples-to-apples comparison, maybe next week we’ll try switching back and see what happens, but this was Tuesday, on the Redarc controller:

You can see that overnight, the Meanwell 240V charger was active until a little after 5AM, when my power controller decided the sun should take over. There’s a bit of discharging, until the sun crept up over the roof of our back-fence-neighbour’s house at about 8AM. The Redarc basically started in “float” mode, because the Meanwell had done all the hard work overnight. It remains so until the sun drops down over the horizon around 4PM, and the power controller kicks the mains back on around 6PM.

I figured that, if the Redarc controller saw the battery get below the float voltage at around sunrise, it should boost the voltage.

The SSR controlling the Meanwell was “powered” by the solar, meaning that by default, the charge controller would not be able to inhibit the mains charger at night as there was nothing to power the SSR. I changed that last night, powering it from the battery. Now, the power controller only brings in the mains charger when the battery is below about 12.75V. It’ll remain on until it’s been at above 14.4V for 30 minutes, then turn off.

In the last 24 hours, this is what the battery voltage looks like.

I made the change at around 8PM (can you tell?), and so the battery immediately started discharging, then the charge-discharge cycles began. I’m gambling on the power being always available to give the battery a boost here, but I think the gamble is a safe one. You can see what happened 12 hours later when the sun started hitting the panels: the Redarc sprang into action and is on a nice steady trend to a boost voltage of 14.6V.

We’re predicted to get rain and storms tomorrow and Saturday, but maybe Monday, I might try swapping back to the Powertech controller for a few days and we’ll be able to compare the two side-by-side with the same set-up.


It’s switched to float mode now having reached a peak boost voltage of 14.46V.  As Con the fruiterer would say … BEEEAAUUTIFUUUL!

Solar Cluster: Return of the Redarc BCDC1225

Well, I’ve now had the controller working for a week or so now… the solar output has never been quite what I’d call, “great”, but it seems it’s really been on the underwhelming side.

One of the problems I had earlier before moving to this particular charger was that the Redarc wouldn’t reliably switch between boosting from 12V to MPPT from solar.  It would get “stuck” and not do anything.  Coupled with the fact that there’s no discharge protection, and well, the results were not a delight to the olfactory nerves at 2AM on a Sunday morning!

It did okay as a MPPT charger, but I needed both functions.  Since the thinking was I could put a SSR between the 12V PSU and the Redarc charger, we tried going the route of buying the Powertech MP3735 solar charge controller to handle the solar side.

When it wants to work, it can put over 14A in.  The system can run on solar exclusively.  But it’s as if the solar controller “hesitates”.

I thought maybe the other charger was confusing it, but having now set up a little controller to “turn off” the other charger, I think I can safely put that theory to bed.  This was the battery voltage yesterday, where there was pretty decent sunshine.

There’s an odd blip at about 5:40AM, I don’t know what that is, but the mains charger drops its output by a fraction for about 50 seconds.  At 6:37AM, the solar voltage rises above 14V and the little ATTiny24A decides to turn off the mains charger.

The spikes indicate that something is active, but it’s intermittent.  Ultimately, the voltage winds up slipping below the low voltage threshold at 11:29AM and the mains charger is brought in to give the batteries a boost.  I actually made a decision to tweak the thresholds to make things a little less fussy and to reduce the boost time to 30 minutes.

The charge controller re-booted and turned off the mains charger at that point, and left it off until sunset, but the solar controller really didn’t get off its butt to keep the voltages up.

At the moment, the single 120W panel and 20A controller on my father’s car is outperforming my 3-panel set-up by a big margin!

Today, no changes to the hardware or firmware, but still a similar story:

The battery must’ve been sitting just on the threshold, which tripped the charger for the 30 minutes I configured yesterday.  It was pretty much sunny all day, but just look at that moving average trend!  It’s barely keeping up.

A bit of searching suggests this is not a reliable piece of kit, with one thread in particular suggesting that this is not MPPT at all, and many people having problems.

Now, I could roll the dice and buy another.

I could throw another panel on the roof and see if that helps, we’re considering doing that actually, and may do so regardless of whether I fix this problem or not.

There’s several MPPT charger projects on this very site.  DIY is a real possibility.  A thought in the back of my mind is to rip the Powertech MP3735 apart and re-purpose its guts, and make it a real MPPT charger.

Perhaps one with Modbus RTU/RS-485 reporting so that I can poll it from the battery monitor computer and plot graphs up like I’m doing now for the battery voltage itself.  There’s a real empty spot for 12V DC energy meters that speak Modbus.

If I want a 240V mains energy meter, I only have to poke my head into the office of one of my colleagues (who works for the sister company selling this sort of kit) and I could pick up a little CET PMC-220 which with the addition of some terminating resistors (or just run comms at 4800 baud), work just fine.  Soon as you want DC, yeah, sure there’s some for solar set-ups that do 300V DC, but not humble 12V DC.

Mains energy meters often have extra features like digital inputs/outputs, so this could replace my little charge controller too.  This would be a separate project.

But that would leave me without a solar controller, which is not ideal, and I need to shut everything down before I can extract the existing one.  So for now, I’ve left the Powertech one in-place, disconnected its solar input so that now it just works as a glorified VSR and voltmeter/ammeter, as that bit works fine.

The Redarc is now hooked up to solar, with its output going into a spare socket going to the batteries.  This will cost me nothing to see if it’s the solar controller or not.  If it is, then I think some money on a VSR to provide the low-voltage protection, and re-instating the Redarc charger for solar duty will be the next step.  Then I can tear down the Powertech one at my leisure and figure out what they did wrong, or if it can be re-programmed.

The Meanwell charger is taking care of things as I type this, but tomorrow morning, we should hopefully see the solar set-up actually do some work…

… maybe. 🙂

Solar Cluster: Return of the power controller

Well, I’ve been tossing up how to control the mains charger for a while now.

When I first started the project, my thinking was to use an old Xantrex charger we had kicking around, and just electrically disconnect it from the batteries when we wanted to use the solar power.  I designed a 4-layer PCB which sported a ATTiny24A microcontroller, MOSFETs (which I messed up) and some LEDs.

This was going to be a combined fan controller and power management unit.  It had the ability (theoretically) to choose a supply based on the input voltage, and to switch if needed between supplies.

It didn’t work out, the charger got really confused by the behaviour of the controller.  I was looking to re-instate it using the Redarc solar controller, but I never got there.  In the end, it was found that the Redarc controller had problems switching sources and would do nothing whilst the batteries went flat.

We’ve now replaced both ends of the system.  The solar controller is a Powertech MP3735 and integrates over-discharge protection.  The mains charger is now a MeanWell HEP-600C-12 (which has not missed a beat since the day it was put in).

Unlike my earlier set-up, this actually has a 5V logic signal to disable it, and my earlier controller could theoretically generate that directly.

Looking at the PCB of my earlier power controller attempt, it looks like this could still work.

Above is the PCB artwork.  I’ve coloured in the sections and faded out the parts I can omit.

In green up the top-left we have the mains control/monitoring circuitry.  We no longer see the mains voltage, so no point in monitoring it, so we can drop the resistor divider that fed the ADC.  This also means we no longer need the input socket P2.

Q2 and Q7 were the footprints of the two P-channel MOSFETs.  We don’t need the MOSFETs themselves, but the signals we need can be found on pin 1 of Q2.  This is actually the open-drain output of Q1, which we may be able to hook directly to the REMOTE+ pin on the charger.  A pull-up between there and the charger’s 5V rail, and we should be in business.

In yellow, bottom left is the solar monitoring interface.  This is still useful, but we won’t be connecting solar to the battery ourselves, so we just keep the monitoring parts.  The LED can stay as an indicator to show when solar is “good enough”.

In purple, occupying most of the board, is the controller itself.  It stays for obvious reasons.

In red, is the fan control circuitry.  No reason why this can’t stay.

In blue is the circuitry for monitoring the battery voltage.  Again, this stays.

The main advantage of doing this is I already have the boards, and a number of microcontrollers already present.  There’s a board with all except the big MOSFETs populated: with the MOSFETs replaced by 3-pin KK sockets.

How would the logic work?  Much the same as the analogue version I was pondering.

  • If battery voltage is low, OR, the sun has set, enable the mains charger.

What concerned me about an analogue solution was what would happen once the charger got to the constant-voltage stage.  We want to give it a bit of time to keep the battery topped up.  Thus it makes sense to shut down the charger after a fixed delay.

This is easy to do in a microcontroller.  Not hard with analogue electronics either, it’s fundamentally just a one-shot, but doing it with an MCU is a single-chip solution.  I can make the delay as long as I like.  So likely once the battery is “up to voltage”, I can let it float there for an hour, or until sunrise if it’s at night.

Solar Cluster: Fine Tuning

So yesterday I wound back the mains charger so that the solar would take on the load during the day.  Seems I wound it back a bit far, and the mains charger did almost no work overnight, leaving the battery somewhere around 11.8V.

That’s a wee bit low for my comfort.  Yes, they are deep cycle AGMs, but I’d rather not get that low.

Thus, I wound it up a bit, float at 12.8V, so Vboost at 13.6V.  That looks to be the sweet spot.  Now that the sun is up, I’m getting nice healthy amps of current down the wire from the roof:

The cluster is drawing about 8A, so that’s the cluster powered, and about 6A going to the batteries. It intermittently peaks about 15A or so.

I also found myself fine tuning the Ethernet settings on the border router. For some reason, its Realtek RTL8139 was happy to talk to the Cisco SG-200-08 it was connected to before, but didn’t quite get along with the Linksys LGS326-AU. I’ve told the switch to force 100Mbps full-duplex MDIX (evidently, it’s a cross-over cable), and so far, that seems to have settled things down.

Solar Cluster: Suspicions with the charger…

So, at present I’ve been using a two-charger solution to keep the batteries at full voltage.  On the solar side is the Powertech MP3735, which also does over-discharge protection.  On the mains side, I’m using a Xantrex TC2012.

One thing I’ve observed is that the TC2012, despite being configured for AGM batteries, despite the handbook saying it charges AGM batteries to a maximum 14.3V, has a happy knack of applying quite high charging voltages to the batteries.

I’ve verified this… every meter I’ve put across it has reported it at one time or another, more than 15V across the terminals of the charger.  I’m using SB50 connectors rated at 50A and short runs of 6G cable to the batteries.  So a nice low-resistance path.

The literature I’ve read says 14.8V is the maximum.  I think something has gone out of calibration!

This, and the fact that the previous set-up over-discharged the batteries twice, are the factors that lead to the early failure of both batteries.

The two new batteries (Century C12-105DA) are now sitting in the battery cases replacing the two Giant Energy batteries, which will probably find themselves on a trip to the Upper Kedron recycling facility in the near future.

The Century batteries were chosen as I needed the replacements right now and couldn’t wait for shipping.  This just happened to be what SuperCheap Auto at Keperra sell.

The Giant Energy batteries took a number of weeks to arrive: likely because the seller (who’s about 2 hours drive from me) had run out of stock and needed to order them in (from China).  If things weren’t so critical, I might’ve given those batteries another shot, but I really didn’t have the time to order in new ones.

I have disconnected the Xantrex TC2012.  I really am leery about using it, having had one bad experience with it now.  The replacement batteries cost me $1000.  I don’t want to be repeating the exercise.

I have a couple of options:

  1. Ditch the idea of using mains power and just go solar.
  2. Dig out the Redarc BCDC1225 charger I was using before and hook that up to a regulated power supply.
  3. Source a new 20A mains charger to hook in parallel with the batteries.
  4. Hook a dumb fixed-voltage power supply in parallel with the batteries.
  5. Hook a dumb fixed-voltage power supply in parallel with the solar panel.

Option (1) sounds good, but what if there’s a run of cloudy days?  This really is only an option once I get some supervisory monitoring going.  I have the current shunts fitted and the TI INA219Bs for measuring those shunts arrived a week or so back, just haven’t had the time to put that into service.  This will need engineering time.

Option (2) could be done right now… and let’s face it, its problem was switching from solar to mains.  In this application, it’d be permanently wired up in boost mode.  Moreover, it’s theoretically impossible to over-discharge the batteries now as the MP3735 should be looking after that.

Option (3) would need some research as to what would do the job.  More money to spend, and no guarantee that the result will be any better than what I have now.

Option (4) I’m leery about, as there’s every possibility that the power supply could be overloaded by inrush current to the battery.  I could rig up a PWM circuit in concert with the monitoring I’m planning on putting in, but this requires engineering time to figure out.

Option (5) I’m also leery about, not sure how the panels will react to having a DC supply in parallel to them.  The MP3735 allegedly can take an input DC supply as low as 9V and boost that up, so might see a 13.8V switchmode PSU as a solar panel on a really cloudy day.  I’m not sure though.  I can experiment, plug it in and see how it reacts.  Research gives mixed advice, with this Stack Exchange post saying yes and this Reddit thread suggesting no.

I know now that the cluster averages about 7A.  In theory, I should have 30 hours capacity in the batteries I have now, if I get enough sun to keep them charged.

This I think, will be a week-end experiment, and maybe something I’ll try this weekend.  Right now, the cluster itself is running from my 40A switchmode PSU, and for now, it can stay there.

I’ll let the solar charger top the batteries up from the sun this week.  With no load, the batteries should be nice and full, ready come Friday evening, when I can, gracefully, bring the cluster down and hook it up to the solar charger load output.  If, at the end of the weekend, it’s looking healthy, I might be inclined to leave it that way.

Solar Cluster: Solar controller replaced, upgrading RAM and VM management

So, this morning I decided to shut the whole lot down and switch to the new solar controller.  There’s some clean-up work to be done, but for now, it’ll do.  The new controller is a Powertech MP3735.  Supposedly this one can deliver 30A, and has programmable float and bulk charge voltages.  A nice feature is that it’ll disconnect the load when it drops below 11V, so finding the batteries at 6V should be a thing of the past!  We’ll see how it goes.

I also put in two current shunts, one on the feed into/out of the battery, and one to the load.  Nothing is connected to monitor these as yet, but some research suggested that while in theory it is just an op-amp needed, that op-amp has to deal with microvolt differences and noise.

There are instrumentation amplifiers designed for that, and a handy little package is TI’s INA219B.  This incorporates aforementioned amplifier, but also adds to that an ADC with an I²C interface.  Downside is that I’ll need an MCU to poll it, upside is that by placing the ADC and instrumentation amp in one package, it should cut down noise, further reduced if I mount the chip on a board bolted to the current shunt concerned.  The ADC measures bus voltage and temperature as well.  Getting this to work shouldn’t be hard.  (Yes, famous last words I know.)

A few days ago, I also placed an order for some more RAM for the two compute nodes.  I had thought 8GB would be enough, and in a way it is, except I’ve found some software really doesn’t work properly unless it has 2GB RAM available (Gitea being one, although it is otherwise a fantastic Git repository manager).  By bumping both these nodes to 32GB each (4×8GB) I can be less frugal about memory allocations.

I can in theory go to 16GB modules in these boxes, but those were hideously expensive last time I looked, and had to be imported.  My debit card maxes out at $AU999.99, and there’s GST payable on anything higher anyway, so there goes that idea.  64GB would be nice, but 32GB should be enough.

The fun bit though, Kingston no longer make DDR3 ECC SO-DIMMs.  The mob I bought the last lot though informed me that the product is no longer available, after I had sent them the B-Pay payment.  Ahh well, I’ve tossed the question back asking what do they have available that is compatible.

Searching for ECC SODIMMs is fun, because the search engines will see ECC and find ECC DIMMs (i.e. full-size).  When looking at one of these ECC SODIMM unicorns, they’ll even suggest the full-size version as similar.  I’d love to see the salespeople try to fit the suggested full-size DIMM into the SODIMM socket and make it work!

The other thing that happens is the search engine sees ECC and see that that’s a sub-string of non-ECC.  Errm, yeah, if I meant non-ECC, I’d have said so, and I wouldn’t have put ECC there.

Crucial and Micron both make it though, here’s hoping mixing and matching RAM from different suppliers in the same bank won’t cause grief, otherwise the other option is I pull the Kingston sticks out and completely replace them.

The other thing I’m looking at is an alternative to OpenNebula.  Something that isn’t a pain in the arse to deploy (like OpenStack is, been there, done that), that is decentralised, and will handle KVM with a Ceph back-end.

A nice bonus would be being able to handle cross-architecture QEMU VMs, in particular for ARM and MIPS targets.  This is something that libvirt-based solutions do not do well.

I’m starting to think about ways I can DIY that solution.  Blockchain was briefly looked at, and ruled out on the basis that while it’d be good for an audit log, there’s no easy way to index it: reading current values would mean a full-scan of the blockchain, so not a solution on its own.

CephFS is stable now, but I’m not sure how file locking works on it.  Then there’s object storage itself, librados.  I’m not sure if there’s a database engine that can interface to that, or maybe to Amazon S3 storage (radosgw can emulate that), that’ll be the next step.  Lots to think about.