battery-charging

Solar Cluster: Re-wiring the rack

It’s been on my TO-DO list now for a long time to wire in some current shunts to monitor the solar input, replace the near useless Powertech solar controller with something better, and put in some more outlets.

Saturday, I finally got around to doing exactly that. I meant to also add a low-voltage disconnect to the rig … I’ve got the parts for this but haven’t yet built or tested it — I’d like to wait until I have done both,but I needed the power capacity. So I’m running a risk without the over-discharge protection, but I think I’ll take that gamble for now.

Right now:

  • The Powertech MP-3735 is permanently out, the Redarc BCDC-1225 is back in.
  • I have nearly a dozen spare 12V outlet points now.
  • There are current shunts on:
    • Raw solar input (50A)
    • Solar controller output (50A)
    • Battery (100A)
    • Load (100A)
  • The Meanwell HEP-600C-12 is mounted to the back of the server rack, freeing space from the top.
  • The janky spade lugs and undersized cable connecting the HEP-600C-12 to the battery has been replaced with a more substantial cable.

This is what it looks like now around the back:

Rear of the rack, after re-wiring

What difference has this made? I’ll let the graphs speak. This was the battery voltage this time last week:

Battery voltage for 2019-05-22

… and this was today…

Battery voltage 2019-05-29

Chalk-and-bloody-cheese! The weather has been quite consistent, and the solar output has greatly improved just replacing the controller. The panels actually got a bit overenthusiastic and overshot the 14.6V maximum… but not by much thankfully. I think once I get some more nodes on, it’ll come down a bit.

I’ve gone from about 8 hours off-grid to nearly 12! Expanding the battery capacity is an option, and could see the cluster possibly run overnight.

I need to get the two new nodes onto battery power (the two new NUCs) and the Netgear switch. Actually I’m waiting on a rack-mount kit for the Netgear as I have misplaced the one it came with, failing that I’ll hack one up out of aluminium angle — it doesn’t look hard!

A new motherboard is coming for the downed node, that will bring me back up to two compute nodes (one with 16 cores), and I have new 2TB HDDs to replace the aging 1TB drives. Once that’s done I’ll have:

  • 24 CPU cores and 64GB RAM in compute nodes
  • 28 CPU cores and 112GB RAM in storage nodes
  • 10TB of raw disk storage

I’ll have to pull my finger out on the power monitoring, there’s all the shunts in place now so I have no excuse but to make up those INA-219 boards and get everything going.

2019/05/29 by Computing Projects Public Syndication Solar-powered Cloud Computing 0

Solar Cluster: Dusty solar panels?

So recently, I had a melt-down with some of the monitor wiring on the cluster… to counteract that, I have some parts on order (RS Components annoyingly seem to have changed their shipping policies, so I suspect I’ll get them Monday)… namely some thermocouple extension cable, some small 250mA fast-blow fuses and suitable in-line holders.

In the meantime, I’m doing without the power controller, just turning the voltage down on the mains charger so the solar controller did most of the charging.

This, isn’t terribly reliable… and for a few days now my battery voltage has just sat at a flat 12.9V, which is the “boost” voltage set on the mains charger.

Last night we had a little rain, and today I see this:

Battery voltage today… the solar charger is doing some work.

Something got up and boogied this morning, and it was nothing I did to make that happen.  I’ll re-instate that charger, or maybe a control-only version of the #High-power DC-DC power supply which I have the parts for, but haven’t yet built.

2018/12/14 by Computing High-power DC-DC Converter Projects Public Syndication Solar-powered Cloud Computing 0

Solar Cluster: Reverting back to the Powertech MP-3735

So, for the past few weeks I’ve been running a Redarc BCDC-1225 solar controller to keep the batteries charged.  I initially found I had to make my little power controller back off on the mains charger a bit, but was finally able to prove conclusively that the Redarc was able to operate in both boost and float modes.

In the interests of science, I have plugged the Powertech back in.  I have changed nothing else.  What I’m interested to see, is if the Powertech in fact behaves itself, or whether it will go back to its usual tricks.

The following is the last 6 hours.

Next week, particularly Thursday and Friday, are predicted to have similar weather patterns to today. Today’s not a good test, since the battery started at a much higher voltage, so I expect that the solar controller will be doing little more than keeping the battery voltage up to the float set-point.

For reference, the settings on the MP-3735 are: Boost voltage 14.6V, Float voltage 13.8V. These are the recommended settings according to Century’s datasheets for the batteries concerned.

Interestingly, no sooner do I wire this up, but the power controller reaches for the mains. The MP-3735 definitely likes to flip-flop. Here’s a video of its behaviour shortly after connecting up the solar (and after I turned off the mains charger at the wall).

Now looking, it’s producing about 10A, much better than the 2A it was doing whilst filming.  So it can charge properly, when it wants to, but it’s intermittent, and inside you can sometimes hear a quiet clicking noise as if it’s switching a relay.  At 2A it’s wasting time, as the cluster draws nearly 5× that.

The hesitation was so bad, the power controller kicked the mains charger in for about 30 minutes, after that, the MP-3735 seems to be behaving itself.  I guess the answer is, see what it does tomorrow, and later this week without me intervening.

If it behaves itself, I’m happy to leave it there, otherwise I’ll be ordering a VSR, pulling out the Powertech MP-3735 and re-instating the Redarc BCDC-1225 with the VSR to protect against over-discharge.

Update 2018-10-28… okay, overcast for a few hours this morning, but by 11AM it had fined up.  The solar performance however was abysmal.

Let’s see how it goes this week… but I think I might be ordering that VSR and installing the Redarc permanently now.

Today’s effort:

Each one of those vertical lines was accompanied by a warning email.

2018/10/27 by Computing Projects Public Syndication Solar-powered Cloud Computing 0

Solar Cluster: Making the BCDC1225 get up and boogy!

So, I’ve been running the Redarc controller for a little while now, and we’ve had some good days of sunshine to really test it out.

Recall in an earlier posting with the Powertech solar controller I was getting this in broad daylight:

Note the high amount of “noise”, this is the Powertech solar controller PWMing its output. I’m guessing output filtering is one of the corners they cut, I expect to see empty footprints for juicy big capacitors that would have been in the “gold” model sent for emissions testing. It’ll be interesting to tear that down some day.

I’ve had to do some further tweaks to the power controller firmware, so this isn’t an apples-to-apples comparison, maybe next week we’ll try switching back and see what happens, but this was Tuesday, on the Redarc controller:

You can see that overnight, the Meanwell 240V charger was active until a little after 5AM, when my power controller decided the sun should take over. There’s a bit of discharging, until the sun crept up over the roof of our back-fence-neighbour’s house at about 8AM. The Redarc basically started in “float” mode, because the Meanwell had done all the hard work overnight. It remains so until the sun drops down over the horizon around 4PM, and the power controller kicks the mains back on around 6PM.

I figured that, if the Redarc controller saw the battery get below the float voltage at around sunrise, it should boost the voltage.

The SSR controlling the Meanwell was “powered” by the solar, meaning that by default, the charge controller would not be able to inhibit the mains charger at night as there was nothing to power the SSR. I changed that last night, powering it from the battery. Now, the power controller only brings in the mains charger when the battery is below about 12.75V. It’ll remain on until it’s been at above 14.4V for 30 minutes, then turn off.

In the last 24 hours, this is what the battery voltage looks like.

I made the change at around 8PM (can you tell?), and so the battery immediately started discharging, then the charge-discharge cycles began. I’m gambling on the power being always available to give the battery a boost here, but I think the gamble is a safe one. You can see what happened 12 hours later when the sun started hitting the panels: the Redarc sprang into action and is on a nice steady trend to a boost voltage of 14.6V.

We’re predicted to get rain and storms tomorrow and Saturday, but maybe Monday, I might try swapping back to the Powertech controller for a few days and we’ll be able to compare the two side-by-side with the same set-up.

It’s switched to float mode now having reached a peak boost voltage of 14.46V.  As Con the fruiterer would say … BEEEAAUUTIFUUUL!

2018/10/04 by Computing Projects Public Syndication Solar-powered Cloud Computing 0

Solar Cluster: Jury still out on solar controller, thinking of PSU designs

So, the last few days it’s been overcast.  Monday I had a firmware glitch that caused the mains supply to be brought in almost constantly, so I’d disregard that result.

Basically, the moment the battery dropped below ~12.8V for even a brief second, the mains got brought in.  We were just teetering on the edge of 12.8V all day.  I realised that I really did need a delay on firing off the timer, so I’ve re-worked the logic:

  • If battery drops below V_L, start a 1-hour timer
  • If battery rises above V_L, reset the 1-hour timer
  • If the battery drops below V_CL or the timer expires, turn on the mains charger

That got me better results.  It means V_CL can be quite low, without endangering the battery supply, and V_L can be at 12.8V where it basically ensures that the battery is at a good level for everything to operate.

I managed to get through most of Tuesday until about 4PM, there was a bit of a hump which I think was the solar controller trying to extract some power from the panels.  I really need a good sunny day like the previous week to test properly.

This is leading me to consider my monitoring device.  At the moment, it just monitors voltage (crudely) and controls the logic-level enable input on the mains charger.  Nothing more.  It has done that well.

A thought is that maybe I should re-build this as a Modbus-enabled energy meter with control.  This idea has evolved a bit, enough to be its own project actually.  The thought I have now is a more modular design.

If I take the INA219B and a surface-mount current shunt, I have a means to accurately measure input voltage and current.  Two of these, and I can measure the board’s output too.  Stick a small microcontroller in between, some MOSFETs and other parts, and I can have a switchmode power supply module which can report on its input and output power and vary the PWM of the power supply to achieve any desired input or output voltage or current.

The MCU could be the ATTiny24As I’m using, or a ATTiny861.  The latter is attractive as it can do high-speed PWM, but I’m not sure that’s necessary in this application, and I have loads of SOIC ATTiny24As.  (Then again, I also have loads of PDIP ATTiny861s.)

The board would expose the ICSP pins plus two more for interrupt and chip select, allowing for a simple jig for reprogramming.  I haven’t decided on a topology yet, but the split-pi is looking attractive.  I might start with a buck converter first though.

This would talk to a “master” microcontroller which would provide the UI and Modbus interface.  If the brains of the PSU MCU aren’t sufficient, this could do the more grunty calculations too.

This would allow me to swap out the PSU boards to try out different designs.

2018/09/27 by Computing Projects Public Syndication Solar-powered Cloud Computing 0

Solar Cluster: Return of the Redarc BCDC1225

Well, I’ve now had the controller working for a week or so now… the solar output has never been quite what I’d call, “great”, but it seems it’s really been on the underwhelming side.

One of the problems I had earlier before moving to this particular charger was that the Redarc wouldn’t reliably switch between boosting from 12V to MPPT from solar.  It would get “stuck” and not do anything.  Coupled with the fact that there’s no discharge protection, and well, the results were not a delight to the olfactory nerves at 2AM on a Sunday morning!

It did okay as a MPPT charger, but I needed both functions.  Since the thinking was I could put a SSR between the 12V PSU and the Redarc charger, we tried going the route of buying the Powertech MP3735 solar charge controller to handle the solar side.

When it wants to work, it can put over 14A in.  The system can run on solar exclusively.  But it’s as if the solar controller “hesitates”.

I thought maybe the other charger was confusing it, but having now set up a little controller to “turn off” the other charger, I think I can safely put that theory to bed.  This was the battery voltage yesterday, where there was pretty decent sunshine.

There’s an odd blip at about 5:40AM, I don’t know what that is, but the mains charger drops its output by a fraction for about 50 seconds.  At 6:37AM, the solar voltage rises above 14V and the little ATTiny24A decides to turn off the mains charger.

The spikes indicate that something is active, but it’s intermittent.  Ultimately, the voltage winds up slipping below the low voltage threshold at 11:29AM and the mains charger is brought in to give the batteries a boost.  I actually made a decision to tweak the thresholds to make things a little less fussy and to reduce the boost time to 30 minutes.

The charge controller re-booted and turned off the mains charger at that point, and left it off until sunset, but the solar controller really didn’t get off its butt to keep the voltages up.

At the moment, the single 120W panel and 20A controller on my father’s car is outperforming my 3-panel set-up by a big margin!

Today, no changes to the hardware or firmware, but still a similar story:

The battery must’ve been sitting just on the threshold, which tripped the charger for the 30 minutes I configured yesterday.  It was pretty much sunny all day, but just look at that moving average trend!  It’s barely keeping up.

A bit of searching suggests this is not a reliable piece of kit, with one thread in particular suggesting that this is not MPPT at all, and many people having problems.

Now, I could roll the dice and buy another.

I could throw another panel on the roof and see if that helps, we’re considering doing that actually, and may do so regardless of whether I fix this problem or not.

There’s several MPPT charger projects on this very site.  DIY is a real possibility.  A thought in the back of my mind is to rip the Powertech MP3735 apart and re-purpose its guts, and make it a real MPPT charger.

Perhaps one with Modbus RTU/RS-485 reporting so that I can poll it from the battery monitor computer and plot graphs up like I’m doing now for the battery voltage itself.  There’s a real empty spot for 12V DC energy meters that speak Modbus.

If I want a 240V mains energy meter, I only have to poke my head into the office of one of my colleagues (who works for the sister company selling this sort of kit) and I could pick up a little CET PMC-220 which with the addition of some terminating resistors (or just run comms at 4800 baud), work just fine.  Soon as you want DC, yeah, sure there’s some for solar set-ups that do 300V DC, but not humble 12V DC.

Mains energy meters often have extra features like digital inputs/outputs, so this could replace my little charge controller too.  This would be a separate project.

But that would leave me without a solar controller, which is not ideal, and I need to shut everything down before I can extract the existing one.  So for now, I’ve left the Powertech one in-place, disconnected its solar input so that now it just works as a glorified VSR and voltmeter/ammeter, as that bit works fine.

The Redarc is now hooked up to solar, with its output going into a spare socket going to the batteries.  This will cost me nothing to see if it’s the solar controller or not.  If it is, then I think some money on a VSR to provide the low-voltage protection, and re-instating the Redarc charger for solar duty will be the next step.  Then I can tear down the Powertech one at my leisure and figure out what they did wrong, or if it can be re-programmed.

The Meanwell charger is taking care of things as I type this, but tomorrow morning, we should hopefully see the solar set-up actually do some work…

… maybe. 🙂

2018/09/23 by Computing Projects Public Syndication Solar-powered Cloud Computing 0

Solar Cluster: Power controller installed

Well, I finally got around to installing that power controller.

Yes, the top of that rack is getting to be a pigsty. Even the controller isn’t my best work:

You can see above I’ve just tacked wires onto the points I need and brought those out to a terminal strip.  There’s provision there for some PWM-controlled fans, but right now this is unused.  I’ve omitted the parts not required for the application.  If this works out, I might consider doing another board, this time better dedicated to the task at hand.

With that controller in place, I’ve now wound the charger back up.  In fact I made a whoopsie at first: I forgot that the Vout pot on the HEF-600C-12 sets the float voltage and wound that right up to 14.4V which meant a boost voltage of 15V!

Thankfully I looked over at the volt meter on the solar controller and realised my mistake quick.  15 seconds won’t hurt anything, but it is now set at 13.6V.  You don’t even see it on the 40-sample average.  The controller should let the mains charger sit there for an hour before it reconsiders the need for mains.

I think my next step … there’s a yard that could do with a hair cut… I’ll drag the mower out and chase that around the yard for a bit.  Then we’ll see what it looks like.

Okay, back from a little mowing… and sure enough, the controller is mostly doing the right thing.  I think I’ll need to tweak some set-points, maybe set the solar threshold lower.  Thankfully the “inhibit” LED is just an indication that it considers the solar voltage low, the solar is going to be on no matter what.

Yes, that SSR is massive for the job. It’s what I had on hand at the time. I’ll probably replace it with something small, maybe a reed relay since they’re cheap.

Right at this point, I have the SSR’s inputs connected between the solar V+ and the BC-547B on the board, so when the sun *does* go down, the mains power will be turned back on no matter what the controller thinks.

A close-up look of the status LEDs shows what mode we’re in:

We’ll ignore the temperature ones. Ultimately they indicate the state of the fan controller, and in this state, it’d be running the fans, as indicated by the Fan PWM LED to the left. Temperature is measured by the sensor in the ATTiny24A on-board, so not highly accurate.

The other mystery “LED” is the shiny surface on the BC-547B to the left of the two source status LEDs.

Here, I suspect the sun ducked behind a cloud so the voltage dropped, hence both “inhibit” LEDs came on. Earlier, “Float” was lit (you can sort-of make it out in the previous photo), the charger was actually actively trying to charge the battery, but to the controller it looked to be done. It left it go for the hour as programmed, then turned off the mains charger to let the solar panel take over.

The idea is that during the day, if it gets low, give it a boost from mains, then go back to solar. We only want to rely on mains at night.

Now, it should stay in that state until tonight, when the lack of sun should bring the mains charger online (by sheer fact that the solar panels power the “coil” of the relay).

So, I saw that, and had a look… sure enough, the controller is still asserting that the mains charger should be off. I think I need to bump the battery thresholds up a bit, although that’s still safely above the danger zone, it’s lower than I’m comfortable with.

Right up until 5:58PM, it seems the MCU just held on, thinking the battery voltage was “good enough”, so no need for a charge yet. I might want to drop the solar threshold down some so it doesn’t “flap” when broken cloud passes over, then raise the minimum battery threshold a bit.

Even now, the thing that’s turning the mains charger on is the fact that the 1.5V coming off the panels is not sufficient for activating the solid-state relay I’m using. I’m thankful I wired the SSR to Vsolar and made the MCU output open-collector. This is a useful little safety feature, making it impossible for the MCU to latch-up and hold the mains charger off, as the sun will eventually set, and that will force the mains charger to turn on like it did tonight.

2018/09/16 by Projects Public Syndication Solar-powered Cloud Computing 0

Solar Cluster: Return of the power controller

Well, I’ve been tossing up how to control the mains charger for a while now.

When I first started the project, my thinking was to use an old Xantrex charger we had kicking around, and just electrically disconnect it from the batteries when we wanted to use the solar power.  I designed a 4-layer PCB which sported a ATTiny24A microcontroller, MOSFETs (which I messed up) and some LEDs.

This was going to be a combined fan controller and power management unit.  It had the ability (theoretically) to choose a supply based on the input voltage, and to switch if needed between supplies.

It didn’t work out, the charger got really confused by the behaviour of the controller.  I was looking to re-instate it using the Redarc solar controller, but I never got there.  In the end, it was found that the Redarc controller had problems switching sources and would do nothing whilst the batteries went flat.

We’ve now replaced both ends of the system.  The solar controller is a Powertech MP3735 and integrates over-discharge protection.  The mains charger is now a MeanWell HEP-600C-12 (which has not missed a beat since the day it was put in).

Unlike my earlier set-up, this actually has a 5V logic signal to disable it, and my earlier controller could theoretically generate that directly.

Looking at the PCB of my earlier power controller attempt, it looks like this could still work.

Above is the PCB artwork.  I’ve coloured in the sections and faded out the parts I can omit.

In green up the top-left we have the mains control/monitoring circuitry.  We no longer see the mains voltage, so no point in monitoring it, so we can drop the resistor divider that fed the ADC.  This also means we no longer need the input socket P2.

Q2 and Q7 were the footprints of the two P-channel MOSFETs.  We don’t need the MOSFETs themselves, but the signals we need can be found on pin 1 of Q2.  This is actually the open-drain output of Q1, which we may be able to hook directly to the REMOTE+ pin on the charger.  A pull-up between there and the charger’s 5V rail, and we should be in business.

In yellow, bottom left is the solar monitoring interface.  This is still useful, but we won’t be connecting solar to the battery ourselves, so we just keep the monitoring parts.  The LED can stay as an indicator to show when solar is “good enough”.

In purple, occupying most of the board, is the controller itself.  It stays for obvious reasons.

In red, is the fan control circuitry.  No reason why this can’t stay.

In blue is the circuitry for monitoring the battery voltage.  Again, this stays.

The main advantage of doing this is I already have the boards, and a number of microcontrollers already present.  There’s a board with all except the big MOSFETs populated: with the MOSFETs replaced by 3-pin KK sockets.

How would the logic work?  Much the same as the analogue version I was pondering.

  • If battery voltage is low, OR, the sun has set, enable the mains charger.

What concerned me about an analogue solution was what would happen once the charger got to the constant-voltage stage.  We want to give it a bit of time to keep the battery topped up.  Thus it makes sense to shut down the charger after a fixed delay.

This is easy to do in a microcontroller.  Not hard with analogue electronics either, it’s fundamentally just a one-shot, but doing it with an MCU is a single-chip solution.  I can make the delay as long as I like.  So likely once the battery is “up to voltage”, I can let it float there for an hour, or until sunrise if it’s at night.

2018/09/01 by Computing Projects Public Syndication Solar-powered Cloud Computing 0

Solar Cluster: Kernel driver now up on Github

So, I’m happy enough with the driver now that I’ll collapse down the commits and throw it up onto the Github repository.  I might take another look at kernel 4.18, but for now, you’ll find them on the ts7670-4.14.67 branch.

Two things I observe about this voltage monitor:

  1. The voltage output is not what you’d call, accurate.  I think it’s only a 10-bit ADC, which is still plenty good enough for this application, but the reading I think is “high” by about 50mV.
  2. There’s significant noise on the reading, with noticeable quantisation steps.

Owing to these, and to thwart the possibility of using this data in side-channel attacks using power analysis, I’ve put a 40-sample moving-average filter on the “public” data.

Never the less, it’s a handy party trick, and not one I expected these devices to be able to do.  My workplace manages a big fleet of these single-board computers in the residential towers at Barangaroo where they spend all day polling Modbus and M-Bus meters.  In the event we’re at all suspicious about DC power supplies though, it’s a simple matter to load this kernel tree (they already run U-Boot) and configure collectd (which is also installed).

I also tried briefly switching off the mains power to see that I was indeed reading the battery voltage and not just a random number that looked like the voltage.  That yielded an interesting effect:

You can see where I switched the mains supply off, and back on again.  From about 8:19PM the battery voltage predictably fell until about 8:28PM where it was at around 12.6V.

Then it did something strange, it rose about 100mV before settling at 12.7V.  I suspect if I kept it off all night it’d steadily decrease: the sun has long set.  I’ve turned the mains charger back on now, as you can see by the step-rise shortly after 8:44PM.

The bands on the above chart are the alert zones.  I’ll get an email if the battery voltage strays outside of that safe region of 12-14.6V.  Below 12V, and I run the risk of deep-cycling the batteries.  Above 14.6V, and I’ll cook them!

The IPMI BMCs on the nodes already sent me angry emails when the battery got low, so in that sense, Grafana duplicates that, but does so with pretty charts.  The BMCs don’t see when the battery gets too high though, for the simple matter that what they see is regulated by LDOs.

2018/08/30 by Computing Kernel hacking Linux Development Projects Public Syndication Solar-powered Cloud Computing 0

Solar Cluster: Getting the battery voltage into Grafana

I’ve succeeded in getting a working battery monitor kernel module. This is basically taking the application note by Technologic Systems and spinning that into a power supply class driver that reports the voltage via sysfs.

As it happens, the battery module in collectd does not see this as a “battery”, something I’ll look at later. For now the exec plug-in works well enough. This feeds through eventually to an InfluxDB database with Grafana sitting on top.

https://netmon.longlandclan.id.au/d/IyZP-V2mk/battery-voltage?orgId=1

2018/08/30 by Computing Kernel hacking Linux Development Projects Public Syndication Solar-powered Cloud Computing 0

Calendar

March 2023
S M T W T F S
 1234
567891011
12131415161718
19202122232425
262728293031  

Battery voltage

Mastodon

Mastodon