Oct 212018
 

So, I placed an order to Mouser the other day to actually get some parts into my hands so I can better design the boards.

In that, I discovered the screw terminals I was planning on using, are discontinued.  So, I found something that was able to take the same gauge wire: Phoenix 1017526s.  Turns out, these will not fit along side the current shunts on the board as planned.

There’s just no way I’ll be fitting these on a 5×5cm board and have room to spare for a shunt in between.  Since this is really application-specific, it might be better off board.  We’ll put the INA219 and PCA9615 together on the board so we have a nice self-contained sensor board that can be mounted close to the current shunt, wherever that lives, and have nice noise-resistant links back to the controller.

This does mean I can do things like put a current shunt in the fuse box where the solar panels connect, and run CAT5 down to the controller from there.

To make routing easier, I’ve gone to a 4-layer board.  The board has solder-jumpers for setting the I²C address of the INA219, and I’ve documented all the termination and pull resistors.  I’m not sure what ones are needed yet, so there’s space at every point where I could envisage one being needed.

There’s two power planes in the inner layers, one for VCC the other for 0V.

Next step, I’ll print out the board designs and test fit everything before ordering the boards, which I hope to have ordered this afternoon.

Oct 062018
 

So, I’ve designed the sensor board, this is basically a break-out of an INA219 coupled with a PCA9615, for extended I²C range.  If I was to use one of these on the cluster, it’s theoretically possible for me to put one up in the fuse box on the back deck, and run CAT5e down to the server rack to help measure voltage drop across that long run.  Doing that with regular I²C would be insane.

Again, I’ve gone crazy with pull-up, pull-down and termination resistances, not knowing what would be needed.  The schematic is nothing special.

The board wound up bigger than I’d expected, but largely because it had to accommodate fairly heavy power traces.  I think I’ve got the footprint for the screw terminal blocks right.  I’ve managed to cram it onto a 5cm×5cm board (two layer).

As always, you’ve got two ways of dealing with the current shunt, either hook one up externally, which means you don’t bother with the beefy power connection footprints, or you fit a surface-mount shunt on.

You’ve got full flexibility there, as well as what address to set the board to via the jumpers.

I’ll probably order some of the connectors and other parts in question, print out the board layout and test-fit everything.  I’m not happy about the fact that NXP only make the PCA9615 in TSSOP, but I guess I should be thankful the part has legs.

Sep 302018
 

So this is what I’ve come up with for the core controller.

There’s provisions for two versions on this board, one with an ATTiny861 which does high-speed (250kHz) PWM and can drive a buck, boost or buck-boost DC-DC converter.  It features differential I²C interfaces for the input and output INA219 boards, and LVDS for controlling the MOSFET boards.

The other version is built around the ATTiny24A, and just features the ability to turn on and off MOSFETs.  It can drive two statically, or PWM one (at a much lower speed), with the user supplying the driver logic.  Due to the the fact that this device does not do high-speed switching, I’ve forgone the LVDS control over a simple current loop.  The I²C is still differential though as that could be some distance away and is still somewhat high frequency.

The layout of the board is a small 5×5cm 4-layer PCB.

I had to go 4-layer as I needed to route signals both sides and didn’t want to interrupt the power planes.  The two inner layers are VCC and GND.  There’s de-coupling capacitors galore, although the two power planes will probably function as a decent capacitor in their own right.  ICSP is via the interface header at the bottom.

Sep 302018
 

I was originally thinking of one monolithic board which would have everything it needed.

There was provision for the lot, including a separate ATTiny24A so that you could omit all but one of the MOSFETs, swap the remaining MOSFET for a P-channel, drop the MOSFET drivers, one of the INA219s, and the ATTiny861, and you’d have just a monitoring board with a (low-speed) PWMable switch.  It’d plug into the same place and use the same host interface.  The one board could be made into just a boost, or just a buck.  Flexibility.

There was just one snag.  That’ll work for small power supplies with maybe up to 5A capability (~100W) but not for the 50A version.  The MOSFETs will fit, but the tracks will need to be huge, the board will be hideously expensive to make, and they don’t make inductors big enough.

Looking around for inductors, I did see these.  They’re not massive like the 10uH one I saw, and they’re not expensive.  The downside is they’re about 10% of what I really need.  I guess I’ll just make do.  They’re also not PCB-mount (mind you, a 40kg inductor doesn’t PCB-mount either).

Thus, it may be more sensible to separate the MOSFETs and high-power stuff from the controller.  Now here’s the rub.  We’re dealing with sub-15ns pulses.  PWM.

Years ago for my industrial experience, I did work on an electric harvester platform.  The system ran 48V.  The motors were rated at 20kW, and were made in house using windings wound from 5mm diameter enamelled copper wire and neodymium magnets.

We had loads of issues with MOSFETs blowing.  The MOSFET driver was mounted close to the MOSFETs, as I’m proposing to do here, but between DSP and driver, was a long-ish run of ribbon cable. @Bil Herd posted this article covering the challenges involving inductance on PCB layout.  That same problem applies to “long” cable runs too.

10 years ago when we were working on this project, I remember asking about if we had considered maybe using coax cable instead of a ribbon cable.  The idea was rubbished at the time.  Given we were PWMing 400A, I think there might’ve been something in that suggestion.

That ribbon (10~20cm of it) would have had a lovely inductance all of its own, and while I have no idea what frequency the PWM was running at (I might have the code somewhere but I can’t be stuffed digging it up), and we were fundamentally driving a single-ended signal over a fairly long distance.  Yes, ground was close, but probably not enough, a twisted pair would have been better, but even then not perfect.  We blew many MOSFETs on that project.  Big TO-263s!

An earlier article on differential signalling got me thinking: why not use LVDS for the PWM?  A quick search has revealed this receiver and transmitter (Mouser says two receivers on it, but I think that’s a typo).  The idea being that I send the PWM down a differential pair using LVDS.  155Mbps should be plenty fast enough (the ATTiny861 can only do 64MHz) and these parts will run at the 5V needed for fast switching.  In fact they require it.

Using twisted pairs, the inductance should cancel.  I’ll make a MOSFET board that just has these signal pairs:

  • +12V (for the MOSFET driver) and 0V
  • +5V (for the LVDS receiver) and 0V
  • High side PWM + and –
  • Low side PWM + and –

There’s a ground-loop I need to be wary of between the 12V and 5V rails, really it’s the same 0V rail for both.  I suspect they’ll still need to be connected at both ends.  Add in more of those screw terminals to take the input and output power off-board, and I think we should be set.

Similarly, the INA219 should probably be a separate board, with scope for having a chassis-mounted current shunt.  The connection to the current shunt’s sense output is a low-power connection, so no issue there.  You want to keep it short for accuracy reasons, but a simple twisted pair will work fine.

Sep 272018
 

So, the last few days it’s been overcast.  Monday I had a firmware glitch that caused the mains supply to be brought in almost constantly, so I’d disregard that result.

Basically, the moment the battery dropped below ~12.8V for even a brief second, the mains got brought in.  We were just teetering on the edge of 12.8V all day.  I realised that I really did need a delay on firing off the timer, so I’ve re-worked the logic:

  • If battery drops below V_L, start a 1-hour timer
  • If battery rises above V_L, reset the 1-hour timer
  • If the battery drops below V_CL or the timer expires, turn on the mains charger

That got me better results.  It means V_CL can be quite low, without endangering the battery supply, and V_L can be at 12.8V where it basically ensures that the battery is at a good level for everything to operate.

I managed to get through most of Tuesday until about 4PM, there was a bit of a hump which I think was the solar controller trying to extract some power from the panels.  I really need a good sunny day like the previous week to test properly.

This is leading me to consider my monitoring device.  At the moment, it just monitors voltage (crudely) and controls the logic-level enable input on the mains charger.  Nothing more.  It has done that well.

A thought is that maybe I should re-build this as a Modbus-enabled energy meter with control.  This idea has evolved a bit, enough to be its own project actually.  The thought I have now is a more modular design.

If I take the INA219B and a surface-mount current shunt, I have a means to accurately measure input voltage and current.  Two of these, and I can measure the board’s output too.  Stick a small microcontroller in between, some MOSFETs and other parts, and I can have a switchmode power supply module which can report on its input and output power and vary the PWM of the power supply to achieve any desired input or output voltage or current.

The MCU could be the ATTiny24As I’m using, or a ATTiny861.  The latter is attractive as it can do high-speed PWM, but I’m not sure that’s necessary in this application, and I have loads of SOIC ATTiny24As.  (Then again, I also have loads of PDIP ATTiny861s.)

The board would expose the ICSP pins plus two more for interrupt and chip select, allowing for a simple jig for reprogramming.  I haven’t decided on a topology yet, but the split-pi is looking attractive.  I might start with a buck converter first though.

This would talk to a “master” microcontroller which would provide the UI and Modbus interface.  If the brains of the PSU MCU aren’t sufficient, this could do the more grunty calculations too.

This would allow me to swap out the PSU boards to try out different designs.

Jan 242018
 

So, I now have my little battery monitoring computer.  Shipping wound up being a little more than I was expecting… about US$80… but never mind.  It’s here, arrived safely:

HTLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLFLC
>> TS-BOOTROM - built Jan 26 2017 12:29:21
>> Copyright (c) 2013, Technologic Systems
LLCLLLLLLLFLCLLJUncompressing Linux... done, booting the kernel.
/ts/fastboot file present.  Booting to initramfs instead
Booted from eMMC in 3.15s
Initramfs Web Interface: http://ts7670-498476.local
Total RAM: 128MB
# exit
INIT: version 2.88 booting
[info] Using makefile-style concurrent boot in runlevel S.
[ ok ] Starting the hotplug events dispatcher: udevd.
[ ok ] Synthesizing the initial hotplug events...done.
[ ok ] Waiting for /dev to be fully populated...done.
[ ok ] Activating swap...done.
[....] Checking root file system...fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
/dev/mmcblk2p2: clean, 48540/117600 files, 282972/469760 blocks
done.
[ ok ] Cleaning up temporary files... /tmp /lib/init/rw.
…
ts7670-498476 login: root
Linux ts7670-498476 2.6.35.3-571-gcca29a0+ #1 PREEMPT Mon Nov 27 11:05:10 PST 2017 armv5tejl
TS Root Image 2017-11-27

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

root@ts7670-498476:~# 

The on-board 2GB eMMC has a version of Debian Wheezy on it.  That’ll be going very soon.  For now, all I’ve done is pop the cover, shove a 8GB MicroSD card into one of the on-board slots, wired up a 12V power brick temporarily to the unit, hooked a USB cable into the console port (/dev/ttyAMA0 is wired up to an on-board CP2103 USB-serial chip) and verified that it is alive.

Next step will be to bootstrap Gentoo.  I could use standard ARMv5 stages, or I can build my own, which I might do.  I’ve done this before for mips64el n64 using glibc.  Modern glibc is a goliath on a machine with 128MB RAM though, so I’ll be looking at either µClibc/µClibc-ng or musl… most likely the latter.

That said, 20 years ago, we had the same computing power in a desktop. 🙂

I have a few options for interfacing to the power meters…

  • I²C, SPI, a number of GPIOs and a spare UART on a 2.54mm header inside the case.
  • Another spare UART on the footprint for the GPS module (which my unit does not have)
  • Two RS-232 serial ports with RTS/CTS control lines, exposed via RJ-45 jacks
  • Two CANbus ports on a single RJ-45 jack
  • RS-485 on a port marked “Modbus”

In theory, I could just skip the LPC810s and hook this up directly to the INA219Bs.  I’d have to double check what the TTL voltage is… Freescale love their 1.8V logic… but shifting that up to 3.3V or 5V is not hard.  The run is a little longer than I’m comfortable running I²C though.

The LPC810s don’t feature CANbus, so I think my original plan of doing Modbus is going to be the winner.  I can either do a single-ended UART using a resistor/diode in parallel to link RX and TX to the one UART line, or use RS-485.

I’m leaning towards the latter, if I decide to buy a little mains energy meter to monitor power, I can use the same RS-485 link to poll that.  I have some RS-485 transceivers coming for that.

For now though, I’ll at least get Debian Stretch going… this should not be difficult, as I’ll just use the images I’ve built for work to get things going.  I’m downloading a Jessie image now:

root@ts7670-498476:~# curl https://bne.vrt.com.au/technologicsys/ts7670d-jessie-4.4.1-20160226.dd.xz | xzcat | dd of=/dev/mmcblk0 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  113M    0  544k    0     0   114k      0  0:16:48  0:00:04  0:16:44  116k

Once that is done, I can reboot, re-format the eMMC and get debootstrap going.  I might even publish an updated image while I’m at it.

Dec 252017
 

So, I’m home now for the Christmas break… and the fan in my power supply decided it would take a Christmas break itself.

The power supply was purchased brand new in June… it still works as a power supply, but with the fan seized up, it represents an overheating risk.  Unfortunately, the only real options I have are the Xantrex charger, which cooked my last batteries, or a 12V 20A linear PSU I normally use for my radio station.  20A is just a touch light-on, given the DC-DC converter draws 25A.  It’ll be fine to provide a top-up, but I wouldn’t want to use it for charging up flat batteries.

Now, I can replace the faulty fan.  However, that PSU is under warranty still, so I figure, back it goes!

In the meantime, an experiment.  What happens if I just turn the mains off and rely on the batteries?  Well, so far, so good.  Saturday afternoon, the batteries were fully charged, I unplugged the mains supply.  Battery voltage around 13.8V.

Sunday morning, battery was down to 12.1V, with about 1A coming in off the panels around 7AM (so 6A being drained from batteries by the cluster).

By 10AM, the solar panels were in full swing, and a good 15A was being pumped in, with the cluster drawing no more than 8A.  The batteries finished the day around 13.1V.

This morning, batteries were slightly lower at 11.9V.   Just checking now, I’m seeing over 16A flowing in from the panels, and the battery is at 13.2V.

I’m in the process of building some power meters based on NXP LPC810s and TI INA219Bs.  I’m at two minds what to use to poll them, whether I use a Raspberry Pi I have spare and buy a case, PSU and some sort of serial interface for it… or whether I purchase a small industrial PC for the job.

The Technologic Systems TS-7670 is one that I am considering, given they’ll work over a wide range of voltages and temperatures, they have plenty of UARTs including RS-485 and RS-232, and while they ship with an old Linux kernel, yours truly has ported both U-Boot and the mainline Linux kernel.  Yes, it’s ARMv5, but it doesn’t need to be a speed demon to capture lots of data, and they work just fine for Barangaroo where they poll Modbus (via pymodbus) and M-bus (via python-mbus).

Nov 192017
 

So, this weekend I did plan to run from solar full time to see how it’d go.

Mother nature did not co-operate.  I think there was about 2 hours of sunlight!  This is what the 24 hour rain map looks like from the local weather radar (image credit: Bureau of Meteorology):

In the end, I opted to crimp SB50 connectors onto the old Redarc BCDC1225 and hook it up between the battery harness and the 40A power supply. It’s happily keeping the batteries sitting at about 13.2V, which is fine. The cluster ran for months off this very same power supply without issue: it’s when I introduced the solar panels that the problems started. With a separate controller doing the solar that has over-discharge protection to boot, we should be fine.

I also have mostly built-up some monitoring boards based on the TI INA219Bs hooked up to NXP LPC810s. I have not powered these up yet, plan is to try them out with a 1ohm resistor as the stand-in for the shunt and a 3V rail… develop the firmware for reporting voltage/current… then try 9V and check nothing smokes.

If all is well, then I’ll package them up and move them to the cluster. Not sure of protocols just yet. Modbus/RTU is tempting and is a protocol I’m familiar with at work and would work well for this application, given I just need to represent voltage and current… both of which can be scaled to fit 16-bit registers easy (voltage in mV, current in mA would be fine).

I just need some connectors to interface the boards to the outside world and testing will begin. I’ve ordered these and they’ll probably turn up some time this week.

Nov 132017
 

So, at present I’ve been using a two-charger solution to keep the batteries at full voltage.  On the solar side is the Powertech MP3735, which also does over-discharge protection.  On the mains side, I’m using a Xantrex TC2012.

One thing I’ve observed is that the TC2012, despite being configured for AGM batteries, despite the handbook saying it charges AGM batteries to a maximum 14.3V, has a happy knack of applying quite high charging voltages to the batteries.

I’ve verified this… every meter I’ve put across it has reported it at one time or another, more than 15V across the terminals of the charger.  I’m using SB50 connectors rated at 50A and short runs of 6G cable to the batteries.  So a nice low-resistance path.

The literature I’ve read says 14.8V is the maximum.  I think something has gone out of calibration!

This, and the fact that the previous set-up over-discharged the batteries twice, are the factors that lead to the early failure of both batteries.

The two new batteries (Century C12-105DA) are now sitting in the battery cases replacing the two Giant Energy batteries, which will probably find themselves on a trip to the Upper Kedron recycling facility in the near future.

The Century batteries were chosen as I needed the replacements right now and couldn’t wait for shipping.  This just happened to be what SuperCheap Auto at Keperra sell.

The Giant Energy batteries took a number of weeks to arrive: likely because the seller (who’s about 2 hours drive from me) had run out of stock and needed to order them in (from China).  If things weren’t so critical, I might’ve given those batteries another shot, but I really didn’t have the time to order in new ones.

I have disconnected the Xantrex TC2012.  I really am leery about using it, having had one bad experience with it now.  The replacement batteries cost me $1000.  I don’t want to be repeating the exercise.

I have a couple of options:

  1. Ditch the idea of using mains power and just go solar.
  2. Dig out the Redarc BCDC1225 charger I was using before and hook that up to a regulated power supply.
  3. Source a new 20A mains charger to hook in parallel with the batteries.
  4. Hook a dumb fixed-voltage power supply in parallel with the batteries.
  5. Hook a dumb fixed-voltage power supply in parallel with the solar panel.

Option (1) sounds good, but what if there’s a run of cloudy days?  This really is only an option once I get some supervisory monitoring going.  I have the current shunts fitted and the TI INA219Bs for measuring those shunts arrived a week or so back, just haven’t had the time to put that into service.  This will need engineering time.

Option (2) could be done right now… and let’s face it, its problem was switching from solar to mains.  In this application, it’d be permanently wired up in boost mode.  Moreover, it’s theoretically impossible to over-discharge the batteries now as the MP3735 should be looking after that.

Option (3) would need some research as to what would do the job.  More money to spend, and no guarantee that the result will be any better than what I have now.

Option (4) I’m leery about, as there’s every possibility that the power supply could be overloaded by inrush current to the battery.  I could rig up a PWM circuit in concert with the monitoring I’m planning on putting in, but this requires engineering time to figure out.

Option (5) I’m also leery about, not sure how the panels will react to having a DC supply in parallel to them.  The MP3735 allegedly can take an input DC supply as low as 9V and boost that up, so might see a 13.8V switchmode PSU as a solar panel on a really cloudy day.  I’m not sure though.  I can experiment, plug it in and see how it reacts.  Research gives mixed advice, with this Stack Exchange post saying yes and this Reddit thread suggesting no.

I know now that the cluster averages about 7A.  In theory, I should have 30 hours capacity in the batteries I have now, if I get enough sun to keep them charged.

This I think, will be a week-end experiment, and maybe something I’ll try this weekend.  Right now, the cluster itself is running from my 40A switchmode PSU, and for now, it can stay there.

I’ll let the solar charger top the batteries up from the sun this week.  With no load, the batteries should be nice and full, ready come Friday evening, when I can, gracefully, bring the cluster down and hook it up to the solar charger load output.  If, at the end of the weekend, it’s looking healthy, I might be inclined to leave it that way.

Oct 282017
 

So, this morning I decided to shut the whole lot down and switch to the new solar controller.  There’s some clean-up work to be done, but for now, it’ll do.  The new controller is a Powertech MP3735.  Supposedly this one can deliver 30A, and has programmable float and bulk charge voltages.  A nice feature is that it’ll disconnect the load when it drops below 11V, so finding the batteries at 6V should be a thing of the past!  We’ll see how it goes.

I also put in two current shunts, one on the feed into/out of the battery, and one to the load.  Nothing is connected to monitor these as yet, but some research suggested that while in theory it is just an op-amp needed, that op-amp has to deal with microvolt differences and noise.

There are instrumentation amplifiers designed for that, and a handy little package is TI’s INA219B.  This incorporates aforementioned amplifier, but also adds to that an ADC with an I²C interface.  Downside is that I’ll need an MCU to poll it, upside is that by placing the ADC and instrumentation amp in one package, it should cut down noise, further reduced if I mount the chip on a board bolted to the current shunt concerned.  The ADC measures bus voltage and temperature as well.  Getting this to work shouldn’t be hard.  (Yes, famous last words I know.)

A few days ago, I also placed an order for some more RAM for the two compute nodes.  I had thought 8GB would be enough, and in a way it is, except I’ve found some software really doesn’t work properly unless it has 2GB RAM available (Gitea being one, although it is otherwise a fantastic Git repository manager).  By bumping both these nodes to 32GB each (4×8GB) I can be less frugal about memory allocations.

I can in theory go to 16GB modules in these boxes, but those were hideously expensive last time I looked, and had to be imported.  My debit card maxes out at $AU999.99, and there’s GST payable on anything higher anyway, so there goes that idea.  64GB would be nice, but 32GB should be enough.

The fun bit though, Kingston no longer make DDR3 ECC SO-DIMMs.  The mob I bought the last lot though informed me that the product is no longer available, after I had sent them the B-Pay payment.  Ahh well, I’ve tossed the question back asking what do they have available that is compatible.

Searching for ECC SODIMMs is fun, because the search engines will see ECC and find ECC DIMMs (i.e. full-size).  When looking at one of these ECC SODIMM unicorns, they’ll even suggest the full-size version as similar.  I’d love to see the salespeople try to fit the suggested full-size DIMM into the SODIMM socket and make it work!

The other thing that happens is the search engine sees ECC and see that that’s a sub-string of non-ECC.  Errm, yeah, if I meant non-ECC, I’d have said so, and I wouldn’t have put ECC there.

Crucial and Micron both make it though, here’s hoping mixing and matching RAM from different suppliers in the same bank won’t cause grief, otherwise the other option is I pull the Kingston sticks out and completely replace them.

The other thing I’m looking at is an alternative to OpenNebula.  Something that isn’t a pain in the arse to deploy (like OpenStack is, been there, done that), that is decentralised, and will handle KVM with a Ceph back-end.

A nice bonus would be being able to handle cross-architecture QEMU VMs, in particular for ARM and MIPS targets.  This is something that libvirt-based solutions do not do well.

I’m starting to think about ways I can DIY that solution.  Blockchain was briefly looked at, and ruled out on the basis that while it’d be good for an audit log, there’s no easy way to index it: reading current values would mean a full-scan of the blockchain, so not a solution on its own.

CephFS is stable now, but I’m not sure how file locking works on it.  Then there’s object storage itself, librados.  I’m not sure if there’s a database engine that can interface to that, or maybe to Amazon S3 storage (radosgw can emulate that), that’ll be the next step.  Lots to think about.