Mar 312018
 

So last post, I mentioned about the installation of the new battery charger, which is fed from 240V mains. Over the last few days this charger has held the batteries at a rock-solid 14.4V. Not once did the batteries drop below that voltage setpoint.

So good in fact, the solar charger does no work at all.

By the way, this is what the install looks like. I promised pictures last post.

That’s the DC end … and the nasty AC end is all sealed up…

I will eventually move this to a spot on the back of the rack, but it can sit here for now.

Ultimately, the proper fix to this will be to have the mains-powered charger power off when the sun is up. On the DC output connector, the two rightmost screw terminals go to an opto-isolator that, when powered, shuts off the charger, putting it into stand-by mode. This was one of the reasons I bought this particular unit. The other was the wide range of voltage adjustment.

The question is when to turn on, and when to go to stand-by. Basically if the following expression is true, then turn off the mains:

(V_{batt} > 12.8) \\wedge (V_{solar} > 15)

We do not want solar if the battery is very low, as there’s a possibility that the solar output will not be sufficient.  Likewise, if the sun’s out, we need the mains to keep the battery topped up.

The solar output is nearly always above 15V when the sun is up, so there’s our first clue.  We can safely get to 12.8V before things start going pear shaped on the cluster, so we can use that as our low-voltage safety net.  If both of these conditions are met, then it’s safe to turn off the mains power and rely on solar only.

We need a +5V signal when both these conditions are met.  This very much sounds like the job of a dual-comparator with diode-OR outputs pulling on a 5V pull-up.  Maybe a wee bit of hysteresis on those to prevent flapping, and we should be good.

Unfortunately, to do that, I need to unscrew terminals to feed some wires in.  I don’t feel like doing that just now… we’re packing up to go away for a while, and I think this sort of job can wait until we return.

In the meantime, I’ve done something of a hack.  I mentioned the PSU is adjustable.  I wound Vfloat back to 12V… thus Vboost has gone to 12.8V.  Right now, the mains PSU is showing a green LED, meaning it is in floating mode.

We have good sun right now, and the solar controller is currently boosting the battery.  When the battery gets low, the charging circuitry of the mains PSU should kick in, and bring the battery voltage up, holding it at 12.8V until the sun comes up.  I’ll leave it for now and see how this hack goes.

On other news… I might need to re-consider my NTP server arrangements.  I’m not sure if it’s a quirk of OpenBSD, or of the network here, but it seems OpenNTPD struggles to keep good time.  Never tried using the Advantech PC as a NTP server until now, and I’m also experimenting with using my VPS at Vultr as a NTP server.

http://www.pool.ntp.org/user/Redhatter

Both are drifting like crazy.  I have a GPS module lying around that I might consider hooking up to the TS-7670… perhaps make it a Stratum 1 NTP server on the NTP server pool, then the Advantech can sync to that.

This won’t help the VPS though, and I’m at a loss to explain why a Geode LX800 running on an ADSL link in my laundry, outperforms a VPS in a nicely climate-controlled data centre with gigabit Internet.

But at least now that’s one less job for my aging server.  I’ve also moved mail server duties off the old box onto a VM, so I’ll be looking at the BIOS settings there to see if I can get the box to wake up some time in the evening, let cron run the back-up jobs, then power the whole lot back down again, save some juice.

Mar 282018
 

So tonight I finally got my shiny new power supply installed.

Tuesday night I took it with me along with a cable gland to HSBNE with three items on my agenda:

  • Hooking up a mains power lead to the power input.
  • Getting the newly hooked up lead inspected for electrical safety.
  • Putting some sort of cover over the screw terminals to prevent accidental contact.

I did some digging around in the HSBNE bone yard, and managed to come out with a 10A 240V mains lead, the chassis of a Sonoff TH-10, and a bit of off-cut perspex from the laser cutter to cover the gaping hole in the TH-10 casing.

The 240V mains lead came from someone’s long abandoned project.  Not sure what it was, but it basically was housed in take away food containers, so losing its mains lead is probably a good thing!  The rest of it is there if they want it … whatever it is.

I terminated the 240V lead with fork lugs, ready to go into the screw terminals on the power supply.  A small square of perspex was cut out of the off-cut, and that was sliced into three parts to be glued to the TH-10 case.

The back panel of the TH-10 case had an opening cut in it to allow the screw terminal block to pass through the back.  One of the pieces of perspex had a 14mm hole drilled through it and the cable gland was fitted.  All that was left to do was some hot glue to fix the perspex panels into place over the hole, attach the mains lead and get it checked.

Sadly, I couldn’t find anyone about with an electrical ticket to actually install the cable, so I did that bit myself in the end.  There also weren’t any glue sticks for the glue gun around, and I still had to think about how I was going to secure the TH-10 case to the aluminium of the PSU.

I brought it into my workplace this morning and got one of the people there to check it over (there’s two at my work who have a current electrical ticket).  My cabling job was given the tick of approval, and as a bonus, we had some silicon glue which could fix the TH-10 case to the aluminium panel on the PSU.  Perfect, two birds with one stone.

Once home, I set to work on the 12V end of it.  I needed to go from 4 smallish screw terminals to an Anderson SB50 connector which was intended for 8AWG cable.  In the end, the solution was to use two lengths of twin-12AWG.  One end was terminated with fork lugs, the other was twisted together and soldered into a SB50 connector.  I had to solder it because even doubled over, it was too thin to crimp into the pins securely.

I used about 10cm of 12AWG.  To that SB-50 I made a patch lead with two SB-50s out of figure-8 8AWG cable, about 50cm long to reach the charger input on the battery bank.

I’ll put some pictures up later, but already the silence of this new charger is deafening.  It happily boosted the batteries up to 14.3V and is now letting them sit in constant voltage mode.

We shall see what happens when the sun comes up tomorrow.  Hopefully it just backs right off and lets mother nature do all the work.

Mar 242018
 

So, I’ve now moved the ADSL and router onto the battery supply. This has added an extra amp of load, but really, the solar panel handles this easy.

I dug up one of my spare switchmode PSU modules and then got to thinking about how I’d mount the thing. In the end, double sided tape… to keep the terminals of the adjustment pot from shorting, to a piece of old copper clad PCB from the project graveyard, with some wires soldered on.

The donor PCB already had regions cut out for terminals around the edge, so I could use those for drilling mounting holes. I just made additional terminal pads for soldering the input and output supply rails. Initially I tried putting a 1mF capacitor across the output, but evidently the one I grabbed was crook as it presented a 10Ω load. I don’t think the cause was due to it charging. The PSU has a 220µF there already, so let’s see how it fares.

Fairly simple, +12V comes in via the orange wire into IN+, the “LM2596” steps that down to 5V, comes out the red wire. Screw terminals allow me to swap input and output.

Before hooking it up to the ADSL modem, I made sure to dial it in to 5V.

Meh… who’s going to care about 3mV. 🙂

As it happens, the original PSU puts out 5.3V. I think I’m closer. I can always dial it up if needed.

I put the lid on the case and made up the rest of my wiring harness. One 5A blade fuse, a bit of work around the back of the rack, and it was installed.

In the meantime, I have my old server busy pushing its last daily back-up across to a newly provisioned virtual machine on the cluster.

One problem this presents is that this one VM occupies about 70% of my usable storage cluster capacity. The cases can take one 2.5″ HDD, which unless you’re willing to risk it with Seagate (I’ve had too many of them fail), top-out at 2TB.

There are SSDs too, but I’m not made of money, and I’ve already spent the cost of a small car on this cluster as it is. My thinking is I might look at modifying the cases with a new lid to accept a 3.5″ HDD. If I make the case a wee bit taller, a 3.5″ HDD would fit in the lid, and I could add fans around it to cool it.

The other option is to make external eSATA 3.5″ DIN-rail mounted cases. I did look online, but didn’t see any for sale. That said, space is getting squeezy on that DIN rail, and I do have to be mindful of cooling.

Mar 232018
 

So last week, I came home to no power, which of course meant no Internet because the ADSL service is still on mains power.

This is something that’s been on my TO-DO list for a while now, and I’ve been considering how to go about it.

One way was to run 12V from the server rack to the study where the ADSL is. I’d power the study switch (a Cisco SG-208), the ADSL modem/router (a TP-Link TD-8817) and the border router (an Advantech UNO-1150G).

The border router, being a proper industrial PC is happy with any voltage between 9 and 32V, but will want up to 24W, so there’s 2A. The ADSL modem needs 5V 1A… easy enough, and the switch needs 12V, not sure what power rating. I’m not sure if it’ll take 15V, I’d be more comfortable putting it on an LDO like I did for the Linksys switch and the cluster nodes. (Thanks to @K.C. Lee for the suggestion on those LDOs.)

With all that, we’re looking at 3-4A of current at 12V, over a distance of about 5 metres. The 6 AWG cable I used to hook panels to solar controller is obviously massive overkill here, but CAT5e is not going to cut it… it needs to be something around the realm of 12 AWG… 20 at the smallest.

I have some ~14AWG speaker cable that could do it, but that sounds nasty.

The other approach is to move the ADSL. After finding a CAT3 6P4C keystone insert, I dug out some CAT5e (from a box that literally fell off the back of a truck), slapped my headlamp onto my hard hat, plonked that on my head and got to work.

It took me about an hour to install the new cable. I started by leaving the network-end unterminated, but with enough loose cable to make the distance… worked my way back to the socket location, cut my cable to length, fitted the keystone insert, then went back to the ADSL splitter and terminated the new run.

There was a momentary blip on the ADSL (or maybe that was co-incidence), then all was good.

After confirming I still had ADSL on the old socket, I shut down the router and ADSL modem, and re-located those to sit on top of the rack. Rather than cut new cables, I just grabbed a power board and plugged that in behind the rack, and plugged the router and modem into it. I rummaged around and found a suitably long telephone cable (with 6P6C terminations), and plugged that in. Lo and behold, after a minute or two, I had Internet.

The ugly bit though is that the keystone insert didn’t fit the panel I had, so for now, it’s just dangling in the air. No, not happy about that, but for now, it’ll do. At worst, it only has to last another 3 years before we’ll be ripping it out for the NBN.

The other 3 pairs on that CAT5e are spare.  If I want a 56kbps PSTN modem port, I can wire up one of those to the voice side of the ADSL splitter and terminate it here.

I think tomorrow, I’ll make up a lead that can power the border router directly from the battery.  I have two of these “LM2596HV” DC-DC converter modules.  I’m thinking put an assortment of capacitors (a few beefy electrolytics and some ceramics) to smooth out the DC output, and I can rummage around for a plug that fits the ADSL modem/router and adjust the supply for 5V.  I’ll daisy-chain this off the supply for the border router.

We’re slated for Hybrid Fibre Coax for NBN, when that finally arrives.  I’ll admit I am nowhere near as keen as I was on optic fibre.  Largely because the coax isn’t anywhere near as future-proofed, plus in the event of a lightning strike hitting the ground, optic fibre does not conduct said lightning strike into your equipment; anything metallic, will.

By moving the ADSL to here though, switching to the NBN in the next 12-24 months should be dead easy.  We just need to run it from the junction box outside, nailing it to the joists under the floor boards in our garage through to where the rack is.  No ceiling/wall cavities or confined spaces to worry about.  If the NBN modem needs a different voltage or connector, we just give that DC-DC converter a tweak and replace the output cable to suit.

We of course wait before switching the DC supply until after we’ve proven it working from mains power in the presence of the installer.  Keep the original PSU handy and intact for “debugging” purposes. 😉

There is an existing Foxtel cable, from the days when Foxtel was an analogue service, and I remember the ol’e tug-o-war the installer had with that cable.  It is installed in the lounge room, which is an utterly useless location for the socket, and given the abuse the cable suffered (a few channels were a bit marginal after install), I have no faith in it for an Internet connection.  Thus, a new cable would be best.  I’ll worry about that when the time comes.

On the power supply front… I have my replacement.  The big hold-up with installing it though is I’ll need to get a suicide lead wired up to the mains end, then I need to figure out some way to protect that from accidental contact.  There’s a little clear plastic cover that slips over the contacts, but it is minimal at best.

I’m thinking a 3D printed or molded two-part cover, one part which is glued to the terminal block and provides the anchor point for the second part which can house a grommet and screw into the first block.  That will make the mains end pretty much as idiot-resistant as it’s possible to be.  We’ll give that some thought over the weekend.

The other end, is 15V at most, I’m not nearly so worried about that, as it won’t kill you unless you do something incredibly stupid.

Mar 192018
 

So, on Friday, I had a job to update some documentation.  Specifically, I had to update the code examples on a Confluence document.

No problem… or so I thought.  The issue I faced was that it seems the Confluence application is getting too clever for its own good.  Honestly, I’d be happier with a plain textarea which took some Wiki syntax such as Markdown… or heck… plain HTML!  I use WordPress on this blog here, and while the editor here isn’t bad, I’m thankful that going to the source editor is just a click away, as there’s some things the WYSIWYG editor can’t do well (inline code), or even at all (tables).

The editor in Confluence is much less polished.  Navigating with the arrow keys is an unpredictable experience, sometimes it moves by single lines, sometimes it jumps a page.  Sometimes, starting several lines deep in a code block, a single up-arrow will move you to the line above, sometimes it moves you to some line in a paragraph above the code block.  It’s an exercise in frustration.

Fine, I thought, I’ll just copy and paste the code into qvim.  Highlight… copy… paste… ohh brilliant, it’s now all stuffed onto one line!  Thankfully what I was editing, was JSON, so it’s real easy to re-format that, vim makes it real easy to pipe the buffer contents through an arbitrary external program such as python -m json.tool.  This lacked the flexibility to auto-format the JSON the way the code examples were formatted though, so I made a work-alike that made use of Python’s OrderedDict to sort the keys a bit more logically, and told json.dump to indent the code with 2-space indentation (this is how the existing examples were formatted).

Having done this, I thought I’d make mention to Atlassian about the issues with their editor.  I hit the Feedback link up the top of the page.  I pointed out the issues I was having.  In closing I also pointed out how sluggish their system was.  The desktop PC at work is a 8-core AMD Ryzen 7 1700 with 16GB of DDR4.  Not a slow machine.  Maybe it’s rose-coloured glasses, but I recall having a smoother editing experience with Microsoft Word for Windows 6.0 on my 33MHz 486/DX, which sported a whopping 8MB RAM.  Hot stuff back in 1994.  My present desktop does fine with LibreOffice, and this WordPress blog works fine in it, so I know it’s not my browser or hardware.  Yet Confluence struggles, on a PC that has 8 times the CPU cores, each running at nearly 10 times the clock speed, and with 2048 times the amount of RAM to boot.

I composed my feedback and sent it Friday afternoon.  I left the browser window open while I submitted the feedback, and went home.  This morning, I get in, enter my password to unlock the workstation, and see this:

Atlassian feedback … *still* sending after a whole week-end!

Yep, about 2kB of plain text has taken more than 50 hours to make its way from my desktop to their back-end servers.  Did a feral cat interrupt their RFC-1149 based Internet link?

Mar 172018
 

Last night, I got home, having made a detour on my way into work past Jaycar Wooloongabba to replace the faulty PSU.
It was a pretty open-and-shut case, we took it out of the box, plugged it in, and sure enough, no fan.  After the saleswoman asked the advice of a co-worker, it was confirmed that the fan should be running.
It took some digging, but they found a replacement, and so it was boxed up (in the box I supplied, they didn’t have one), and I walked out the door with PSU No. 3.
I had to go straight to work, so took the PSU with me, and that evening, I loaded it into the top box to transport home on the bicycle.
I get home, and it’s first thing on my mind.  I unlock the top box, get it out, and still decked out in my cycling gear, helmet and all (needed the headlight to see down the back of the rack anyway), I get to work.
I put the ring lugs on, plug it into the wall socket and flick the switch.
Nothing.
Toggle the switch on the front, still nothing.
Tried the other socket on the outlet, unplugging the load, still nothing.  Did the 10km trip from Milton to The Gap kill it?
Frustrated, I figure I’ll switch a light on.  Funny… no lights.
I wander into the study… sure enough, the router, modem and switch are dead as doornails.  Wander out to the MDB outside, saw the main breaker was still on, and tried hitting the test button.  Nothing.
I wander back inside, switching the bike helmet for my old hard hat, since it looks as if I’ll need the headlight a bit longer, then take a sticky beak down the road to see if anyone else is facing the same issue.
Sure enough, I look down the street, everyone’s out.
So there goes my second attempt at bootstrapping Gentoo, and my old server’s uptime.
The power did return about an hour or so later.  The PSU was fine, you don’t think of the mains being out as the cause of your problems.
I’ll re-start my build, but I’m not going to lose another build to failing power.  Nope, had enough of that for a joke.
I could have rigged up a UPS to the TS-7670, but I already have one, and it’s in the very rack where it’ll get installed anyway.  Thus, no time like the present to install it.
I’ll have to configure the switch to present the right VLANs to the TS-7670, but once I do that, it’ll be able to take over the role of routing between the management VLAN and the main network.
I didn’t want to do this in a VM because that means exposing the hosts and the VMs to the management VLAN, meaning anyone who managed to compromise a host would have direct access to the BMCs on the other nodes.
This is not a network with high bandwidth demands, and so the TS-7670 with its 100Mbps Ethernet (built into the SoC; not via USB) is an ideal machine for this task.
Having done this, all that’s left to do is to create a 2GB dual-core VM which will receive the contents of the old server, then that server can be shut down, after 8 years of good service.  I’ll keep it around for storing the on-site backups, but now I can keep it asleep and just wake it up with Wake-on-LAN when I want to make a back-up.
This should make a dint in our electricity bill!
Other changes…

  • Looks like we’ll be upgrading the solar with the addition of another 120W panel.
  • I will be hooking up my other network switches, the ADSL router and ADSL modem up to the battery bank on the cluster, just got to get some suitable cable for doing so.
  • I have no faith in this third PSU, so already, I have a MeanWell HEP-600C coming.  We’ll wire up a suicide lead to it, and that can replace the Powertech MP-3089 + Redarc BCDC1225, as the MeanWell has a remote on/off feature I can use to control it.
Mar 152018
 

Perhaps literally… it has bitten the dust.  Although I wouldn’t call its installed location, dusty.  Once again, the fan in the mains power supply has carked it.

Long-term followers of this project may remember that the last PSU failed the same way.

The reason has me miffed.  All I did with the replacement, was take the PSU out of its box, loosen the two nuts for the terminals, slip the ring lugs for my power lead over the terminals, returned the nuts, plugged it in and turned it on.

While it is running 24×7, there is nothing in the documentation to say this PSU can’t run that way.  This is what the installation looks like.

If it were dusty, I’d expect to be seeing hardware failures in my nodes.

This PSU is barely 4 months old, and earlier this week, the fan started making noises, and requiring percussive maintenance to get started. Tonight, it failed. Completely, no taps on the case will convince it to go.

Now, I need to keep things running until the weekend. I need it to run without burning the house down.

Many moons ago, my father bought a 12V fan for the caravan. Cheap and nasty. It has a slider switch to select between two speeds; “fast” and “slow”, which would be better named “scream like a banshee” and “scream slightly less like a banshee”. The speed reduction is achieved by passing current through a 10W resistor, and achieves maybe a 2% reduction in motor RPM. As you can gather, it proved to be a rather unwelcome room mate, and has seen its last day in the caravan.

This fan, given it runs off 12V, has proven quite handy with the cluster. I’ve got my SB-50 “load” socket hanging out the front of the cluster. A little adaptor to bring that out to a cigarette lighter socket, and I can run it off the cluster batteries. When a build job has gotten a node hot and bothered, sitting this down the bottom of the cluster and aiming it at a node has cooled things down well.

Tonight, it has another task … to try and suck the hot air out of the PSU.

That’s the offending power supply.  A PowerTech MP-3089.  It powers the RedARC BCDC-1225 right above it.  And you can see my kludge around the cooling problem.  Not great, but it should hold for the next 24 hours.

Tomorrow, I think we’ll call past Aspley and pick up another replacement.  I’m leery of another now, but I literally have no choice … I need it now.  Sadly, >250W 12V switchmode PSUs are somewhat rare beasts here in Brisbane.  Altronics don’t sell them that big.  The grinning glasses are no more, and I’m not risking it with the Xantrex charger again.

Long term, I’m already looking at the MeanWell SP-480-12.  This is a PSU module, and will need its own case and mains wiring… but I have no faith in the MP-3089 to not fail and cremate my home of 34 years.

The nice feature of the SP-480-12 is that it does have a remote +12V power-off feature.  Presumably I can drive this with a comparator/output MOSFET, so that when the battery voltage drops below some critical threshold, it kicks in, and when it rises above a high set-point, it drops out.  Simple control, with no MCU involved.  I don’t see a reason to get more fancy than that on the control side, anything more is a liability.

On other news, my gcc build on the TS-7670 failed … so much for the wait.  We’ll try another version and see how we go.

Mar 132018
 

So the house got momentarily power-cycled this morning… I’m at work, minding my own business, next thing the access point emails me this:

Mar 13 09:04:23 Syslogd start up

Now, it only does that for two reasons.  Either someone told it to reboot (not I), or it got hard reset.  Sure enough, log into the old server, and it’s reporting an uptime of 15 minutes.  I get home this evening, and clocks all around are on the blink … literally.

The cluster course is going, power outage?  What power outage?

I did consider wiring up the ADSL modem, router, study switch, and the TS-7670 up to the cluster’s power rails, but haven’t gotten around to doing that.  Alas, I’m not quite there yet.

In any case, even if the TS-7670 had been powered from the solar, I’d have still have temporarily lost the build as the HDD dock I have the hard drive sitting in is mains powered.  It also doesn’t remember its state after a power cycling.  I’d have re-started the build from work, but the HDD remained off when the power came back on.

Never mind.  The downside is now I get to re-start a multi-day build.  The good news though, is that knowing the ebuild file that Portage picked out for compiling gcc; I can resume where it left off.  In this case, it’s using an ebuild from the musl overlay; /root/musl/sys-devel/gcc/gcc-6.4.0-r1.ebuild.

ebuild /root/musl/sys-devel/gcc/gcc-6.4.0-r1.ebuild package will preserve the current working tree and will resume where it was, hopefully without incident.  I’ll be left with a .tbz2; which will be picked up when I run emerge –keep-going -ekv @system.

Mar 122018
 

Well, in my last post I discussed getting OpenADK to build a dev environment on the TS-7670.  I had gotten Gentoo’s Portage installed, and started building packages.

The original plan was to build everything into /tmp/seed, but that requires that all the dependencies are present in the chroot.  They aren’t.  In the end, I decided to go the ill-advised route of compiling Gentoo over the top of OpenADK.

This is an ugly way to do things, but it so far is bearing fruit.  Initially there were some hiccups, and I had to restore some binaries from my OpenADK build tree.  When Gentoo installed python-exec; that broke Portage and I found I had to unpack a Python 2.7 binary I had built earlier then use that to re-install Portage.  I could then continue.

Right now, it’s grinding away at gcc; which was my nemesis from the beginning.  This time though, it successfully built xgcc and xg++; which means it has compiled itself using the OpenADK-supplied gcc; and now is building itself using its self-built binaries.  I think it does two or three passes at this.

If it gets through this, there’s about 65 packages to go after that.  Mostly small ones.  I should be able to do a ROOT=/tmp/seed emerge -ek @system then tar up /tmp/seed and emerge catalyst.  I have some wrapper scripts around Catalyst that I developed back when I was responsible for doing the MIPS stages.  These have been tweaked to do musl builds, and were used to produce these x86 stages.  The same will work for ARMv5.

It might be another week of grinding away, but we should get there. 🙂