May 242019
 

Recently, I had a failure in the cluster, namely one of my nodes deciding to go the way of the dodo. I think I’ve mostly recovered everything from that episode.

I bought some new nodes which I can theoretically deploy as spare nodes, Core i5 Intel NUCs, and for now I’ve temporarily decommissioned one of my compute nodes (lithium) to re-purpose its motherboard to get the downed storage node back on-line. Whilst I was there, I went and put a new 2TB HDD in… and of course I left the 32GB RAM in, so it’s pretty much maxxed out.

I’d like to actually make use of these two new nodes, however I am out of switch capacity, with all 26 ports of the Linksys LGS-326AU occupied or otherwise reserved. I did buy a Netgear GS748T with the intention of moving across to it, but never got around to doing so.

The principle matter here being that the Netgear requires a wee bit more power. AC power ratings are 100-250V, 1.5A max. Now, presumably the 1.5A applies at the 100V scale, that’s ~150W. Some research suggested that internally, they run 12V, that corresponds to about 8.5A maximum current.

This is a bit beyond the capabilities of the MIC29712s.

I wound up buying a DC-DC power supply, an isolated one as that’s all I could get: the Meanwell SD-100A-12. This theoretically can take 9-18V in, and put out 12V at up to 8.5A. Perfect.

Due to lack of time, it sat there. Last week-end though, I realised I’d probably need to consider putting this thing to use. I started by popping open the cover and having a squiz inside. (Who needs warranties?)

The innards of the GS-748Tv5, ruler for scale

I identified the power connections. A probe around with the multimeter revealed that, like the Linksys, it too had paralleled conductors. There were no markings on the PSU module, but un-plugging it from the mainboard and hooking up the multimeter whilst powering it up confirmed it was a 12V output, and verified the polarity. The colour scheme was more sane: Red/Yellow were positive, Black/Blue were negative.

I made a note of the pin-out inside the case.

There’s further DC-DC converters on-board near the connector, what their input range is I have no idea. The connector on the mainboard intrigued me though… I had seen that sort of connector before on ATX power supplies.

The power supply connector, close up.

At the other end of the cable was a simple 4-pole “KK”-like connector with a wider pin spacing (I think ~3mm). Clearly designed with power capacity in mind. I figured I had three options:

  1. Find a mating connector for the mainboard socket.
  2. Find a mating header for the PSU connector.
  3. Ram wires into the plug and hot-glue in place.

As it happens, option (1) turned out easier than I thought it would be. When I first bought the parts for the cluster, the PicoPSU modules came with two cables: one had the standard SATA and Molex power connectors for powering disk drives, the other came out to a 4-pin connector not unlike the 6-pole version being used in the switch.

Now you’ll note of those 6 poles, only 4 are actually populated. I still had the 4-pole connectors, so I went digging, and found them this evening.

One of my 4-pole 12V connectors, with the target in the background.

As it happens, the connectors do fit un-modified, into the wrong 4 holes — if used unmodified, they would only make contact with 2 of the 4 pins. To make it fit, I had to do a slight modification, putting a small chamfer on one of the pins with a sharp knife.

After a slight modification, the connector fits where it is needed.

The wire gauge is close to that used by the original cable, and the colour coding is perfect… black corresponds to 0V, yellow to +12V. I snipped off the JST-style connector at the other end.

I thought about pulling out the original PSU, but then realised that there was a small hole meant for a Kensington-style lock which I wasn’t using. No sharp edges, perfect for feeding the DC cables through. I left the original PSU in-situ, and just unplugged its DC output.

The DC input leads snake through the hole that Netgear helpfully provided.

Bringing the DC power input to the outside.

Before putting the screws in, I decided to give this a test on the bench supply. The switch current fluctuates a bit when booting, but it seems to settle on about 1.75A or so. Not bad.

Testing the switch running on 12V

Terminating this, I decided to use XT-60 connectors. I wanted something other than the 30A “powerpoles” and their larger 50A cousins that are dotted throughout the cluster, as this needed to be regulated 12V. I did not want to get it mixed up with the raw 12V feed from the batteries.

I ran some heavier gauge cable to the DC-DC PSU, terminated with the mating XT-60 connector and hooked that up to my PSU. Providing it with 12V, I dialled the output to 12V exactly. I then gave it a no-load test: it held the output voltage pretty good.

Next, I hooked the switch up to the new PSU. It fired up and I measured the voltage now under load: it still remained at 12V. I wound the voltage down to 9V, then up to 15V… the voltage output never shifted. At 9V, the current consumption jumps up to about 3.5A, as one would expect.

Otherwise, it seemed to be content to draw under 2A so the efficiency of the DC-DC converter is pretty good.

I’ll need to wire in a new fuse box to power everything, but likely the plan will be to decommission the 16-port 100Mbps switch I use for the management network, slide the 48-port switch in its place, then gradually migrate everything across to the new switch.

Overall, the modding of this model switch was even less invasive than that of the Linksys. It’s 100% reversible. I dare say having posted this, there’ll be a GS748Tv6 that’ll move the 240V PSU to the mainboard, but for now at least, this is definitely a switch worth looking at if 12V operation is needed.

May 242019
 

So, in my workplace we’re developing a small energy/water metering device, which runs on a 6LoWPAN network and runs OpenThread-based firmware. The device itself is largely platform-agnostic, requiring a simple CoAP gateway to provide it with a configuration blob and history write end-point. The gateway service I’m running is intended to talk to WideSky.

One thorny issue we need to solve before deploying these things in the wild, is over-the-air updates. So we need a way to transfer the firmware image to the device over the mesh network. Obviously, this firmware image needs to be digitally signed and hashed using a strong cryptographic hash — I’ve taken care of this already. My problem is downloading an image that will be up to 512kB in size.

Thankfully, the IETF has thought of this, the solution to big(gish) files over CoAP is Block-wise transfers (RFC-7959). This specification gives you the ability to break up a large payload into smaller chunks that are powers of two in size between 16 and 2048 bytes.

6LoWPAN itself though has a limitation: the IEEE 802.15.4 radio specification it is built on cannot send frames bigger than 128 bytes. Thus, any message sent via this network must be that size or smaller. IPv6 has a minimum MTU of 1280 bytes, so how do they manage? They fragment the IPv6 datagram into multiple 802.15.4 frames. The end device re-assembles the fragments as it receives them.

The catch is, if a fragment is lost, you lose the entire datagram, there’s no repeats of individual fragments, the entire datagram must be re-sent. The question in my mind was this: Is it faster to rely on block-wise transfers to break the payload up and make lots of small requests, or is it faster to rely on 6LoWPAN fragmentation?

The test network here has a few parts:

  • The target device, which will be downloading a 512kB firmware image to a separate SPI flash chip.
  • The border router, which provides a secure IPv6 tunnel to a cloud host.
  • The cloud server which runs the CoAP service we’ll be talking to.

The latency between the office (in Brisbane) and the cloud server (in Sydney) isn’t bad, about 30~50ms. The CoAP service is built using node-coap and coap-polka.

My CoAP requests have some overheads:

  • The path being downloaded is about 19 bytes long.
  • There’s an authentication token given as a query string, so this adds an additional 12 bytes.

The data link is not 100% reliable, with the device itself dropping some messages. This is leading to some retransmits. The packet loss is not terrible, but probably in the region of around 5%. Over this slightly lossy link, I timed the download of my 512kB firmware image by my device with varying block size settings.

Note that node-coap seems to report a “Bad Option” error for szx=0 and szx=7, even though both are legitimately within specification. (I’d expect node-coap to pass szx=7 through and allow the application to clamp it to 6, but it seems node-coap‘s behaviour is to report “Bad option”, but then pass the payload through anyway.)

Size Exponent (szx)Block sizeStart time (UTC)End time (UTC)Effective data rate
6102403:27:0403:37:52809B/s
551203:41:2503:53:40713B/s
425603:57:1504:16:16458B/s
312804:17:4604:54:17239B/s
26404:56:0905:54:53150B/s
13223:31:3301:39:4468B/s

It remains to be seen how much multiple hops and outdoor atmospherics affect the situation. A factor here is how quickly the device can turn-around between processing a response and sending the next request, which in this case is governed by the speed of the SPI flash and its driver.

Effects on “busy” networks

So, I haven’t actually done any hard measurements here, but doing testing on a busier network with about 30 nodes, the block size equation tips more in favour of a smaller block size.

I’ll try to quantify the delays at some point, but right now 256 byte blocks are the clear winner, with 512 and 1024 byte block transfers proving highly unreliable. The speed advantage between 1k and 512 bytes in ideal conditions was a big over 10%… which really doesn’t count for much. At 256 bytes, the speed difference was about 43%, quite significant. You’re better off using 512-byte blocks if the network is quiet.

On a busy network, with all the retransmissions, smaller is better. No hard numbers yet, but right now at 256 byte blocks, the effective rate is around 118 bytes/sec. I’ll have to analyse the logs a bit to see where 512/1024 byte block sizes sat, but the fact they rarely completed says it all IMO.

Slow and steady beats fast and flakey!