May 2016

Solar Cluster: ATTiny24A charge controller: Fail so far

Well, last weekend I had finally acquired the bits after some delays getting everything together. This included some ATTiny24As, some P-channel MOSFETs and some 5V DC-DC PSUs.

Last weekend I got around to building the PCB, and yes, I’m not as handy with a dalo pen as I used to be. That, and the ferric chloride I was using had seen better days (the bottle has “Dick Smith Electronics” printed on it — enough said).

So I spent much time last weekend finding shorts between tracks and attacking those with a sharp knife. Last Sunday I managed to get something built, but not tested.

Today, I got around to testing it. At first I plugged it into a bank of 6 AA cells, and got 0.8V across the power input. WTF? Okay, maybe the DC-DC converter needs a little more current. So I wire up a cable loom with a 5A blade fuse and 30A Andersen connector. Plug everything together: *BOOM*, a tantalum capacitor blows up!

The tantalum was a 330µF that was scavenged from an old computer motherboard, probably only rated for 10V, and when you over-voltage a tantalum, they do throw a tantrum!

Lesson learned: don’t use those scavenged parts for 12V! I swap that out for an electrolytic (rated at 16V).

The reason for my high voltage drop though? A faulty MOSFET. I found that by de-soldering the tab on both input MOSFETs until the problem disappeared. Then pressing down the faulty one caused the short to re-appear. Strange, as the MOSFET has never seen use.

I proceeded minus a MOSFET for a bit, see if I could get some code into the MCU. I was able to program that okay, but then fun came when I tried to use the timer interrupt: including the timer ISR would cause the MCU to not boot. It’d just sit there.

The problem disappeared if I compiled my code without optimisation, or at -O1, but would return at -O2 or -Os (I was using -Os). So something in my toolchain was broken for the ATTiny24A.

Whilst waiting for a toolchain re-build, I decided to tackle the faulty MOSFET. I had one spare, so I carefully soldered that into place, only for the original issue to re-appear, so now I’m completely lost.

I guess next weekend, I’ll take a closer look and the cause will become obvious, but right now I’m more confused than a moth in a light shop!

Improved Helmets: Case Study: Phillip Hughes

It appears that Cricket Australia might release the findings of a safety review into the death of Phillip Hughes. This could be an interesting report to read when it comes out, in that it hints at what parts of the head are particularly vulnerable to hits from hard objects.

If you consider a cricket helmet for a moment, and contrast that to the coverage area of a bicycle helmet. Both are designed with the assumption of a collision with an object coming from in front, and so the back of the neck was not an area given much thought.

Motorcycle helmets do a lot better here in almost all cases except the “half helmets”: tick-a-box helmets if ever I saw one! More recent designs come down much lower on the head. Interestingly, professional cricket did originally start with motorcycle helmets, but found them too heavy.

Within the construction of even a motorcycle helmet however, it’s interesting to note where the foam exists: and it’s thickest around the top of the head. The sides where one’s ears would be, the foam gets thinner. Possibly a compromise for acoustic reasons, but is it significant?

Bicycle helmets however, with the exception of MTB type ones, don’t seem to cover this very well from what I’ve seen. Perhaps the back and sides needs a bit more attention. I guess I’ll have to have a gander at the report when it is released.

Solar cluster: Software stack beginning to take shape.

So, after putting aside the charge controller for now, I’ve taken some time to see if I can get the software side of things into shape.

In the midst of my development, I found a small wiring fault that was responsible for blowing a couple of fuses. A small nick in the sheath of the positive wire in a power cable was letting the crimp part of a DC barrel connector contact +12V. A tweak of that crimp and things are back to normal. I’ve swapped all the 10A fuses for 5A ones, since the regulators are only rated at 7.5A.

The VLANs are assigned now, and I have bonding going between the two pairs of Ethernet devices. In spite of the switch only supporting 4 LAGs, it seems fine with me doing LACP on effectively 10 LAGs. I’ll see how it goes.

The switch has 5 ports spare after plugging in all 5 nodes and a 16-port switch for the IPMI subnet. One will be used for a management interface so I can plug a laptop in, and the others will be paired with LACP for linking to my two existing Cisco SG200-8s.

One of the goals of this project is to try and push the performance of Ceph. In the office, we tried bare Ceph, and found that, while it’s fine for sequential I/O, it suffers a bit with random read/writes, and Windows-based HyperV images like to do a lot of random reads/writes.

Putting FlashCache in the mix really helped, but I note now, it’s no longer maintained. EnhanceIO had only just forked when I tried FlashCache, now it seems that’s the official successor.

There are two alternatives to FlashCache/EnhanceIO: bcache and dm-cache.

I’ll rule out bcache now as it requires the backing image be “formatted” for use. In other words, the backing image is not a raw image, but some proprietary (to bcache) format. This isn’t unworkable, but it raises concerns with me about portability: if I migrate a VM, do I need to migrate its cache too, or is it sufficient to cleanly shut down and detach the bcache device before re-assembling it on the new host?

By contrast, dm-cache and EnhanceIO/FlashCache work with raw backing images, making them much more attractive. Flush the cache before migration or use writethru mode, and all should be fine. dm-cache does however require a separate metadata device: messy, but not unworkable. We can provision the cache-related devices we need using LVM2, and use the kernel-mode Rados block device as our backing image.

So I think my caching subsystem is a two-horse race: dm-cache or EnhanceIO. I guess we’ll give them a try and see how they go.

For those following along at home, if you’re running kernel >4.3, you might want use this fork of EnhanceIO due to changes in the kernel block I/O layer.

To manage the OpenNebula master node, I’ve installed corosync/pacemaker. Normally these are used with DR:BD, however I figure Ceph can fulfil that role. The concepts are similar: it’s a shared block device. I’m not sure if it’ll be LXC, Docker or a VM at this point that “contains” the server, but whatever it is, it should be possible for it to have its root FS and data on Ceph.

I’m leaning towards LXC for this. Time for some more experimentation.

Improved Helmets: Project background

Well, looks like this project is very much thrust into the spotlight having been covered in Hacklet 105 . Mine’s probably the least technical of the lot, it’s definitely worth having a look at what the others are doing, as there’s some really innovative ideas there. Many thanks to @Mike Szczys and @Adam Fabio for the shout-out. 🙂

One thing I haven’t done with this project yet, is to actually post the background of why I’ve started this. A big part of this was I wanted to get permission from the family of a work colleague of mine so that I could mention him by name, but at this stage, permission has not been given, so I have to keep things anonymous.

On the 12th of February, a colleague of mine was cycling to work over the Go Between Bridge here in Brisbane when he lost control on a bend as the bridge joins the Bicentennial Bikeway. This is an off-road, dedicated cycleway, so no cars, and supposedly no pedestrians, however many seem to not understand what a sign with a bicycle symbol and the letters O, N, L, Y mean. (I usually ride past and comment: “Funny bike you’re riding!”. Since this accident though, I intend to be a lot more assertive.)

(Above: the crash scene. That blood smear is still visible on the path today.)

I’m no crash investigator, but I did study physics, and I cycle as my sole means of transport myself, having no driver’s license. (And no interest in getting one either.) I’m familiar with what that bridge is like to cycle over, having done it many times shortly after it opened when I worked at West End.

Looking at the scene though, it was apparent to me that my colleague was going much faster than was sensible for that stretch of road, and something caused him to lose control just prior to the bend.

The resulting impact with the railing was devastating: in addition to a few broken bones elsewhere in the body, he suffered skull fractures, and what I understand now to be a Coup-Contrecoup injury to the brain.

I remember that morning arriving at work early (we both were early birds, and had he not crashed, he would have beaten me that morning), sitting down at my desk and preparing to do battle with U-Boot and an industrial PC, when at 6:34AM, the office phone rings. It was then I learned that my colleague was in a serious condition in hospital, and I then found myself frantically looking for contact details for his wife. (Which were nowhere to be found.)

We later learned he’d never regain consciousness, having lost all executive function in the brain. The only bits that worked, were the bits responsible for low-level muscle control. From bright mind, to persistent vegetative state. He passed away about a fortnight after his accident.

During his brief time in ICU, we were told by one of the people there that these sorts of injuries were common in bicycle and motorcycle accidents. That worried me.

That tells me that perhaps, something is wrong with these blocks of foam we insist on strapping to our heads, and that we’ve missed something. This is one of the first goals I’d like to pinpoint, but so far, has been the most difficult: trying to get hold of data that would statistically prove or disprove how “common” these injuries are.

There’s no point in protecting the skull itself if the brain is to get shaken around to the point that the person winds up with total mental incapacitation.

Research seems to suggest that helmets have had a big hand in reducing the incidents of these injuries, but the fact that it’s still “common”, seems to suggest there’s lots more work to be done.

The standards are focussed on linear acceleration, and single impacts at no more than about 20km/hr. Is that sufficient? I regularly find myself doing 40, and I’m no speed demon. (Hell, I’ve accidentally found myself doing 71km/hr once!) I think it’s time the standards were revised. The question is: how?

My colleague was a key member of our team, and one of the brighter minds I know. While he shouldn’t have taken that bend at such speed and expect to get away with it, he did not deserve to die. I can’t save him, but perhaps I can help save someone else. That’s what this project is about.