Mar 232018
 

So last week, I came home to no power, which of course meant no Internet because the ADSL service is still on mains power.

This is something that’s been on my TO-DO list for a while now, and I’ve been considering how to go about it.

One way was to run 12V from the server rack to the study where the ADSL is. I’d power the study switch (a Cisco SG-208), the ADSL modem/router (a TP-Link TD-8817) and the border router (an Advantech UNO-1150G).

The border router, being a proper industrial PC is happy with any voltage between 9 and 32V, but will want up to 24W, so there’s 2A. The ADSL modem needs 5V 1A… easy enough, and the switch needs 12V, not sure what power rating. I’m not sure if it’ll take 15V, I’d be more comfortable putting it on an LDO like I did for the Linksys switch and the cluster nodes. (Thanks to @K.C. Lee for the suggestion on those LDOs.)

With all that, we’re looking at 3-4A of current at 12V, over a distance of about 5 metres. The 6 AWG cable I used to hook panels to solar controller is obviously massive overkill here, but CAT5e is not going to cut it… it needs to be something around the realm of 12 AWG… 20 at the smallest.

I have some ~14AWG speaker cable that could do it, but that sounds nasty.

The other approach is to move the ADSL. After finding a CAT3 6P4C keystone insert, I dug out some CAT5e (from a box that literally fell off the back of a truck), slapped my headlamp onto my hard hat, plonked that on my head and got to work.

It took me about an hour to install the new cable. I started by leaving the network-end unterminated, but with enough loose cable to make the distance… worked my way back to the socket location, cut my cable to length, fitted the keystone insert, then went back to the ADSL splitter and terminated the new run.

There was a momentary blip on the ADSL (or maybe that was co-incidence), then all was good.

After confirming I still had ADSL on the old socket, I shut down the router and ADSL modem, and re-located those to sit on top of the rack. Rather than cut new cables, I just grabbed a power board and plugged that in behind the rack, and plugged the router and modem into it. I rummaged around and found a suitably long telephone cable (with 6P6C terminations), and plugged that in. Lo and behold, after a minute or two, I had Internet.

The ugly bit though is that the keystone insert didn’t fit the panel I had, so for now, it’s just dangling in the air. No, not happy about that, but for now, it’ll do. At worst, it only has to last another 3 years before we’ll be ripping it out for the NBN.

The other 3 pairs on that CAT5e are spare.  If I want a 56kbps PSTN modem port, I can wire up one of those to the voice side of the ADSL splitter and terminate it here.

I think tomorrow, I’ll make up a lead that can power the border router directly from the battery.  I have two of these “LM2596HV” DC-DC converter modules.  I’m thinking put an assortment of capacitors (a few beefy electrolytics and some ceramics) to smooth out the DC output, and I can rummage around for a plug that fits the ADSL modem/router and adjust the supply for 5V.  I’ll daisy-chain this off the supply for the border router.

We’re slated for Hybrid Fibre Coax for NBN, when that finally arrives.  I’ll admit I am nowhere near as keen as I was on optic fibre.  Largely because the coax isn’t anywhere near as future-proofed, plus in the event of a lightning strike hitting the ground, optic fibre does not conduct said lightning strike into your equipment; anything metallic, will.

By moving the ADSL to here though, switching to the NBN in the next 12-24 months should be dead easy.  We just need to run it from the junction box outside, nailing it to the joists under the floor boards in our garage through to where the rack is.  No ceiling/wall cavities or confined spaces to worry about.  If the NBN modem needs a different voltage or connector, we just give that DC-DC converter a tweak and replace the output cable to suit.

We of course wait before switching the DC supply until after we’ve proven it working from mains power in the presence of the installer.  Keep the original PSU handy and intact for “debugging” purposes. 😉

There is an existing Foxtel cable, from the days when Foxtel was an analogue service, and I remember the ol’e tug-o-war the installer had with that cable.  It is installed in the lounge room, which is an utterly useless location for the socket, and given the abuse the cable suffered (a few channels were a bit marginal after install), I have no faith in it for an Internet connection.  Thus, a new cable would be best.  I’ll worry about that when the time comes.

On the power supply front… I have my replacement.  The big hold-up with installing it though is I’ll need to get a suicide lead wired up to the mains end, then I need to figure out some way to protect that from accidental contact.  There’s a little clear plastic cover that slips over the contacts, but it is minimal at best.

I’m thinking a 3D printed or molded two-part cover, one part which is glued to the terminal block and provides the anchor point for the second part which can house a grommet and screw into the first block.  That will make the mains end pretty much as idiot-resistant as it’s possible to be.  We’ll give that some thought over the weekend.

The other end, is 15V at most, I’m not nearly so worried about that, as it won’t kill you unless you do something incredibly stupid.

Jul 232017
 

So, having got some instances going… I thought I better sort out the networking issues proper.  While it was working, I wanted to do a few things:

  1. Bring a dedicated link down from my room into the rack directly for redundancy
  2. Define some more VLANs
  3. Sort out the intermittent faults being reported by Ceph

I decided to tackle (1) first.  I have two 8-port Cisco SG-200 switches linked via a length of Cat5E that snakes its way from our study, through the ceiling cavity then comes up through a small hole in the floor of my room, near where two brush-tail possums call home.

I drilled a new hole next to where the existing cable entered, then came the fun of trying to feed the new cable along side the old one.  First attempt had the cable nearly coil itself just inside the cavity.  I tried to make a tool to grab the end of it, but it was well and truly out of reach.  I ended up getting the job done by taping the cable to a section of fibreglass tubing, feeding that in, taping another section of tubing to that, feed that in, etc… but then I ran out of tubing.

Luckily, a rummage around, and I found some rigid plastic that I was able to tape to the tubing, and that got me within a half-metre of my target.  Brilliant, except I forgot to put a leader cable through for next time didn’t I?

So more rummaging around for a length of suitable nylon rope, tape the rope to the Cat5E, haul the Cat5E out, then grab another length of rope and tape that to the end and use the nylon rope to haul everything back in.

The rope should be handy for when I come to install the solar panels.

I had one 16-way patch panel, so wound up terminating the rack-end with that, and just putting a RJ-45 on the end in my room and plugging that directly into the switch.  So on the shopping list will be some RJ-45 wall jacks.

The cable tester tells me I possibly have brown and white-brown switched, but never mind, I’ll be re-terminating it properly when I get the parts, and that pair isn’t used anyway.

The upshot: I now have a nice 1Gbps ring loop between the two SG-200s and the LGS326 in the rack.  No animals were harmed in the formation of this ring, although two possums were mildly inconvenienced.  (I call that payback for the times they’ve held the Marsupial Olympics at 2AM when I’m trying to sleep!)

Having gotten the physical layer sorted out, I was able to introduce the upstairs SG-200 to the new switch, then remove the single-port LAG I had defined on the downstairs SG-200.  A bit more tinkering going, and I had a nice redundant set-up: setting my laptop to ping one of the instances in the cluster over WiFi, I could unplug my upstairs trunk, wait a few seconds, plug it back in, wait some more, unplug the downstairs trunk, wait some more again, then plug in back in again, and not lose a single ICMP packet.

I moved my two switches and my AP over to the new management VLAN I had set up, along side the IPMI interfaces on the nodes.  The SG-200s were easy, aside from them insisting on one port being configured with a PVID equal to the management VLAN (I guess they want to ensure you don’t get locked out), it all went smoothly.

The AP though, a Cisco WAP4410N… not so easy.  In their wisdom, and unlike the SG-200s, the management VLAN settings page is separate from the IP interface page, so you can’t change both at the same time.  I wound up changing the VLAN, only to find I had locked myself out of it.  Much swearing at the cantankerous AP and wondering how could someone overlook such a fundamental requirement!  That, and the switch where the AP plugs in, helpfully didn’t add the management VLAN to the right port like I asked of it.

Once that was sorted out, I was able to configure an IP on the old subnet and move the AP across.

That just left dealing with the intermittent issues with Ceph.  My original intention with the cluster was to use 802.3AD so each node had two 2Gbps links.  Except: the LGS326-AU only supports 4 LAGs.  For me to do this, I need 10!

Thankfully, the bonding support in the Linux kernel has several other options available.  Switching from 802.3ad to balance-tlb, resolved the issue.

slaves_bond0="enp0s20f0 enp0s20f1"
slaves_bond1="enp0s20f2 enp0s20f3"
config_bond0="null"
config_bond1="null"
config_enp0s20f0="null"
config_enp0s20f1="null"
config_enp0s20f2="null"
config_enp0s20f3="null"
rc_net_bond0_need="net.enp0s20f0 net.enp0s20f1"
rc_net_bond1_need="net.enp0s20f2 net.enp0s20f3"
mode_bond0="balance-tlb"
mode_bond1="balance-tlb"

I am now currently setting up a core router instance (with OpenBSD 6.1) and a OpenNebula instance (with Gentoo AMD64/musl libc).