Jan 282019
 

My cloud computing cluster like all cloud computing clusters of course needs a storage back-end. There were a number of options I could have chosen, but the one I went with in the end was Ceph, and so far, it’s ran pretty well.

Lately though, I was starting to get some odd crashes out of ceph-osd. I was running release 10.2.3, which is quite dated now, this is one of the earlier Jewel releases. Adding to the fun, I’m running btrfs as my filesystem on the OS and the OSD, and I’m running it all on Gentoo. On top of this, my monitor nodes are my OSDs as well.

Not exactly a “supported” configuration, never mind the hacks done at hardware level.

There was also a nagging issue about too many placement groups in the Ceph cluster. When I first established the cluster, I christened it by dragging a few of my lxc containers off the old server and making them VMs in the cluster. This was done using libvirt and virt-manager. These got thrown into a storage pool called transitional-inst, with a VLAN set aside for the VMs to use. When I threw OpenNebula on, I created another Ceph pool, one for its images. The configuration of these lead to the “too many placement groups” warning, which until now, I just ignored.

This weekend was a long weekend, for controversial reasons… and so I thought I’ll take a snapshot of all my VMs, download those snapshots to a HDD as raw images, then see if I can fix these issues, and migrate to Ceph Luminous (v12.2.10) at the same time.

Backing up

I was going to be doing some nasty things to the cluster, so I thought the first thing to do was to back up all images. This was done by using rbd snap create pool/image@date to create a snapshot of an image, then rbd export pool/image@date /path/to/storage/pool-image.img before blowing away the snapshot with rbd snap rm pool/image@date.

This was done for all images on the Ceph cluster, stashing them on a 4TB hard drive I had bought for the purpose.

Getting things ready

My cluster is actually set up as a distcc cluster, with Apache HTTP server instances sharing out distfiles and binary package repositories, so if I build packages on one, I can have the others fetch the binary packages that it built. I started with a node, and got it to update all packages except Ceph. Made sure everything was up-to-date.

Then, I ran emerge -B =ceph-10.2.10-r2. This was the first step in my migration, I’d move to the absolute latest Jewel release available in Gentoo. Once it built, I told all three storage nodes to install it (emerge -g =ceph-10.2.10-r2). This was followed up by a re-start of the mon daemons on each node (one at a time), then the mds daemons, finally the osd daemons.

Resolving the “too many placement groups” warning

To resolve this, I first researched the problem. An Internet search lead me to this Stack Overflow post. In it, it was suggested the problem could be alleviated by making a new pool with the correct settings, then copying the images over to it and blowing away the old one.

As it happens, I had an easier solution… move the “transitional” images to OpenNebula. I created empty data blocks in OpenNebula for the three images, then used qemu-img convert -p /path/to/image.img rbd:pool/image to upload the images.

It was then a case of creating a virtual machine template to boot them. I put them in a VLAN with the other servers, and when each one booted, edited the configuration with the new TCP/IP settings.

Once all those were moved across, I blew away the old VMs and the old pool. The warning disappeared, and I was left with a HEALTH_OK message out of Ceph.

The Luminous moment

At this point I was ready to try migrating. I had a good read of the instructions beforehand. They seemed simple enough. I prepared as I did before by updating everything on the system except Ceph, then, telling Portage to build a binary package of Ceph itself.

Then I deployed the binary to the three nodes.

First step was to re-start the monitors… this went smoothly, I just did a /etc/init.d/ceph-mon.${HOST} restart on each one individually, and after a brief moment, quorum was re-established. I then deployed a manager daemon to each one — basically I just “copied” my monitor symbolic link, changing mon to mgr, added it to OpenRC’s list, then started them. No problems.

The OSDs though were still running the Jewel release.

I proceeded as before, trying a re-start of the first OSD. After a while it hadn’t come back…

2019-01-27 14:42:59.745860 7f28fac06e00 -1 filestore(/var/lib/ceph/osd/ceph-0) _detect_fs(1197): deprecated btrfs support is not ena
bled

Ohh bugger, so no btrfs support. This is where the fun began. At this point I was a bit flustered and thought I’d have to either migrate these nodes to XFS, or to BlueStore. So immediately I started looking at the BlueStore migration documentation, as I did not want to risk re-starting the other two OSDs and losing access to my data!

A hasty BlueStore migration

So, I started this by doing the ceph osd set out 0 to start my now downed OSD 0 on the path of migration. The fact it was already down didn’t click with me. I then tried running ceph osd safe-to-destroy 0, only to be told Error EINVAL: (22) Invalid argument.

Uhh ohh, this isn’t good. I waited a bit, but also part of me said: there should be a copy of everything on this node, on at least one of the other two nodes. I had configured it to maintain at least two copies of everything, so even if this node went up in smoke, the data should be recoverable.

With great trepidation, I continued and tried destroying the OSD, then creating a BlueStore one in its place… only to have the ceph-volume command blow up. It couldn’t find the keyring, then when I got that sorted out, it was failing to talk to systemd, then when I found the --no-systemd argument, it still failed because of LVM. I therefore realised I needed two things:

  1. I needed the bootstrap-osd keyring that ceph-deploy normally creates.
  2. The lvmetad daemon must be running.

For (1), this is taken care of with the following commands:

# ceph auth add client.bootstrap-osd --cap mon 'profile bootstrap-osd
# mkdir /var/lib/ceph/bootstrap-osd
# ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

As for (2), install sys-fs/lvm and add lvmetad to your start-up services. Also add lvm, as you’ll want that at boot. (I learned this later.)

After doing that, the following command worked:

ceph-volume lvm create --bluestore --data /dev/sdb \
--osd-id 0 --no-systemd

The --no-systemd is important on Gentoo with OpenRC as there is no systemctl binary. Once I did that, I found I could start my OSD again. Data recovery began at once. The data recovery was an overnight effort — it took with my hardware until 3PM today to migrate all the placement groups over to the newly re-formatted OSD.

Migrating the other nodes

For now, they still run btrfs. In my “ohh crap” state, I didn’t see the little hint given:

2019-01-27 14:40:55.147888 7f8feb7a2e00 -1 *** experimental feature 'btrfs' is not enabled ***
This feature is marked as experimental, which means it
 - is untested
 - is unsupported
 - may corrupt your data
 - may break your cluster is an unrecoverable fashion
To enable this feature, add this to your ceph.conf:
  enable experimental unrecoverable data corrupting features = btrfs

2019-01-27 14:40:55.147901 7f8feb7a2e00 -1 filestore(/var/lib/ceph/osd/ceph-0) _detect_fs(1197): deprecated btrfs support is not enabled
2019-01-27 14:40:55.147906 7f8feb7a2e00 -1 filestore(/var/lib/ceph/osd/ceph-0) mount(1523): error in _detect_fs: (1) Operation not permitted
2019-01-27 14:40:55.147926 7f8feb7a2e00 -1 osd.0 0 OSD:init: unable to mount object store

Not feeling like a 24-hour wait, I did as it told me:

osd pool default size = 2  # Write an object n times.
osd pool default min size = 1 # Allow writing n copy in a degraded state.
osd pool default pg num = 128
osd pool default pgp num = 128
osd crush chooseleaf type = 1
osd max backfills = 10

# Allow btrfs to work:
enable experimental unrecoverable data corrupting features = btrfs

Now, my other OSDs re-started successfully, and I could finally finish off by restarting the metadata daemons and completing the migration. I’m now left with two OSDs with BTRFS and one with BlueStore.

For now, I’ll leave it that way, next week end, I might migrate a second node to BlueStore.

The reboot test

I needed to ensure the nodes would come back without my intervention. So starting with the two BTRFS nodes, I rebooted each one individually. The OSD on that node first went offline, then the monitor, finally the cluster noticed the metadata and manager services had gone. Then, upon successful boot, the services returned.

So far so good. Now the BlueStore node.

First reboot, my OSD didn’t come back. On investigation, I saw the following logs:

2019-01-28 16:25:59.312369 7fd58d4f0e00 -1  ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-0: (2) No such file or
directory
2019-01-28 16:26:14.865883 7fe92f942e00 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-0: (2) No such file or
directory
2019-01-28 16:26:30.419863 7fd4fa026e00 -1 ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-0: (2) No such file or directory

/var/lib/ceph/osd/ceph-0 was completely empty! Bugger, do I have to endure those 24 hours again? As it happened, no. I don’t know how the files in that directory disappeared, I did observe a tmpfs pseudovolume mounted at that directory earlier when trying to create the OSD … maybe that didn’t get unmounted before OSD creation, anyway, the files were gone.

A bit of digging revealed a ceph-bluestore-tool utility, with options like repair. At first I tried to wing it using that, but no dice. Then looking at the man page I noticed the sub-command prime-osd-dir. BINGO.

At first I threw the raw device at it, but as it happens, ceph-volume had deployed LVM to the raw disk, then put BlueStore on top of that. Starting lvm got the volume group recognised, so I added that to my boot-up services (see why I mentioned it earlier). It had created a sym-link to the LVM volume in /dev/ceph-${UUID1}/osd-block-${UUID2}.

No idea where the two UUIDs came from, but I tried this:

# ceph-bluestore-tool prime-osd-dir \
    --dev /dev/ceph-d62d0d95-2e13-4c59-834d-03a87b88c85e/osd-block-62b4be3e-3935-4d51-ab5c-dde077f99ea3 \
    --path /var/lib/ceph/osd/ceph-0

That populated the directory with files, so I tried again starting the OSD.

2019-01-28 16:59:23.680039 7fd93fcbee00 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied
2019-01-28 16:59:23.680082 7fd93fcbee00 -1  ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-0: (2) No such file or directory
2019-01-28 16:59:39.229888 7f4a585b4e00 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (13) Permission denied
2019-01-28 16:59:39.229918 7f4a585b4e00 -1  ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-0: (2) No such file or directory

Ah ha, chown -R ceph:ceph /var/lib/ceph/osd/ceph-0, and all sprang to life. The OSD came up.

Testing the fixes, a second re-boot

Since the OSD now was starting, and working, I did a second re-boot test, only to have history partially repeat itself.

The files were still there this time, but it was failing with a permissions error opening the block device. Sure enough, it was now owned by root.

Changed the permissions, and the OSD came up.

Fixing this was a job for udev:

cat /etc/udev/rules.d/99ceph.rules
SUBSYSTEM=="block", KERNEL=="sda7", OWNER="ceph", GROUP="ceph", MODE="0600"
SUBSYSTEM=="block", ENV{DM_VG_NAME}=="ceph-*", OWNER="ceph", GROUP="ceph", MODE="0600"

The first line is left-over from when /dev/sda7 was my journal. Not sure what I’ll do with this partition now, I’ll think of something (maybe Docker). The second line tells udev to change the permissions on the volume group that Ceph created.

Having done this, I rebooted again. This time, all worked. The OSD came up without my intervention.

Recap

So, the pitfalls I ran across in my Jewel-Luminous migration on Gentoo.

btrfs OSDs

I had btrfs volumes for my OSDs, which are now frowned upon and considered experimental. It isn’t necessary to migrate to BlueStore or XFS straight away, but for the OSDs to boot, you will need the following line in your /etc/ceph/ceph.conf before restarting your OSDs:

enable experimental unrecoverable data corrupting features = btrfs

ceph-volume expects the bootstrap-osd key.

To use ceph-volume, it for some reason expects to see the bootstrap-osd key in a hard-coded location. It won’t work with the default admin key.

This bootstrap key can be generated as follows:

# ceph auth add client.bootstrap-osd --cap mon 'profile bootstrap-osd
# mkdir /var/lib/ceph/bootstrap-osd
# ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

Before creating a BlueStore OSD, make sure lvmetad and lvm are started (and set to start at boot)

You can get away with just lvmetad for the initial creation, but you’ll want lvm running at boot anyway to ensure all the logical volume groups get started at boot before Ceph goes looking for them.

So before attempting OSD creation, ensure LVM is installed, and set to start at boot.

ceph-osd runs as the ceph user

So your udev rules need to reflect that. Luckily, ceph-volume seems to prefer creating LVM volume groups named ceph-${UUID}. I don’t know what decides the UUID value, but thankfully udev supports globbing. The following udev rule (put it in /etc/udev/rules.d/99ceph.rules or wherever seems appropriate) will keep permissions in check:

SUBSYSTEM=="block", ENV{DM_VG_NAME}=="ceph-*", OWNER="ceph", GROUP="ceph", MODE="0600"

(The above should be all on one line.)

Before rebooting a BlueStore node, back up your OSD data directories

Shouldn’t be strictly necessary, but now I’ve been bitten, I’m going to be taking extra care of that data directory on my other two nodes when I migrate them. I don’t fancy playing around with ceph-bluestore-tool frantically trying to get an OSD back up again.

Jan 192019
 

Recently, I’ve been looking at the problem of how to retrieve IPv6 traffic from the network stack of my workstation and manipulate it for transmission over AX.25.

My last experiments focussed on the TUN/TAP interface in Linux. Using this interface, I could create a virtual network interface that piped its traffic to a file descriptor in a program written in C.

One advantage of using the C language for this is that, as binding to the TAP interface requires root privileges, the binary could be installed setuid root. Thus, any time it started, it would be running as root. From there, it could do what it needed, then drop privileges back to a regular user.

The program would just run as a child process… when there was traffic received from the kernel, it would just spit that out to stdout. If my parent application had something to send, it would feed that into stdin.

6lhagent is an implementation of that idea. It’s pretty rough, but it seems to work. It uses a simple protocol to frame the Ethernet packets so that it can maintain synchronisation with the parent process. All frames are ACKed or NAKed, depending on whether they were understood or not. The protocol is analogous to KISS or SLIP in concept. The framing is very different to these protocols, but the concept is that of frames delimited by a byte sequence, with occurrences of the special byte sequences replaced with place-holders to prevent the parser getting confused.

I then wrote this Python script which uses the asyncio IO loop to run 6lhagent and dump the packets it receives:

$ python3 demo/dumper.py 
Interface data: b'V\xc7\x05\\yA\x05\x00\x00\x00\x00\xca\x04tap0'
Interface: MAC=[86, 199, 5, 92, 121, 65] MTU=1280 IDX=202 NAME=tap0
Ethernet traffic: b'33330000001656c7055c794186dd600000000024000100000000000000000000000000000000ff0200000000000000000000000000163a000502000001008f00f5ec0000000104000000ff0200000000000000000001ff5c7941'
From: 33:33:00:00:00:16
To:   56:c7:05:5c:79:41
Protocol: 86dd
IPv6: Priority 0, Flow 000000
From: ::
To:   ff02::16
Length: 36, Next header: 0, Hop Limit: 1
Payload: b':\x00\x05\x02\x00\x00\x01\x00\x8f\x00\xf5\xec\x00\x00\x00\x01\x04\x00\x00\x00\xff\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\xff\\yA'
Ethernet traffic: b'33330000001656c7055c794186dd600000000024000100000000000000000000000000000000ff0200000000000000000000000000163a000502000001008f00f5ec0000000104000000ff0200000000000000000001ff5c7941'
From: 33:33:00:00:00:16
To:   56:c7:05:5c:79:41
Protocol: 86dd
IPv6: Priority 0, Flow 000000
From: ::
To:   ff02::16
Length: 36, Next header: 0, Hop Limit: 1
Payload: b':\x00\x05\x02\x00\x00\x01\x00\x8f\x00\xf5\xec\x00\x00\x00\x01\x04\x00\x00\x00\xff\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\xff\\yA'
Ethernet traffic: b'3333ff5c794156c7055c794186dd6000000000203aff00000000000000000000000000000000ff0200000000000000000001ff5c79418700bebb00000000fe8000000000000054c705fffe5c79410e01a02d5c9a6698'
From: 33:33:ff:5c:79:41
To:   56:c7:05:5c:79:41
Protocol: 86dd
IPv6: Priority 0, Flow 000000
From: ::
To:   ff02::1:ff5c:7941
Length: 32, Next header: 58, Hop Limit: 255
ICMP Type 135, Code 0, Checksum bebb
Data: b'\x00\x00\x00\x00\xfe\x80\x00\x00'
Payload: b'\x00\x00\x00\x00T\xc7\x05\xff\xfe\\yA\x0e\x01\xa0-\\\x9af\x98'
Ethernet traffic: b'33330000001656c7055c794186dd6000000000240001fe8000000000000054c705fffe5c7941ff0200000000000000000000000000163a000502000001008f0025070000000104000000ff0200000000000000000001ff5c7941'
From: 33:33:00:00:00:16
To:   56:c7:05:5c:79:41
Protocol: 86dd
IPv6: Priority 0, Flow 000000
From: fe80::54c7:5ff:fe5c:7941
To:   ff02::16
Length: 36, Next header: 0, Hop Limit: 1
Payload: b':\x00\x05\x02\x00\x00\x01\x00\x8f\x00%\x07\x00\x00\x00\x01\x04\x00\x00\x00\xff\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\xff\\yA'
Ethernet traffic: b'33330000001656c7055c794186dd6000000000240001fe8000000000000054c705fffe5c7941ff0200000000000000000000000000163a000502000001008f009cab0000000104000000ff0200000000000000000000000000fb'
From: 33:33:00:00:00:16
To:   56:c7:05:5c:79:41
Protocol: 86dd
IPv6: Priority 0, Flow 000000
From: fe80::54c7:5ff:fe5c:7941
To:   ff02::16
Length: 36, Next header: 0, Hop Limit: 1
Payload: b':\x00\x05\x02\x00\x00\x01\x00\x8f\x00\x9c\xab\x00\x00\x00\x01\x04\x00\x00\x00\xff\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xfb'

The thinking is that the bulk of the proof-of-concept will be done in Python. My reasoning for this is that it’s usually easier to prototype in a higher-level language than in C, and in this application, speed is not important. At best our network interface will be running at 9600 baud — Python will keep up just fine. Most of it will be at 1200 baud.

The Python code will do some packet filtering (e.g. filtering out the multicast NS messages, which are a no-no in RFC-6775) and to add options where required. It’ll also be responsible for rate-limiting the firehose-like output of the tap interface from the host so the AX.25 network doesn’t get flooded.

The proof of concept is coming together. Next steps are to implement an IPv6 stack of sorts in Python to dissect the datagrams.

Jan 122019
 

For 6LoWHAM, it could work that we just use the link-local address space to directly communicate between stations and leave it at that.

If I want to send a message to VK4BWI-5 from my station VK4MSL-9, I could just fire off a packet to fe80::6894:49ff:feae:7318 directed to my 6LoWHAM interface and be done with it. This then requires one of two things:

  1. that VK4BWI-5 can directly communicate with me
  2. that the intermediate stations know to forward my message on to that station

(1) is easy enough. (2) raises the question of “what is local”?

Supposing that this protocol took off, and suddenly the WIA decides to earmark special frequencies on a few bands for 6LoWHAM, with a fairly complete network stretching up the eastern seaboard of Australia. If my station sends a router solicitation from my home QTH in Brisbane, does someone in Melbourne really care to hear it? I’d wager this is a recipe for a very clogged packet network!

In Thread, the “link local” scope only gets you as far as the nodes that can directly hear you. It does mean that protocols like mDNS, which rely on the “link-local” multicast scope aren’t going to reach all nodes, but it also means that far flung nodes don’t need to listen to all the low-level chatter. For communications between nodes, an “on-mesh” prefix is used, and for mesh-wide multicast, a “realm-local” prefix of ff03::/64 is defined.

In truth, it’s highly unlikely that we’d have “one” single network. More likely it’ll be a mesh of interconnected networks with trunk links going via some other band (or perhaps VPNs over the Internet). For that to work, we can’t rely on just link-local networking, we actually need a routable network address for the mesh.

The Thread “mesh local” prefix is actually defined by the network’s extended IEEE-802.15.4 PAN ID, which is a 64-bit number that you define when setting up the network. Thread simply takes the most significant 40 bits of this, slaps fd in front and pads it out with zeros to 64-bits. The PAN ID 0x0123456789abcdef forms the subnet fd01:2345:6789::/64. This can be seen in the OpenThread sources.

This wastes 16-bits of address space normally reserved for the ULA subnet ID and throws away 24-bits of the PAN ID. For our network, we don’t need 16-bits worth of subnets, we just need one. We also don’t have a PAN ID in AX.25.

The thinking is, we’ll use a “group” address. This will be a regular AX.25 SSID, which will translate to a MAC which has the group bit set. (Exactly how I’ll differentiate between a station SSID and a group SSID I’m not sure. Probably will look at the destination IP, if it’s multicast then the group bit gets set.)

Supposing we were to use this for the International Rally of Queensland (an event which is now defunct), we might create a 6LoWHAM network with a group address of “IROQ19”. The MAC address used for group-wide communications would be 03:01:cd:e5:a9:f8.

We can derive a prefix from this MAC address. A ULA normally consists of a 7-bit ULA prefix, a 1-bit “global/local” bit, a 40-bit global ID, and a 16-bit subnet ID.

The ULA prefix is fc::/7. The global/local bit is always set to 1 (local) because no one has come up with a way that ULAs can be globally administered. 40 bits is a bit tight, we could truncate our MAC to 40 bits and ignore the subnet ID like Thread do, that gives us a subnet of fd03:1cd:5ea9::/64.

The last 3 bits of the SSID though, are like a subnet ID. So if we move those 3 bits to set the last 3 bits of the prefix, we can make some use of that subnet ID, but still waste 13 bits with zeros.

Alternatively, we can consider the global ID and subnet ID to be one 56-bit field. We effectively shrink the subnet ID to 3 bits. That gives us a 53-bit global ID, which now fits the remaining 45-bits of our MAC and leaves us with 8 bits left over.

We can discard the lowest two bits in the first byte of the MAC as those (the group and local bits) will be the same for all groups, so that gives us another two bits. 10 bits isn’t a lot, but it’s enough to encode “AR” (amateur radio) in ITA-2, thus giving us a recognisable subnet mask for all 6LoWHAM networks. We wind up with the following:

┌─ULA─┐L┌──"AR"──┐┌───────────── Network Address ──────────────┐
1111110100010010100000000000000111001101111001011010100111111000
└──┼───┼───┼───┼───┼───┼───┼───┼───┼───┼───┼───┼───┼───┼───┼───┤
   f   d   1   2 : 8   0   0   1 : c   d   e   5 : a   9   f   8 /64

This actually has me thinking whether the call-sign part of the SSID should be right-padded out to make the network address consistent. Maybe my SSID to MAC algorithm could do with a tweak there as it may make routing easier as it’ll put all those zeros to the right.

In Thread, the mesh-local prefix isn’t route-able beyond the mesh, there’s a separate prefix handed out by border routers for that. In our case, I don’t think there’s any point in complicating matters by having more than one route-able prefix for a mesh. If a station participates in two networks that share a frequency, then sure, that node may have an address on each network, but each network should share a common identity.

Thus in the contrived example of having a large network along the coastline: it’d be an “inter-network” of smaller meshes, linked together via router nodes which know how to hop between them. Those routes may be via point-to-point microwave links, HF, Internet tunnels, etc.

The subnets used for these other networks may be assigned a “context identifier” which is 4-bits. I’ll have to figure out if there’s a sane way to do that on a given network. Most 802.15.4 networks have a “PAN co-ordinator” which could be looking after that. Thread networks elect a “leader” node.

Given the small number of identifiers, and the low probability of this being used, this should be manually administered. Even without a context ID being assigned, one can still route between the subnets, just that the full IPv6 address needs to be given for the foreign node, so you incur a 16-byte penalty doing so. Thus the context IDs will probably be handed out for “popular routes”, with the mesh prefix being “context 0”.

I haven’t yet given thought to how this “context” would be disseminated over the mesh or kept updated. That is a can of worms for another day.

Jan 122019
 

One of the aims of 6LoWHAM was to provide a means to send IPv6 traffic between user applications and the AX.25 network.

In order to do this, the applications have to have some way of injecting their IP traffic. The canonical way this is done is through the operating system’s TCP/IP stack. This requires that we have an interface to the operating system kernel in order to receive that IP traffic destined for the airwaves.

Now, we could write a kernel driver for this, but it’s going the long way around to do it. Especially as we intend to interface to software that runs in userspace for the actual transmission. Our driver at best would be just taking the raw Ethernet frame, extracting the IP part, and forwarding that back to our program running in userspace.

There’s a driver that does that for us: TUN/TAP. This driver can either create a TUNnel device, which forwards IP datagrams, or a TAP device, which forwards Ethernet frames. We’ll focus on the TUN mode of this driver here.

The idea is this will create an IP tunnel, with one side exposing a network device to the kernel, and the other side being a file descriptor in a userspace application that just reads and writes raw IP frames. How it generates and processes those frames is entirely up to the software author. Most famous uses for this device are VPNs, so taking the IP datagram, encrypting it, then encapsulating it in an IP datagram (usually UDP) to be sent over the Internet to some other peer, which reverses the process and writes the original packet to its tunnel file descriptor.

In our case, we’ll be dissecting it a bit to extract the key fields, then applying our own “compression” defined in the 6LoWHAM specs, then forwarding it on to our AX.25 stack (probably LinBPQ or Direwolf) to be sent as an AX.25 UI frame.

The first step in this journey was actually figuring out what the packets look like on a tunnel device. I created this little program to explore the idea.

It just needs the usual C toolchain and libraries on a Linux system. I tested with Gentoo and Linux kernel 4.15. Building it is a simple make command. If you then run the resulting binary as root, you’ll find a tun0 device (or maybe some other number) created.

Bring the interface up, and you should start to see some traffic as the host tries to talk to is new (and very much mute) peer:

RC=0 stuartl@rikishi ~/projects/6lowham/packetdumper $ make 
cc    -c -o linuxtun.o linuxtun.c
cc    -c -o main.o main.c
cc -o packetdumper linuxtun.o main.o
RC=0 stuartl@rikishi ~/projects/6lowham/packetdumper $ sudo ./packetdumper 
Password: 
^Z
[1]+  Stopped(SIGTSTP)        sudo ./packetdumper
RC=148 stuartl@rikishi ~/projects/6lowham/packetdumper $ sudo ip link set dev tun0 up
RC=0 stuartl@rikishi ~/projects/6lowham/packetdumper $ fg
sudo ./packetdumper
Flags: 0x0000  Protocol: 0x86dd
  48:  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15
   0: 60 00 00 00 00 08 3a ff fe 80 00 00 00 00 00 00
  16: 5e be 89 41 7b 19 d5 60 ff 02 00 00 00 00 00 00
  32: 00 00 00 00 00 00 00 02 85 00 44 bd 00 00 00 00
Flags: 0x0000  Protocol: 0x86dd
  48:  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15
   0: 60 00 00 00 00 08 3a ff fe 80 00 00 00 00 00 00
  16: 5e be 89 41 7b 19 d5 60 ff 02 00 00 00 00 00 00
  32: 00 00 00 00 00 00 00 02 85 00 44 bd 00 00 00 00

I didn’t bother to decode the IP datagram further, but if you look at the Wikipedia IPv6 Packet article, it isn’t difficult to see what’s going on. In this case, we can see it’s an IPv6 packet both from the Protocol field (0x86dd is the Ethertype for IPv6), and from the first 4 bits of the frame payload.

The traffic class and flow label are both 0s here. The IPv6 payload length is just 8 bytes, so most of this is in fact IPv6 header data. Next header is type 0x3a (IPv6 ICMP) and the hop limit is 255. This is followed by the source address (my laptop’s link-local address fe80::5ebe:8941:7b19:d560) and the destination address (all link-local routers multicast address ff02::2).

The ICMPv6 message is the last 8 bytes; and in this case, it’s type is 0x85 (router solicitation), the code is 0x00, the two bytes after that are the checksum and the message (4 bytes) is all zeros.

Quite how that address was chosen is something I’ll have to get to grips with. Yes, it’s SLAAC, but where did it get the hardware address from? That I’ll have to figure out.

The alternative is to use a TAP interface, which means I choose the MAC address, and thus can control what the SLAAC-derived address becomes. Ohh, and it goes without saying that the privacy extensions will be a big no no on the air: we’re relying on the fact that we can derive the IPv6 address from the SSID of the station both for technical reasons and to legally meet the requirements for stations to “identify” who they are and whom they are talking to. SLAAC privacy will make a mess of that.

So controlling this link-local address is a must. I guess next stop: let’s look at a tap device. I’ve just made some changes to explore the differences from the application end. There isn’t a lot of difference here.

RC=130 stuartl@rikishi ~/projects/6lowham/packetdumper $ sudo ./packetdumper -tap
Password: 
^Z
[1]+  Stopped(SIGTSTP)        sudo ./packetdumper -tap
RC=148 stuartl@rikishi ~/projects/6lowham/packetdumper $ sudo ip link set tap0 up
RC=0 stuartl@rikishi ~/projects/6lowham/packetdumper $ fg
sudo ./packetdumper -tap
Flags: 0x0000  Protocol: 0x86dd
  90:  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15
   0: 33 33 00 00 00 16 ce 65 0c 34 48 34 86 dd 60 00
  16: 00 00 00 24 00 01 00 00 00 00 00 00 00 00 00 00
  32: 00 00 00 00 00 00 ff 02 00 00 00 00 00 00 00 00
  48: 00 00 00 00 00 16 3a 00 05 02 00 00 01 00 8f 00
  64: 27 22 00 00 00 01 04 00 00 00 ff 02 00 00 00 00
  80: 00 00 00 00 00 01 ff 34 48 34
Flags: 0x0000  Protocol: 0x86dd
  86:  0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15
   0: 33 33 ff 34 48 34 ce 65 0c 34 48 34 86 dd 60 00
  16: 00 00 00 20 3a ff 00 00 00 00 00 00 00 00 00 00
  32: 00 00 00 00 00 00 ff 02 00 00 00 00 00 00 00 00
  48: 00 01 ff 34 48 34 87 00 af 03 00 00 00 00 fe 80
  64: 00 00 00 00 00 00 cc 65 0c ff fe 34 48 34 0e 01
  80: 61 78 48 c1 ac aa

The big difference is now we have an Ethernet header prepended. The proto field in the packet information now duplicates what we can see in the Ethernet frame header (bytes 12 and 13), and the IPv6 packet starts from byte 14.

I think this is the mode 6LoWHAM will use. It’s possible to set the MAC address on the created tap0 device to whatever 46 bits we like, the remaining two bits in the MAC address are for defining whether the address is global or local (we’ll set ours to “local”), and the other sets whether this is a multicast or unicast address. The SLAAC address will closely match this address with two differences:

  1. The MAC will have the bytes 0xff 0xfe inserted into the middle.
  2. The “global/local” bit is inverted. So for the 2001:db8::/64 prefix:
    • aa:bb:cc:dd:ee:ff becomes 2001:db8::a8bb:ccff:fedd:eeff
    • a8:bb:cc:dd:ee:ff becomes 2001:db8::aabb:ccff:fedd:eeff

That latter point had me confused at first, I thought it might’ve been that a bit got cleared, but instead it’s just inverted, so completely reversible.