Recently, I noticed my network monitoring was down… I hadn’t worried about it because I had other things to keep me busy, and thankfully, my network monitoring, whilst important, isn’t mission critical.

I took a look at it today. The symptom was an odd one, influxd was running, it was listening on the back-up/RPC port 8088, but not 8086 for queries.

It otherwise was generating logs as if it were online. What gives?

Tried some different settings, nothing… nada… zilch. Nothing would make it listen to port 8086.

Tried updating to 1.8 (was 1.1), still nothing.

Tried manually running it as root… sure enough, if I waited long enough, it started on its own, and did begin listening on port 8086. Hmmm, I wonder. I had a look at the init scripts:

#!/bin/bash -e

/usr/bin/influxd -config /etc/influxdb/influxdb.conf $INFLUXD_OPTS &
PID=$!
echo $PID > /var/lib/influxdb/influxd.pid

PROTOCOL="http"
BIND_ADDRESS=$(influxd config | grep -A5 "\[http\]" | grep '^  bind-address' | cut -d ' ' -f5 | tr -d '"')
HTTPS_ENABLED_FOUND=$(influxd config | grep "https-enabled = true" | cut -d ' ' -f5)
HTTPS_ENABLED=${HTTPS_ENABLED_FOUND:-"false"}
if [ $HTTPS_ENABLED = "true" ]; then
  HTTPS_CERT=$(influxd config | grep "https-certificate" | cut -d ' ' -f5 | tr -d '"')
  if [ ! -f "${HTTPS_CERT}" ]; then
    echo "${HTTPS_CERT} not found! Exiting..."
    exit 1
  fi
  echo "$HTTPS_CERT found"
  PROTOCOL="https"
fi
HOST=${BIND_ADDRESS%%:*}
HOST=${HOST:-"localhost"}
PORT=${BIND_ADDRESS##*:}

set +e
max_attempts=10
url="$PROTOCOL://$HOST:$PORT/health"
result=$(curl -k -s -o /dev/null $url -w %{http_code})
while [ "$result" != "200" ]; do
  sleep 1
  result=$(curl -k -s -o /dev/null $url -w %{http_code})
  max_attempts=$(($max_attempts-1))
  if [ $max_attempts -le 0 ]; then
    echo "Failed to reach influxdb $PROTOCOL endpoint at $url"
    exit 1
  fi
done
set -e

Ahh right, so start the server, check every second to see if it’s up, and if not, just abort and let systemd restart the whole shebang. Because turning the power on-off-on-off-on-off is going to make it go faster, right?

I changed max_attempts to 360 and the sleep to 10.

Having fixed this, I am now getting data back into my system.

So, the situation: I have two boxes that must replicate data between themselves and generally keep in contact with one another over a network (Ethernet or WiFi) that I do not control. I want the two to maintain a peer-to-peer VPN over this potentially hostile network: ensuring confidentiality and authenticity of data sent over the tunnelled link.

The two nodes should be able to try and find each-other via other means, such as mDNS (Avahi).

I had thought of just using OpenVPN in its P2P mode, but I figured I’d try something new, WireGuard. Both machines are running Debian 10 (Buster) on AMD64 hardware, but this should be reasonably applicable to lots of platforms and Linux-based OSes.

This assumes WireGuard is in fact, installed: sudo apt-get install -y wireguard will do the deed on Debian/Ubuntu.

Initial settings

First, having installed WireGuard, I needed to make some decisions as to how the VPN would be addressed. I opted for using an IPv6 ULA. Why IPv6? Well, remember I mentioned I do not control the network? They could be using any IPv4 subnet, including the one I hypothetically might choose for my own network. This is also true of ULAs, however the probabilities are ridiculously small: parts per billion chance, enough to ignore!

So, I trundled over to a ULA generator site and generated a ULA. I made up a MAC address for this purpose. For the purposes of this document let’s pretend it gave me 2001:db8:aaaa::/48 as my address (yes, I know this is not a ULA, this is in the documentation prefix). For our VPN, we’ll statically allocate some addresses out of 2001:db8:aaaa:1000::/64, leaving the other address blocks free for other use as desired.

For ease of set-up, we also picked a port number for each node to listen on, WireGuard’s Quick Start guide uses port 51820, which is as good as any, so we used that.

Finally, we need to choose a name for the VPN interface, wg0 seemed as good as any.

Summarising:

ULA: 2001:db8:aaaa::/48
VPN subnet: 2001:db8:aaaa:1000::/64
Listening port number: 51820
WireGuard interface: wg0

Generating keys

Each node needs a keypair for communicating with its peers. I did the following:

( umask 077 ; wg genkey > /etc/wg.priv )
wg pubkey < /etc/wg.priv > /etc/wg.pub

I gathered all the wg.pub files from my nodes and stashed them locally

Creating settings for all nodes

I then made some settings files for some shell scripts to load. First, a description of the VPN settings for wg0, I put this into /etc/wg0.conf:

INTERFACE=wg0
SUBNET_IP=2001:db8:aaaa:1000::
SUBNET_SZ=64
LISTEN_PORT=51820
PERSISTENT_KEEPALIVE=60

Then, in a directory called wg.peers, I added a file with the following content for each peer:

pubkey=< node's /etc/wg.pub content >
ip=<node's VPN IP >

The VPN IP was just allocated starting at ::1 and counting upwards, do whatever you feel is appropriate for your virtual network. The IPs only need to be unique and within that same subnet.

Both the wg.peers and wg0.conf were copied to /etc on all nodes.

The VPN clean-up script

I mention this first, since it makes debugging the set-up script easier since there’s a single command that will bring down the VPN and clean up /etc/hosts:

#!/bin/bash

. /etc/wg0.conf

if [ -d /sys/class/net/${INTERFACE} ]; then
	ip link set ${INTERFACE} down
	ip link delete dev ${INTERFACE}

	sed -i -e "/^${SUBNET_IP}/ d" /etc/hosts
fi

This checks for the existence of wg0, and if found, brings the link down and deletes it; then cleans up all VPN IPs from the /etc/hosts file. Copy this to /usr/local/sbin, make permissions 0700.

The VPN set-up script

This is what establishes the link. The set-up script can take arguments that tell it where to find each peer: e.g. peernode.local=10.20.30.40 to set a static IP, or peernode.local=10.20.30.40:12345 if an alternate port is needed.

Giving peernode.local=listen just tells the script to tell WireGuard to listen for an incoming connection from that peer, where-ever it happens to be.

If a peer is not mentioned, it tries to discover the address of the peer using getent: the peer must have a non-link-local, non-VPN address assigned to it already: this is due to getent not being able to tell me which interface the link-local address came from. If it does, and it can ping that address, it considers the node up and adds it.

Nodes that do not have a static address configured, are set to listen, or are not otherwise locatable and reachable, are dropped off the list for VPN set-up. For two peers, this makes sense, since we want them to actively seek each-other out; for three nodes you might want to add these in “listen” mode, an exercise I leave for the reader.

#!/bin/bash

set -e

. /etc/wg0.conf

# Pick up my IP details and private key
ME=$( uname -n ).local
MY_IP=$( . /etc/wg.peers/${ME} ; echo ${ip} )

# Abort if we already have our interface
if [ -d /sys/class/net/${INTERFACE} ]; then
	exit 0
fi

# Gather command line arguments
declare -A static_peers
while [ $# -gt 0 ]; do
	case "$1" in
	*=*)	# Peer address
		static_peers["${1%=*}"]="${1#*=}"
		shift
		;;
	*)
		echo "Unrecognised argument: $1"
		exit 1
	esac
done

# Gather the cryptography configuration settings
peers=""
for peerfile in /etc/wg.peers/*; do
	peer=$( basename ${peerfile} )
	if [ "${peer}" != ${ME} ]; then
		# Derive a host name for the endpoint on the VPN
		host=${peer%.local}
		vpn_hostname=${host}.vpn

		# Do we have an endpoint IP given on the command line?
		endpoint=${static_peers[${peer}]}

		if [ -n "${endpoint}" ] && [ "${endpoint}" != listen ]; then
			# Given an IP/name, add brackets around IPv6, add port number if needed.
			endpoint=$(
				echo "${endpoint}" | sed \
					-e 's/^[0-9a-f]\+:[0-9a-f]\+:[0-9a-f:]\+$/[&]/' \
					-e "s/^\\(\\[[0-9a-f:]\\+\\]\\|[0-9\\.]\+\\)\$/\1:${LISTEN_PORT}/"
			)
		elif [ -z "${endpoint}" ]; then
			# Try to resolve the IP address for the peer
			# Ignore link-local and VPN tunnel!
			endpoint_ip=$(
				getent hosts ${peer} \
					| cut -f 1 -d' ' \
					| grep -v "^\(fe80:\|169\.\|${SUBNET_IP}\)"
			)

			if ping -n -w 20 -c 1 ${endpoint_ip}; then
				# Endpoint is reachable.  Construct endpoint argument
				endpoint=$( echo ${endpoint_ip} | sed -e '/:/ s/^.*$/[&]/' ):${LISTEN_PORT}
			fi
		fi

		# Test reachability
		if [ -n "${endpoint}" ]; then
			# Pick up peer pubkey and VPN IP
			. ${peerfile}

			# Add to peers
			peers="${peers} peer ${pubkey}"
			if [ "${endpoint}" != "listen" ]; then
				peers="${peers} endpoint ${endpoint}"
			fi
			peers="${peers} persistent-keepalive ${PERSISTENT_KEEPALIVE}"
			peers="${peers} allowed-ips ${SUBNET_IP}/${SUBNET_SZ}"

			if ! grep -q "${vpn_hostname} ${host}\\$" /etc/hosts ; then
				# Add to /etc/hosts
				echo "${ip} ${vpn_hostname} ${host}" >> /etc/hosts
			else
				# Update /etc/hosts
				sed -i -e "/${vpn_hostname} ${host}\\$/ s/^[^ ]\+/${ip}/" \
					/etc/hosts
			fi
		else
			# Remove from /etc/hosts
			sed -i -e "/${vpn_hostname} ${host}\\$/ d" \
				/etc/hosts
		fi
	fi
done

# Abort if no peers
if [ -z "${peers}" ]; then
	exit 0
fi

# Create the interface
ip link add ${INTERFACE} type wireguard

# Configre the cryptographic settings
wg set ${INTERFACE} listen-port ${LISTEN_PORT} \
	private-key /etc/wg.priv ${peers}

# Bring the interface up
ip -6 addr add ${MY_IP}/${SUBNET_SZ} dev ${INTERFACE}
ip link set ${INTERFACE} up

This is run from /etc/cron.d/vpn:

* * * * * root /usr/local/sbin/vpn-up.sh >> /tmp/vpn.log 2>&1

S	M	T	W	T	F	S
					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30
31

Blogospheric Refraction

Making InfluxDB’s init script more patient

Peer-to-peer LAN VPN using WireGuard

Initial settings

Generating keys

Creating settings for all nodes

The VPN clean-up script

The VPN set-up script