Nov 282019
 

This is more of a brain dump for yours truly, since as a day job, I’m dealing with OpenThread a lot, and so there’s lots of little tricks out there for building it on various platforms and configurations. The following is not so much a how-to guide, but a quick brain dump of different things I’ve learned over the past year or so of messing with OpenThread.

Verbose builds

To get a print out of every invocation of the toolchain with all flags, specify V=1 on the call to make:

$ make -f examples/Makefile-${PLATFORM} …${ARGS}… V=1

Running one step at a time

To disable the parallel builds when debugging the build system, append BuildJobs=1 to your make call:

$ make -f examples/Makefile-${PLATFORM} …${ARGS}… BuildJobs=1

Building a border router NCP image

General case

$ make -f examples/Makefile-${PLATFORM} …${ARGS}… BORDER_AGENT=1 BORDER_ROUTER=1 COMMISSIONER=1 UDP_FORWARD=1

For TI CC2538

# Normal CC2538 (e.g. Zolertia Firefly)
$ make -f examples/Makefile-cc2538 BORDER_AGENT=1 BORDER_ROUTER=1 COMMISSIONER=1 UDP_FORWARD=1

# CC2538 + CC2592
$ make -f examples/Makefile-cc2538 BORDER_AGENT=1 BORDER_ROUTER=1 COMMISSIONER=1 UDP_FORWARD=1 CC2592=1

If you want to run the latter image on a WideSky Hub, you’ll need to edit examples/platform/cc2538/uart.c and comment out line 117 before compiling as it uses a simple diode for 5V to 3.3V level shifting, and this requires the internal pull-up to be enabled:

115     // rx pin
116     HWREG(IOC_UARTRXD_UART0) = IOC_PAD_IN_SEL_PA0;
117     HWREG(IOC_PA0_OVER)      = IOC_OVERRIDE_DIS; // ← comment out this to allow UART RX to work
118     HWREG(GPIO_A_BASE + GPIO_O_AFSEL) |= GPIO_PIN_0;

For Nordic nRF52840

# Nordic development board (PCA10056) via J2 (near battery)
$ make -f examples/Makefile-nrf52840 BORDER_AGENT=1 BORDER_ROUTER=1 COMMISSIONER=1 UDP_FORWARD=1

# Ditto, but instead using J3 (near middle of bottom edge)
$ make -f examples/Makefile-nrf52840 BORDER_AGENT=1 BORDER_ROUTER=1 COMMISSIONER=1 UDP_FORWARD=1 USB=1

# Nordic dongle (PCA10059)
$ make -f examples/Makefile-nrf52840 BORDER_AGENT=1 BORDER_ROUTER=1 COMMISSIONER=1 UDP_FORWARD=1 USB=1 BOOTLOADER=USB

I’m working on what needs to be done for the Fanstel BT840X… watch this space.

Building certification test images

For CC2538

$ make -f examples/Makefile-cc2538 BORDER_ROUTER=1 COMMISSIONER=1 DHCP6_CLIENT=1 JOINER=1

Running CI tests outside of Travis CI

You will need:

  • Python 3.5 or later. 3.6 recommended.
  • pycryptodome (not pycrypto: if you get an AttributeError referencing AES.MODE_CCM, that’s why!
  • enum34 (for now… I suspect this will disappear once the Python 2.7 requirement is dropped for Android tools)
  • ipaddress
  • pexpect

The test suites work by running the POSIX ot-cli-${TYPE} (where ${TYPE} is ftd or mtd).

Running tests

From the root of the OpenThread tree:

$ make -f examples/Makefile-posix check

Making tests

The majority of the tests lurk under tests/scripts/thread-cert. Don’t be fooled by the name, lots of non-Thread tests live there.

The file node.py wraps pexpect up in an object with methods for calling the various CLI commands or waiting for things to be printed to the console.

The tests themselves are written using using the standard Python unittest framework, with setUp creating a few Node objects (node.py) and tearDown cleaning them up. The network is simulated.

The test files must be executable, and call unittest.main() after checking if the script is called directly.

Logs during the test runs

During the tests, the logs are stashed in build/${CHOST}/tests/scripts/thread-cert and will be named ${SCRIPTNAME}.log. (e.g test_coap_observe.py.log). Also present is the .pcap file (Packet dump).

Logs after the tests

The test suite will report the logs are at tests/scripts/thread-cert/test-suite.log. This is relative to the build directory… so look in build/${CHOST}/tests/scripts/thread-cert/test-suite.log.

make pretty with clang 8.0

Officially this isn’t supported, but you can “fool” OpenThread’s build system into using clang 8.0 anyway:

$ cat ~/bin/clang-format-6.0 
#!/bin/bash

if [ "$1" == "--version" ]; then
        echo "clang-format version 6.0"
else
        clang-format "$@"
fi

Put that file in your ${PATH}, make it executable, then OpenThread will think you’ve got clang-format version 6 installed. This appears to work without ill effect, so maybe a future release of OpenThread will support it.

Nov 282018
 

The binutils linker is able to generate a map file when it links your binaries.  This provides a lot of detail on how the functions and variables have been arranged into the program memory space, which is crucial information when dealing with embedded devices.

Unfortunately, looking around I didn’t see any decent tools for extracting this information.  I wound up cooking my own Python script up to do this.  It’s very crude, just takes a map file on standard input, and dumps a report to standard output.  It seems to work okay with ARM, and sorta works with AVR but might need some more work.

import re
from sys import stdin, stdout

WIDTH = 8
SPARSE_SKIP = 4*WIDTH
SYMBOL_ONLY_RE = re.compile(\
        r'^ \.([a-zA-Z0-9]+)\.([a-zA-Z0-9_\.]+)$')
ADDR_ONLY_RE = re.compile(\
        r'^ {16}(0x[0-9a-f]+) +(0x[0-9a-f]+) (.*)$')
ADDR_CXXSYM_RE = re.compile(\
        r'^ {16}(0x[0-9a-f]+) {16}([a-zA-Z_][a-zA-Z0-9_:()*\[\]\.]+)$')
SYMBOL_ADDR_RE = re.compile(\
        r'^ \.([a-zA-Z0-9]+)\.([a-zA-Z0-9_\.]+) +(0x[0-9a-f]+) +(0x[0-9a-f]+) (.*)$')
FILL_RE = re.compile('^ \*fill\* +(0x[0-9a-f]+) +(0x[0-9a-f]+) +(\d+)$')
REGION_RE = re.compile('^([a-zA-Z0-9_]+) +(0x[0-9a-f]+) (0x[0-9a-f]+) ([rwx]+)$')

regions = []
last = None
objects = []

def on_symbol_only(match):
    global last
    if match:
        (section, symbol) = match.groups()
        last = {
            'type': 'symbol',
            'section': section,
            'symbol': symbol
        }
        objects.append(last)
    return match


def on_addr_only(match):
    if match:
        (address, size, loc) = match.groups()
        if last is None:
            return match

        assert last['type'] == 'symbol'
        assert 'address' not in last
        assert 'size' not in last
        assert 'loc' not in last
        last['address'] = int(address, base=16)
        last['size'] = int(size, base=16)
        last['loc'] = loc
    return match


def on_symbol_addr(match):
    global last
    if match:
        (section, symbol, address, size, loc) = match.groups()
        last = {
            'type': 'symbol',
            'section': section,
            'symbol': symbol,
            'address': int(address, base=16),
            'size': int(size, base=16),
            'loc': loc
        }
        objects.append(last)
    return match


def on_addr_cxxsym(match):
    if match:
        (address, cxxsym) = match.groups()
        if last is None:
            return match

        assert last['type'] == 'symbol'
        if last['address'] != int(address, base=16):
            return match
        if 'cxxsyms' not in last:
            last['cxxsyms'] = set()
        last['cxxsyms'].add(cxxsym)
    return match


def on_fill(match):
    if match:
        (address, size, data) = match.groups()
        objects.append({
            'type': 'fill',
            'address': int(address, base=16),
            'size': int(size, base=16),
            'data': int(data, base=16)
        })
    return match

def on_region(match):
    if match:
        (region, origin, length, attrs) = match.groups()
        regions.append({
            'region': region,
            'address': int(origin, base=16),
            'size': int(length, base=16),
            'attrs': attrs
        })
    return match


for line in stdin:
    line = line.rstrip()

    try:
        if line == 'Memory Configuration':
            break
    except:
        print ('# Failed at line %r' % line)
        raise


for line in stdin:
    line = line.rstrip()

    try:
        if line == 'Linker script and memory map':
            break
        if on_region(REGION_RE.match(line)):
            continue
    except:
        print ('# Failed at line %r' % line)
        raise


for line in stdin:
    line = line.rstrip()
    try:
        if on_symbol_only(SYMBOL_ONLY_RE.match(line)):
            continue

        if on_addr_only(ADDR_ONLY_RE.match(line)):
            continue

        if on_addr_cxxsym(ADDR_CXXSYM_RE.match(line)):
            continue

        if on_fill(FILL_RE.match(line)):
            continue

        last = None
    except:
        print ('Failure context:')
        print ('# last = %r' % last)
        print ('# line = %r' % line)
        raise

for region in regions:
    region['end'] = region['address'] + region['size']
regions.sort(key=lambda r : r['address'])

for obj in objects:
    if 'cxxsyms' in obj:
        obj['cxxsyms'] = list(sorted(obj['cxxsyms']))

    if ('address' in obj) and ('size' in obj):
        obj['end'] = obj['address'] + obj['size']

        for region in regions:
            if (obj['end'] <= region['end']) and \ (obj['address'] >= region['address']):
                obj['region'] = region['region']
                break
objects.sort(key=lambda o : o.get('address', -1))

for region in regions:
    address = region['address']
    sym_idx = 0
    row_rem = 0
    row_syms = []
    seen = set()

    region_objects = list(filter(
        lambda obj : obj.get('region') == region['region'],
        objects))
    if not region_objects:
        continue

    for obj in region_objects:
        if obj['type'] == 'symbol':
            sym = '%02d' % (sym_idx % 100)
            sym_idx += 1
        elif obj['type'] == 'fill':
            sym = '--'
        else:
            sym = '??'

        while address < obj['address']: if not row_rem: if (obj['address'] - address) > SPARSE_SKIP:
                    end = obj['address'] - (obj['address'] % SPARSE_SKIP)
                    stdout.write('\n%16s 0x%08x -- 0x%08x (%d bytes)' % (
                        region['region'], address, end, end - address))
                    address = end
                    continue

                stdout.write('\n%16s 0x%08x: ' % (region['region'], address))
                row_rem = WIDTH

            stdout.write(' ..')
            row_rem -= 1
            address += 1

        while address < obj['end']:
            if not row_rem:
                if row_syms:
                    stdout.write(' | %s\n' % row_syms.pop(0))
                else:
                    stdout.write('\n')
                stdout.write('%16s 0x%08x: ' % (region['region'], address))
                row_rem = WIDTH

            stdout.write(' %s' % sym)
            row_rem -= 1
            address += 1
            if ('symbol' in obj) and (obj['symbol'] not in seen):
                row_syms.append('%s: %s' % (sym, obj['symbol']))
                seen.add(obj['symbol'])

            if not row_rem:
                if row_syms:
                    stdout.write(' | %s\n' % row_syms.pop(0))
                else:
                    stdout.write('\n')
                stdout.write('%16s 0x%08x: ' % (region['region'], address))
                row_rem = WIDTH

    while row_rem:
        stdout.write(' ..')
        row_rem -= 1
        address += 1

    stdout.write('\n%16s 0x%08x -- 0x%08x (%d bytes)\n' % (
        region['region'], address, region['end'], region['end'] - address))
    stdout.write('%16s %d bytes remaining\n' % (
        region['region'], region['end'] - address))

Continue reading »