Gentoo Development

Solar Cluster: Gentoo Linux MUSL for ARMv5 finally built

It’s taken several months and had a few false starts, but at long last I have some stage tarballs for Gentoo Linux MUSL for ARMv5 processors.  I’m not the only one wanting such a port, even looking for my earlier thread on the matter, I stumbled on this post.  (Google translate is hopeless with Russian, but I can get the gist of what’s being said.)

This was natively built on the TS-7670 using an external hard drive connected over USB 2.0 as both swap and the chroot.  It took two passes to clean everything up and get something that’s in a “release-able” state.

I think my next steps now will be:

  • Build an updated kernel … I might see if I can expose that I²C register via a sysfs file or something that collectd can pick up whilst I’m at it.  I have the kernel sources and bootloader sources.
  • Prepare the 32GB MicroSD card I bought a few weeks back with the needed partitions and load Gentoo onto that.
  • Install the MicroSD card and boot off it.
  • Back up the eMMC
  • Re-format the eMMC and copy the MicroSD card to it.

It’s supposed to be wet this weekend, so it sounds like a good project for indoors.

Solar Cluster: arm-unknown-linux-musleabi… saga part IV

So, at long last, I finally saw this in my chroot‘s /var/log/emerge.log:

1524887925: Started emerge on: Apr 28, 2018 03:58:45
1524887926:  *** emerge --oneshot sys-devel/gcc::musl
1524888211:  >>> emerge (1 of 1) sys-devel/gcc-7.3.0 to /
1524888212:  === (1 of 1) Cleaning (sys-devel/gcc-7.3.0::/root/musl/sys-devel/gcc/gcc-7.3.0.ebuild)
1524888307:  === (1 of 1) Compiling/Packaging (sys-devel/gcc-7.3.0::/root/musl/sys-devel/gcc/gcc-7.3.0.ebuild)
1525472690:  === (1 of 1) Merging (sys-devel/gcc-7.3.0::/root/musl/sys-devel/gcc/gcc-7.3.0.ebuild)
1525472838:  >>> AUTOCLEAN: sys-devel/gcc:7.3.0
1525473358:  === (1 of 1) Post-Build Cleaning (sys-devel/gcc-7.3.0::/root/musl/sys-devel/gcc/gcc-7.3.0.ebuild)
1525473358:  ::: completed emerge (1 of 1) sys-devel/gcc-7.3.0 to /
1525473360:  *** Finished. Cleaning up...
1525473373:  *** exiting successfully.

That’s 6 days, 18 hours and 32 minutes, of solid compiling. BUT WE GOT THERE!

What’s left? This:

Calculating dependencies... done!
[ebuild     U  ] sys-libs/musl-1.1.19 [1.1.18]
[binary   R    ] sys-libs/zlib-1.2.11-r1
[binary   R    ] app-arch/xz-utils-5.2.3
[ebuild     U  ] sys-libs/ncurses-6.1-r2 [6.0-r1]
[binary   R    ] sys-libs/readline-7.0_p3
[binary   R    ] virtual/libintl-0-r2
[binary   R    ] dev-lang/python-exec-2.4.5
[binary   R    ] virtual/libiconv-0-r2
[binary   R    ] sys-apps/gentoo-functions-0.12
[binary   R    ] dev-libs/libpcre-8.41-r1
[binary   R    ] sys-apps/sed-4.2.2
[binary   R    ] app-arch/bzip2-1.0.6-r8
[binary   R    ] dev-libs/gmp-6.1.2
[binary   R    ] app-shells/bash-4.4_p12
[binary   R    ] sys-apps/file-5.32
[binary   R    ] sys-devel/gnuconfig-20170101
[binary   R    ] dev-libs/mpfr-3.1.6
[binary   R    ] app-misc/c_rehash-1.7-r1
[binary   R    ] app-misc/mime-types-9
[binary   R    ] app-arch/tar-1.29-r3
[binary   R    ] app-arch/gzip-1.8
[binary   R    ] dev-libs/mpc-1.0.3
[binary   R    ] sys-devel/gcc-config-1.8-r1
[binary   R    ] app-misc/editor-wrapper-4
[binary   R    ] sys-apps/less-529
[binary   R    ] sys-apps/debianutils-4.8.3
[binary   R    ] net-libs/libmnl-1.0.4
[binary   R    ] sys-libs/libseccomp-2.3.2
[binary   R    ] dev-libs/popt-1.16-r2
[binary   R    ] sys-libs/e2fsprogs-libs-1.43.6
[binary   R    ] sys-devel/binutils-config-5-r4
[binary   R    ] dev-libs/libffi-3.2.1
[binary   R    ] virtual/libffi-3.0.13-r1
[binary   R    ] sys-apps/sysvinit-2.88-r9
[binary   R    ] sys-apps/opentmpfiles-0.1.3
[binary   R    ] virtual/tmpfiles-0
[binary   R    ] app-text/manpager-1
[binary   R    ] sys-libs/cracklib-2.9.6-r1
[binary   R    ] sys-apps/install-xattr-0.5
[binary   R    ] app-editors/nano-2.8.7
[binary   R    ] app-portage/elt-patches-20170815
[binary   R    ] sys-devel/m4-1.4.17
[binary   R    ] app-arch/unzip-6.0_p21-r2
[binary   R    ] sys-devel/autoconf-wrapper-13
[binary   R    ] sys-devel/bison-3.0.4-r1
[binary   R    ] sys-devel/flex-2.6.4-r1
[binary   R    ] dev-libs/libltdl-2.4.6
[binary   R    ] sys-devel/automake-wrapper-10
[binary   R    ] app-text/sgml-common-0.6.3-r6
[binary   R    ] dev-libs/libgpg-error-1.27-r1
[ebuild  N     ] dev-lang/perl-5.24.3-r1  USE="-berkdb -debug -doc -gdbm -ithreads"
[ebuild  N     ] sys-kernel/linux-headers-4.13  USE="-headers-only"
[ebuild  N     ] virtual/perl-Data-Dumper-2.160.0-r1
[ebuild  N     ] virtual/perl-Test-Harness-3.360.100_rc-r3
[ebuild  N     ] perl-core/File-Temp-0.230.400-r1
[ebuild  N     ] virtual/perl-File-Temp-0.230.400-r5
[ebuild  N     ] perl-core/File-Path-2.130.0
[ebuild  N     ] virtual/perl-File-Path-2.130.0
[binary   R    ] virtual/os-headers-0
[ebuild  N     ] sys-devel/autoconf-2.69-r4  USE="-emacs"
[ebuild  N     ] sys-apps/attr-2.4.47-r2  USE="-nls -static-libs"
[ebuild   R    ] sys-apps/coreutils-8.28-r1
[ebuild     U  ] app-admin/eselect-1.4.12 [1.4.8]
[ebuild     U  ] app-eselect/eselect-python-20171204 [20160516]
[ebuild     U  ] sys-devel/patch-2.7.6-r1 [2.7.5]
[ebuild  N     ] sys-apps/shadow-4.5  USE="cracklib xattr -acl -audit -nls -pam (-selinux) -skey"
[binary   R    ] virtual/shadow-0
[ebuild  N     ] virtual/perl-ExtUtils-MakeMaker-7.100.200_rc-r4
[ebuild  N     ] sys-libs/libcap-2.24-r2  USE="-pam -static-libs"
[ebuild  N     ] dev-perl/Text-Unidecode-1.270.0
[ebuild  N     ] dev-perl/libintl-perl-1.240.0-r2
[ebuild  N     ] sys-apps/help2man-1.47.4  USE="-nls"
[ebuild  N     ] sys-devel/automake-1.15.1-r2  USE="{-test}"
[ebuild  N     ] sys-devel/libtool-2.4.6-r3  USE="-vanilla"
[ebuild  N     ] dev-libs/expat-2.2.5  USE="unicode -examples -static-libs"
[ebuild   R    ] sys-process/psmisc-22.21-r3
[ebuild  N     ] sys-libs/gdbm-1.13-r2  USE="readline -berkdb -exporter -nls -static-libs"
[ebuild  N     ] sys-apps/groff-1.22.2  USE="-X -examples" L10N="-ja"
[ebuild  N     ] dev-libs/libelf-0.8.13-r2  USE="-debug -nls"
[ebuild  N     ] virtual/libelf-2
[ebuild  N     ] dev-libs/libgcrypt-1.8.1  USE="-doc -static-libs"
[ebuild  N     ] dev-perl/XML-Parser-2.440.0
[ebuild  N     ] virtual/perl-File-Spec-3.630.100_rc-r4
[ebuild  N     ] dev-perl/Unicode-EastAsianWidth-1.330.0-r1
[ebuild  N     ] sys-apps/texinfo-6.3  USE="-nls -static"
[ebuild  N     ] dev-libs/iniparser-3.1-r1  USE="-doc -examples -static-libs"
[ebuild  N     ] app-portage/portage-utils-0.64  USE="-nls -static"
[ebuild  N     ] dev-libs/openssl-1.0.2o  USE="asm sslv3 tls-heartbeat zlib -bindist -gmp -kerberos -rfc3779 -sctp -sslv2 -static-libs {-test} -vanilla"
[binary  N     ] dev-lang/python-2.7.14-r1  USE="ipv6 ncurses readline ssl (threads) (wide-unicode) xml (-berkdb) -build -doc -examples -gdbm -hardened -libressl -sqlite -tk -wininst"
[binary  N     ] sys-apps/openrc-0.34.11  USE="ncurses netifrc unicode -audit -debug -newnet -pam (-prefix) (-selinux) -static-libs" 
[ebuild  N     ] net-misc/netifrc-0.5.1
[binary   R    ] sys-apps/grep-3.0
[binary   R    ] sys-apps/findutils-4.6.0-r1
[binary   R    ] sys-apps/kbd-2.0.4
[ebuild  N     ] sys-apps/busybox-1.28.0  USE="ipv6 static -debug -livecd -make-symlinks -math -mdev -pam -savedconfig (-selinux) -sep-usr -syslog (-systemd)"
[binary   R    ] virtual/service-manager-0
[binary   R    ] sys-devel/binutils-2.29.1-r1
[ebuild  N     ] sys-apps/net-tools-1.60_p20161110235919  USE="arp hostname ipv6 -nis -nls -plipconfig (-selinux) -slattach -static" 
[binary   R    ] sys-apps/gawk-4.1.4
[binary   R    ] virtual/editor-0
[binary   R    ] sys-devel/make-4.2.1
[binary   R    ] sys-process/procps-3.3.12-r1
[binary   R    ] virtual/dev-manager-0-r1
[binary   R    ] sys-apps/which-2.21
[ebuild  N     ] net-misc/iputils-20171016_pre  USE="arping filecaps ipv6 openssl ssl -SECURITY_HAZARD -caps -clockdiff -doc -gcrypt
 (-idn) -libressl -nettle -rarpd -rdisc -static -tftpd -tracepath -traceroute"
[binary   R    ] virtual/pager-0
[binary   R    ] sys-apps/diffutils-3.5
[binary   R    ] sys-apps/baselayout-2.4.1-r2
[binary   R    ] virtual/libc-1
[binary   R   ~] sys-devel/gcc-7.3.0
[binary   R    ] virtual/pkgconfig-0-r1
[ebuild  N     ] dev-lang/python-3.5.5  USE="ipv6 ncurses readline ssl (threads) xml -build -examples -gdbm -hardened -libressl -sqlite {-test} -tk -wininst"
[ebuild  N     ] app-misc/ca-certificates-20170717.3.36.1  USE="-cacert -insecure_certs"
[ebuild  N     ] sys-apps/util-linux-2.30.2-r1  USE="cramfs ncurses readline suid unicode -build -caps -fdformat -kill -nls -pam -python (-selinux) -slang -static-libs (-systemd) {-test} -tty-helpers -udev" PYTHON_SINGLE_TARGET="python3_5 -python2_7 -python3_4 -python3_6" PYTHON_TARGETS="python2_7 python3_5 -python3_4 -python3_6"
[ebuild     U  ] app-misc/pax-utils-1.2.3 [1.1.7]
[ebuild     U  ] sys-apps/sandbox-2.13 [2.10-r4]
[ebuild     U  ] net-misc/rsync-3.1.3 [3.1.2-r2]
[ebuild  N     ] net-firewall/iptables-1.6.1-r3  USE="ipv6 -conntrack -netlink -nftables -pcap -static-libs"
[ebuild     U  ] dev-libs/libpipeline-1.4.2 [1.4.0]
[ebuild  N     ] sys-apps/man-db-2.7.6.1-r2  USE="gdbm manpager zlib -berkdb -nls (-selinux) -static-libs"
[ebuild     U  ] sys-apps/kmod-24 [23] PYTHON_TARGETS="-python3_6%"
[ebuild  N     ] dev-python/pyblake2-1.1.0  PYTHON_TARGETS="python2_7 python3_5 (-pypy) -python3_4 -python3_6"
[ebuild  N     ] net-misc/openssh-7.5_p1-r4  USE="hpn pie ssl -X -X509 -audit -bindist -debug -kerberos -ldap -ldns -libedit -libressl -livecd -pam -sctp (-selinux) -skey -ssh1 -static {-test}"
[ebuild  N     ] dev-util/gtk-doc-am-1.25-r1
[ebuild  N     ] dev-libs/libxml2-2.9.7  USE="ipv6 readline -debug -examples -icu -lzma -python -static-libs {-test}" PYTHON_TARGETS="python2_7 python3_5 -python3_4 -python3_6"
[ebuild  N     ] sys-devel/gettext-0.19.8.1  USE="cxx ncurses openmp -acl -cvs -doc -emacs -git -java (-nls) -static-libs"
[ebuild  N     ] app-text/build-docbook-catalog-1.19.1
[ebuild  N     ] dev-libs/libxslt-1.1.30-r2  USE="crypt -debug -examples -python -static-libs" PYTHON_TARGETS="python2_7"
[ebuild  N     ] app-text/docbook-xsl-stylesheets-1.79.1-r2  USE="-ruby"
[ebuild  N     ] app-text/docbook-xml-dtd-4.1.2-r6
[ebuild  N     ] dev-util/intltool-0.51.0-r2
[ebuild  N     ] dev-libs/glib-2.52.3  USE="mime xattr -dbus -debug (-fam) (-selinux) -static-libs -systemtap {-test} -utils" PYTHON_TARGETS="python2_7"
[ebuild  N     ] x11-misc/shared-mime-info-1.9  USE="{-test}"
[ebuild  N     ] dev-python/setuptools-36.7.2  USE="{-test}" PYTHON_TARGETS="python2_7 python3_5 (-pypy) (-pypy3) -python3_4 -python3_6"
[ebuild  N     ] dev-python/certifi-2017.4.17  PYTHON_TARGETS="python2_7 python3_5 (-pypy) (-pypy3) -python3_4 -python3_6"
[ebuild  N     ] dev-python/pyxattr-0.5.5  USE="-doc {-test}" PYTHON_TARGETS="python2_7 python3_5 (-pypy) -python3_4"
[ebuild  N     ] sys-apps/portage-2.3.24-r1  USE="(ipc) native-extensions xattr -build -doc -epydoc -gentoo-dev (-rsync-verify) (-selinux)" PYTHON_TARGETS="python2_7 python3_5 (-pypy) -python3_4 -python3_6"
[ebuild  N     ] app-admin/perl-cleaner-2.25
[binary   R    ] virtual/man-0-r1
[binary   R    ] virtual/modutils-0
[ebuild  N     ] sys-fs/e2fsprogs-1.43.6  USE="-fuse (-nls) -static-libs"
[ebuild     U  ] virtual/package-manager-1 [0]
[ebuild  N     ] sys-apps/iproute2-4.14.1-r2  USE="iptables ipv6 -atm -berkdb -minimal (-selinux)"
[binary   R    ] virtual/ssh-0
[ebuild  N     ] net-misc/wget-1.19.1-r2  USE="ipv6 pcre ssl zlib -debug -gnutls -idn -libressl -nls -ntlm -static {-test} -uuid"
[ebuild   R    ] dev-util/pkgconfig-0.29.2  USE="-internal-glib*"

!!! The following binary packages have been ignored due to non matching USE:

    =dev-util/pkgconfig-0.29.2 internal-glib
    =sys-apps/attr-2.4.47-r2 nls
    =sys-apps/man-db-2.7.6.1-r2 nls
    =dev-libs/libelf-0.8.13-r2 nls
    =sys-apps/shadow-4.5 -linguas_cs -linguas_da -linguas_de -linguas_es -linguas_fi -linguas_fr -linguas_hu -linguas_id -linguas_it -linguas_ja -linguas_ko -linguas_pl -linguas_pt_BR -linguas_ru -linguas_sv -linguas_tr -linguas_zh_CN -linguas_zh_TW nls

NOTE: The --binpkg-respect-use=n option will prevent emerge
      from ignoring these binary packages if possible.
      Using --binpkg-respect-use=y will silence this warning.

I think that’s broken the back of the job.  Of course when I come to running Catalyst, I’ll have to do it all over again, but at least now the environment is clean.

Solar Cluster: Next steps, better control of the charger

So, a few weeks ago I installed a new battery charger, and tweaked it so that the solar did most of the leg work during the day, and the charger kept the batteries topped up at night.

I also discussed the addition of a new industrial PC to perform routing and system monitoring functions… which was to run Gentoo Linux/musl. For now, that little PC is still running Debian Stretch, but for 45 days, it was rock solid. The addition of this box, and taking on the role of router to the management network meant I could finally achieve one of my long-term goals for the project: decommissioning the old server.

The old server is still set up with all my data and software… but now the back-up cron job calls /sbin/poweroff when it’s done, and the BIOS is set to wake the machine up in the evening ready to receive a back-up late at night.

In its place, a virtual machine clone of the box, handles my email and all the old functions of that server. This was all done just prior to my father and I leaving for a 3 week holiday in the Snowy Mountains.

I did have a couple of hiccups with Ceph OSDs crashing … but basically re-starting the daemons (done remotely whilst travelling through Cowra) got everything back up. A bit of placement group cleaning, and everything was back online again. I had another similar hiccup coming out of Maitland, but once again, re-starting the daemons fixed it. No idea why it crashed, that’s something I’ll have to investigate.

Other than that, the cluster itself has run well.

One thing that did momentarily kill the industrial PC though: I wandered down to the rack with a small bus-powered 2.5″ HDD with the intent of re-starting my Gentoo builds. This HDD had the same content as the 3.5″ HDD I had plugged in before. I figured being bus powered, I would not be dependent on mains, and it could just chug away to its heart’s content.

No such luck, the moment I plugged that drive in, the little machine took great umbrage to the spinning rust now vacuuming the electrons away from its core functions, and shut down abruptly. I’ve now brought my 3.5″ drive and dock down, plugged that into the wall, and have my builds resuming. If power goes off, hopefully the machine either handles the loss of swap gracefully. If it does crash, the watchdog will take care of it.

Thus, I have the little TS-7670 first attempting a build of gcc, to see how we go. Finger’s crossed our power should remain up. There was at least one outage in the time we were away, but hopefully we should get though this next build!

The next step I think should be to add some control of the mains charger to allow the batteries to be boosted to full charge overnight. The thinking is a simple diode-OR arrangement. Many comparators such as the LM393 have an open-collector output, which gives us this for free.

The theory is this.

The battery bank powers a simple circuit which runs of a 5V regulator. That regulator powers a dual comparator IC and provides a reference voltage. The comparator draws bugger all power, so I’m happy to use a linear PSU here. It’s mainly there as a voltage reference.

Precision isn’t really the aim here, so adjustable pots will make life easier.

The voltages from the battery bank and the solar panel are fed through voltage dividers to bring the voltages down to below 5V, then those voltages are individually fed into separate pots that control the hysteresis. I can adjust all points of the system.

The idea is that should the batteries get too low, or the sun go down, one or the other (or both) comparators will go low and pull down on R2. If the batteries are high and the sun is up, nothing pulls on R2 so the REMOTE+ pin on the HEP-600C-12 is allowed to float to +5V, turning off the mains charger.

The advantage of this is there’s no programming of a microcontroller, it’s just analogue electronics. The LM393s are pretty hardy things, the datasheet says they’ll run at 36V and can accept a maximum voltage of VCC-1.5V; so if I run at 5V, 3.5V is my recommended maximum. The adjustment pots should let me set a threshold voltage that avoids going above this.

I mainly need 5V for the HEP-600C-12, and for providing that stable known voltage reference. The LM78C05 should be fine for this.

Once I’ve done that, I should be able to wind that charger back up to its factory setting of 14.4V, which will mean that overnight the batteries will be charged back to full charge.

Solar Cluster: Beware, I’m ARMed

Last night, I got home, having made a detour on my way into work past Jaycar Wooloongabba to replace the faulty PSU.
It was a pretty open-and-shut case, we took it out of the box, plugged it in, and sure enough, no fan.  After the saleswoman asked the advice of a co-worker, it was confirmed that the fan should be running.
It took some digging, but they found a replacement, and so it was boxed up (in the box I supplied, they didn’t have one), and I walked out the door with PSU No. 3.
I had to go straight to work, so took the PSU with me, and that evening, I loaded it into the top box to transport home on the bicycle.
I get home, and it’s first thing on my mind.  I unlock the top box, get it out, and still decked out in my cycling gear, helmet and all (needed the headlight to see down the back of the rack anyway), I get to work.
I put the ring lugs on, plug it into the wall socket and flick the switch.
Nothing.
Toggle the switch on the front, still nothing.
Tried the other socket on the outlet, unplugging the load, still nothing.  Did the 10km trip from Milton to The Gap kill it?
Frustrated, I figure I’ll switch a light on.  Funny… no lights.
I wander into the study… sure enough, the router, modem and switch are dead as doornails.  Wander out to the MDB outside, saw the main breaker was still on, and tried hitting the test button.  Nothing.
I wander back inside, switching the bike helmet for my old hard hat, since it looks as if I’ll need the headlight a bit longer, then take a sticky beak down the road to see if anyone else is facing the same issue.
Sure enough, I look down the street, everyone’s out.
So there goes my second attempt at bootstrapping Gentoo, and my old server’s uptime.
The power did return about an hour or so later.  The PSU was fine, you don’t think of the mains being out as the cause of your problems.
I’ll re-start my build, but I’m not going to lose another build to failing power.  Nope, had enough of that for a joke.
I could have rigged up a UPS to the TS-7670, but I already have one, and it’s in the very rack where it’ll get installed anyway.  Thus, no time like the present to install it.
I’ll have to configure the switch to present the right VLANs to the TS-7670, but once I do that, it’ll be able to take over the role of routing between the management VLAN and the main network.
I didn’t want to do this in a VM because that means exposing the hosts and the VMs to the management VLAN, meaning anyone who managed to compromise a host would have direct access to the BMCs on the other nodes.
This is not a network with high bandwidth demands, and so the TS-7670 with its 100Mbps Ethernet (built into the SoC; not via USB) is an ideal machine for this task.
Having done this, all that’s left to do is to create a 2GB dual-core VM which will receive the contents of the old server, then that server can be shut down, after 8 years of good service.  I’ll keep it around for storing the on-site backups, but now I can keep it asleep and just wake it up with Wake-on-LAN when I want to make a back-up.
This should make a dint in our electricity bill!
Other changes…

  • Looks like we’ll be upgrading the solar with the addition of another 120W panel.
  • I will be hooking up my other network switches, the ADSL router and ADSL modem up to the battery bank on the cluster, just got to get some suitable cable for doing so.
  • I have no faith in this third PSU, so already, I have a MeanWell HEP-600C coming.  We’ll wire up a suicide lead to it, and that can replace the Powertech MP-3089 + Redarc BCDC1225, as the MeanWell has a remote on/off feature I can use to control it.

Solar Cluster: History repeats: another PSU fan bites the dust

Perhaps literally… it has bitten the dust.  Although I wouldn’t call its installed location, dusty.  Once again, the fan in the mains power supply has carked it.

Long-term followers of this project may remember that the last PSU failed the same way.

The reason has me miffed.  All I did with the replacement, was take the PSU out of its box, loosen the two nuts for the terminals, slip the ring lugs for my power lead over the terminals, returned the nuts, plugged it in and turned it on.

While it is running 24×7, there is nothing in the documentation to say this PSU can’t run that way.  This is what the installation looks like.

If it were dusty, I’d expect to be seeing hardware failures in my nodes.

This PSU is barely 4 months old, and earlier this week, the fan started making noises, and requiring percussive maintenance to get started. Tonight, it failed. Completely, no taps on the case will convince it to go.

Now, I need to keep things running until the weekend. I need it to run without burning the house down.

Many moons ago, my father bought a 12V fan for the caravan. Cheap and nasty. It has a slider switch to select between two speeds; “fast” and “slow”, which would be better named “scream like a banshee” and “scream slightly less like a banshee”. The speed reduction is achieved by passing current through a 10W resistor, and achieves maybe a 2% reduction in motor RPM. As you can gather, it proved to be a rather unwelcome room mate, and has seen its last day in the caravan.

This fan, given it runs off 12V, has proven quite handy with the cluster. I’ve got my SB-50 “load” socket hanging out the front of the cluster. A little adaptor to bring that out to a cigarette lighter socket, and I can run it off the cluster batteries. When a build job has gotten a node hot and bothered, sitting this down the bottom of the cluster and aiming it at a node has cooled things down well.

Tonight, it has another task … to try and suck the hot air out of the PSU.

That’s the offending power supply.  A PowerTech MP-3089.  It powers the RedARC BCDC-1225 right above it.  And you can see my kludge around the cooling problem.  Not great, but it should hold for the next 24 hours.

Tomorrow, I think we’ll call past Aspley and pick up another replacement.  I’m leery of another now, but I literally have no choice … I need it now.  Sadly, >250W 12V switchmode PSUs are somewhat rare beasts here in Brisbane.  Altronics don’t sell them that big.  The grinning glasses are no more, and I’m not risking it with the Xantrex charger again.

Long term, I’m already looking at the MeanWell SP-480-12.  This is a PSU module, and will need its own case and mains wiring… but I have no faith in the MP-3089 to not fail and cremate my home of 34 years.

The nice feature of the SP-480-12 is that it does have a remote +12V power-off feature.  Presumably I can drive this with a comparator/output MOSFET, so that when the battery voltage drops below some critical threshold, it kicks in, and when it rises above a high set-point, it drops out.  Simple control, with no MCU involved.  I don’t see a reason to get more fancy than that on the control side, anything more is a liability.

On other news, my gcc build on the TS-7670 failed … so much for the wait.  We’ll try another version and see how we go.

Solar Cluster: Resuming a build

So the house got momentarily power-cycled this morning… I’m at work, minding my own business, next thing the access point emails me this:

Mar 13 09:04:23 Syslogd start up

Now, it only does that for two reasons.  Either someone told it to reboot (not I), or it got hard reset.  Sure enough, log into the old server, and it’s reporting an uptime of 15 minutes.  I get home this evening, and clocks all around are on the blink … literally.

The cluster course is going, power outage?  What power outage?

I did consider wiring up the ADSL modem, router, study switch, and the TS-7670 up to the cluster’s power rails, but haven’t gotten around to doing that.  Alas, I’m not quite there yet.

In any case, even if the TS-7670 had been powered from the solar, I’d have still have temporarily lost the build as the HDD dock I have the hard drive sitting in is mains powered.  It also doesn’t remember its state after a power cycling.  I’d have re-started the build from work, but the HDD remained off when the power came back on.

Never mind.  The downside is now I get to re-start a multi-day build.  The good news though, is that knowing the ebuild file that Portage picked out for compiling gcc; I can resume where it left off.  In this case, it’s using an ebuild from the musl overlay; /root/musl/sys-devel/gcc/gcc-6.4.0-r1.ebuild.

ebuild /root/musl/sys-devel/gcc/gcc-6.4.0-r1.ebuild package will preserve the current working tree and will resume where it was, hopefully without incident.  I’ll be left with a .tbz2; which will be picked up when I run emerge –keep-going -ekv @system.

Solar Cluster: arm-unknown-linux-musleabi… saga part III

So, after a longish wait… my laptop finally coughed up an image with a C/C++ compiler and almost all the bits necessary to make Gentoo Portage tick.

Almost everything… wget built, but it segfaults on start-up.  No matter, it seems curl works.  We do have an issue though: Portage no longer supports customising the downloader like it used to, or at least I couldn’t see how to do it, it used to be settings in make.conf.

Thankfully, I know shell scripts, and can make my own wget using the working curl:

bash-4.4# cat > /usr/bin/wget
#!/bin/bash

OUT=
while [ $# -gt 0 ]; do
    case "$1" in
        -O) OUT="$2"; shift;;
        -t) shift;;
        -T) shift;;
        --passive-ftp) : ;;
        *) break ;;
    esac
    shift
done

set -ex
curl --progress-bar -o "${OUT}" "$1"

Okay, it’s a little (a lot) braindead, but it beats downloading the lot by hand!

I was able to get Gentoo installed by hand using these instructions.  I have an old 1TB HDD plugged into a USB dock, formatted with a 10GB swap partition and the rest btrfs.  Sure, it’s only USB 2.0, but I’d sooner just put up with some CPU overhead than wear out my eMMC.

Next step; ROOT=/tmp/seed emerge -ev system

Solar Cluster: arm-unknown-linux-musleabi… a saga

So, fun and games with the TS-7670.

At present, I have it up and running:

root@ts7670:~# uname -a
Linux ts7670 4.14.15-vrt-ts7670-00031-g1a006273f907-dirty #2 Sun Jan 28 20:21:08 EST 2018 armv5tejl GNU/Linux

That’s booted up into Debian Stretch right now.  debootstrap did its deed a few days ago on the eMMC, and I was able to boot up this new image.  Today I built a new kernel, and tweaked U-Boot to boot from eMMC.

Thus now the unit can boot without any MicroSD cards fitted.

There’s a lot of bit rot to address.  U-Boot was forked from some time in 2014.  I had a crack at rebasing the code onto current U-Boot, but there’s a lot of clean-up work to do just to get it to compile.  Even the kernel needed some fixes to get the newer devicetree sources to build.

As for getting Gentoo working… I have a cross-compiling toolchain that works.  With it, I’ve been able to compile about 99% of a seed stage needed for catalyst.  The 1% that eludes me, is GCC (compiled to run on ARMv5).  GCC 4.9.4 will try to build, but fails near the end… anything newer will barf complaining that my C++ compiler is not working.  Utter bollocks, both AMD64 and ARM toolchains have working C++ compilers, just it’s looking for a binary called “g++” rather than being specific about which one.  I suspect it wants the AMD64 g++, but then if I symlink that to /usr/bin/g++, it throws in ARM CFLAGS, and AMD64 g++ barfs on those.

I’ve explored other options.  I can compile GCC by hand without C++ support, and this works, but you can’t build modern GCC without a C++ compiler … and people wonder why I don’t like C++ on embedded!

buildroot was my next thought, but as it happens, they’ve stripped out the ability to compile a native GCC on the target.

crosstool-ng is the next logical choice, but I’ll have to fiddle with settings to get the compiler to build.

I’ve also had OpenADK suggested, which may be worth a look.  Other options are OpenEmbedded/Yocto, and Cross Linux from Scratch.  I think for the latter, cross is what I’ll get, this stuff can be infuriatingly difficult.

Solar Cluster: arm-unknown-linux-musleabi… saga part II

So, last time I was trying to get Gentoo’s portage to cross-build gcc so that I’d have a C/C++ compiler in my ARMv5 musl environment.

It is literally the last piece of the puzzle.  Once compiled, that is the last step I need before I can throw the shiny new environment onto an ARMv5 VM (or real ARMv5 CPU), do an emerge -e world on it then tar the lot up and throw it at Catalyst.

Building an entire OS on a 454MHz ARMv5 machine with 128MB RAM does not faze me one bit… I used to do it regularly on a (Gateway-branded) Cobalt Qube II server appliance, which sports a 250MHz QED RM5231 and 128MB RAM.  The other compile workhorse I used in those days was an SGI O2; 300MHz RM5200, again 128MB RAM.

Yes, Linux and its userland has bulked up a bit in the last 10 years, but not so much so that a build on these is impossible.

Certainly, native building is easier than cross-compiling.  Cross-compilers have always been a voodoo art for me.  Getting one that will build a Linux kernel or U-Boot, usually isn’t too hard… but get userland involved and it gets complex.  Throw in C++ and complexity skyrockets!

I’m taking OpenADK for a spin now, and in concept, it’s exactly what I remember buildroot used to be.  It’s a tool for generating a fully fledged embedded Linux system with a wide package selection including development tools.  I also find that you have to hold your tongue just right to get stuff to compile.

Selecting a generic arm926ej-s; it succeeded to build a x86-64 hosted cross-toolchain once, but then silently refused to build anything else.  I told it instead to build for a Versatile PB with an arm926ej-s CPU… it failed to build the cross-toolchain, even though it pretty much is the exact same target.

A make cleandirs later, and it happily started building everything, but then hiccupped on permissions, so against my better judgement, I’m running it now with sudo, and things are progressing.  With some luck, I should have something that will give me a working native gcc/g++ for musl on ARMv5.

Bootstrapping Gentoo Linux

So, in amongst my pile of crusty old hardware is the old netbook I used to use in the latter part of my univerity days. It is a Lemote Yeeloong, and sports a ~700MHz Loongson 2F CPU (MIPS III little endian ISA) and 1GB RAM.

Back in the day it was a brilliant little machine. It came out of the box running a localised (for China) version of Debian, and had pretty much everything you’d need. I natually repartitioned the machine, setting up Gentoo and I had a separate partition for Debian, so I could actually dual-boot between them.

Fast forward 10 years, the machine runs, but the battery is dead, and Debian no longer supports MIPS-III machines. Debian Jessie does, but Stretch, likely due for release some time this year, will not, if you haven’t got a CPU that supports mips32r2 or mips64r2, you’re stuffed.

I don’t want to throw this machine away.  Being as esoteric as it is, it is an unlikely target for theft, as to the casual observer, it’ll just be “some crappy netbook”.  If someone were to try and steal it, there’s a very high probability I’ll recover it with my data because the day its PMON2000 boot firmware successfully boots a x86-64 OS like Ubuntu or Windows without the assistance of a VM of some kind would be the day Satan puts a requisition order in for anti-freeze and winter mittens.

My use case is for a machine I can take with me on the bicycle.  My needs aren’t huge: I won’t be playing video on this thing, it’ll be largely for web browsing and email.  The web browser needs to support JavaScript, so that rules out options like ELinks or Dillo, my preferred browser is Firefox but I’ll settle for something Webkit-based if that’s all that’s out there.

So what operating systems do I have for a machine that sports a MIPS-III CPU and 1GB RAM?  Fedora has a MIPS port, but that, like Debian, is for the newer MIPS systems.  Arch Linux too is for newer architectures.

I could bootstrap Alpine Linux… and maybe that’s worth looking into, they seem to be doing some nice work in producing a small and capable Linux distribution.  They don’t yet support MIPS though.

Linux From Scratch is an option, if a little labour intensive.  (Been there, done that.)

OpenBSD directly supports this machine, and so I gave OpenBSD 6.0 a try.  It’s a very capable OS, and while it isn’t Linux, there isn’t much that an experienced Linux user like myself needs to adapt to in order to effectively use the OS.  pkgsrc is a great asset to OpenBSD, with a large selection of pre-built packages already available.  Using that, it is possible to get a workable environment up and running very quickly.  OpenBSD/loongson uses the n64 ABI.

Due to licensing worries, they use a particularly old version of binutils as their linker and assembler.  The plan seems to be they wish to wean themselves off the GNU toolchain in favour of LLVM.  At this time though, much of the system is built using the GNU toolchain with some custom patches.  I found that, on the Yeeloong, 1GB RAM was not sufficient for compiling LLVM, even after adding additional swap files, and some packages I needed weren’t available in pkgsrc, nor would they build with the version of GNU tools available.

Maybe as they iron out the kinks in their build environment with LLVM, this will be worth re-visiting.  They’ve done a nice job so far, but it’s not quite up to where I need it to be.

Gentoo actually gives me the choice of two possible ABIs: o32 and n32o32 is the old 32-bit ABI, and suffers a number of performance problems, but generally works.  It’s what Debian Jessie and earlier supplies, and what their mips32 port will produce from Stretch onwards.

n32 is the MIPS equivalent of what some of you may know as x32 on AMD64 platforms, it is a 32-bit environment with 64-bit long pointers… the idea being that very few applications actually benefit from the use of 64-bit data types, and so the usual quantities like int and long remain the same as what they’d be on o32, saving memory.  The long long data type gets a boost because, although “32-bit”, the 64-bit operations are still available for use.

The trouble is, some applications have problems with this mode.  Either the code sees “mips64” in the CHOST and assumes a full 64-bit system (aka n64), or it assumes the pointers are the same width as a long, or the build system makes silly assumptions as to where things get put.  (virtualenv comes to mind, which is what started me on this journey.  The same problem affects x32 on AMD64.)

So I thought, I’d give n64 a try.  I’d see if I can build a cross-compiler on my AMD64 host, and bootstrap Gentoo from that.

Step 1: Cross-compiler

For the cross-compiler, Gentoo has a killer feature that I have not seen in too many other distributions: crossdev.  This is a toolchain build tool that can generate cross-compiler toolchains for most processor architectures and environments.

This is installed by running emerge sys-devel/crossdev.

A gotcha with hardened

I run “hardened” AMD64 stages on my machines, and there’s a little gotcha to be aware of: the hardened USE flag gets set by crossdev, and that can cause fun and games if, like on MIPS, the hardening features haven’t been ported.  My first attempt at this produced a n64 userland where pretty much everything generated a segmentation fault, the one exception being Python 2.7.  If I booted with init=/bin/bash (or init=/bin/bb), my virtual environment died, if I booted with init=/usr/bin/python2.7, I’d be dropped straight into a Python shell, where I could import the subprocess module and try to run things.

Cleaning up, and forcing crossdev to leave off hardened support, got things working.

Building the toolchain

With the above gotcha in mind:

# crossdev --abis n64 \
           --env 'USE="-hardened"' \
           -s4 -t mips64el-unknown-linux-gnu

The --abis n64 tells crossdev you want a n64 ABI toolchain, and the --env will hopefully keep the hardened flag unset. Failing that, try this:

# cat > /etc/portage/package.use/mips64 <<EOF
cross-mips64el-unknown-linux-gnu/binutils -hardened
cross-mips64el-unknown-linux-gnu/gcc -hardened
cross-mips64el-unknown-linux-gnu/glibc -hardened
EOF

If you want a combination of specific toolchain components to try, I’m using:

  • Binutils: 2.28
  • GCC: 5.4.0-r3
  • glibc: 2.25
  • headers: 4.10

Step 2: Checking our toolchain

This is where I went wrong the first time, I tried building the entire OS, only to discover I had wasted hours of CPU time building non-functional binaries. Save yourself some frustration. Start with a small binary to test.

A good target for this is busybox. Run mips64el-unknown-linux-gnu-emerge busybox, and wait for a bit.

When it completes, you should hopefully have a busybox binary:

RC=0 stuartl@beast ~ $ file /usr/mips64el-unknown-linux-gnu/bin/busybox 
/usr/mips64el-unknown-linux-gnu/bin/busybox: ELF 64-bit LSB executable, MIPS, MIPS-III version 1 (SYSV), statically linked, for GNU/Linux 3.2.0, stripped

Testing busybox

There is qemu-user-mips64el, but last time I tried it, I found it broken. So an easier option is to use real hardware or QEMU emulating a full system. In either case, you’ll want to ensure you have your system-of-choice running with a working 64-bit kernel already, if your real hardware isn’t already running a 64-bit Linux kernel, use QEMU.

For QEMU, the path-of-least-resistance I found was to use Debian. Aurélien Jarno has graciously provided QEMU images and corresponding kernels for a good number of ports, including little-endian MIPS.

Grab the Wheezy disk image and the corresponding kernel, then run the following command:

# qemu-system-mips64el -M malta \
    -kernel vmlinux-3.2.0-4-5kc-malta \
    -hda debian_wheezy_mipsel_standard.qcow2 \
    -append "root=/dev/sda1 console=ttyS0,115200" \
    -serial stdio -nographic -net nic -net user

Let it boot up, then log in with username root, password root.

Install openssh-client and rsync (this does not ship with the image):

# apt-get update
# apt-get install openssh-client rsync

Now, you can create a directory, and pull the relevant files from your host, then try the binary out:

# mkdir gentoo
# rsync -aP 10.0.2.2:/usr/mips64el-unknown-linux-gnu/ gentoo/
# chroot gentoo bin/busybox ash

With luck, you should be in the chroot now, using Busybox.

Step 3: Building the system

Having done a “hello world” test, we’re now ready to build everything else. Start by tweaking your /usr/mips64el-unknown-linux-gnu/etc/portage/make.conf to your liking then adjust /usr/mips64el-unknown-linux-gnu/etc/portage/make.profile to point to one of the MIPS profiles. For reference, on my system:

RC=0 stuartl@beast ~ $ ls -l /usr/mips64el-unknown-linux-gnu/etc/portage/make.profile
lrwxrwxrwx 1 root root 49 May  1 09:26 /usr/mips64el-unknown-linux-gnu/etc/portage/make.profile -> /usr/portage/profiles/default/linux/mips/13.0/n64
RC=0 stuartl@beast ~ $ cat /usr/mips64el-unknown-linux-gnu/etc/portage/make.conf 
CHOST=mips64el-unknown-linux-gnu
CBUILD=x86_64-pc-linux-gnu
ARCH=mips

HOSTCC=x86_64-pc-linux-gnu-gcc

ROOT=/usr/${CHOST}/

ACCEPT_KEYWORDS="mips ~mips"

USE="${ARCH} -pam"

CFLAGS="-O2 -pipe -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}"

FEATURES="-collision-protect sandbox buildpkg noman noinfo nodoc"
# Be sure we dont overwrite pkgs from another repo..
PKGDIR=${ROOT}packages/
PORTAGE_TMPDIR=${ROOT}tmp/

ELIBC="glibc"

PKG_CONFIG_PATH="${ROOT}usr/lib/pkgconfig/"
#PORTDIR_OVERLAY="/usr/portage/local/"

Now, you should be ready to start building:

# mips64el-unknown-linux-gnu-emerge -e \
    --keep-going -j6 --load-average 12.0 @system

Now, go away, and do something else for several hours.  It’ll take that long, depending on the speed of your machine.  In my case, the machine is an AMD Phenom II x6 with 8GB RAM, which was brand new in 2010.  It took a good day or so.

Step 4: Testing our system

We should have enough that we can boot our QEMU VM with this image instead.  One way of trying it would be to copy across the userland tree the same way we did for pulling in busybox and chrooting back in again.

In my case, I took the opportunity to build a kernel specifically for the VM that I’m using, and made up a disk image using the new files.

Building a kernel

Your toolchain should be able to cross-build a kernel for the virtual machine.  To get you started, here’s a kernel config file.  Download it, decompress it, then drop it into your kernel source tree as .config.

Having done that, run make olddefconfig ARCH=mips to set the defaults, then make menuconfig ARCH=mips and customise to your hearts content. When finished, run make -j6 vmlinux modules CROSS_COMPILE=mips64el-unknown-linux-gnu- to build the kernel and modules.

Finally, run make modules_install firmware_install INSTALL_MOD_PATH=$PWD/modules CROSS_COMPILE=mips64el-unknown-linux-gnu- to install the kernel modules and firmware into a convenient place.

Making a root disk

Create a blank, raw disk image using qemu-img, then partition it as you like and mount it as a loopback device:

# qemu-img create -f raw gentoo.raw 8G
# fdisk gentoo.raw
(do your partitioning here)
# losetup -P /dev/loop0 $PWD/gentoo.raw

Now you can format the partitions /dev/loop0pX as you see fit, then mount them in some convenient place. I’ll assume that’s /mnt/vm for now. You’re ready to start copying everything in:

# rsync -aP /usr/mips64el-unknown-linux-gnu/ /mnt/vm/
# rsync -aP /path/to/kernel/tree/modules/ /mnt/vm/

You can use this opportunity to make some tweaks to configuration files, like updating etc/fstab, tweaking etc/portage/make.conf (changing ROOT, removing CBUILD), and setting up a getty on ttyS0. I also like to symlink lib to lib64 in non-multilib environments such as this: Don’t symlink lib and lib64! See below.

# cd /mnt/vm
# mv lib/* lib64
# rmdir lib
# ln -s lib64 lib
# cd usr
# mv lib/* lib64
# rmdir lib
# ln -s lib64 lib

When you’re done, unmount.

First boot

Run QEMU with the following arguments:

# qemu-system-mips64el -M malta \
    -kernel /path/to/your/kernel/vmlinux \
    -hda /path/to/your/gentoo.raw \
    -append "root=/dev/sda1 console=ttyS0,115200 init=/bin/bash" \
    -serial stdio -nographic -net nic -net user

It should boot straight to a bash prompt. Mount the root read/write, and then you can make any edits you need to do before boot, such as changing the root password. When done, re-mount the root as read-only, then exec /sbin/init.

# mount / -o rw,remount
# passwd
… etc
# mount / -o ro,remount
# exec /sbin/init

With luck, it should boot to completion.

Step 5: Making the VM a system service

Now, it’d be real nice if libvirt actually supported MIPS VMs, but it doesn’t appear that it does, or at least I couldn’t get it to work.  virt-manager certainly doesn’t support it.

No matter, we can make do with a telnet console (on loopback), and supervisord to daemonise QEMU.  I use the following supervisord configuration file to start my VMs:

[unix_http_server]
file=/tmp/supervisor.sock   ; (the path to the socket file)

[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false               ; (start in foreground if true;default false)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

[program:qemu-mips64el]
command=/usr/bin/qemu-system-mips64el -cpu MIPS64R2-generic -m 2G -spice disable-ticketing,port=5900 -M malta -kernel /home/stuartl/kernels/qemu-mips/vmlinux -hda /var/lib/libvirt/images/gentoo-mips64el.raw -append "mem=256m@0x0 mem=1792m@0x90000000 root=/dev/sda1 console=ttyS0,115200" -chardev socket,id=char0,port=65223,host=::1,server,telnet,nowait -chardev socket,id=char1,port=65224,host=::1,server,telnet,nowait -serial chardev:char0 -mon chardev=char1,mode=readline -net nic -net bridge,helper=/usr/libexec/qemu-bridge-helper,br=br0

The following creates two telnet sockets, port 65223 is the VM’s console, 65224 is the QEMU control console. The VM has the maximum 2GB RAM possible and uses bridged networking to the network bridge br0. There is a graphical console available via SPICE.

All telnet and SPICE interfaces are bound to loopback, so one must use SSH tunnelling to reach those ports from another host. You can change the above command line to use VNC if that’s what you prefer.

At this point, the VM should be able to boot on its own. I’d start with installing some basic packages, and move on from there. You’ll find the environment is very sparse (my build had no Perl binary for example) but the basics for building everything should be there.

You may also find that what is there, isn’t quite installed right… I found that sshd wasn’t functional due to missing users… a problem soon fixed by doing an emerge -K openssh (the earlier step will have produced binary packages).

In my case, that’s installing a decent text editor (vim) and GNU screen so I can start a build, then detach.  Lastly, I’ll need catalyst, which is Gentoo’s release engineering tool.

At the moment, this is where I’m at.  GNU screen has indirectly pulled in Perl as a dependency, and that is building as I type this.  It is building faster than the little netbook does, and I have the bonus that I can throw more RAM at the problem than I can on the real hardware. The plan from here:

  1. emerge -ek @system, to build everything that got missed before.
  2. ROOT=/tmp/seed emerge -eK @system, to bundle everything up into a staging area
  3. populating /tmp/seed/dev with device files
  4. tar-ing up /tmp/seed to make my initial “seed” stage for catalyst.
  5. building the first n64 stages for Gentoo using catalyst
  6. building the packages I want for the netbook in a chroot
  7. transferring the chroot to the netbook

Symlinking lib and lib64… don’t do it!

So, I was doing this years ago when n32 was experimental.  I recall it being necessary then as this was before Portage having proper multilib support.  The earlier mipsel n32 stages I built, which started out from kanaka‘s even more experimental multilib stages, required this kludge to work-around the lack of support in Portage.

Portage has changed, it now properly handles multilib, and so the symlink kludge is not only not necessary, it breaks things rather badly, as I discovered.  When packages merge files to /lib, rather than following the symlink, they’ll replace it with a directory.  At that point, all hell breaks loose, because stuff that “appeared” in /lib before is no longer there.

I was able to recover by rsync-ing /lib64 to /lib, which isn’t a pretty solution, but it’ll be enough to get an initial “seed” stage.  Running that seed stage through Catalyst will clean up the remnants of that bungle.