Sep 232018

Well, I’ve now had the controller working for a week or so now… the solar output has never been quite what I’d call, “great”, but it seems it’s really been on the underwhelming side.

One of the problems I had earlier before moving to this particular charger was that the Redarc wouldn’t reliably switch between boosting from 12V to MPPT from solar.  It would get “stuck” and not do anything.  Coupled with the fact that there’s no discharge protection, and well, the results were not a delight to the olfactory nerves at 2AM on a Sunday morning!

It did okay as a MPPT charger, but I needed both functions.  Since the thinking was I could put a SSR between the 12V PSU and the Redarc charger, we tried going the route of buying the Powertech MP3735 solar charge controller to handle the solar side.

When it wants to work, it can put over 14A in.  The system can run on solar exclusively.  But it’s as if the solar controller “hesitates”.

I thought maybe the other charger was confusing it, but having now set up a little controller to “turn off” the other charger, I think I can safely put that theory to bed.  This was the battery voltage yesterday, where there was pretty decent sunshine.

There’s an odd blip at about 5:40AM, I don’t know what that is, but the mains charger drops its output by a fraction for about 50 seconds.  At 6:37AM, the solar voltage rises above 14V and the little ATTiny24A decides to turn off the mains charger.

The spikes indicate that something is active, but it’s intermittent.  Ultimately, the voltage winds up slipping below the low voltage threshold at 11:29AM and the mains charger is brought in to give the batteries a boost.  I actually made a decision to tweak the thresholds to make things a little less fussy and to reduce the boost time to 30 minutes.

The charge controller re-booted and turned off the mains charger at that point, and left it off until sunset, but the solar controller really didn’t get off its butt to keep the voltages up.

At the moment, the single 120W panel and 20A controller on my father’s car is outperforming my 3-panel set-up by a big margin!

Today, no changes to the hardware or firmware, but still a similar story:

The battery must’ve been sitting just on the threshold, which tripped the charger for the 30 minutes I configured yesterday.  It was pretty much sunny all day, but just look at that moving average trend!  It’s barely keeping up.

A bit of searching suggests this is not a reliable piece of kit, with one thread in particular suggesting that this is not MPPT at all, and many people having problems.

Now, I could roll the dice and buy another.

I could throw another panel on the roof and see if that helps, we’re considering doing that actually, and may do so regardless of whether I fix this problem or not.

There’s several MPPT charger projects on this very site.  DIY is a real possibility.  A thought in the back of my mind is to rip the Powertech MP3735 apart and re-purpose its guts, and make it a real MPPT charger.

Perhaps one with Modbus RTU/RS-485 reporting so that I can poll it from the battery monitor computer and plot graphs up like I’m doing now for the battery voltage itself.  There’s a real empty spot for 12V DC energy meters that speak Modbus.

If I want a 240V mains energy meter, I only have to poke my head into the office of one of my colleagues (who works for the sister company selling this sort of kit) and I could pick up a little CET PMC-220 which with the addition of some terminating resistors (or just run comms at 4800 baud), work just fine.  Soon as you want DC, yeah, sure there’s some for solar set-ups that do 300V DC, but not humble 12V DC.

Mains energy meters often have extra features like digital inputs/outputs, so this could replace my little charge controller too.  This would be a separate project.

But that would leave me without a solar controller, which is not ideal, and I need to shut everything down before I can extract the existing one.  So for now, I’ve left the Powertech one in-place, disconnected its solar input so that now it just works as a glorified VSR and voltmeter/ammeter, as that bit works fine.

The Redarc is now hooked up to solar, with its output going into a spare socket going to the batteries.  This will cost me nothing to see if it’s the solar controller or not.  If it is, then I think some money on a VSR to provide the low-voltage protection, and re-instating the Redarc charger for solar duty will be the next step.  Then I can tear down the Powertech one at my leisure and figure out what they did wrong, or if it can be re-programmed.

The Meanwell charger is taking care of things as I type this, but tomorrow morning, we should hopefully see the solar set-up actually do some work…

… maybe. 🙂

Sep 132018

A few months back now, I had the misfortune of overshooting my Internet quota, and winding up with a AU$380 bill for the month (and that was capped… in truth it was more like AU$3000).  In fact, it happened a couple of times until I finally nailed down the cause.

Part of it was NTP traffic (seems lots of cowboys write SNTP clients now and point them at, some was the Spambot Hunter Project and related activity.  In short, I invested some money into upping the quota, and some time into better monitoring.

I wanted to do the monitoring anyway to keep an eye on operations, as well as things like the solar panel voltages, etc.  Since I got it in place, I’ve been able to get much faster notifications of when things go awry.  Much sooner than the 120% quota usage alarm that Internode sends you.

I’m glad I did that now, last night I left a few tabs open on the site.  I noticed this evening they were still trying to load something and got suspicious… then I saw this:

Double checking, sure enough, something on one of those pages made Chromium get its knickers into a twist, and chew through all that data.

It took me a bit of tinkering to get the right query to extract the above chart.  Essentially there was a sustained 1.5MB/sec download for over 21 hours which would account for the 113.1GB that Internode recorded.

It’s a bit co-incidental that the usage dropped the moment I re-started Chromium.  Not sure why it was continually re-loading pages, but never mind.

The above data is collected using a combination of collectd and InfluxDB, with Grafana doing the dashboarding and alarms, and a small Perl script pulling the usage data off Internode’s API.

Sep 012018

Well, I’ve been tossing up how to control the mains charger for a while now.

When I first started the project, my thinking was to use an old Xantrex charger we had kicking around, and just electrically disconnect it from the batteries when we wanted to use the solar power.  I designed a 4-layer PCB which sported a ATTiny24A microcontroller, MOSFETs (which I messed up) and some LEDs.

This was going to be a combined fan controller and power management unit.  It had the ability (theoretically) to choose a supply based on the input voltage, and to switch if needed between supplies.

It didn’t work out, the charger got really confused by the behaviour of the controller.  I was looking to re-instate it using the Redarc solar controller, but I never got there.  In the end, it was found that the Redarc controller had problems switching sources and would do nothing whilst the batteries went flat.

We’ve now replaced both ends of the system.  The solar controller is a Powertech MP3735 and integrates over-discharge protection.  The mains charger is now a MeanWell HEP-600C-12 (which has not missed a beat since the day it was put in).

Unlike my earlier set-up, this actually has a 5V logic signal to disable it, and my earlier controller could theoretically generate that directly.

Looking at the PCB of my earlier power controller attempt, it looks like this could still work.

Above is the PCB artwork.  I’ve coloured in the sections and faded out the parts I can omit.

In green up the top-left we have the mains control/monitoring circuitry.  We no longer see the mains voltage, so no point in monitoring it, so we can drop the resistor divider that fed the ADC.  This also means we no longer need the input socket P2.

Q2 and Q7 were the footprints of the two P-channel MOSFETs.  We don’t need the MOSFETs themselves, but the signals we need can be found on pin 1 of Q2.  This is actually the open-drain output of Q1, which we may be able to hook directly to the REMOTE+ pin on the charger.  A pull-up between there and the charger’s 5V rail, and we should be in business.

In yellow, bottom left is the solar monitoring interface.  This is still useful, but we won’t be connecting solar to the battery ourselves, so we just keep the monitoring parts.  The LED can stay as an indicator to show when solar is “good enough”.

In purple, occupying most of the board, is the controller itself.  It stays for obvious reasons.

In red, is the fan control circuitry.  No reason why this can’t stay.

In blue is the circuitry for monitoring the battery voltage.  Again, this stays.

The main advantage of doing this is I already have the boards, and a number of microcontrollers already present.  There’s a board with all except the big MOSFETs populated: with the MOSFETs replaced by 3-pin KK sockets.

How would the logic work?  Much the same as the analogue version I was pondering.

  • If battery voltage is low, OR, the sun has set, enable the mains charger.

What concerned me about an analogue solution was what would happen once the charger got to the constant-voltage stage.  We want to give it a bit of time to keep the battery topped up.  Thus it makes sense to shut down the charger after a fixed delay.

This is easy to do in a microcontroller.  Not hard with analogue electronics either, it’s fundamentally just a one-shot, but doing it with an MCU is a single-chip solution.  I can make the delay as long as I like.  So likely once the battery is “up to voltage”, I can let it float there for an hour, or until sunrise if it’s at night.

Aug 302018

So, I’m happy enough with the driver now that I’ll collapse down the commits and throw it up onto the Github repository.  I might take another look at kernel 4.18, but for now, you’ll find them on the ts7670-4.14.67 branch.

Two things I observe about this voltage monitor:

  1. The voltage output is not what you’d call, accurate.  I think it’s only a 10-bit ADC, which is still plenty good enough for this application, but the reading I think is “high” by about 50mV.
  2. There’s significant noise on the reading, with noticeable quantisation steps.

Owing to these, and to thwart the possibility of using this data in side-channel attacks using power analysis, I’ve put a 40-sample moving-average filter on the “public” data.

Never the less, it’s a handy party trick, and not one I expected these devices to be able to do.  My workplace manages a big fleet of these single-board computers in the residential towers at Barangaroo where they spend all day polling Modbus and M-Bus meters.  In the event we’re at all suspicious about DC power supplies though, it’s a simple matter to load this kernel tree (they already run U-Boot) and configure collectd (which is also installed).

I also tried briefly switching off the mains power to see that I was indeed reading the battery voltage and not just a random number that looked like the voltage.  That yielded an interesting effect:

You can see where I switched the mains supply off, and back on again.  From about 8:19PM the battery voltage predictably fell until about 8:28PM where it was at around 12.6V.

Then it did something strange, it rose about 100mV before settling at 12.7V.  I suspect if I kept it off all night it’d steadily decrease: the sun has long set.  I’ve turned the mains charger back on now, as you can see by the step-rise shortly after 8:44PM.

The bands on the above chart are the alert zones.  I’ll get an email if the battery voltage strays outside of that safe region of 12-14.6V.  Below 12V, and I run the risk of deep-cycling the batteries.  Above 14.6V, and I’ll cook them!

The IPMI BMCs on the nodes already sent me angry emails when the battery got low, so in that sense, Grafana duplicates that, but does so with pretty charts.  The BMCs don’t see when the battery gets too high though, for the simple matter that what they see is regulated by LDOs.

Aug 302018

I’ve succeeded in getting a working battery monitor kernel module. This is basically taking the application note by Technologic Systems and spinning that into a power supply class driver that reports the voltage via sysfs.

As it happens, the battery module in collectd does not see this as a “battery”, something I’ll look at later. For now the exec plug-in works well enough. This feeds through eventually to an InfluxDB database with Grafana sitting on top.

Aug 282018

So, I successfully last night, parted the core bits out of ts_wdt.c and make ts-mcu-core.c.  This is a multi-function device, and serves to provide a shared channel for the two drivers that’ll use it.

Tonight, I took a stab at writing the PSU part of it.  Suffice to say, I’ve got work to do:

[  158.712960] Unable to handle kernel NULL pointer dereference at virtual address 00000005
[  158.721328] pgd = c3854000
[  158.724089] [00000005] *pgd=4384f831, *pte=00000000, *ppte=00000000
[  158.730629] Internal error: Oops: 1 [#3] ARM
[  158.734947] Modules linked in: 8021q garp mrp stp llc nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables flexcan can_dev
[  158.755812] CPU: 0 PID: 2059 Comm: cat Tainted: G      D         4.14.67-vrt-ts7670+ #3
[  158.763840] Hardware name: Freescale MXS (Device Tree)
[  158.769008] task: c68f3a20 task.stack: c3846000
[  158.773598] PC is at ts_mcu_transfer+0x1c/0x48
[  158.778073] LR is at 0x3
[  158.780630] pc : []    lr : [<00000003>]    psr: 60000013
[  158.786918] sp : c3847e44  ip : 00000000  fp : 014000c0
[  158.792165] r10: c5035000  r9 : c5305900  r8 : c777b428
[  158.797412] r7 : c0a7fa80  r6 : c777b400  r5 : c5035000  r4 : c3847e6c
[  158.803961] r3 : c3847e58  r2 : 00000001  r1 : c3847e4c  r0 : c07b8c68
[  158.810512] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  158.817671] Control: 0005317f  Table: 43854000  DAC: 00000051
[  158.823440] Process cat (pid: 2059, stack limit = 0xc3846190)
[  158.829212] Stack: (0xc3847e44 to 0xc3848000)
[  158.833611] 7e40:          c05bd150 00000001 00010000 00000004 c3847e48 00000150 c5035000
[  158.841833] 7e60: c777b420 c05bcaa4 c4f70c60 c777e070 c0a7fa80 c05bca20 00000fff c07ad4b0
[  158.850051] 7e80: c777b428 c04e46dc c4f52980 00001000 00000fff c01b1994 c4f52980 c4f70c60
[  158.858268] 7ea0: c3847ec8 ffffe000 00000000 c3847f88 00000001 c0164eac c3847fb0 c4f529b0
[  158.866487] 7ec0: 00020000 b6e3d000 00000000 00000000 c4f73f70 00000800 00000000 c01b0f60
[  158.874703] 7ee0: 00020000 c4f70c60 ffffe000 c3847f88 00000000 00000000 00000000 c013eb84
[  158.882918] 7f00: 000291ac 00000000 00000000 c0009344 00000077 b6e3c000 00000022 00000022
[  158.891135] 7f20: c686bdc0 c0117838 000b6e3c c3847f80 00022000 c686be14 b6e3c000 00000000
[  158.899354] 7f40: 00000000 00022000 b6e3d000 00020000 c4f70c60 ffffe000 c3847f88 c013ed0c
[  158.907571] 7f60: 00000022 00000000 000b6e3c c4f70c60 c4f70c60 b6e3d000 00020000 c000a9e4
[  158.915786] 7f80: c3846000 c013f2d8 00000000 00000000 00000000 00000000 00000000 00000000
[  158.924002] 7fa0: 00000003 c000a820 00000000 00000000 00000003 b6e3d000 00020000 00000000
[  158.932217] 7fc0: 00000000 00000000 00000000 00000003 00020000 00000000 00000001 00000000
[  158.940434] 7fe0: be8a62c0 be8a62ac b6eb77c4 b6eb6b9c 60000010 00000003 00000000 00000000
[  158.948691] [] (ts_mcu_transfer) from [] (ts_psu_get_prop+0x38/0xb0)
[  158.956847] [] (ts_psu_get_prop) from [] (power_supply_show_property+0x84/0x220)
[  158.966036] [] (power_supply_show_property) from [] (dev_attr_show+0x1c/0x48)
[  158.974974] [] (dev_attr_show) from [] (sysfs_kf_seq_show+0x84/0xf0)
[  158.983129] [] (sysfs_kf_seq_show) from [] (seq_read+0xcc/0x4f4)
[  158.990930] [] (seq_read) from [] (__vfs_read+0x1c/0x11c)
[  158.998117] [] (__vfs_read) from [] (vfs_read+0x88/0x158)
[  159.005304] [] (vfs_read) from [] (SyS_read+0x3c/0x90)
[  159.012232] [] (SyS_read) from [] (ret_fast_syscall+0x0/0x28)
[  159.019766] Code: e52de004 e281300c e590e004 e25cc001 (e1dee0b2) 
[  159.026278] ---[ end trace 2807dc313991fd87 ]---

The good news is the machine didn’t crash.c

Aug 262018

So, I had a brief look after getting kernel 4.18.5 booting… sure enough the problem was I had forgotten the watchdog, although I did see btrfs trigger a deadlock warning, so I may not be out of the woods yet.  I’ve posted the relevant kernel output to the linux-btrfs list.

Anyway, as it happens, that watchdog driver looks like it’ll need some re-factoring as a multi-function device.  At the moment, ts-wdt.c claims it based on this binding.

If I try to add a second driver, they’ll clash, and I expect the same if I try to access it via userspace.  So the sensible thing to do here, is to add a ts-companion.c MFD driver here, then re-factor ts-wdt.c to use it.  From there, I can write a ts-psu.c module which will go right here.

I think I’ll definitely be digging into those older sources to remind myself how that all worked.

Aug 252018

So, after some argument, and a bit of sitting on a concrete floor with the netbook, I managed to get Gentoo loaded onto the TS-7670.  Right now it’s running off the MicroSD card, I’ll get things right, then shift it across to eMMC.

ts7670 ~ # emerge --info
Portage 2.3.40 (python 3.5.5-final-0, default/linux/musl/arm/armv7a, gcc-6.4.0, musl-1.1.19, 4.14.15-vrt-ts7670-00031-g1a006273f907-dirty armv5tejl)
System uname: Linux-4.14.15-vrt-ts7670-00031-g1a006273f907-dirty-armv5tejl-ARM926EJ-S_rev_5_-v5l-with-gentoo-2.4.1
KiB Mem:      111532 total,     13136 free
KiB Swap:    4194300 total,   4191228 free
Timestamp of repository gentoo: Fri, 17 Aug 2018 16:45:01 +0000
Head commit of repository gentoo: 563622899f514c21f5b7808cb50f6e88dbd7d7de
sh bash 4.4_p12
ld GNU ld (Gentoo 2.30 p2) 2.30.0
app-shells/bash:          4.4_p12::gentoo
dev-lang/perl:            5.24.3-r1::gentoo
dev-lang/python:          2.7.14-r1::gentoo, 3.5.5::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1-r2::gentoo
sys-apps/openrc:          0.34.11::gentoo
sys-apps/sandbox:         2.13::musl
sys-devel/autoconf:       2.69-r4::gentoo
sys-devel/automake:       1.15.1-r2::gentoo
sys-devel/binutils:       2.30-r2::gentoo
sys-devel/gcc:            6.4.0-r1::musl
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.13::musl (virtual/os-headers)
sys-libs/musl:            1.1.19::gentoo

    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://
    priority: -1000
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-max-age: 24

CFLAGS="-Os -pipe -march=armv5te -mtune=arm926ej-s -mfloat-abi=soft"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-Os -pipe -march=armv5te -mtune=arm926ej-s -mfloat-abi=soft"
FCFLAGS="-O2 -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
USE="arm bindist cli crypt cxx dri fortran iconv ipv6 modules ncurses nls nptl openmp pam pcre readline seccomp ssl tcpd unicode xattr zlib" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="musl" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-0" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby23" USERLAND="GNU" VIDEO_CARDS="dummy fbdev v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"

I still have to update the kernel.  I actually did get kernel 4.18 to boot, but I forgot to add in support for the watchdog, so U-Boot tickled it, then the watchdog got hungry and kicked the reset half way through the boot sequence.

Rolling back to my older 4.14 kernel works.  I’ll try again with 4.18.5 in a moment.  Failing that, I have also brought the 4.14 patches up to 4.14.69 which is the latest LTS release of the kernel.

I’ve started looking at the power supply sysfs device class, with a view to exposing the supply voltage via sysfs.  The thinking here is that collectd supports reading this via the “battery” module (and realistically, it is a battery that is being measured: two 105Ah AGMs).

Worst case is I do something a little proprietary and deal with it in user space.  I’ll have to dig up the Linux kernel tree I did for Jacques Electronics all those years ago, as that had some examples of interfacing sysfs to a Cypress PSOC device that was acting as an I²C slave.  Rather than using an off-the-shelf solution, they programmed up a MCU that did power management, touchscreen sensing, keypad sensing, RGB LED control and others, all in one chip.  (Fun to try and interface that to the Linux kernel.)

Technologic Systems appear to have done something similar.  The device ID 0x78 implies a 10-bit device, but I think they’re just squatting on that 7-bit address.  They hail 0x78 then read out 4 bytes, which the last two bytes are the supply voltage ADC readings.  They do their own byte swapping before scaling the value to get mV.

Aug 222018

It’s taken several months and had a few false starts, but at long last I have some stage tarballs for Gentoo Linux MUSL for ARMv5 processors.  I’m not the only one wanting such a port, even looking for my earlier thread on the matter, I stumbled on this post.  (Google translate is hopeless with Russian, but I can get the gist of what’s being said.)

This was natively built on the TS-7670 using an external hard drive connected over USB 2.0 as both swap and the chroot.  It took two passes to clean everything up and get something that’s in a “release-able” state.

I think my next steps now will be:

  • Build an updated kernel … I might see if I can expose that I²C register via a sysfs file or something that collectd can pick up whilst I’m at it.  I have the kernel sources and bootloader sources.
  • Prepare the 32GB MicroSD card I bought a few weeks back with the needed partitions and load Gentoo onto that.
  • Install the MicroSD card and boot off it.
  • Back up the eMMC
  • Re-format the eMMC and copy the MicroSD card to it.

It’s supposed to be wet this weekend, so it sounds like a good project for indoors.

Aug 212018

I have a bad habit where it comes to updating systems, I tend to do it less frequently than I should, and that can sometimes snowball like it has for my mail server.  Even if it’s a fresh install, sometimes there’s a large number of packages that need installing.

Now Portage does report where it’s up to, but often that has long scrolled past the buffer on your terminal.  You can look at /var/log/emerge.log for this information, but sometimes it’s nice to just see a percentage progress and a pseudo graphical representation.

With this in mind, I cooked up a little script which just tails /var/log/emerge.log and displays a progress bar along with the last message reported. The script is quite short:


shopt -s checkwinsize

stdbuf -o L tail -n 0 -F /var/log/emerge.log | while read line; do
	eval $( echo ${line} | \
		sed -ne '/[0-9]\+ of [0-9]\+/ { s:^.*(\([0-9]\+\) of \([0-9]\+\)).*$:done=\1 total=\2 changed=1:; p; }' )

	if [ "${changed}" = 1 ]; then
		case "${line}" in
			*"::: completed emerge"*)
				done=$(( ${done} - 1 ))

		percent=$(( ( ${done}*100 ) / ${total} ))
		width=$(( ${COLUMNS:-80} - 8 ))
		progress=$(( ( ${done}*${width} ) / ${total} ))
		remain=$(( ${width} - ${progress} ))

		progressbar="$( for n in $( seq 1 ${progress} ); do echo -n '#'; done )"
		remainbar="$( for n in $( seq 1 ${remain} ); do echo -n ':'; done )"

		printf '\033[2A\033[2K%s\n\033[2K\033[1G[\033[1m%s\033[0m%s] \033[1m%3d%%\033[0m\n' \
			"${line:0:${COLUMNS:-80}}" "$progressbar" "$remainbar" "$percent"
		printf '\033[2A\033[2K%s\n\n' "${line:0:${COLUMNS:-80}}"

	if echo "${line}" | grep -q '*** terminating.'; then

What’s it look like?

It works well with GNU Screen as seen above.