Going low and slow with bochs

 Computing, Public Syndication  No Responses »
Jan 302018
 

So, today I had a problem… I needed to solve a race condition in a test case for my workplace’s WideSky system.  The test case was meant to ensure that, if the AMQP broker crashed or was restarted, it would re-connect and resume operations as quickly as possible.

On my desktop (an 8-core AMD Rysen 7), the test case always passed.  On the CI server (a VM running on a dual-core Core i3), it failed.  I figured the desktop here was running too quickly for me to see the problem.  I needed a machine that ran more like the CI server to see the problem.

Looking around, I couldn’t see any way to reliably slow down QEMU, KVM or VirtualBox… but I do remember one old project from the mid-late 90s that could: Bochs.

Bochs in action… emulating a P4 Prescott on a Rysen 7

Turns out, far from what it could do back in 1998 when it was strictly a 386 emulator (and a slow one at that!) it now has AMD64 emulation capabilities.  Thus, I can run the software stack inside this VM, and have it throttle the CPU speed down so that hopefully, the problem arises.  The first problem I needed to solve was trying to get the network running.  We have a PXE boot server which can serve up Ubuntu, no problem.  I just needed to bridge the Bochs VM onto the network somehow.

I already have bridge interfaces configured on my two physical network interfaces, and these work great with KVM.  Sadly, Bochs is rather primitive in what it supports… tap-mode networking just did not work, it complained that tap0 was not “running” even if created beforehand by iproute2, but I did find I could bind it directly to one of the enslaved network interfaces (enp36s0.200; yes, a VLAN interface).

e1000 worked for network booting, but then Ubuntu couldn’t retrieve an IP address for whatever reason. ne2k is working fine, and presently, I have the VM installing.  To make it network bootable, you need a boot ROM image, which you can download from the iPXE rom-o-matic service.  The magic PCI IDs you need are 10ec 8029 for ne2k, or (if it gets fixed) 8086 10de for e1000.

The following is my Bochs config file:

# configuration file generated by Bochs
plugin_ctrl: unmapped=1, biosdev=1, speaker=1, extfpuirq=1, parallel=1, serial=1, gameport=1, ne2k=1
config_interface: textconfig
display_library: x
debug: action=report
memory: host=2048, guest=2048
romimage: file="/usr/share/bochs/BIOS-bochs-latest", address=0x0, options=none
vgaromimage: file="/usr/share/bochs/VGABIOS-lgpl-latest"
boot: disk, network
floppy_bootsig_check: disabled=0
# no floppya
# no floppyb
ata0: enabled=1, ioaddr1=0x1f0, ioaddr2=0x3f0, irq=14
ata0-master: type=disk, path="/tmp/wstest.raw", mode=flat, cylinders=0, heads=0, spt=0, model="Generic 1234", biosdetect=auto, translation=auto
ata0-slave: type=none
ata1: enabled=1, ioaddr1=0x170, ioaddr2=0x370, irq=15
ata1-master: type=none
ata1-slave: type=none
ata2: enabled=0
ata3: enabled=0
optromimage1: file=none
optromimage2: file=none
optromimage3: file=none
optromimage4: file=none
optramimage1: file=none
optramimage2: file=none
optramimage3: file=none
optramimage4: file=none
pci: enabled=1, chipset=i440fx, slot1=ne2k, slot2=cirrus
vga: extension=cirrus, update_freq=5, realtime=1
cpu: count=1:1:1, ips=40000000, quantum=16, model=p4_prescott_celeron_336, reset_on_triple_fault=1, cpuid_limit_winnt=0, ignore_bad_msrs=1, mwait_is_nop=0
print_timestamps: enabled=0
port_e9_hack: enabled=0
private_colormap: enabled=0
clock: sync=none, time0=local, rtc_sync=0
# no cmosimage
# no loader
log: -
logprefix: %t%e%d
debug: action=ignore
info: action=report
error: action=report
panic: action=ask
keyboard: type=mf, serial_delay=250, paste_delay=100000, user_shortcut=none
mouse: type=ps2, enabled=0, toggle=ctrl+mbutton
speaker: enabled=1, mode=system
parport1: enabled=1, file=none
parport2: enabled=0
com1: enabled=1, mode=null
com2: enabled=0
com3: enabled=0
com4: enabled=0
ne2k: enabled=1, mac=fe:fd:de:ad:be:ef, ethmod=linux, ethdev=enp36s0.200, script=/bin/true, bootrom="/tmp/10ec8029.rom"

Create your hard drive image using qemu-img, then run bochs -f yourfile.cfg and it should, hopefully, work.

Solar Cluster: arm-unknown-linux-musleabi… a saga

 Computing, Gentoo Development, Linux Development, Projects, Public Syndication, Solar-powered Cloud Computing  No Responses »
Jan 292018
 

So, fun and games with the TS-7670.

At present, I have it up and running:

root@ts7670:~# uname -a
Linux ts7670 4.14.15-vrt-ts7670-00031-g1a006273f907-dirty #2 Sun Jan 28 20:21:08 EST 2018 armv5tejl GNU/Linux

That’s booted up into Debian Stretch right now.  debootstrap did its deed a few days ago on the eMMC, and I was able to boot up this new image.  Today I built a new kernel, and tweaked U-Boot to boot from eMMC.

Thus now the unit can boot without any MicroSD cards fitted.

There’s a lot of bit rot to address.  U-Boot was forked from some time in 2014.  I had a crack at rebasing the code onto current U-Boot, but there’s a lot of clean-up work to do just to get it to compile.  Even the kernel needed some fixes to get the newer devicetree sources to build.

As for getting Gentoo working… I have a cross-compiling toolchain that works.  With it, I’ve been able to compile about 99% of a seed stage needed for catalyst.  The 1% that eludes me, is GCC (compiled to run on ARMv5).  GCC 4.9.4 will try to build, but fails near the end… anything newer will barf complaining that my C++ compiler is not working.  Utter bollocks, both AMD64 and ARM toolchains have working C++ compilers, just it’s looking for a binary called “g++” rather than being specific about which one.  I suspect it wants the AMD64 g++, but then if I symlink that to /usr/bin/g++, it throws in ARM CFLAGS, and AMD64 g++ barfs on those.

I’ve explored other options.  I can compile GCC by hand without C++ support, and this works, but you can’t build modern GCC without a C++ compiler … and people wonder why I don’t like C++ on embedded!

buildroot was my next thought, but as it happens, they’ve stripped out the ability to compile a native GCC on the target.

crosstool-ng is the next logical choice, but I’ll have to fiddle with settings to get the compiler to build.

I’ve also had OpenADK suggested, which may be worth a look.  Other options are OpenEmbedded/Yocto, and Cross Linux from Scratch.  I think for the latter, cross is what I’ll get, this stuff can be infuriatingly difficult.

Solar Cluster: TS-7670 showed up today

 Computing, Linux Development, Projects, Public Syndication, Solar-powered Cloud Computing  No Responses »
Jan 242018
 

So, I now have my little battery monitoring computer.  Shipping wound up being a little more than I was expecting… about US$80… but never mind.  It’s here, arrived safely:

HTLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLFLC
>> TS-BOOTROM - built Jan 26 2017 12:29:21
>> Copyright (c) 2013, Technologic Systems
LLCLLLLLLLFLCLLJUncompressing Linux... done, booting the kernel.
/ts/fastboot file present.  Booting to initramfs instead
Booted from eMMC in 3.15s
Initramfs Web Interface: http://ts7670-498476.local
Total RAM: 128MB
# exit
INIT: version 2.88 booting
[info] Using makefile-style concurrent boot in runlevel S.
[ ok ] Starting the hotplug events dispatcher: udevd.
[ ok ] Synthesizing the initial hotplug events...done.
[ ok ] Waiting for /dev to be fully populated...done.
[ ok ] Activating swap...done.
[....] Checking root file system...fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
/dev/mmcblk2p2: clean, 48540/117600 files, 282972/469760 blocks
done.
[ ok ] Cleaning up temporary files... /tmp /lib/init/rw.
…
ts7670-498476 login: root
Linux ts7670-498476 2.6.35.3-571-gcca29a0+ #1 PREEMPT Mon Nov 27 11:05:10 PST 2017 armv5tejl
TS Root Image 2017-11-27

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

root@ts7670-498476:~#

The on-board 2GB eMMC has a version of Debian Wheezy on it.  That’ll be going very soon.  For now, all I’ve done is pop the cover, shove a 8GB MicroSD card into one of the on-board slots, wired up a 12V power brick temporarily to the unit, hooked a USB cable into the console port (/dev/ttyAMA0 is wired up to an on-board CP2103 USB-serial chip) and verified that it is alive.

Next step will be to bootstrap Gentoo.  I could use standard ARMv5 stages, or I can build my own, which I might do.  I’ve done this before for mips64el n64 using glibc.  Modern glibc is a goliath on a machine with 128MB RAM though, so I’ll be looking at either µClibc/µClibc-ng or musl… most likely the latter.

That said, 20 years ago, we had the same computing power in a desktop. 🙂

I have a few options for interfacing to the power meters…

  • I²C, SPI, a number of GPIOs and a spare UART on a 2.54mm header inside the case.
  • Another spare UART on the footprint for the GPS module (which my unit does not have)
  • Two RS-232 serial ports with RTS/CTS control lines, exposed via RJ-45 jacks
  • Two CANbus ports on a single RJ-45 jack
  • RS-485 on a port marked “Modbus”

In theory, I could just skip the LPC810s and hook this up directly to the INA219Bs.  I’d have to double check what the TTL voltage is… Freescale love their 1.8V logic… but shifting that up to 3.3V or 5V is not hard.  The run is a little longer than I’m comfortable running I²C though.

The LPC810s don’t feature CANbus, so I think my original plan of doing Modbus is going to be the winner.  I can either do a single-ended UART using a resistor/diode in parallel to link RX and TX to the one UART line, or use RS-485.

I’m leaning towards the latter, if I decide to buy a little mains energy meter to monitor power, I can use the same RS-485 link to poll that.  I have some RS-485 transceivers coming for that.

For now though, I’ll at least get Debian Stretch going… this should not be difficult, as I’ll just use the images I’ve built for work to get things going.  I’m downloading a Jessie image now:

root@ts7670-498476:~# curl https://bne.vrt.com.au/technologicsys/ts7670d-jessie-4.4.1-20160226.dd.xz | xzcat | dd of=/dev/mmcblk0 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  113M    0  544k    0     0   114k      0  0:16:48  0:00:04  0:16:44  116k

Once that is done, I can reboot, re-format the eMMC and get debootstrap going.  I might even publish an updated image while I’m at it.

Solar Cluster: arm-unknown-linux-musleabi… saga part II

 Computing, Gentoo Development, Linux Development, Projects, Public Syndication, Solar-powered Cloud Computing  No Responses »
Jan 232018
 

So, last time I was trying to get Gentoo’s portage to cross-build gcc so that I’d have a C/C++ compiler in my ARMv5 musl environment.

It is literally the last piece of the puzzle.  Once compiled, that is the last step I need before I can throw the shiny new environment onto an ARMv5 VM (or real ARMv5 CPU), do an emerge -e world on it then tar the lot up and throw it at Catalyst.

Building an entire OS on a 454MHz ARMv5 machine with 128MB RAM does not faze me one bit… I used to do it regularly on a (Gateway-branded) Cobalt Qube II server appliance, which sports a 250MHz QED RM5231 and 128MB RAM.  The other compile workhorse I used in those days was an SGI O2; 300MHz RM5200, again 128MB RAM.

Yes, Linux and its userland has bulked up a bit in the last 10 years, but not so much so that a build on these is impossible.

Certainly, native building is easier than cross-compiling.  Cross-compilers have always been a voodoo art for me.  Getting one that will build a Linux kernel or U-Boot, usually isn’t too hard… but get userland involved and it gets complex.  Throw in C++ and complexity skyrockets!

I’m taking OpenADK for a spin now, and in concept, it’s exactly what I remember buildroot used to be.  It’s a tool for generating a fully fledged embedded Linux system with a wide package selection including development tools.  I also find that you have to hold your tongue just right to get stuff to compile.

Selecting a generic arm926ej-s; it succeeded to build a x86-64 hosted cross-toolchain once, but then silently refused to build anything else.  I told it instead to build for a Versatile PB with an arm926ej-s CPU… it failed to build the cross-toolchain, even though it pretty much is the exact same target.

A make cleandirs later, and it happily started building everything, but then hiccupped on permissions, so against my better judgement, I’m running it now with sudo, and things are progressing.  With some luck, I should have something that will give me a working native gcc/g++ for musl on ARMv5.

Toy Synthesizer: Second revision I/O board nearly complete

 Projects, Public Syndication, Toy Synthesizer  No Responses »
Jan 212018
 

So last week, some more parts arrived for this project.  Crucially, some capacitors, and some MOSFETs.

Turns out, the MOSFETs I got last time were not I²PAK … I thought that’s what I ordered last time, but clearly not, because that’s what these are, and they’re bigger than the previous ones.

No matter, lean them back like dominoes and they still fit.  I’ve got a tube of 50 of them.

I was able to put the pull-down resistor in, and I basically fitted the MOSFETs along a track, scoring the track between the legs so they didn’t short out.  For the 12ohm resistor to the drain, I’m doing half-veroboard-PCB, half-point-to-point construction to link the drain pin (annoyingly in the centre) to the outside world.

I’ll need to rustle up a 2-pin KK to act as the power input, and that board is done.  I might add two in parallel on here so a short lead can link from here to the mainboard to supply +12V.  This will go on the right-hand side (lower photo) just past where that jumper connects.

After that, the next step is wiring up the buttons and switches.  The use of 4-pin connectors should greatly simplify the wiring since everything we need is on the one connector.

Solar Cluster: Battery monitor computer ordered

 Computing, Projects, Public Syndication, Solar-powered Cloud Computing  No Responses »
Jan 172018
 

I’ve taken the plunge and gotten a TS-7670 ordered in a DIN-rail mount for monitoring the battery.  Not sure what the shipping will be from Arizona to here, but I somehow doubt I’m up for more than AU$300 for this thing.  The unit itself will cost AU$250.

Some will argue that a Raspberry Pi or BeagleBone would be cheaper, and that would be correct, however by the time you’ve added a DIN-rail mount case, an RS-485 control board and a 12V to 5V step-down power converter, you’d be around that figure anyway.  Plus, the Raspberry Pi doesn’t give you schematics.  The BeagleBone does, but is also a more sophisticated beast.

The plan is I’ll spin a version of Gentoo Linux on it… possibly using the musl C library to keep memory usage down as I’ve gone the base model with 128MB RAM.  I’ll re-spin the kernel and U-Boot patches I have for the latest release.

There will be two functions looked after:

  • Access to the IPMI/L2 management network
  • Polling of the two DC power meters (still to be fully designed) via Modbus

It can report to a VM running on one of the hosts.  I believe collectd has the necessary bits and pieces to do this.  Failing that, I’ve written code before that polls Modbus… I write such code for a day job.

Ahh… SEO Spam

 Public Syndication, Rants  No Responses »
Jan 132018
 

Part of my day job involves being the technical contact for their website, which means we get lots of offers from people offering to put us on the “first page of Google”.

Hmm, last time I checked, the first page of Google was, strangely, Google.  Somehow, I don’t think they outsource their SEO strategy to get there… they wrote the bloody code!

These emails go straight to Spamcop generally… and they send nastygrams to the people hosting the email servers they used.  In some cases, I’ve taken the extraordinary step of blocking frequently abused hosts.

# Block Centrilogic and SmartMailer because they don't act on spam reports.
-A INPUT -s 173.240.14.0/24 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 199.43.203.0/24 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
# Block OVH because they don't act on spam reports.
# List taken from https://mxtoolbox.com/SuperTool.aspx?action=asn%3aAS16276&run=toolpage
-A INPUT -s 5.39.0.0/17 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 5.135.0.0/16 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 5.196.0.0/16 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 8.7.244.0/24 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 8.18.128.0/24 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 8.18.136.0/21 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 8.18.172.0/24 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 8.20.110.0/24 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 8.21.41.0/24 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 8.24.8.0/21 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 8.26.94.0/24 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 8.29.224.0/24 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 8.30.208.0/21 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
-A INPUT -s 8.33.96.0/21 -p tcp --dport 25 -j REJECT --reject-with icmp-host-prohibited
…

That is not an exhaustive list.  Sorry to people who use OVH for hosting and were trying to contact VRT/CETA legitimately, but OVH have shown themselves to be grossly incompetent with regard to management of network abuse.  Centrilogic/SmartMailer are more recent additions.

Of course, they keep trying, and thankfully, it takes longer for them to write the email than it does for me to deal with it. This doesn’t stop them claiming little gems like this:

Note: We are not spammers and are against spamming of any kind. If you are not interested then you can reply with a simple “NO”.

Errm, hate to disagree (actually no, in this case, I love disagreement)… but a few points:

  1. Your sending me an unsolicited content…
  2. … without my consent… (no listing in domain registration or scraping from a website is not consent)
  3. … that is advertising a paid-for service or otherwise something you’re hoping to make money from…
  4. … by electronic messaging.

That by definition is an Unsolicited Commercial Email… aka SPAM.  If you claim to be an Australian business, you better have a look at this.  If your ISP is complaining that you are abusing their services by sending spam, then perhaps you need to realise the people you are contacting are not interested!  You have your NO.