May 152015
 

… or how to emulate Red Hat’s RPM dependency hell in Debian with Python.

There are times I love open source systems and times when it’s a real love-hate relationship. No more is this true than trying to build Python module packages for Debian.

On Gentoo this is easy: in the past we had g-pypi. I note that’s gone now and replaced with a gsourcery plug-in called gs-pypi. Both work. The latter is nice because it gives you an overlay potentially with every Python module.

Building packages for Debian in general is fiddly, but not difficult, but most Python packages follow the same structure: a script, setup.py, calls on distutils and provides a package builder and installer. You call this with some arguments, it builds the package, plops it in the right place for dpkg-buildpackage and the output gets bundled up in a .deb.

Easy. There’s even a helper script: stdeb that plugs into distutils and will do the Debian packaging all for you. However, stdeb will not source dependencies for you. You must do that yourself.

So quickly, building a package for Debian becomes reminiscent of re-living the bad old days with early releases of Red Hat Linux prior to yum/apt4rpm and finding the RPM you just obtained needs another that you’ll have to hunt down from somewhere.

Then you get the people who take the view, why have just one package builder when you can have two. fysom needs pybuilder to compile. No problems, I’ll just grab that. Checked it out of github, uhh ohh, it uses itself to build, and it needs other dependencies.

Lovely. It gets better though, those dependencies need pybuilder to build. I just love circular dependencies!

So as it turns out, in order to build this, you’ll need to enlist pip to install these behind Debian’s back (I just love doing that!) then you’ll have the dependencies needed to actually build pybuilder and ultimately fysom.

Your way out of this maze is to do the following:

  • Ensure you’ve got the python-stdeb, dh-python and python-pip packages installed.
  • Use pip to install the dependencies for pybuilder and its dependencies: pip install fluentmock pybuilder pyassert pyfix pybuilder-external-plugin-demo pybuilder_header_plugin pybuilder_release_plugin
  • Now you should be able to build pybuilder, do pyb publish in the directory, then look under target/dist/pybuilder-${VERSION} you should see the Python sources with a setup.py you can use with stdeb.

Any other dependencies are either in Debian repositories, or you can download the sources yourself and use the stdeb technique to build them.

Apr 112015
 

To whom it may concern,

There have been reports of web browser sessions from people outside China to websites inside China being hijacked and having malware injected.  Dubbed “Great Cannon”, this malware having the sole purpose of carrying out distributed denial of service attacks on websites that the Chinese Government attempts to censor from its people.  Whether it be the Government there itself doing this deliberately, or someone hijacking major routing equipment is fundamentally irrelevant here, either way the owner of the said equipment needs to be found, and a stop put to this malware.

I can understand you wish to prevent people within your borders from accessing certain websites, but let me make one thing abundantly clear.

COUNT ME OUT!

I will not accept my web browser which is OUTSIDE China being hijacked and used as a mule for carrying out your attacks.  It is illegal for me to carry out these attacks, and I do not authorise the use of my hardware or Internet connection for this purpose.  If this persists, I will be blocking any and all Chinese-owned websites’ executable code in my browser.

This will hurt Chinese business more than it hurts me.  If you want to ruin yourselves economically, go ahead, it’ll be like old times before the Opium Wars.

Apr 102015
 

This afternoon, whilst waiting for a build job to complete I thought I’d do some further analysis on my annual mileage.

Now I don’t record my odometer readings daily (perhaps I should), but I do capture them every Sunday morning.  So I can possibly assume that the distance done for each day of a “run” is the total distance divided by the number of days.  I’m using a SQLite3 database to track this, question is, how do I extract this information?

This turned out to be the key to the answer.  I needed to enumerate all the days between two points.  SQLite3 has a julianday function, and with that I have been able to extract the information I need.

My database schema is simple. There are two tables:
CREATE TABLE bikes (id integer primary key not null, description varchar(64));
CREATE TABLE odometer (timestamp datetime not null default current_timestamp, action char(8) not null, bike_id integer not null, odometer real not null, constraint duplicate_log unique (timestamp, action, bike_id) on conflict replace);

Then there are the views.
CREATE VIEW run_id as select s.rowid as start_id, (select rowid from odometer where bike_id=s.bike_id and timestamp > s.timestamp and action='stop' order by timestamp asc limit 1) as stop_id from odometer as s where s.action='start';
CREATE VIEW "run" AS select start.timestamp as start_timestamp, stop.timestamp as stop_timestamp, start.bike_id as bike_id, start.odometer as start_odometer, stop.odometer as stop_odometer, stop.odometer-start.odometer as distance,julianday(start.timestamp) as start_day, julianday(stop.timestamp) as stop_day from (run_id join odometer as start on run_id.start_id=start.rowid) join odometer as stop on run_id.stop_id=stop.rowid;

The first view breaks up the start and stop events, and gives me row IDs for where each “run” starts and stops. I then use that in my run view to calculate distances and timestamps.

Here’s where the real voodoo lies, to enumerate days, I start at the very first timestamp in my dataset, find the Julian Day for that, then keep adding one day on until I get to the last timestamp. That gives me a list of Julian days that I can marry up to the data in the run view.

CREATE VIEW distance_by_day as
SELECT day_of_year, avg_distance FROM (
SELECT days.day - julianday(date(days.day,'start of year')) as day_of_year, sum(run.distance/max((run.stop_day-run.start_day),1))/count(*) as avg_distance
FROM run,
(WITH RECURSIVE
days(day) as (
SELECT julianday((select min(timestamp) from odometer))
union all
SELECT day+1 from days
limit cast(round(julianday((select max(timestamp) from odometer))-julianday((select min(timestamp) from odometer))) as int)
) SELECT day from days) as days
where
run.start_day < = days.day AND run.stop_day >= days.day
group by day_of_year) dist_by_doy;

This is the result.

Distance by Day Of Year

Distance by Day Of Year

Apr 062015
 

I’ve been a long time user of PGP, had a keypair since about 2003.  OpenPGP has some nice advantages in that it’s a more social arrangement in that verification is done by physically meeting people.  I think it is more personal that way.

However, you still can get isolated islands, my old key was a branch of the strong set, having been signed by one person who did do a lot of key-signing, but sadly thanks to Heartbleed, I couldn’t trust it anymore.  So I’ve had to start anew.

The alternate way to ensure communications is to use some third party like a certificate authority and use S/MIME.  This is the other side of the coin, where a company verifies who you are.  The company is then entrusted to do their job properly.  If you trust the company’s certificate in your web browser or email client, you implicitly trust every non-revoked valid certificate that company has signed.  As such, there is a proliferation of companies that act as a CA, and a typical web browser will come with a list as long as your arm/leg/whatever.

I’ve just set up one such certificate for myself, using StartCOM‘s CA as the authority.  If you trust StartCOM, and want my GPG key, you’ll find a S/MIME signed email with my key here.  If you instead trust my GPG signature and want my S/MIME public key, you can get that here.  If you want to throw caution to the wind, you can get the bare GPG key or S/MIME public key instead.

Update: I noticed GnuPG 2.1 has been released, so I now have an ECDSA key; fingerprint B8AA 34BA 25C7 9416 8FAE  F315 A024 04BC 5865 0CF9.  You may use it or my existing RSA key if your software doesn’t support ECDSA.

Mar 192015
 

Hi all,

This is more a note to self for future reference.  Qt has a nice handy reference counting memory management system by means of QSharedPointer and QWeakPointer.  The system is apparently thread-safe and seems to be totally transparent.

One gotcha though, is two QSharedPointer objects cannot share the same pointer unless one is cloned from the other (either directly or via QWeakPointer).  The other is that you must leave deletion of the object to QSharedPointer.  You’ve given it your precious pointer, it has adopted it and while you may call the object, it is no longer yours, so don’t go deleting it.

So you create an object, you want to pass a reference to yourself to some other object.  How?  Like this?

QSharedPointer<MyClass> MyClass::ref() {
    return QSharedPointer<MyClass>(this); /* NO! */
}

No, not like that! That will create QSharedPointer instances left right and centre. Not what you want to do at all. What you need to do, is create the initial reference, but then store a weak reference to it. Then all future calls, you simply call the toStrongRef method of the weak reference to get a QSharedPointer that’s linked to the first one you handed out.

Then, having done this, when you create your new object, you create it with the new keyword as normal, take a QSharedPointer reference to it, then forget all about the original pointer! You can get it back by calling the data method of the pointer object.

To make it simple, here’s a base class you can inherit to do this for you.

    #include <QWeakPointer>
    #include <QSharedPointer>

    /*!
     * Self-Reference helper.  This allows for objects to maintain
     * references to "this" via the QSharedPointer reference-counting
     * smart pointers.
     */
    template<typename T>
    class SelfRef {
        public:
            /*!
             * Get a strong reference to this object.
             */
            QSharedPointer<T>    ref()
            {
                QSharedPointer<T> this_ref(this->this_weak);
                if (this_ref.isNull()) {
                    this_ref = QSharedPointer<T>((T*)this);
                    this->this_weak = this_ref.toWeakRef();
                }
                return this_ref;
            }

            /*!
             * Get a weak reference to this object.
             */
            QWeakPointer<T>        weakRef() const
            {
                return this->this_weak;
            }
        private:
            /*! A weak reference to this object */
            QWeakPointer<T>        this_weak;
    };

Example usage:

#include <iostream>
#include <stdexcept>
#include "SelfRef.h"

class Test : public SelfRef<Test> {
        public:
                Test()
                {
                        std::cout << __func__ << std::endl;
                        this->freed = false;
                }
                ~Test()
                {
                        std::cout << __func__ << std::endl;
                        this->freed = true;
                }

                void test() {
                        if (this->freed)
                                throw std::runtime_error("Already freed!");
                        std::cout
                                << "Test object is at "
                                << (void*)this
                                << std::endl;
                }

                bool                    freed;
                QSharedPointer<Test>    another;
};

int main(void) {
        Test* a = new Test();
        if (a != NULL) {
                QSharedPointer<Test> ref1 = a->ref();
                if (!ref1.isNull()) {
                        QSharedPointer<Test> ref2 = a->ref();
                        ref2->test();
                }
                ref1->test();
        }
        a->test();
        return 0;
}

Note that the line before the return is a deliberate use after free bug to prove the pointer really was freed.  Also note that the idea of setting a boolean flag to indicate the constructor has been called only works here because nothing happens between that use after free attempt and the destructor being called.  Don’t rely on this to see if your object is being called after destruction.  This is what the output session from gdb looks like:

RC=0 stuartl@rikishi /tmp/qtsp $ make CXXFLAGS=-g
g++ -c -g -I/usr/share/qt4/mkspecs/linux-g++ -I. -I/usr/include/qt4/QtCore -I/usr/include/qt4/QtGui -I/usr/include/qt4 -I. -I. -o test.o test.cpp
g++ -Wl,-O1 -o qtsp test.o    -L/usr/lib64/qt4 -lQtGui -L/usr/lib64 -L/usr/lib64/qt4 -L/usr/X11R6/lib -lQtCore -lgthread-2.0 -lglib-2.0 -lpthread 
RC=0 stuartl@rikishi /tmp/qtsp $ gdb ./qtsp 
GNU gdb (Gentoo 7.7.1 p1) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./qtsp...done.
(gdb) r
Starting program: /tmp/qtsp/qtsp 
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Test
Test object is at 0x555555759c90
Test object is at 0x555555759c90
~Test
terminate called after throwing an instance of 'std::runtime_error'
  what():  Already freed!

Program received signal SIGABRT, Aborted.
0x00007ffff5820775 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff5820775 in raise () from /lib64/libc.so.6
#1  0x00007ffff5821bf8 in abort () from /lib64/libc.so.6
#2  0x00007ffff610cd75 in __gnu_cxx::__verbose_terminate_handler() ()
   from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
#3  0x00007ffff6109ec8 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
#4  0x00007ffff6109f15 in std::terminate() () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
#5  0x00007ffff610a2e9 in __cxa_throw () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
#6  0x0000555555555cea in Test::test (this=0x555555759c90) at test.cpp:20
#7  0x0000555555555315 in main () at test.cpp:41
(gdb) up
#1  0x00007ffff5821bf8 in abort () from /lib64/libc.so.6
(gdb) up
#2  0x00007ffff610cd75 in __gnu_cxx::__verbose_terminate_handler() ()
   from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
(gdb) up
#3  0x00007ffff6109ec8 in ?? () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
(gdb) up
#4  0x00007ffff6109f15 in std::terminate() () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
(gdb) up
#5  0x00007ffff610a2e9 in __cxa_throw () from /usr/lib/gcc/x86_64-pc-linux-gnu/4.8.3/libstdc++.so.6
(gdb) up
#6  0x0000555555555cea in Test::test (this=0x555555759c90) at test.cpp:20
20                                      throw std::runtime_error("Already freed!");
(gdb) up
#7  0x0000555555555315 in main () at test.cpp:41
41              a->test();
(gdb) quit
A debugging session is active.

        Inferior 1 [process 17906] will be killed.

Quit anyway? (y or n) y

You’ll notice it fails right on that second last line because the last QSharedPointer went out of scope before this.  This is why you forget all about the pointer once you create the first QSharedPointer.

To remove the temptation to use the pointer directly, you can make all your constructors protected (or private) and use a factory that returns a QSharedPointer to your new object.

A useful macro for doing this:

/*!
 * Create an instance of ClassName with the given arguments
 * and immediately return a reference to it.
 *
 * @returns	QSharedPointer<ClassName> object
 */
#define newRef(ClassName, args ...)	\
	((new ClassName(args))->ref().dynamicCast<ClassName>())
Feb 012015
 

How do software companies get things so wrong?  I aim this at both Google and Apple here, as both are equally as guilty of this.  Maybe Microsoft can learn from this?

So you see something on an “app store” that might be useful on the device you’re using.  Ohh, a free download, great!  Let’s download.  You click the link, and immediately get asked for a login and password.  There’s no option to proceed without.  They insist you create an account with them even if it’s the one and only thing you’re interested from them.

In the past my gripe has been with the Google Play store.  Even the “free” apps, require you to log in.  Ohh, and to add insult to injury, the Google Play store doesn’t just expect any Google account, it has to be one of their Gmail accounts.  Back in the late 90s I had an email address with most providers as the average quota was about 5MB.  I’ve had a mailbox of my own with a multi-gigabyte (actually limited only by disk capacity) “quota” since 2002, I have no use for Gmail, and only keep my old Yahoo! address (created in 1999) for historic reasons.

I have an Android phone (release 4.1: thanks to ZTE’s backward thinking and short attention span), and thankfully there’s the F-Droid store which has sufficient applications for my use.  So I can work around the Google Play store most of the time and so far, haven’t needed anything from there.

Today, my gripe is at Apple, and the “app” in question is MacOS X, which cannot be obtained anywhere else.

With all the high-profile attacks on websites that store user accounts, one has to ask, why?  It’s one extra username and password, which given the frequency I’m likely to use it, will have to be written down and stored somewhere secure as it won’t get sufficient use to commit it to memory.  Before people point out password managers, I’d like to point out one thing: it’s still writing it down!

There’s absolutely no need for an “app store” to know your email address, usernames, passwords, or any details.  If you are actually purchasing an application, they only need enough information to process a payment.

Usually this is by a debit/credit card, so they need to know the details on the card.  An alternative might be direct deposit through a bank, at which point they need to supply you with details on how to make the payment — details that include the information they need to match your payment in their ledger to your store purchase.  At no point do they need anything else.

For convenience an email address might be supplied so they can confirm your order or contact you if there’s a problem, however for debit/credit cards, this happens so quickly that it can be achieved via the web browser.

Despite this, they insist on you providing just about everything.

I’m no stranger to the “app store” concept.  Linux and BSD distributions have had this sort of concept for years.  BSD has had ports for as long as I can remember.  Debian had apt since 1998, Gentoo has had portage since its inception in 2003 and RPM-based distributions have had yum for some time too.

None of these actually need to know who you are in order to download a package.  Admittedly none of these are geared toward commercial sales of software, and so lack the ability to prompt for credentials or payment information.

Since both Google Play and the Apple App store have solved the former problem, I see no reason why they couldn’t solve the latter.  I don’t want to post anything to the site, I don’t want to leave feedback as I can hardly comment on something I haven’t received yet, and I don’t know when I’ll next visit the site.

If I was going to be back repeatedly, sure, I’ll make an account.  It’ll make everyone’s lives easier. (Including the blackhats!)  But I’m not.  I have a late-2008 model MacBook, probably the oldest machine that Apple support for their latest OS.  The machine dual-boots MacOS X 10.6 and Gentoo Linux, and spends 99% of its time in the latter OS.

Given the age of the machine and the frequency at which I use its native OS, it is not worth me spending a lot of time or expense updating it.  A 2GHz Core 2 Duo with 8GB RAM and a 750GB HDD is good enough for many tasks under Linux, but is the bare minimum to run OS X 10.10.  The only reason this machine doesn’t grace my desk at work anymore is the fact the lack of ports (USB in particular) proved to be a right pain.

Why update?  Well, applications these days seem to expect at least MacOS X 10.7 now.  I either have to build everything myself or update the OS, so I’m investigating the possibility of updating the OS to see if it’s feasible.  Apparently it’s a free download, so why not?

Well, why not indeed!  Instead of having a simple http, https or ftp link to the file in question (maybe a .dmg image) for software they’re not actually selling to me in the traditional sense, they instead insist on making me jump through hoops like requiring their “app store” client — so I can’t just grab the link, tell the web server here to download the file then grab it from there when I’m ready.

Since I can’t do the download any other way than via their “app store” client, I have to remain booted in MacOS X in order to download it regardless of what I might otherwise wish to do the machine and what OS that requires.

However, before I can even think about starting the download, I’ve got to register an account, supplying a username and password for something that will probably be used exactly once.  Details that they have to pay people big money to store securely.

Instead of spending some money paying someone to add an extra one-off button and form to their “app store” clients, they instead spend significantly more on infrastructure designed to meet the privacy requirements of various laws to store user information that simply is not necessary for the transaction to proceed.

In light of the sophistication of the modern cracker and the cut-throat nature of the mobile market, is this such a wise use of company funds?

Dec 042014
 

Just recently I’ve been looking into asynchronous programming.

Previously I had an aversion to asynchronous code due to the ugly twisted web of callback functions that it can turn into. However, after finding that having a large number of threads blocking on locks and semaphores still manages to thrash a machine, I’ve come to the conclusion that I should put aside my feelings and try it anyway.

Our codebase is written in Python 2.7, sadly, not new enough to have asyncio. However we do plan to eventually move to Python 3.x when things are a bit more stable in the Debian/Ubuntu department (Ubuntu 12.04 didn’t support it and there are a few sites that still run it, one or two still run 10.04).

That said, there’s thankfully a port of what became asyncio in the form of Trollius.

Reading through the examples though still had me lost and the documentation is not exactly extensive. In particular, coroutines and yielding. The yield operator is not new, it’s been in Python for some time, but until now I never really understood it or how it was useful in co-operative programming.

Thankfully, Sahand Saba has written a guide on how this all works:
http://sahandsaba.com/understanding-asyncio-node-js-python-3-4.html

I might put some more notes up as I learn more, but that guide explained a lot of the fundamentals behind a lot of event loop frameworks including asyncio.

Sep 292014
 

Well, it’s been a busy year so far for security vulnerabilities in open-source projects.  Not that those have been the only two bugs, they’re just two high-profile ones that are getting a lot of media attention.

Now, a number of us do take sheer delight in pointing and laughing when one of the big boys, whether they be based in Redmond or California, makes a security balls-up on a big scale.  After all, people pay big dollars to use some of that software, and many are dependent on it for their livelihoods.

The question does get raised though, what do you trust more?  A piece of software whose code is a complete secret, or the a piece of software anyone can audit?  Some argue the former, because anyone can find the holes in the latter and exploit them.  Some argue the latter, since anyone can find the holes and fix them.  Not being able to see the code doesn’t guarantee a lack of security issues however, and these last two headline-making bugs is definitely evidence that having the code isn’t a guarantee to a bug-free utopia.

There is no guarantee either way.

I’ve seen both open-source systems and high-end commercial systems both perform well and I’ve seen both make a dismal failure.  Bad code is bad code, no matter what the license, and even having the source available doesn’t mean you can fix it as first one must be able to understand what its intent is.  Information Technology in particular seems to attract the technologically inept but socially capable types that are able to talk their way into nearly any position, and so you wind up with the monstrosities that you might see on The Daily WTF.  These same people lurk amongst open-source circles too, and there are those who just make an honest mistake.  Security is hard, and it can be easy to overlook a possible hole.

I run Gentoo here, have done so now since 2004 (damn, 10 years already, but I digress…).  I’ve been building my own stage 3 tarballs from scratch since 2010.  July 2010 I bought my current desktop, a 6-core AMD Phenom machine, and so combined with the 512Kbps ADSL I had at the time, it was faster for me to compile stage 3 tarballs for the various systems (i386, AMD64 and about 6 different MIPS builds) than to download the sources.  If I wanted an up-to-date stage 3, I just took my last build, ran it through Gentoo Catalyst, and out came a freshly built tarball.

I still obtain my operating systems that way.  Even though I’ve upgraded the ADSL, I still use the same scripts that used to produce the official Gentoo/MIPS media.

This means I could audit every piece of software that forms my core system.  I have the source code there, all of it.  Not many Linux users have this, most have it at arms reach (i.e. an apt-get source ${PACKAGE} away), or at worst, a polite email/letter to their supplier (e.g. Netcomm will supply sources for their routers for a ~AU$10 fee), however I already have it.

So did I do any audits?  Have I done any audits?  No.  Ultimately I just blindly trust what comes down the wire, and to some, that is arguably no better than just blindly trusting what Apple and Microsoft produce.

Those who say that, do have a point.  I didn’t pick up on HeartBleed, nor on ShellShock, and I probably haven’t spotted what will become the next headline-grabbing bug.  There’s a lot of source code that goes into a GNU/Linux system, and if I were to sit there and audit it, myself, it’d take me a lifetime.  It’d cost me a fortune to pay a team to analyse it.

However, I at least have the choice of auditing parts of it.  I’ll never be able to audit the copies of Microsoft Windows, or the one copy of Apple MacOS X I have.  For those, I’m reliant on the upstream vendors to audit, test and patch their code, I cannot do it myself.

For the open-source software though, it’s ultimately my choice.  I can do it myself, I can also pay someone to do it, I’ve simply chosen not to at this time.  This is an important distinction that the anti-open-source camp seem to forget.

As for the quality factor: well I’ve spent more time arguing with some piece of proprietary software and having trouble getting it to do something I need it to do, or fixing up some cock up caused by a bug in the said software.  One option, I spend hours arguing with it to make it work, and have to pay good money for the privilege.  The other, they money stays in my pocket, and in theory I can re-build it to make it work if needed.  One will place arbitrary restrictions on how I use the software as an end user, forcing me to spend money on more expensive licenses, the other will happily let me keep pushing it until I hit my system’s technical limits.

Neither offer me any kind of warranty regarding to losses I might suffer as a result of their software (I’m sorry, but US$5.00 is as good as worthless), so the money might as well stay in my pocket while I learn something about the software I use.

I remain in control of my destiny that way, and that is the way I’d like to keep it.

Aug 062014
 

Unix-to-Unix Copy is a rather old way of sending files between Unix systems.  Before SMTP was invented, it was the de-facto way to shunt email around, and prior to NNTP, was also the backbone of Usenet.

Fast forward to today, we’ve got a lot of options open to us.  So why would I use a crusty old one like UUCP?

UUCP has one big advantage, it doesn’t assume your system is always online.

  • It might be a workstation at your workplace which is behind a corporate firewall.
  • It might be a more powerful desktop computer at home that’s usually in “sleep” mode to save power.

Because the initial connection can be established in either direction, it is ideal for a system that may not be directly reachable, but is able to poll on a regular schedule for instructions.  It’s also useful, since UUCP assumes some steps need to be taken to bring a link up, to perform tasks such as powering on a system using IPMI or Wake-on-LAN, wait for it to come up, perform a task, then have the machine power back down when finished.

UUCP over the Internet

Now, UUCP can and does work directly over the Internet.  in.uucpd runs from inetd, and basically fires up uucico each time someone connects to it. But: it is unencrypted and insecure. It’s not what you want exposed on today’s public Internet.

UUCP does support SSL, and there are ways to make stunnel work with packages like Taylor UUCP. This still requires some significant setup and an additional open port.

There’s another way. Thanks to the OpenBSD community, we have OpenSSH, and it is very trivial to set up a secure UUCP link using public key authentication, to lock down the public key to only be used with uucico, and to effectively secure communications between your hosts.

Generating the SSH key

Since this is going to be used with an automated service, one needs to make it a passwordless key. Go for something secure in terms of the key length and algorithm: I used 4096-bit RSA. To do this, log in as root then:

# su uucp -
$ ssh-keygen -t rsa -b 4096 -N '' -C 'UUCP key'
Generating public/private rsa key pair.
Enter file in which to save the key (/var/spool/uucp/.ssh/id_rsa): 
Your identification has been saved in /var/spool/uucp/.ssh/id_rsa.
Your public key has been saved in /var/spool/uucp/.ssh/id_rsa.
The key fingerprint is:
c3:42:5d:77:a9:c2:3a:da:bd:98:6a:5d:03:62:79:19 UUCP key
The key's randomart image is:
+--[ RSA 4096]----+
|          . . .. |
|       .E. . ..  |
|      ...+   .   |
|     .+.+ o .    |
|     ..oSo .     |
|       .o.o      |
|       + + .     |
|      o oo.      |
|     ...o ..     |
+-----------------+
$

You have a choice. You can either: make a keypair for each host, and set up authorized_keys so the hosts can log into eachother, or you can use the same keypair for all hosts. I went the latter route, as I’m not that paranoid. Whilst still logged in as the UUCP user:

$ echo 'command="/usr/sbin/uucico -l" '$(< .ssh/id_rsa.pub ) > .ssh/authorized_keys

Now, securely transfer the UUCP user’s .ssh directory between your hosts. This will allow uucp to log in.

Populating known_hosts

The easiest way to do this, is to log into each host as the UUCP user, then run a script like this:

$ for h in host1 host2 host3 ; do ssh $host true; done

Check each key carefully, answer yes if you’re satisfied.

UUCP Log-in script

Taylor UUCP, has the ability to define a “port” that runs an arbitrary application. You could put a call to SSH here, but there’s another trick I use. As root:

# cat < /usr/local/bin/uussh
#!/bin/sh
echo -n 'Address: '
read user host wake

if [ -n "${wake}" ]; then
        timeout=60
        until ping6 -c 1 -w 1 -n "${host}" 2>&1 >/dev/null \
                        || ping -c 1 -w 1 -n "${host}" 2>&1 >/dev/null \
                        || [ $timeout -le 0 ]; do
                timeout=$(( ${timeout} - 1 ))
                /usr/bin/wol ${wake} 2>&1 > /dev/null
        done
fi

exec /usr/bin/ssh -x -o StrictHostKeyChecking=no -o batchmode=yes ${user}@${host}
EOF
# chmod 755 /usr/local/bin/uussh

Now we can define one “SSH” port, that will automatically wake a computer if needed, wait for it to become alive, then initiate the SSH link. The chat script will specify the host name.

Taylor UUCP configuration

Now we come to UUCP itself. First, let’s create this special port. Edit /etc/uucp/port and add the following:

port ssh
type pipe
command /usr/local/bin/uussh

Now, we’ll set up login usernames and passwords for each host. The easiest way is to do this from a local shell, then distribute the generated passwords.

$ for src in host1 host2 host3 host4; do
   [ -d $src ] || mkdir $src
   for dest in host1 host2 host3 host4; do
      [ -d $dest ] || mkdir $dest
      if [ $src != $dest ]; then
         passwd=$( dd if=/dev/urandom bs=12 count=1 2>/dev/null | base64 )
         echo "$dest $src $passwd" >> $src/call
         echo "$src $passwd" >> $dest/passwd
      fi
   done
done
$ for h in host1 host2 host3 host4; do scp $h/* root@$h:/etc/uucp/; done

Now we have separate usernames and passwords on each host. We can finish up with the /etc/uucp/sys file:

commands rmail rnews gruntreceive-uucp
chat-timeout 120

These are some initial instructions that apply to all hosts. Here, I give permission to run rnews, rmail and gruntreceive-uucp, and I tell it to wait 2 minutes before giving up.

The following is an example of a host that sleeps and needs to be woken first:

system host1
chat Address: uucp\shost1.local\saa:bb:cc:dd:ee:ff\n\c login: \L Password: \P
port ssh
time any
call-login *
call-password *
protocol t
forward-from ANY
forward-to ANY

The following, is an always-on host.

system host2
chat Address: uucp\shost2.some.domain\n\c login: \L Password: \P
port ssh
time any
call-login *
call-password *
protocol t
forward-from ANY
forward-to ANY

Phoning home and scheduling retries

In the case of satellite systems behind some resticted network, assuming you have a way of tunnelling out of the network, you can “phone home” on a regular basis. You also want to periodically call uucico on all hosts to check if there’s any scheduled work on. This is done via /etc/crontab:

* * * * * uucp /usr/sbin/uucico -r1 -q
0 * * * * uucp /usr/sbin/uucico -r1 -s main_hub -c

The first line is a good idea on all hosts. It checks each minute for work to do, and calls uucico to do it.

The second line is the phone-home bit. In this case, it phones home to a system called main_hub, which in my case, is my public web server. You’ll want this second line on your satellite systems. It basically unconditionally phones home, and checks for instructions.

Great, UUCP works, what now?

Well, now you have a way of sending files between hosts. Two services that run well over UUCP worth investigating:

  • grunt: is a tool for securely running commands on another host. It can work over email or UUCP and uses GnuPG signature verification for authentication.
  • Many MTAs support UUCP as a back-end, such as Postfix. Very handy for sending reminders to yourself in a manner that is guaranteed to be noticed and not get buried in spam.
May 282014
 

Entered into an eBay contact form.

Hi, Just a short note.

I am closing my account: the form that asks why didn’t really capture the true reason why I’m closing.

It’s not quite “identity theft”, but it is security-related.

I haven’t been using my eBay account, so I thought I’d set the password to something nice and *strong*. On the password change form, I noticed a 20-character maximum limit.

This was red flag no. 1.

Then I pasted a randomised password from a generator. The site complained I had forbidden characters.

This was red flag no. 2.

By placing limits on the size of password and its content, it is clear to me that eBay is *not* serious about making its systems truly secure, and that breaches like the one experienced recently will be a recurring event.

By hiding behind “proprietary encryption” it isn’t even serious about reassuring the public: good crypto doesn’t need secret algorithms to work well.

As there’s now very little I buy off eBay, I feel the time has come to say goodbye. If you ever do get your act together, I might consider returning, but until then, farewell.