No Sound (or: What the Plumber Taught Me About Linux)

Last week ended with our downstairs neighbor knocking at our door, to complain about a water leak from our apartment. He's a soft spoken fella, always calm, and always right. He annoys me to no end.

He was right of course. It took a minute to find the leak, close to the apartment's master water tap. It was a slow, but steady, dripping, that caused the trouble. The pipe itself looked all rusted and about to blow.

Quite a bit of water - and a hefty sum of money - went down the drain during the next few days, as a pair of nitwit plumbers worked on fixing the leak. I got a living proof that if anything gets rewarded it is initiative, not intelligence.

With a looming deadline at work, this was pretty much too much for my taste. Imagine my annoyance when I found, after a casual reboot of my Debian box, that I had no sound.

I noticed this one evening after that reboot, while trying to relax - I was attempting to view animated films over at NFB.ca. I tried turning up the volume with the knob on the speakers, verified that they were turned on and connected properly, and then used the volume multimedia keys on the keyboard to turn the volume up, and then used alsamixer at the command line to turn the volume up, and then... Nothing worked. There was no sound.

There were no obvious errors in the log files at /var/log. lspci listed the device as

00:08.0 Multimedia audio controller: ALi Corporation M5451 PCI AC-Link Controller Audio Device (rev 02)

lsmod told me that the relevant Kernel module, snd_ali5451, was loaded.

All is well, but still no sound.

I rebooted the box. Nothing.

I shut down the box and then turned it on again. Nada.

What now? I was stumped. I was pissed. I was tired. Can't a guy enjoy a cartoon when he feels like it?

Well, as I said, it's initiative that gets rewarded, so I continued fuzting around until I found alsactl:

root@machine-cycle:~# alsactl init
Unknown hardware: "ALI5451" "Analog Devices AD1886" "AC97a:41445361" "0x0e11" "0x00b0"
Hardware is initialized using a guess method

and voilà! sound was working again.

I have no idea what went wrong, and I guess it's likely to happen on the next reboot, but till then I have sound and I really don't care.

Storing PuTTY's Configuration to a File

In the previous post I described how I partitioned my SanDisk Cruzer Micro USB flash drive, and installed grml Linux on it. The first partition on that flash drive is formatted as FAT32, so that I can use the flash drive on Window$ machines, for the normal tasks of moving files around between computers.

I also mentioned that I've placed a copy of PuTTY on that partition, in order to allow me to connect to my home PC, via secure shell (SSH) from any available Window$ box with an Internet connection. The problem here is that PuTTY's configuration is saved to the Registry, and it's lost whenever I switch to a different computer. It's rather easy to configure PuTTY, but it's tedious nonetheless, and I'd like to avoid it if I can.

After going through the PuTTY documentation I found section 4.26 - "Storing configuration in a file", which states that "PuTTY does not currently support storing its configuration in a file instead of the Registry", and then goes on to describe how to do it anyway, including a method for saving to file any modifications made to the configuration during the session.

I've opted for a less complicated setup, since I don't intend to modify the configuration. The first step is to place the following batch file, putty.bat, alongside the PuTTY executables

@ECHO OFF
regedit /s putty.reg
start /w putty.exe
regedit /s puttydel.reg

together with two .reg files - putty.reg and puttydel.reg. The first contains PuTTY's configuration, as saved (once) with the following command:

regedit /ea putty.reg HKEY_CURRENT_USER\Software\SimonTatham\PuTTY

and the second file, puttydel.reg is used to remove this configuration from the Registry of the host machine, once the shell session is over:

REGEDIT4

[-HKEY_CURRENT_USER\Software\SimonTatham\PuTTY]

This should work fine for password based authentication, but not for public-key authentication. The problem is that the full path to the key file is stored in the Registry, which includes the drive letter, and this drive letter is liable to be different when switching computers.

The solution is easy: edit putty.reg with a text editor you like, and modify the value of the key PublicKeyFile to be relative to the PuTTY folder on the flash drive. So that, for example, if you've placed a key file named putty.ppk in a sub-folder named keys under the folder where the rest of the PuTTY files are, then you should set the value of PublicKeyFile to ".\\keys\\putty.ppk"

One last tip: save the PuTTY Registry after you connect at least once to the target PC, thus not only verifying that the configuration actually works, but also letting PuTTY save the target host key to the Registry. This will avoid the need to confirm the host key whenever you connect to it from another PC.

Installing grml Linux on a USB Flash Drive

Last time I mentioned that I needed a USB flash drive for storing the GnuPG encryption keys and Amazon S3 access keys, which are used for backing up my files to the cloud.

The reason should be obvious: if I ever need to restore files from the remote backup, it would probably be because I either lost both my Debian box and Bacula backup disk, or that I can't access them for some reason. So I better have those keys available elsewhere.

So I purchased a 4GB SanDisk Cruzer Micro USB flash drive. But then I decided to install Linux on it, instead of just using it to store those key files.

Sick? I don't think so. After all, if I ever get that desperate, it's likely that I'll need to setup a Linux workstation with, at a minimum, GnuPG and duplicity installed on it. It would be much simpler if I could just boot any available PC with my USB key and be able to restore my files to some local storage device.

The only remaining question was which distro should I install? I wanted a distro that can be installed to a USB flash drive, which can be tailored ("remastered") to include up-to-date versions of s3cmd, duplicity and GnuPG, and support persistency of some sort (for storing the access keys and other configuration files).

DistroWatch.com and Google provided some leads. There are many Live CD distros available, and quite a few minimal distros meant for USB flash drives. I further narrowed down my search by going for a Debian or Ubuntu based distro, which I'm familiar with, and figured I should first try the official releases: Debian Live and Ubuntu.

I've spent a few evenings trying to figure out Debian Live, and failed miserably. Debian Live is extremely configurable - it attempts to be the universal Live CD build system - and as such is pretty complex, at least for my taste and limited free time. The documentation seems to be out of date (e.g. it mentions commands such as lh_config and lh_build, where the actual command is lh) and the Debian Live wiki is a rather messy collection of apparently out-of-date howtos. Bottom line is that I just couldn't get past some error messages and gave up on it.

I ditched Debian Live in favor of the Ubuntu Live CD, which seemed less complicated to customize, and can also be installed to a USB flash drive.

I actually managed to generate a Live CD image with extra packages installed (s3cmd, duplicity and a few others) and some packages removed (such as the Ubuntu documentation). The only problem was that this image failed to boot under QEMU. Well, actually, it just hang during the startup process, showing a throbbing Ubuntu logo. It can very well be that I simply didn't wait long enough, and/or that my trouble had to do with a package that I should not have removed/installed - but I had neither the patience nor time to investigate.

This was rather frustrating. I didn't expect this to be so complicated. As I was about to settle for my original plan of placing the few key files I needed on the USB flash drive, I searched again for a Debian based live distro and found grml - "Linux Live system for sysadmins / texttool-users / geeks".

Grml is developed by several Debian developers, it's based on Debian/Testing, and it's currently actively maintained. Grml already includes s3cmd and duplicity, and a lot more - it even has awesome - ain't that awesome?!

This all looked very nice, and all that was left for me to do was to install it to my USB flash drive:

1. download the grml iso image and burn it to a CD
2. partition your USB flash drive (e.g. with gparted) such that it contains the following partitions, assuming its block device is /dev/sdb (adapted from the grml persistency howto):
   1. /dev/sdb1 - filesystem: FAT32, label: datastore, type: primary, size: 2GB
   2. /dev/sdb2 - filesystem: FAT16, label: grml, type: primary, bootable, size: 800MB (enough for one CD image)
   3. /dev/sdb5 - filesystem: EXT3, label: live-rw, type: logical, size: 1GB
   4. /dev/sdb6 - filesystem: EXT3, label: GRMLCFG, type: logical, size: 200MB
   and tune the EXT3 filesystems:

   tune2fs -i 0 -c 0 -m 1 /dev/sdb5
   tune2fs -i 0 -c 0 -m 1 /dev/sdb6

3. boot a PC with this CD, press "q" at the prompt to enter the shell
4. insert the USB flash drive
5. run

   grml2usb --bootoptions="persistent" /live/image /dev/sdb2
   live-snapshot -d /dev/sdb5 -t ext3
   umount /live/live-snap*

6. shutdown the PC, remove the CD and reboot it from the USB flash drive - if all goes well, grml should come up, with all system modifications loaded from, and subsequently saved to, the persistent live-rw partition

The larger FAT32 partition is used for storing files that should be accessible on a Window$ PC. At the moment I have Putty and TightVNC (viewer only) "installed" on it, allowing me to connect to my Debian box from any windows PC with access to the Internet - but I'll leave this to a future post.

Backup to The Cloud

I've mentioned before that I was looking into off-site backup options. I also reported that I got to the conclusion that the upload bandwidth provided by my ISP is too low to be useful for such a backup plan. My napkin calculations indicated that I'll need 7 days for uploading a full backup snapshot, and 4 to 5 hours to upload an incremental nightly backup.

Well, I'm glad to report that I now have a working off-site backup scheme, based on duplicity, where the nightly backup (to Amazon's S3) takes, typically, just a few minutes to upload. The data transfer is secured with SSL, and the data itself is compressed and encrypted with GnuPG. Furthermore, the off-site backup is an accurate mirror of the contents of my Bacula backup storage, which contains a backup of both my wife's WinXP laptop, and my own Debian box.

The initial full backup was a bit of a pain to setup, though. I first did a full restore from Bacula's backup to an external disk, and used duplicity to generate the initial full backup set on my other external disk. This took a few hours to complete, and then I uploaded the full backup set to S3 using s3cmd's sync command. It took 8 days to complete the upload, with several interruptions, but mostly keeping a pretty constant upload rate of about 60KB/s, which is close to the nominal 64KB/s that it's supposed to be.

Once the initial upload was done, I added a new job to my Bacula configuration that triggers a nightly backup script to run after the last nightly backup job. In the script (shown below) I use BaculaFS to mount Bacula's storage as a filesystem (for each client and fileset) and then I use duplicity to backup its contents. I also backup duplicity's cache directory to an external disk, just in case.

Note the use of duplicity's --archive-dir and --name command line options. These options allow the user to control the location of duplicity's cache directory, instead of the default, which is a path name based on the MD5 signature of the target URL. I needed to do this because I moved the backup set from one URL (local disk) to another (S3), and I didn't want duplicity to rebuild the cache by downloading files from S3, but rather use the cache that was already on my box.

In the few days I've been using this scheme, I haven't hit any communication failure with S3 during backup. I expect I'll have to modify the script in order to handle such a failure, but I'm not sure how - at the moment I just abort the snapshot if the point-to-point network adapter is down.

The next step is to setup a disk-on-key with the relevant GnuPG encryption keys and S3 access keys, in order to allow me to restore files from S3, in case both my box and my Bacula backup go kaput.

To be continued.

#! /bin/bash

export HOME=/root
SCRIPTDIR=$(dirname $0)
ARCHIVE=$HOME/.cache/duplicity
ARCHIVE_BACKUP=/mnt/elements/backup/duplicity
DEST=s3+http://my-bucket
SRC=/mnt/baculafs
DUPLICITY_OPTS="--encrypt-key=XXXXXXXX --sign-key=YYYYYYYY --archive-dir=$ARCHIVE"
DUPLICITY_LOGLEVEL="-v4"

# make sure we're connected to the internet
PPPIFN=$(cat /proc/net/dev | grep ppp | wc -l | awk '{ print $1 }')
if [ "$PPPIFN" == "0" ]
then
    exit 1
fi

snapshot()
{
    umount $SRC/$2 2>/dev/null

    export AWS_ACCESS_KEY_ID=$(grep access_key $HOME/.s3cfg | cut -d\  -f3)
    export AWS_SECRET_ACCESS_KEY=$(grep secret_key $HOME/.s3cfg | cut -d\  -f3)
    export PASSPHRASE=$(cat $SCRIPTDIR/duplicity-gpgpass)

    echo "Generating list of current files in backup ..."
    duplicity list-current-files -v0 $DUPLICITY_OPTS --name=$2 $DEST/$2 > /tmp/$2.list || exit $?

    echo "Mounting BaculaFS ..."
    baculafs -o client=$1-fd,fileset=$2-fileset,cleanup,prefetch_difflist=/tmp/$2.list $SRC/$2 || exit $?

    echo "Prunning remote snaphsot ..."
    duplicity remove-older-than 1W $DUPLICITY_LOGLEVEL $DUPLICITY_OPTS --name=$2 --force $DEST/$2
    
    echo "Updating remote snaphsot ..."
    duplicity $DUPLICITY_LOGLEVEL $DUPLICITY_OPTS --name=$2 $SRC/$2/ $DEST/$2

    unset PASSPHRASE
    unset AWS_SECRET_ACCESS_KEY
    unset AWS_ACCESS_KEY_ID

    fusermount -u $SRC/$2

    echo "Backup local duplicity archive ..." 
    rsync -av --delete $ARCHIVE/$2/ $ARCHIVE_BACKUP/$2/
}

snapshot winxp winxp
snapshot machine-cycle machine-cycle
snapshot machine-cycle catalog