Saturday, June 27, 2009

One Liner: Transcode and Rotate a Video Clip

Here's how I rotated a certain video clip I downloaded from YouTube with clive, along with transcoding it from .mp4 to Xvid .avi:

mencoder video.mp4 -o video.avi -vf rotate=1,expand=256:192 \
-ovc xvid -xvidencopts bitrate=512 -oac mp3lame -lameopts cbr=96

Note the use of video filters -vf:
  1. rotate=1 rotates the video frames by 90 degrees clockwise
  2. expand=256:192 expands the frame from its original size 192x144 pixels to 256x192 pixels, while keeping the aspect ratio.

Friday, June 19, 2009

The Missing Link (or: When UDEV Bites)

It happened again. The nightly backup job failed. What a drag.

After a short investigation I found that the external backup disk disappeared. It was physically there of course, but the symbolic link to its device path was gone.

A while ago I added the following UDEV rule that creates a symbolic link /dev/elements to point to the device path representing the external backup disk (e.g. /dev/sda1) when it is first connected to the computer:
KERNEL=="sd?1", ATTRS{serial}=="574341523030323834303731", ACTION=="add", SYMLINK+="elements"
(see this post).

This allows me to always address the disk as /dev/elements in scripts and configuration files, instead of using the actual device path. This is useful because the device path can and does change, depending on the order of its detection by the kernel (i.e. it may appear as /dev/sdb1 instead of /dev/sda1).

I could've used the UUID link to the device: /dev/disk/by-uuid/306fc694-5328-4a6d-b6ec-4d1310c2feb8, but my custom symlink is shorter to type, easier to remember and I find it to be somehow prettier.

Anyhow, my nice link was gone, but the device path /dev/sda1 was still there, as were the various other links to this device in /dev/disk/by-uuid, /dev/disk/by-path, etc.

I was able to mount the disk via the UUID link, so it didn't appear to be hardware related.

I tried re-triggering UDEV events:
udevadm trigger
and was pleased to find that the link re-appeared.

I chalked it up to my incomplete grasp of reality. It's a fluke. A one-time event. Nothing to worry about.

All was well for a few days, and then I had to reboot my box. I was rather upset to find out that the link was missing again. I had no time to investigate this and simply added udevadm trigger to /etc/init.d/bootmisc.sh and restarted the machine. This seemed to work fine.

Close to two months passed by. Every time I had to reboot my box I recalled that incident but was not inclined to investigate any further - I am trying to curb my pathological dislike to workarounds.

I almost forgot about it, until I tried to mount my encrypted live-HDD.

Epic fail.

The script that I use to do this chore has worked flawlessly for more than a year. I intend to post it here soon, but till then suffice it say that the first step is to open the disk with cryptsetup:
cryptsetup luksOpen /dev/aluminum-crypto aluminum-crypto
and yes, /dev/aluminum-crypto is a custom symlink pointing to the encrypted partition, setup by an UDEV rule similar to the one shown earlier for my backup disk.

And guess what? that link was missing too. I re-triggered UDEV events and saw the symlink re-appear. Been there, done that.

I tried running cryptsetup again, and to my surprise it failed, right after querying me for the encryption password.

And my custom link was gone!

I tried it a few more times, and used strace to find that cryptsetup managed to successfully open and close /dev/aluminum-crypto several times, before it failed to open it. This was weird.

I decided to dig deeper. My plan was simple: run cryptsetup from within a debugger, break close to the failing open statement and step through the code until my link disappeared.

This meant that I needed to get the source code for cryptsetup and build it:

apt-get build-dep cryptsetup
cd /path/to/temporary/dir
apt-get source cryptsetup
cd cryptsetup-1.0.6
fakeroot -- debian/rules build

Unfortunately, cryptsetup failed to build from source (FTBFS - see Debian bug #522338). Fortunately, the bug was already fixed in the unstable package, so I retried with that package:

apt-get source -t unstable cryptsetup
cd cryptsetup-1.0.6+20090405.svn49
fakeroot -- debian/rules build

This time it worked, and I was left with an executable at src/cryptsetup. I launched gdb, stepped through the code and then I realized that the link got removed as soon as the first close function completed.

Did I mention how weird this is? All I did was open and close /dev/aluminum-crypto, which was enough to remove it - this shouldn't happen.

I had to verify that I actually saw what I thought I saw, so I wrote the following little program, named rmlink.c:

/* -*- compile-command:"gcc rmlink.c -g -o rmlink"; -*- */
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{
int fd;

if (argc != 2) {
fprintf(stderr, "usage: rmlink <symlink>\n");
exit(-1);
}

fd = open(argv[1], O_RDWR);
if (fd < 0) {
perror("rmlink");
exit(-1);
}
close(fd);
return 0;
}

This little program successfully removed my custom device symlinks by simply opening and then closing them.

Curiously, I was not able to remove other device symlinks (e.g. the ones under /dev/disk/by-uuid).

What was going on here? is this a personal thing that UDEV has against me? after all, UDEV is responsible for all those symlinks, both standard and custom, and they are all generated using UDEV rules which look similar to mine (see /lib/udev/rules.d/60-persistent-storage.rules).

I used the following magic to persuade UDEV to verbosely log its actions to /var/log/syslog:
udevadm control --log-priority=debug

I then saw the following messages appear, after running ./rmlink /dev/gigapod (yet another external USB disk that I use):
udevd-event[24908]: update old name, '/dev/gigapod' no longer belonging to '/devices/pci0000:00/0000:00:0a.0/0000:02:00.2/usb5/5-1/5-1.4/5-1.4:1.0/host1/target1:0:0/1:0:0:0/block/sda/sda1'
udevd-event[24908]: no reference left, remove '/dev/gigapod'


Gotcha!

But why was this happening? and why now? It used to work...

I scrutinized my own UDEV rules and pondered the rules in /lib/udev/rules.d/60-persistent-storage.rules and noticed two things that I hadn't noticed before:
  1. the standard rules are fired both for "add" and "change" events, while mine only address the "add" event
  2. the standard rules enable an option: OPTIONS+="watch"

I couldn't find any documentation for the "watch" option. But one of the commenters in Ubuntu bug #332270 explained that this option causes the Kernel to trigger "change" events for inotify events (such as closing a block device...), and it seems to be a recent addition...

The fix was easy enough to figure out now: add the "change" action to my rules, so that the symlinks were re-added when the "change" event got triggered:
KERNEL=="sd?1", ATTRS{serial}=="574341523030323834303731", ACTION=="add|change", SYMLINK+="elements"
Later I dropped the ACTION condition altogether.

Finally, I removed the UDEV event triggering workaround from /etc/init.d/bootmisc.sh, and restarted the machine.

All is fine.

For now.

Friday, June 12, 2009

No Floppy? No!

When was the last time you used the floppy disk drive on your box? Oh, you don't have one? Well, I do have a floppy disk drive on my aging laptop. And I'm quite sure that the only use I had for it was when I first installed Debian on it.

Yup! I actually installed Debian/Etch from floppies! that was my only option at the time, because the optical drive on this box is busted, and it doesn't boot from USB. As I said, it's an old box. I've since upgraded to Debian/Lenny and recently to Squeeze, and it seems that installation from floppies is no longer supported. It's kinda sad.

Now, you'd think that if you don't need something it's unlikely to cause any trouble. Well, think again.

Lately, I tried to launch my virtual Window$ PC, after more than a month of not using it, in an attempt to open a Word document inside Micro$oft Word. I was presented with an error message:
The floppy controller cannot attach to the floppy drive (VERR_FILE_NOT_FOUND).
Unknown error creating VM (VERR_FILE_NOT_FOUND)

This was something new. I assumed that the problem was with VirtualBox. And since I only read the second line in the error message at first, I found myself going through the VirtualBox documentation, rummaging through its various directories and files and methodically going through all of its configuration options and menus. No luck.

Finally, after more than an hour of futzing around, I got back to the error message and realized my original sin. I then searched for the error message on the Net, and found a short thread on linuxquestions.org, that suggested that the floppy disk drive should be disabled in the machine settings.

I tried it and it did the trick - I managed to launch my virtual Window$ PC and finish my original business.

But it continued to bother me. The floppy drive exists, so maybe it's just a permissions issue? I tried
ls -l /dev/fd0
and was surprised to find that it doesn't actually exist. WTF?

Time for some more Net searching. I found a thread at the Debian Forums site, and another thread at linuxquestions.org, which seemed rather relevant. Both suggested that the floppy module (device driver) was not loaded, that this is intentional (since floppy disk drives are pretty rare), and that manually loading the floppy module would fix the problem.

I tried
lsmod | grep floppy
as root, and got nothing. Indeed, the floppy module wasn't loaded. I manually loaded the module
modprobe floppy
(again as root) and verified that /dev/fd0 appeared. I then re-enabled the physical floppy drive in VirtualBox, and was able to start the Window$ virtual PC.

Finally, I added the module to /etc/modules in order to ensure that it'll be loaded on the next reboot:
echo floppy >> /etc/modules

Still, I wasn't satisfied. The floppy disk drive never gave me trouble before, so this must be a recent regression. I hate it when things break down like this. I hate it even more when all I can do about it is a workaround and not a solution. But I'm making progress, and am willing to accept workarounds. Sometimes.

In any case, I finally managed to find the root cause of the floppy failure. It's described in Debian bug #521520.

I'm now curious to know if the floppy disk drive works at all, but I don't have any floppy disk to test it with...

Friday, June 5, 2009

Merging Photo Albums with exiv2

More often than not I take a camera (a Canon Powershot A620 that I purchased a few years ago) to family events and take pictures.

To be honest, my camera is better than I am at taking pictures. I usually set the camera on AUTO, aim, half-press the button until I hear a beep and see the green rectangles around the people or stuff that I'm trying to capture, and then shoot. People around me are usually aware that I'm taking their pictures (I guess it's because I ask them to smile and stand still). I'm pretty happy with the results, most of the time.

My wife's dad has a similar approach to family events, but, unlike me, takes photography seriously. He took lessons, he purchased a professional looking camera (namely, Canon Digital Rebel XTi), and is always futzing around with its settings. He tries to blend in, and conceal the fact that he's taking pictures, in an attempt to capture people in their natural state. He's actually quite good at it.

The other difference between us, is that I usually post my pictures on our self-hosted Gallery2 website. I often pester him to send me a CD with the photos so that I can post them too. I take care of quantity, and he takes care of quality.

And here's where I find myself in a need. I like to order the photos in a chronological order on the website. It's rather easy with photos originating from the same camera, because the file name used for each picture contains a serial number, so that sorting them by name is equivalent to chronological ordering. But the numbering of photos from the other camera is, naturally, not synchronized with my own photos.

The first time I hit this, I tried to order the photos in Gallery2 using the time-stamp that's embedded in each photo's EXIF meta-data. But I found out that the internal clocks of the cameras were not synchronized (one of them was still an hour off, on daylight savings time). So I fixed this with exiv2 by running:
exiv2 -v -a 1:06 adjust my_pics_*.jpg
(yeah, a bit more than an hour - the offset was determined by visually matching the photos and subtracting timestamps of corresponding photos from different sessions).

But this didn't get the photos ordered on Gallery2. I tried setting the file creation time to match the EXIF timestamp:
exiv2 -v -T rename *.jpg
but Gallery2 either sorts by name only or rounds the timestamp to the nearest minute. Whatever the reason, I still couldn't get the photos ordered to my liking. I finally renamed all the files so that each file's name matches its timestamp:
exiv2 -v -t rename *.jpg
which, finally, did the trick.