Friday, April 24, 2009

UnDBX v0.13: Breaking the 2GB Barrier

This is the lucky number release.

I finally got some email feedback that pointed to a problem in UnDBX (thanks Darren Veach). In a nutshell: UnDBX v0.12 can't open .dbx files that are larger than 2 gigabytes (GB).

I never bothered to test UnDBX with such files, because the maximum file size of the .dbx files that are used by Outlook Express is 2GB (see Microsoft's KB article 903095). This means that such large files are, by definition, corrupted, and UnDBX was never meant to be used as a recovery tool.

But people do try to use UnDBX for email recovery. And some even complain when it fails. Fair enough.

I decided to look into the 2GB issue, but was not able to get Outlook Express to generate .dbx files larger than 2GB. OE started acting up when a .dbx reached 2GB, spewing error messages whenever I attempted to add messages to a full folder. But the corresponding .dbx file was not corrupted and its size never exceeded 2GB.

In the end I generated a file larger than 2GB on my Linux box by generating a large file full of zeros:
dd if=/dev/zero of=Zero.dbx bs=1024 count=2097152
and then concatenated it to the end of a valid .dbx file:
cat Original.dbx Zero.dbx > Inbox.dbx
To my surprise, UnDBX was not able to even open the file, let alone extract messages from it.

The reason for this is simple: the standard C-library file handling functions fopen, fseek, fread, etc. use signed 32-bit sized offsets to access files. This limits the size of files that can accessed to 2GB. Accessing larger files requires the use of the 64-bit analogues - fopen64, fread64, etc. And if you add the following in your source code, before including stdio.h, then these functions replace the 32-bit functions:
#define _FILE_OFFSET_BITS 64

I made the necessary modifications, and along the way paid some attention to proper use of long and int types so that UnDBX can be compiled and run on 64-bit platforms.

The bottom line is that UnDBX can now open .dbx files larger than 2GB. But whether or not it can actually extract some or any messages from these files, depends on how badly corrupted they are. Don't expect miracles.

So UnDBX v0.13 is up for grubs.

Enjoy.

[05 Sep 2009] UPDATE: autoconf has a macro for enabling large file support - just add the following line to the project's configure.ac:
AC_SYS_LARGEFILE
and make sure you include config.h before including stdio.h

Friday, April 17, 2009

It Hit the Fan, part II (or: a New MAC Address)

Last time on Machine-Cycle: my wife's laptop broke down, so I sent it to be fixed.

It took about a week to get the laptop fixed. I scrutinized the accompanying paperwork in an attempt to find out what was actually fixed, but the lab report was all too succinct: "fixed".

What I wanted to do now was copy only the files that my wife edited since she started working on my box, in a virtual machine, to her laptop. It was easy enough to find the files using find -mtime 5, and then zip these into an archive. What I found difficult to do was copy this archive to the fixed laptop.

After following a few false trails I realized that its IP address was not 10.0.0.4, but rather 10.0.0.200. I looked at /etc/dhcp3/dhcpd.conf, which I've setup in the past, and figured that the only way this could happen was if the laptop's network adapter has been replaced.

And indeed, a quick check confirmed that the network adapter on the laptop had a new MAC address:
  1. enter Start->Control Panel->Network Connections (I'm using the so called "classic" control panel).
  2. Double click the network adapter, or right-click the network adapter you want to check, select Status from the pop-up menu
  3. Press the "Details..." button in the "Support" tab
  4. The MAC address is listed as the "Physical Address"
I can only assume that the laptop's main board was replaced too.

This caused the DHCP server to treat the laptop as an unknown client, allotting it an IP address from the "unknown clients" address pool, which starts at, as per its configuration, 10.0.0.200.

The fix was easy enough - I just replaced the old MAC address in /etc/dhcp3/dhcpd.conf with the new one, restarted the DHCP server and disabled/enabled the network adapter on the laptop.

I unpacked the updated documents archive to the original documents folder on the laptop, updated the anti-virus virus signatures, re-enabled the nightly backup job, and manually initiated a backup job.

Phew, what a week.

Now back to our regular schedule.

We still have half a year of warranty.

I'm a bit worried.

Friday, April 10, 2009

It Hit the Fan, part I (or: How to Move My Documents)

My wife's laptop broke down. It was sudden and unprovoked. I found it idly blinking its LEDs one evening, completely non responsive.

I tried turning it off and on, removing the battery and putting it back again, and a few other voodoo tricks. But I got the same result each time: the fan started, the LEDs turned on, but the screen remained black, and after a while the fan stopped, the LEDs turned off and after a second the fan started again, and the LEDs turned on, and after a while turned off, and ... well, you get the picture.

One of the wisest decisions I took regarding this laptop, was to pay for a 3-year extended warranty - this is the second time it paid off - much too frequent for my taste.

The other wise decision was to setup a real backup solution. Thank you Bacula.

It took several minutes to restore my wife's files from backup to a directory in my home directory on my Debian box. I then fired up virtualbox ose, running the virtual Window$ machine I've setup the last time her laptop broke down. My objective was to allow my wife to work on my machine, with her documents directory fully available, as if nothing has happened.

Here's how to replace the original "My Documents" folder with a shared virtualbox directory:
  1. open the vitualbox menu Devices->Shared Folders...
  2. add the directory to the list of directories shared with the virtual machine
  3. in Window$ - start the file explorer, and open the menu Tools->Map Network Drive...
  4. map the shared documents folder to some drive letter (I used Z:), and make sure that the option "Reconnect at logon" is tick-marked
  5. now right-click the "My Documents" folder, select the "Properties" menu item from the pop-up menu
  6. select the "Target" tab and enter the drive letter of the mapped network drive
  7. hit OK etc. until you exit all menus


I disabled the nightly backup job by adding Enabled = no to the relevant job section in /etc/bacula/bacula-dir.conf and restarting the backup server:
invoke-rc.d bacula-director restart
and then disconnected the machine and packed it for delivery to the repair lab.

And now we wait.

Friday, April 3, 2009

Running Ubuntu 8.04 LTS from an Encrypted USB Drive

A live HDD is a complete mainstream OS, installed on an external USB disk drive of the rotating platter variety. It can be personalized, updated and modified like any desktop installation, with one very important difference: you can use it to boot any PC that can boot from USB (thus earning the title "live").

I've setup such a beast (running Debian) more than a year ago, and, until recently, have used it as a laptop alternative. I think it's neat.

Recently, I've set up a live HDD based on Ubuntu 8.04 LTS. I assumed, erroneously, that it wouldn't be too difficult - after all, Ubuntu is based on Debian, and installing Debian on a live HDD is almost as easy as installing it on a regular desktop PC.

Small print:
  • don't try this on a USB flash memory based disk ("thumb-drive", "disk-on-key") - these devices require special tweaking/distro to make the OS as read-only as possible
  • some USB hard disk drives may also be unsuitable for this because they startup too slowly during powerup for the computer's BIOS to recognize and boot from them. I had this problem with my Western Digital Elements. YMMV.
  • setting up a live HDD requires a working PC with operational USB, networking and optical drive, in order to run the OS installer.
  • please, please, please backup the PC you use for setting up the live HDD, just in case you make a mistake and find yourself installing the OS on the host PC instead.


The installation procedure is tiresome, yet straightforward:
  1. start installing Ubuntu using the alternate install CD, with the external USB disk connected to the computer
  2. select the guided encrypted LVM partitioning option
  3. IMPORTANT: make sure you format the external disk and not any internal disk on the PC being used for running the installer
  4. IMPORTANT: write down the device name of the external disk - you'll need this later
  5. continue the installation until the installer asks you if you want to install GRUB on the internal disk
  6. IMPORTANT: you MUST decline the installer's suggestion
  7. the installer will ask for a destination disk onto which you want GRUB installed: supply the device name you've recorded previously
  8. complete the installation - but don't attempt to boot into your new system yet!
  9. boot the PC with a live CD and mount the external disk (or do that on another PC)
  10. apply a few tweaks to grub/menu.lst on the first (non-encrypted) partition of the disk:
    • modify, if needed, the grub root device like this:
      # groot=(hd0,0)
      (do not remove the comment marker #)
    • add the string rootdelay=10 to the line that starts with # kopt=, and to each line that starts with kernel
    • one last thing: place this file somewhere on this partition - you'll need it soon...
    (in my original live HDD article, a few more files had to be modified, but these modifications are not needed anymore)
  11. reboot the PC, enter the BIOS setup screen and configure it to boot from the USB disk (on some PCs the disk is recognized as another hard-disk, so you can simply select it as the device to boot from)
  12. if this was a Debian installation you'd be done, but it's not - the boot process will most likely fail, with an error message saying that the root partition has not been found, and you'll end up in a shell with (initramfs) command prompt
    • type the following commands to continue the boot process:
      cryptsetup luksOpen /dev/disk/by-uuid/d1a9df24-b5c1-4ea2-985a-2f0fa3655fc2 sda5_crypt
      exit
      (replace the UUID and the volume identifier with the ones reported by the error message you got - the volume identifier matches the device path of the disk during installation)
    • this should get you into your new system, open a terminal and type the following commands to fix the system:
      sudo su -
      cd /
      patch -p0 < /path/to/live-hdd-ubuntu-8.04.2.patch
      update-initramfs -u
    • reboot


The patch file live-hdd-ubuntu-8.04.2.patch is the one that you've downloaded earlier and placed on the non-encrypted partition of the disk.

The patch fixes the following files:
  • /usr/share/initramfs-tools/init from the initramfs-tools package
  • /usr/share/initramfs-tools/scripts/init-premount/udev from the udev package
The fix makes these initialization scripts behave as in Debian, where the system waits for the USB device to settle down before attempting to access it, when the rootdelay parameter is specified at the kernel command line (this seems to be a manifestation of Ubuntu bug #213279).

That's all folks - enjoy your freedom (COUGH).