Sunday, May 4, 2008

Fixing Bacula E-Mail Notifications

I'm a man of habits. For example, every morning, as soon as I get the chance, I go to my computer and check my local mailbox for email notifications from Bacula - it sends me a message for each of the three backup jobs that were run during the night. It's usually OK, but sometimes not, and then I have to deal with the problem.

Actually, I haven't had a problem for quite a while, until last Thursday. It's not that I got an error, I didn't get any message at all - my mailbox was empty.

My initial guess was that the Bacula director daemon was stopped for some reason, so that the backup jobs were never started. I promptly started the Bacula console program bconsole, and used the run command three times, to start each backup job.

And then it hit me: bconsole would've reported an error if it could not connect to the Bacula director daemon. I typed list jobs and sure 'nuff - last night's jobs were on the list, marked as successfully completed.

By this time the backup jobs that I manually launched earlier were also completed, and no - there were no new messages in my Inbox. Something was broken. It wasn't critical - after all, backup jobs ran to completion - but my habits were disrupted, and I didn't like it at all.

That evening I sat down to fix the issue, or at least figure out what went wrong. I suspected that Bacula's configuration was somehow broken, probably due to a recent upgrade. I fired up emacs, opened /etc/bacula/bacula-dir.conf and /etc/bacula/bacula-dir.conf.dist, and compared them with M-x ediff-buffers.

The file with the .dist extension is the one that would've been installed upon a fresh install of Bacula. This file is left behind after an upgrade in order to allow the system administrator to do exactly what I was doing: compare the current configuration with the pristine configuration, that's shipped with the package.

Apart for the obviously different bacup/restore job definitions, there was indeed a difference in the command line used to launch bsmtp - the utility that's shipped with Bacula and is used to send email notifications. Aha! gotcha!

I "fixed" the command line, saved the file, restarted the director daemon
invoke-rc.d bacula-director restart
and launched a backup job using bconsole.

The job was completed with no errors, but I got no email notification. I read the log messages carefully and noticed that bsmtp reported a fatal error (it could not connect to localhost). So I copied the bsmtp command line from the configuration file and pasted it into a root console window. I got the same error message.

So now it looked like a problem either in bsmtp or exim4 (the mail server). I tried restarting the mail server:
invoke-rc.d exim4 restart
and got an error message:
Stopping MTA for restart:.
Restarting MTA:exim: incompatible command-line options or arguments
invoke-rc.d: initscript exim4, action "restart" failed.
So the culprit was exim4, not Bacula. I googled for the error message and hit Debain bug #476987. I read the thread of messages there, which mention several workarounds, till I got to the last message which simply stated that the bug has been fixed in version 4.69-3. I used
apt-show-versions exim4
to find out that the version installed on my machine is 4.69-2.

I could've stayed with this version, getting along with no email notifications from Bacula, until the fix trickled in from Debian/sid. But sticking to my habits is a motivation too strong to ignore - I decided instead to install the fixed (unstable) version of exim4 (please consult my previous post about maintaining a mixed testing/unstable Debian system).

I used the following command to upgrade any already installed package with the string "exim4" in its name, to the version that's in the unstable repository:
aptitude install -t unstable "~nexim4 ~i"
I then restarted exim4, retried the backup job and finally got a notification email. Yay.

Fixing problems caused by recent upgrades is becoming another habit for me - it took maybe half an hour to sort this out, less than it took to write this blog post.

I was once told that pain is not habituating. Maybe I didn't get the point, but if you can't get used to pain then I guess this kind of activity can't be considered painful. But if this isn't pain then it must be fun, right?

No comments:

Post a Comment