Friday, July 29, 2011

Configuring hpodder To Handle Compressed Podcast Feeds

My Nokia 2730 classic dumbphone is surprisingly smart. Unbeknownst to me at the time of purchase, was the fact that it has a 2GB MicroSD card, and can play MP3 files. But eventually I did stumble upon this feature and it wasn't too long before I started tuning in on podcasts.

I use hpodder (launched from a cron job) to follow podcast feeds and download episodes to my computer, and a semi-scripted procedure to move these files to my phone's memory card over a USB connection.

Lately, hpodder started complaining:
*** 12: Failure parsing feed: Lexical error in  file http://escapepod.org/feed/  at line 1 col 1:
  unrecognised token: ^_\213^H^@
So I tried to download the feed manually:
$ curl http://escapepod.org/feed/
and the terminal filled up with gibberish to the point that I had to blindly type reset in order to fix it.

That was weird: after all, the feed is nothing more than an XML file - a text file, which should be perfectly readable with the naked eye.

I saved the feed with
$ curl -o feed.bin http://escapepod.org/feed/
and then determined its type:
$ file -b -i feed.bin
application/x-gzip; charset=binary
$ zcat feed.bin | file -b -i - 
application/xml; charset=utf-8
I.e. a GZIP compressed XML file.

So hpodder choked on a compressed feed.

I consulted the manual and found out that hpodder delegates downloads to cURL. I skimmed through the cURL manpage, found about the --compressed command line option, tried the download again - and got the uncompressed XML contents. Hoozah!

Apparently, the server, that's serving that particular feed, is mis-configured to always compress its replies, even if not specifically asked to do it. The --compressed command line options tells cURL to request the server to compress its replies, and cURL decompresses them.

I tried downloading other feeds with the --compressed added, and it worked fine. So either this option is supported by all the other servers on my list of feeds, or that cURL does nothing when the reply is not compressed. I dunno.

All that I needed now was a way to convince hpodder to launch cURL the same way.

Turns out that hpodder is a rather nice piece of software (and rather well documented too). The hpodder manual pointed me to ~/.hpodder/curlrc:
$ echo compressed >> ~/.hpodder/curlrc
and now hpodder works like a charm again (and probably faster than before, because it always asks for compressed replies).

Friday, July 22, 2011

Cloning a GitHub GIT Repository on Ubuntu 8.04 LTS

At work, we're still running Ubuntu 8.04 LTS on most PCs with Linux. Most of the time the age of the operating system isn't a problem - but sometimes it can be a pain. Case in point: cloning a GIT repository hosted on GitHub. This used to work just fine, until they switched to HTTPS:
$ git clone https://github.com/user/repo.git
Initialized empty Git repository in /current/directory/repo/.git/
Cannot get remote repository information.
Perhaps git-update-server-info needs to be run there?
When this first happened, I shrugged it off as a problem with the remote end, and just downloaded a source tarball from https://github.com/user/repo/tarball/master. But a few weeks later I got this error again with a different repository, and got annoyed. I tried the same command at home (Debian/testing, GIT version 1.7.5.4):
$ git clone https://github.com/user/repo.git
Cloning into repo...
remote: Counting objects: 81, done.
remote: Compressing objects: 100% (72/72), done.
remote: Total 81 (delta 34), reused 55 (delta 8)
Unpacking objects: 100% (81/81), done.
So, this is a problem with either GIT at work, or the Net connection. I downloaded the GIT source tarball and installed it locally in my account (at ~/local):
wget -c http://kernel.org/pub/software/scm/git/git-1.7.6.tar.bz2
tar xvjf git-1.7.6.tar.bz2
cd git-1.7.6
ls
./configure --prefix=$HOME/local
make
make install
and since we're using tcsh at work (don't ask), I also had to type rehash in order to convince the shell to use the newly installed GIT.

Here's what I got this time:
$ git clone https://github.com/user/repo.git
Cloning into repo...
error: SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed while accessing https://github.com/user/repo.git/info/refs

fatal: HTTP request failed
... which is useful: Google directed me to a question on Stack-Overflow. Most of the answers there deal with installing CA certificates, but the following trick works nicely with git version 1.5.4.3 on Ubuntu 8.04.4 LTS:
$ env GIT_SSL_NO_VERIFY=true git clone https://github.com/user/repo.git
Cloning into repo...
remote: Counting objects: 81, done.
remote: Compressing objects: 100% (72/72), done.
remote: Total 81 (delta 34), reused 55 (delta 8)
Unpacking objects: 100% (81/81), done.
(no need for env in bash).

And while we're at it, here's another trick that might be handy in the future:
$ env GIT_CURL_VERBOSE=1 git clone https://github.com/user/repo.git
Cloning into repo...
* Couldn't find host github.com in the .netrc file, using defaults
* About to connect() to github.com port 443 (#0)
*   Trying 207.97.227.239... * Connected to github.com (207.97.227.239) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Expire cleared
* Connection #0 to host github.com left intact
* Couldn't find host github.com in the .netrc file, using defaults
* Connection #0 seems to be dead!
* Closing connection #0
* About to connect() to github.com port 443 (#0)
*   Trying 207.97.227.239... * Connected to github.com (207.97.227.239) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
* Expire cleared
* Connection #0 to host github.com left intact
error: SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed while accessing https://github.com/user/repo.git/info/refs

fatal: HTTP request failed

Friday, April 29, 2011

Backup a Blogger Blog - Revisited

I recently found out that the method I used to backup this blog automatically, has stopped working. It all hinged on the observation that one can retrieve a single web page with the full text of all the posts of a given Blogger blog, by retrieving the link
http://blogname.blogspot.com/search?max-results=N
with a large enough N, e.g. 10000. This is not true anymore - at the moment I can only retrieve the latest 42 posts on this blog, from a total of 178.

I'm not sure when this method stopped working, or, for that matter, if it ever really worked. The fact remains, however, that I have a blog that I want to backup, so I spent the better part of an evening figuring out how to properly do this.

My new Blogger blog backup script, shown below, makes use of the Google Data services API to export and download the blog archive in XML format, and then extracts from it the links of all the posts, and mirrors these pages locally, with HTTrack:


#! /bin/bash

BLOGGER_EMAIL=user@gmail.com
BLOGGER_PASSWD=password
BLOGGER_BLOGID=000000000000000000
BLOGGER_BLOG=blogname

DEST_DIR=/path/to/backup/directory/
mkdir -p ${DEST_DIR}
cd ${DEST_DIR}

eval $( \
    curl -s "https://www.google.com/accounts/ClientLogin" \
    --data-urlencode Email=$BLOGGER_EMAIL --data-urlencode Passwd=$BLOGGER_PASSWD \
    -d accountType=GOOGLE \
    -d source=MachineCycle-cURL-BlogBackup \
    -d service=blogger | grep 'Auth='
)

curl -s "http://www.blogger.com/feeds/$BLOGGER_BLOGID/archive" \
    --header "Authorization: GoogleLogin auth=$Auth" \
    --header "GData-Version: 2" \
    | xml_pp > ${BLOGGER_BLOG}.blogspot.com.archive.xml

grep -o -e '<link href="http://'$BLOGGER_BLOG'.blogspot.com/..../[^\.]*.html" rel="alternate" title=' \
    ${BLOGGER_BLOG}.blogspot.com.archive.xml | \
    sed -e 's@.link href="@@g' -e 's@" rel="alternate" title=@@g' | \
    sort -ur > ${BLOGGER_BLOG}.links

mkdir -p ${BLOGGER_BLOG}
cd ${BLOGGER_BLOG}
httrack \
    -%v0 \
    -%e0 \
    -X0 \
    --verbose \
    --update \
    -%L ../${BLOGGER_BLOG}.links \
    -"http://${BLOGGER_BLOG}.blogspot.com/" \
    -"${BLOGGER_BLOG}.blogspot.com/*widgetType=BlogArchive*" \
    -"${BLOGGER_BLOG}.blogspot.com/search*" \
    -"${BLOGGER_BLOG}.blogspot.com/*_archive.html*" \
    -"${BLOGGER_BLOG}.blogspot.com/feeds/*" \
    -"${BLOGGER_BLOG}.blogspot.com/*.html?showComment=*" \
    +"*.gif" \
    +"*.jpg" \
    +"*.png"
A few comments are in order:
  1. the script contains the Blogger username and password - keep it safe!
  2. the blog id is the number that appears in the URL of most links accessible from the Blogger dashboard, after the blogID= part
  3. the XML blog archive may later be used to restore/migrate the blog
  4. local mirroring isn't really necessary - I just like it that I can view the blog contents offline
  5. another unnecessary step: I use xml_pp to beautify the exported XML file
  6. currently, the script performs no error checking - I may add some checks if and when I observe failures
  7. sources: Using cURL to interact with Google Data services, Blogger export format

Friday, April 15, 2011

Fixing Normalize Audio Feature in K3b

I usually burn and copy optical media with K3b. I don't do this often, but when I do, it usually just works. Except when it doesn't.

My wife asked me to create an audio CD for her, from an assortment of audio tracks she collected from various sources. It was easy enough to accomplish this with K3b. But the resulting audio CD was annoying to listen to, because I had to change the volume setting for each track.

It's a classic noob's mistake: I should've enabled audio normalization.

So I tried it again, but found that I couldn't enable audio normalization in K3b. Turns out that K3b uses an external application (called - surprise! - normalize-audio) to perform this task, and K3b just couldn't find it - a fact that was clearly (?) stated in the programs section of the K3b settings dialog.

I tried launching normalize-audio at the command line, and it seemed to be installed alright. A quick Net search brought me to Debian bug #597155 and Ubuntu bug #45026. The root cause of the problem is that normalize-audio reports its version number as
normalize 0.7.7
while K3b expects
normalize-audio 0.7.7

This can be fixed either in K3b or in normalize-audio, and patches for both sides have already been posted. But neither has been incorporated yet. In the meantime, I've implemented a workaround, based on suggestions in those bug reports:
  1. create (as root) an executable script named normalize-audio under /usr/local/bin/ with the following contents:
    #!/bin/bash
    case "$1" in    
        --version)
            /usr/bin/normalize-audio --version | sed -e 's/normalize/normalize-audio/g'
            ;;
        *)
            /usr/bin/normalize-audio "$@"
            ;;
    esac
  2. make this script executable:
    chmod a+x /usr/local/bin/normalize-audio
  3. this script is supposed to be used as a wrapper for normalize-audio, so make sure that /usr/local/bin/ appears in the PATH environment variable, and that it comes before /usr/bin
  4. launch K3b from a new command shell - it should now detect normalize-audio and allow you to use it

Time to Burn!
[05 Nov 2011] UPDATE: fixed script to work with file names/paths that contain spaces (Thanks anonymous commenter!)

Friday, March 18, 2011

Setting Default Laptop Screen Brightness when Running on Battery Power

A few weeks ago the screen of my "new" laptop (read: my wife's old WinXP laptop) started flickering.

It was almost imperceptible at first, but it got worse over time, to the point that the screen would, at random intervals, suddenly dim gradually and then brighten back, for a second or so.

I suspected that the laptop screen was failing. The prospect of having another headless laptop made me anxious. But after a while I noticed that whenever the screen brightness flickered, a battery icon would show up momentarily in the system tray.

So I came up with a theory: it may be some kind of a power supply problem that causes the laptop, for very brief periods of time, to think that it is disconnected from the mains power, causing it to switch to battery power, which, in turn, causes the OS to lower the screen brightness in order to reduce power consumption.

This theory seemed plausible, but with the warranty long-since expired, and without spare parts (power cord, power supply, battery, motherboard, etc.), my only option was to let it go. After all, other than this issue, the laptop seemed to be as functional as it can be.

The flickering, however, made me crazy, so I searched for a way to prevent Windows from automatically darkening the screen when the laptop is switched to battery power.

Well, it's easy, but it ain't obvious:
  1. disconnect the laptop from the mains power - the screen dims
  2. use the screen brightness controls to brighten the display back to the same brightness level when the mains power is connected
  3. reconnect the mains power
That's it - I kid you not.

A few days ago the laptop switched to battery power and stayed there. I took a little chance and purchased a replacement AC adapter over at eBay. Surprisingly enough, that was it. Happy Happy Joy Joy.

Friday, March 4, 2011

Getting Rid of rkhunter's False Warning about Xzibit Rootkit

I've installed rkhunter a long while ago, mostly because it seemed irresponsible not to install some sort of "protection". But, as is the case with any such tool, I started getting warnings, which, after I got over the induced anxiety attacks, were invariably confirmed as false positives.

It was usually rather simple to silence these warnings from the rkhunter configuration file /etc/rkhunter.conf - most of the time it was just a matter of un-commenting one or more lines, and occasionally updating rkhunter:
rkhunter --propupd
(say, for instance, after upgrading packages).

One false positive that was somewhat more complicated to disable was a warning about the Xzibit Rootkit. This warning is triggered by files containing the string hdparm - it's a known bug (see Debian bug #576680), and the workaround is to "use the RTKT_FILE_WHITELIST option to whitelist initscripts stating this string" - e.g. /etc/init.d/hdparm ...

The comments in the configuration file, suggest that the proper method of whitelisting a file is to also add it to USER_FILEPROP_FILES_DIRS and then update rkhunter. But this makes rkhunter complain that /etc/init.d/hdparm is an executable script, so I had to also add it to SCRIPTWHITELIST.

Bottom line - add the following lines to /etc/rkhunter.conf:
USER_FILEPROP_FILES_DIRS="/etc/init.d/hdparm /etc/init.d/.depend.boot"
SCRIPTWHITELIST=/etc/init.d/hdparm
RTKT_FILE_WHITELIST="/etc/init.d/hdparm /etc/init.d/.depend.boot"
and the run
rkhunter --propupd
Verify by running:
rkhunter --check
I can only hope that I won't hit any false negatives...

Friday, February 25, 2011

One Liner: Pretty Print, Syntax Highlight and Page Online XML Data

Useful when leaching a bunch of video clips from YouTube:
curl -s "http://gdata.youtube.com/feeds/api/playlists/916175AB005C0FB6" | xml_pp | pygmentize -l xml | less -R
(required packages: curl, xml-twig-tools, python-pygments)

Actually, xml-pp isn't really needed with YouTube - it can do the pretty-printing for you:
curl -s "http://gdata.youtube.com/feeds/api/playlists/916175AB005C0FB6?prettyprint=true" | pygmentize -l xml | less -R

Friday, February 18, 2011

Firefox Crash Recovery

Just the other day, I was left with no option other than to power-cycle my Ubuntu workstation at work. After this, Firefox, which was open when my workstation died on me, would not start anymore. It just insisted that "Firefox is already running, but is not responding. To open a new window, you must first close the existing Firefox process, or restart your system." - a useless piece of crap advice.

This happened to me before, but I never bothered to document the recovery process. I learned my lesson, thank you very much:
  1. step #0: find the location of the default Firefox user profile, and then cd to it:
    cd ~/.mozilla/firefox/pefasakk.default/
  2. step #1: remove the previous session's lock file:
    rm .parentlock
    you may be lucky, and all may just work now. I wasn't, and it didn't.
  3. step #2: if Firefox complains that bookmarks and history have been disabled, then you should move away the places database and try again:
    mv places.sqlite places.sqlite.backup
    mv places.sqlite-journal places.sqlite-journal.backup
    the good news is that Firefox will most likely start with your bookmarks intact, the bad news is that you've just reset your browsing history...
  4. step #3: I thought I was out of the woods - everything looked normal enough, until I restarted Firefox - a few hours later - only to discover that Firefox stopped saving cookies, so that I had to re-login to every web site that required it (so many!). I fixed this by moving away the cookies database:
    mv cookies.sqlite cookies.sqlite.backup
    mv cookies.sqlite-journal cookies.sqlite-journal.backup
Now back to our regularly scheduled programming (pun intended).

Friday, February 11, 2011

How To Disconnect from a Network Share on Windows

My wife's employer issued her an Asus Eee PC 1005P Netbook. She rarely needs to use it, so that it's usually turned off. But, whenever it is turned on, I encounter new system administration problems. The latest problem was with network shares.

Windows 7 Starter, which is the OS installed on this netbook, doesn't seem to allow access to multiple network shares, with different credentials. One has to disconnect from the currently connected network share before connecting to another one.

Disconnecting from a network share that's been mapped as a network drive is easy enough (the menu item to look for starts with 'Disconnect' ...) - but there's no obvious way to disconnect from a network share that has not been mapped as a network drive.

I found a solution on this forum thread - run the following command in a command window:
net use \\MACHINE\share /d
where you should replace \\MACHINE\share with the path of the network share you want to disconnect from.

Friday, February 4, 2011

Highlight #if 0 ... #else ... #endif in Emacs

This seems to be a FAQ, and a rather annoying missing feature in Emacs. Especially once you see it working in Vim. I guess it's the closest I can get to penis envy.

Well, envy no more - just add the code below to your .emacs and you're good to go.

It'll highlight, as comments, bits of C/C++ code that are disabled with #if 0 ... #else ... #endif and #if 1 ... #else ... #endif. This is done by configuring cpp-highlight-buffer to recognize "0" as an undefined C preprocessor label and "1" as defined, and then causing cpp-highlight-buffer to be called every time the buffer is highlighted.

If you feel adventurous, I suggest you remove the first few lines - those that start with (setq ... - and use, instead, the Emacs customization interface to configure cpp-highlight-buffer:
M-x customize-group RET cpp RET
It should be pretty easy to recreate the setup below, and then you may want to experiment:
  1. handle any #ifdef ... #else ... #endif blocks, by adding more pre-defined labels (e.g. HAVE_CONFIG_H)
  2. make disabled code read-only
  3. make disabled code completely invisible!
  4. use (background-color . "gray") instead of font-lock-comment-face, so that disabled code is still syntax highlighted as usual, only grayed-out
  5. ...
(don't forget to save your settings once done)

To be fair with Vim, this is a (potentially CPU intensive) hack, but then again, what isn't?

(setq cpp-known-face 'default)
(setq cpp-unknown-face 'default)
(setq cpp-known-writable 't)
(setq cpp-unknown-writable 't)
(setq cpp-edit-list '(("0" font-lock-comment-face default both)
                      ("1" default font-lock-comment-face both)))

(defun my-c-mode-font-lock-if0 (limit)
  (cpp-highlight-buffer t)
  nil)

(defun my-c-mode-common-hook ()
  (font-lock-add-keywords
   nil
   '((my-c-mode-font-lock-if0 (0 font-lock-comment-face prepend)))
   'add-to-end))

Friday, January 28, 2011

Digging Tunnels (part 2)

WARNING: following the instructions below can get you in trouble.

REMINDER: most system administrators can and do google.

As promised, this time I'll show how to tunnel from your workstation at work, to your PC at home, through a protocol aware firewall - only that this time with the client workstation running Windows.

Server setup is the same as in the previous post. If your home machine runs Windows, then you may want to research setting up Cygwin with an SSH daemon and stunnel directing traffic to it. I haven't done it myself, so you're on your own here.

Client side, at work, is easy, if you already have Cygwin installed there:
  1. install stunnel using Cygwin's setup.exe
  2. add the following stanza in ~/.ssh/config (where sshd.example.com stands for the address of your home PC):
    Host sshd.example.com
      Port 443
      ProxyCommand stunnel3 -c -f -r %h:%p   
    
If you don't have Cygwin, then do this:
  1. download and install stunnel for Windows
  2. open stunnel.conf for editing by selecting 'Edit stunnel.conf' from the newly created stunnel sub-menu in the Windows Start menu
  3. replace its contents with the following:
    client = yes
    debug = 7
    [putty]
    accept = localhost:60022
    connect = sshd.exmaple.com:443 
    
    (replace the stuff in red with your own stuff)
  4. start stunnel by selecting 'Run stunnel' from the same stunnel sub-menu as before
  5. use PuTTY to connect to your home PC, by pointing it to localhost:60022
  6. if you hit any problem, then you may be able to troubleshoot it by going over the stunnel log messages, which can be accessed from the stunnel tray icon context menu

Friday, January 21, 2011

Digging Tunnels (part 1)

WARNING: following the instructions below can get you in trouble.

REMINDER: most system administrators can and do google.

You're at work, behind a restrictive protocol-aware firewall, which allows outgoing connections only through HTTP (port 80) and HTTPS (port 443), and blocks other protocols, specifically SSH, regardless of the destination port (read about Deep Packet Inspection, to see how it's done).

And you want to access your Debian/Linux box at home, over SSH.

As long as said firewall allows HTTPS, you can use stunnel to tunnel SSH traffic through the firewall.

  1. Server side (PC at home):
    1. configure your firewall to accept connections on port 443
    2. configure your SSH daemon to listen to (the default) port 22 (note that it need not be accessible to the outside world)
    3. install stunnel:
      aptitude install stunnel4
    4. comment out unwanted services from /etc/stunnel/stunnel.conf and add the following:
      [sshd]
      accept  = 443
      connect = 22
      TIMEOUTclose = 0
      
    5. generate (as root) a new self-signed SSL certificate:
      openssl req -new -x509 -days 999999 -nodes -out /etc/ssl/certs/stunnel.pem -keyout /etc/ssl/certs/stunnel.pem
      openssl gendh >> /etc/ssl/certs/stunnel.pem
      
    6. enable the stunnel daemon in /etc/default/stunnel4 like this:
      ENABLED=1
      
    7. start the daemon:
      invoke-rc.d stunnel4 start
  2. Client side (PC at work):
    1. download, compile and install stunnel in your account
    2. add the following stanza in ~/.ssh/config (where sshd.example.com stands for the address of your home PC):
      Host sshd.example.com
        Port 443
        ProxyCommand stunnel3 -c -f -r %h:%p   
      
You should now be able to connect to your home PC over SSH.

Next time: same scenario, but with your work PC running Windows.