Recording Live Video Streams

Online video can be rather frustrating to watch even with a broadband connection to the Internet. Too often, it's too choppy to be enjoyable.

I've already mentioned the script that I use to scrape the browser's cache for video files. But the real solution for off-line viewing is to download, or record, the streams to files.

One easy way of doing this is clive. It works with several video sites - I use it regularly to download video clips from youtube.com and google video. Highly recommended.

Some video clips, however, are more elusive. The major local news sites around here serve mms:// and rtsp:// streams (typically accessed via URLs linking to files with .asx extension). These streams can usually be recorded with MPlayer, like this:

mplayer -dumpstream -dumpfile video.wmv -nosound -vo null mms://www.example.com/path/to/video.wmv

or

mplayer -dumpstream -dumpfile video.wmv -nosound -vo null -playlist http://www.example.com/path/to/video.asx

The stream URL can be found in several ways, e.g. by inspecting the source code of the web page serving that video stream; or by sniffing out the URLs being accessed by the web browser.

What I am still looking for is a way to dump rtmp streams. If you know how to do it - please leave a comment!

Scraping The Browser Cache for Video Files

I use the following script to find video files in the web browser's (Iceweasel/Firefox) cache directory (or any other directory that's provided as an argument):

#! /bin/sh
cache=${1:-~/.mozilla/firefox/*/Cache/}
find $cache -print0 | xargs --null file | grep -e "[^:]*:.*\([Vv]ideo\|MPEG v4\|Microsoft ASF\)" | cut -d: -f1 | sort -u |
while read file ; do
    mplayer -identify -frames 0 -vo null -nosound "$file" 2>&1
done | 
gawk -F= '
($1=="ID_FILENAME") {
    file=$2
}
($1=="ID_VIDEO_FORMAT") {
    format=$2
}
($1=="ID_VIDEO_WIDTH") {
    w=$2
}
($1=="ID_VIDEO_HEIGHT") {
    h=$2
}
($1=="ID_VIDEO_FPS") {
    fps=$2
}
($1=="ID_LENGTH" && $2 != 0) {
    print file" ("format") "w"x"h" ("$2" sec, "fps" fps)"
}
'
   

(have you noticed that this is actually an overgrown one-liner?)

Enjoy!

Reinventing a Wheel: Launching Applications

Hitting ALT-F2 in GNOME brings up an application launcher dialog. You start typing a command, and the list of matching applications is displayed, narrowed down as characters are typed. The arrow keys or mouse can also be used to select one of the applications from the list. Most of the time it beats opening a terminal and typing the command, or searching for the application through the menus.

A similar functionality is provided by default in awesome (version 2.3), with MOD4-F2 (MOD4 is bound to the Windows key on PC keyboards) - awesome-menu is launched, replacing the statusbar at the top of the screen with an "Execute:" prompt, together with a list of available executable files. This list is narrowed down as you type, and the arrow keys may also be used to navigate the list.

So far so good, but, on my oh-so-slow laptop, it takes quite a while for the menu to come up. I found the following entry for MOD4-F2 in the awesome configuration file ~/.awesomerc:

    key
    {   
        modkey = {"Mod4"}
        key = "F2"
        command = "spawn"
        arg = "find /usr/bin -type f -executable ! -empty | awesome-menu -e 'exec ' Execute:"
    }

The problem here is that /usr/bin is searched for executable files every time MOD4-F2 is pressed. This takes precious time.

I recalled that wmii had a similar functionality which seemed to work much faster somehow. After a short search I found that the trick was pretty simple: during startup wmiirc caches the list of executable files in ~/.proglist. So I added the following lines to my ~/.xprofile:

    ls -lL $(echo $PATH | sed 'y/:/ /') 2>/dev/null \
        | awk '$1 ~ /^[^d].*x/ && NF > 2 { print $NF }' \
        | sort -u > ~/.proglist

and modified ~/.awesomerc:

    key
    {   
        modkey = {"Mod4"}
        key = "F2"
        command = "spawn"
        arg = "cat ~/.proglist | awesome-menu -e 'exec ' Execute:"
    }

Apart for speed this has the advantage of finding all accessible executables in the path, and not just those in /usr/bin.

It has the disadvantage of not being dynamic - e.g. if you install a new package, the new executables will not be shown with MOD4-F2, until ~/.proglist is updated (either manually or by restarting X).

One Liner: Spy HTTP URLs Accessed on A Specific Netwrok Interface

Run this as root to list HTTP URLs being accessed from your machine via a specific network interface (ppp0 in this example), using tcpdump:

tcpdump -n -s 0 -i ppp0 -vvv -A "tcp" | strings | \
gawk '($0 ~ /GET.*HTTP/){path=$2} ($1=="Host:"){print "http://" $2 path;}'

You may want to filter the packets being captured, by replacing "tcp" with "tcp port 80", or maybe even a more restrictive filter, depending on what you already know about the URL you're looking for and the process that's accessing it. Consult the tcpdump manual page for more info.