Friday, December 11, 2009

Anonymous Browser Uploads to Amazon S3

I've joined the Cloud.

I've signed up for the Amazon Simple Storage Service (aka S3). It costs nothing when unused, and almost nothing when used.

My original motivation for signing up was the potential for off-site backup. You know, just in case. The worst case.

But cheap remote storage isn't enough - what I hadn't considered at all, when I signed up, was bandwidth. The upload bandwidth that my ISP provides me, for the price I'm willing to pay, is a measly 512 Mbit/s. Consider uploading a 35GB snapshot via this narrow straw of a connection. I'll let you do the math. Bottom line is that it seems I won't be using S3 for backup.

But now that I've already signed up for the service, I started looking for other ways of using it: file sharing (of the legal kind) of files that are too large to send/receive as e-mail attachments.

After some digging I found S3Fox Organizer, which provides easy access to S3 from within Firefox. It allowed me to create buckets and folders, and then upload files, and then generate time-limited URLs that I could distribute to friends and family members, in order to allow them to download these files.

It works, but it's rather cumbersome when compared to Picasa, YouTube, SkyDrive, etc. And, while cheap, it ain't free.

And it's unidirectional - I could only send files.

Receiving files to my S3 account seemed to require web development karma that I don't posses. Luckily, after some more digging, I found a relevant article at the AWS Developer Community website: Browser Uploads to S3 using HTML POST Forms. The accompanying thread of reader comments is even more useful than the article itself, since it provided a ready made PHP script for generating a working, albeit rather spartan, browser upload interface:
  1. prerequisites:
    1. AWS S3 account
    2. create a storage bucket and an upload folder under that bucket (I did this with S3Fox)
    3. PHP enabled web server (see this howto for example) that will host the upload script (I host my server at my home computer)
  2. download getMIMEtype.js and place it at the document root directory
  3. place the following PHP script at the document root directory as s3upload.php
  4. edit the script and plug in your own AWS access key, AWS secret key, upload bucket name, upload folder name, and maximum file size (currently set at 50MB)
  5. share a link to this script with anyone you want to get files from

And here's the script itself:

// Send a file to the Amazon S3 service with PHP
// Taken, except for some fixes, from
// which refers to the article at
// Puts up a page which allows the user to select a file and sendit directly to S3,
// and calls this same page with the results when completed

// Change the following to correspond to your system:
$S3_BUCKET = 'my-upload-bucket';
$S3_FOLDER = 'uploads/'; // folder within bucket
$MAX_FILE_SIZE = 50 * 1048576; // MB size limit
$SUCCESS_REDIRECT = ' http://' . $_SERVER['SERVER_NAME'] . ($_SERVER['SERVER_PORT']=='' ? '' : ':') .
$_SERVER['SERVER_PORT'] . '/' . 's3upload.php'/*$_SERVER['SERVER_SELF']*/ .
'?ok' ; // s3upload.php is URL from server root

// create document header
echo '
    <title>S3 POST Form</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <script type="text/javascript" src="./getMIMEtype.js"></script>
    <script type="text/javascript">
    function setType(){
        document.getElementById("Content-Type").value = getMIMEtype(document.getElementById("file").value);


// process result from transfer, if query string present
$query = $_SERVER['QUERY_STRING'];
$res = explode('&', $query);
foreach($res as $ss) {
//echo 'ss: ' . $ss . '<BR/>';
if(substr($ss,0,7) == 'bucket=') $qBucket = urldecode(substr($ss,7));
if(substr($ss,0,4) == 'key=') $qKey = urldecode(substr($ss,4));
if($qBucket != '') {
// show transfer results
echo 'File transferred successfully!<BR/><BR/>';
$expires = time() + 1*24*60*60/*$expires*/;
$resource = $qBucket."/".urlencode($qKey);
$stringToSign = "GET\n\n\n$expires\n/$resource";
//echo "stringToSign: $stringToSign<BR/><BR/>";
$signature = urlencode(base64_encode(hash_hmac("sha1", $stringToSign, $AWS_SECRET_KEY, TRUE/*raw_output*/)));
//echo "signature: $signature<BR/><BR/>";
$queryStringPrivate = "<a href='$resource?AWSAccessKeyId=$AWS_ACCESS_KEY&Expires=$expires&Signature=$signature'>$qBucket/$qKey</a>";
$queryStringPublic = "<a href='$qBucket/$qKey'>$qBucket/$qKey</a>";

echo "URL (private read): $queryStringPrivate<BR/><BR/>";
echo "URL (public read) : $queryStringPublic<BR/><BR/>";

// setup transfer form
$expTime = time() + (1 * 60 * 60); // now plus one hour (1 hour; 60 mins; 60secs)
$expTimeStr = gmdate('Y-m-d\TH:i:s\Z', $expTime);
//echo 'expTimeStr: '. $expTimeStr ."<BR/>";

// create policy document
$policyDoc = '
{"expiration": "' . $expTimeStr . '",
  "conditions": [
    {"bucket": "' . $S3_BUCKET . '"},
    ["starts-with", "$key", "' . $S3_FOLDER . '"],
    {"acl": "private"},
    {"success_action_redirect": "' . $SUCCESS_REDIRECT . '"},
    ["starts-with", "$Content-Type", ""],
    ["starts-with", "$Content-Disposition", ""],
    ["content-length-range", 0, ' . $MAX_FILE_SIZE . ']

//echo "policyDoc: " . $policyDoc . '<BR/>';
// remove CRLFs from policy document
$policyDoc = implode(explode('\r', $policyDoc));
$policyDoc = implode(explode('\n', $policyDoc));
$policyDoc64 = base64_encode($policyDoc); // encode to base 64
// create policy document signature
$sigPolicyDoc = base64_encode(hash_hmac("sha1", $policyDoc64, $AWS_SECRET_KEY, TRUE/*raw_output*/));

// create file transfer form
echo '
<form action=" https://' . $S3_BUCKET . '" method="post" enctype="multipart/form-data">
                           <input type="hidden" name="key" value="' . $S3_FOLDER . '${filename}">
                           <input type="hidden" name="AWSAccessKeyId" value="' . $AWS_ACCESS_KEY . '">
                           <input type="hidden" name="acl" value="private">
                           <input type="hidden" name="success_action_redirect" value="' . $SUCCESS_REDIRECT . '">
                           <input type="hidden" name="policy" value="' . $policyDoc64 . '">
                           <input type="hidden" name="signature" value="' . $sigPolicyDoc . '">
                           <input type="hidden" name="Content-Disposition" value="attachment; filename=${filename}">
                           <input type="hidden" name="Content-Type" id="Content-Type" value="">

                     File to upload to S3:
                           <input name="file" id="file" type="file">
                           <input type="submit" value="Upload File to S3" onClick="setType()">

// create document footer
echo '

No comments: