How do I backup to Amazon S3 storage service

All of today I spent putting together all the tools to do an incremental backup to the Amazon S3 storage service.

Amazon is a good brand and providing storage as service is very good utility. But I was not very happy by the fact that Amazon has tried to reinvent storage with this web service. Web Services are a good interface for remote inter process communication. But why do storage operations like function calls. Anyway, cheap cost of the offered storage and good brand were enough for me to pursue it as my backup solution.

I spent whole evening setting up different tools that provide front end for S3 service. The most useful tool would be a NAS server that provides NFS/CIFS mounts and does S3 transactions at the backend. However it was hard to find any such direct tool. As of today there are many under-development or beta scripts written in perl, python, ruby, java. I gave a shot to few of them, but they weren’t very handy for solving my simple backup solution. JungleDisk is very good frontend tool for S3-service. It gives WebDAV frontend interface (again why WebDAV and not NSF?) and it’s available for all platforms. It is good if your needs are manual access to this storage. It is difficult to treat WebDAV URL as a drive so that a backup utility can write to it just like any other local drive. I investigated lot of options, NFS-to-WebDAV bridge, WebDAV CLI clients which can be called by backup scripts, but there isn’t a great solution for all platforms. My current need is Windows desktop backup. Finally I got ‘S3Drive’. It mounts the S3 storage as just another drive on your windows box. This was gr8 for my current needs. One thing to note - JungleDisk does not show the files stored by other front end tools to the same S3 bucket - it didn’t show the files stored by S3Drive. There are some other tools however which show objects stored by other tools as well, e.g. S3Safe.

So storage is ready, now the backup software. I was using EzBack-it-up. But after reading through its docs, I realized that it is very dumb in doing incremental backups. I had considered WinRAR some time back as backup option, but didn’t get its archiving logic. After reading thru some documentation and its CLI options I figured out a way to do incremental backups. When its CLI tool is run with -ao option it “adds files with Archive attribute set”. With -ac option it “Clears Archive attribute after compression or extraction”. Thus “rar a -ao -ac backup.tar “ will backup only the files that have changed since last backup (unless someone else changes the archive flag - I need to check with my CVS tools for this). So I got incremental backup tool.

How to enforce policies and schedules. CronForWindows solves schedules problem. I spent better part of an hour on very minute problem. Being alergic to BATCH files I couldn’t find a programmatic way to create the unique target tar file name that will have timestamp embedded in it. I tried running bash script using cygwin, but backslashes and/or ‘%’ symbols in date command format caused problems. I thought of writing quick java code, but thought that wasn’t really quick. So I decided to give python a shot. After half an hour I came up with following script:

from datetime import date  
import os  
today_str = "%s%s%s"%(today.month,,today.year)  
command = 'rar a -ac -ao'+'b:\e%s'%today_str+'.tar'+' c:\workspace\project'  

I had to put rar.exe in one of the directories in PATH, but it worked. The Cron scheduler can run this script as per the schedule I set.

I have few MBs of source directory that I need to backup. I expect its contents to change under an MB per day. Looks like pretty good deal with cheap prices of S3 service.




Types of Art

Dimensions of Idea communication


Lessons for Game designers from "Structure of Scientific Revolutions"

Bicycle pedals

Insanity and Software

Playing games, Doing Mathematics

Violet end of spectrum

Vipassana Meditation

Why gamers like pixelated games


Collecting Vs Hoarding Netflix

Game Design tip for Programmers - Less Automation

Review Ghost In The Shell - The blinds scene

Why Electron is the best development platform?

Project Dialog


Don't Panic

The Entanglement Pattern

The flying cars analogy

Road Trip of Atlantic Canada

Having done attitude


Before fixing performance issues

No Exams, Yes Video games

Podcasts Français


Quotable Quotes



Uruguay's President

New rules for Formula 1

Software Engineering Code of Ethics and Professional Practice

Something's missing


Coffeescript: To switch or not to switch

Water from air

The single most reason that keeps me using Vim

Time travel fiction

When your git branch diverges from your remote

Something like Olivia - John Mayer

Online courses - New technology, Old rules

Kelsey Grammer is spectacular

Why Patriotism doesn't make sense

Why sea steading is important to pursue?


Rolling back latest git commit

Google Lunar X Prize

iTunes Download stopped (err =9006)

Good essay on user centric software design

Why we need to explore Seasteading?


Master pieces from screen

How to write interactive CLI utility in node.js

The Gods Themselves by Isaac Asimov

Why OpenGL ES is so hard?

How to print stack trace anywhere in Javascript

It's not fair to compare Android with Internet Explorer

How to debug WebWorker threads?

Why can't I criticize Government if I don't vote? I sure can.

Elon Musk quote

Don't label yourself

Why the hi-tech minds are writing Social apps?

Truce with exercise

JavaScript is the next C

Ever growing todo list is a good sign for creative projects

Lesser women in tech... So what?

HTML5 offline cache - use swapCache carefully

How Wikipedia and Flattr can together change the World


Discovering LastPass

Functional programming in Javascipt using Underscore.js

Youtube ETA - Chrome extension

Why secure airports when it's planes that fly?

Mac book red light audio jack problem on Linux

How to save HTML5 canvas image to Google App Engine

Why don't they publish eBooks for free but with embedded ads?

Silicon Valley and Hi-tech

Reality distortion field around

Indirect experiments

Music sharing

ReaderScope 2.0.6 - Find Comments

ReaderScope 2.0

predictably irrational

ReaderScope upcoming changes


Why "Open" is not the first concern of user?

Mutt, Gmail, gdata, notify-send: A Perfect Email solution

Are you the Matt Damon of your game?

ReaderScope 1.9 - OAuth

New app "Find Comments"


ReaderScope 1.8.{2,3} - new authentication and global share

Portable apps for iPhone, Android, Pre, n900 - Can Processing.js help?

ReaderScope 1.8.1 - rethinking features

ReaderScope 1.8 - Random

Android fragmentation : How big a deal is it?

on Dropbox


ReaderScope v1.7 - Widget, new Icons and .... Ads

on Git: tips for SVN users and Git in Dropbox cloud

How to do an online IDE for Android app development

Anti Asserts

Bono's top ten list

Make Android speak your Twits with "TwitSpeak"


Reality of colors

Talking Twitter client

ReaderScope 1.5 - quicker, better, prettier

Preview android.R.drawable.* images

Is definition of "Evil" changing?

on Music: Battle Studies

ReaderScope 1.4 (with a Theme Pack)

On Chrome OS

Black on White

Android AsyncTask template

The AND-OR problem of laptop and mobile phone

ReaderScope v1.3.3 - offline podcasts

ReaderScope v1.3.2 - Social Channels

ReaderScope v1.3 - Beyond Google Reader

Frets on Fire for Maemo - Update

ReaderScope 1.2.7 - Podcasts, Expanded View

Sprint HTC Hero problems with ReaderScope

ReaderScope 1.2.3 - Search

ReaderScope 1.2.1 - Auto Login

ReaderScope v1.2

ReaderScope 1.1.3 - SDCard storage

Crash report for Android App

Wifi and Battery

ReaderScope 1.1.2

ReaderScope 1.1.1 quick-n-easy features

ReaderScope 1.1 - Social Beacons

ReaderScope 1.0.2 - making deep cache simple

ReaderScope 1.0.1

ReaderScope 1.0

Detecting HTC magic on Fedora

ReaderScope beta4 - offline fun

ReaderScope beta3 - Tweet the news

ReaderScope Beta2

ReaderScope beta1 is out

Intercepting Up/Down keys from bash script

Makeshift XML beautifier in python

HTC Magic (Indian version) - expectations and facts

using android AsyncTask

Android development with Vim + Eclim

Cutewit (preview)

Einstein's Dreams

Preprocessing the GUI

Evas + Inkface in v0.2.5

easy source

FoF - PyOpenGL = miniFoF (possibly for n810)

FoF update

Twitter client with inkface-pygame v0.2.2

Frets on Fire on Maemo 5 (Fremantle)

Static IP configuration on Fedora 10

Inkface+pygame v0.2.0

Smooth scrolling with Inkface+pygame

Inkface v0.2 update and automated GUI testing

Testing Android (Koolu Beta 3) on Openmoko Freerunner

Clutter animation paths with Inkscape+Inkface (v0.1.3)

Turning a page

Planning Inkface v0.2

Debian repository on Google App Engine

Inkface + Clutter

Inkface v0.1.2, Twitter-inkface client


Inkface update

Debugging python reference counts

Paul Graham nails it again!

Fedora 10 on MacBook 2,1

A great essay

Thanksgiving at Crater Lake

Digital SLR buyer's guide

more fun with notify-send

(k)ubuntu 8.10 - not ready for my home yet

Migrating to Google App Engine

Android opened - first looks inside

Has it started?

Inkface v0.1

Musical discovery of the week

Die Berlin-Affäre

aus Berlin

Inkface - SVG based GUI design

Script fun

Visualizing source code

OpenMoko photo tour

New connection with the past

George Carlin - favorite clip


80-20 rule

Python fun

OSiM'08 Berlin, Germany

Of games

"Wine" tasting

Today's slashdot and open source

Cost of Open Source

Lions for Lambs

Ubuntu reviewed

Flickr uploads from n810

Cringely and Paul Graham

Accelerometer brainstorming

Clutter Webkit

OSiM USA 2008 - Day 2

OSiM USA 2008 - Day 1

Google code

Gr8 commercial

OSiM USA 2008, see you there

OpenMoko, Android and a great hacking post

two great videos

Blog from nokia tablets


cool n770 hack


Colbie Caillat

Firefox 3 is brilliant!!!

XML RPC server inside apache mod_python

best effort technologies

coding Monalisa

formula of a horror movie

next desktop jump

The "Screen"

Nice review article on virtualization technologies

Once again... a quote from 'Troy'

Story of a random number

quotes from George Carlin

"from time import sleep"

Sea surfing under the Golden Gate

Trip videos


across two hours

What I read...

How do I backup to Amazon S3 storage service

Stranger than fiction

Gas prices... seems like big scam

Another nice quote

Hilarious nerd comment

Programming & Painting

Why do we like fiction?


If Harry Potter were a computer hacker ...

Gentoo, XGL, HGG

Henry Ford

VI - A capable IDE

New blog

Safari Bookshelf


Back to the L4


GCC 2.95 rpms

repeat (ps | grep) = filtered top

Halo fixed on ATI Mobility Radeon 9000

DirectX Installation Problem

Warning: const is a keyword in ANSI C

Not much on L4 front

Last two weeks


Stumbled on HURD

Installed HURD console

Make problem solved


Handy way to create stack trace