The Connected Web

There's been some surprise expressed on the interwebs today that Amazon Web Services has begun encouraging customers to use FedEx to ship it large amounts of data and thus "bypass the Internet." But this is nothing new. Many years ago, storage guru Jim Gray named this technique "Terascale Sneakernet", reporting a highly respectable transfer rate for a terabyte of data — including the time taken to read and write it at each end — of "seven megabytes per second ... UPS takes 24 hours, and 9 hours at each end to do the copy."

I was reminded of the Jim Gray story last year by the tale of a South African IT company that made headlines when it realized it could send data between its offices faster by attaching a USB stick to a carrier pigeon than over a broadband connection. There's been a similar horrified reaction to the Amazon news from commentators who feel that the Internet somehow just ought to be faster at transferring data than something as old-fashioned and non-virtual as a national courier network.

But that can never be a foregone conclusion so long as commodity storage keeps getting bigger and cheaper. Remember too that courier services maintain highly efficient distribution networks, so long as volcanic ash doesn't keep their planes on the ground of course — though don't discount road freight as a fast-enough transfer medium if the data archive is big enough. If anything, the volumes of data being stored and analyzed by enterprises these days is going to accelerate the need for large-scale physical data transfers (doubtless that's why Amazon has opened up its service to general availability).

Far from fading away into history, I reckon terascale sneakernet is here to stay. The only thing that's going to make it obsolete is when the courier-powered sneakernet goes petascale.