A List of Amazon S3 Backup Tools

I'm using BitBucket in python. Works for me, and reliable enough in my experience of dumping 1 gig (and smaller) chunks up. I'm using a variation on the basic sync example, but I'm gpg encrypted tarball snapshots, not individual files.

There is this project which aims to treat S3 as an infinitely large disk (Linux only?):
http://dev.extensibleforge.net/wiki/s3/fuse

You ought to try out Mozy as well.

http://mozy.com

Latest brackup started to have more docs:
http://search.cpan.org/~bradfitz/Brackup/

But yeah, a little weak. Has a nice test suite and clear source code, though. :-)

How much do you trust the hosting service? Even if you have "nothing to hide", is it important that your data be encrypted or not? I'm simply mentioning because this is more important to some people than others (especially if you want to backup Quicken files or some such).

I’ve been thinking of using S3 to store backups of various machines (basically all linux/OSX ones), but what’s been holding me back is the inability of S3 to do rsync on the server side. rsync really needs an instance of rsync running “near” where the data is stored in order to do its cleverest compression/do-not-transmit smarts. Rsync is basically a win if you have a high-bandwidth link between the rsync server and the backing store, and a lower bandwidth link between the rsync server and client. With S3, you’d have to run the rsync server side yourself, remote from S3, which kind of defeats the purpose of rsync… But then I had a brainstorm.

Amazon’s ECC service, which parallels S3, allows you to create a virtual machine and turn it on/off as needed in the amazon compute cloud. The ECC instances have high-bandwidth connectivity to S3 storage, and so would be ideal for running an rsync server! You can set up an ECC instance which serves rsync, and then your backup script can turn the instance on, do the rsync, then shut the instance down when it’s done.

Now all I have to do is actually create the ECC instance, then create some kind of wrapper around the whole thing which does the startup-backup-shutdown wrapping around the ECC API, and voila!

Craig:

Brillian! I really like this idea...

How much upstream bandwidth do you have to make this satisfactory? My measly 512K upstream makes backing up online sound like a drag (though I do do appreciate the reliability and general peace-of-mind).

As much as I'm willing to pay for. Again, I'm mainly backing up data that's already hosted in a data center somewhere.

I'm using s3sync for my newly-minted nightly backups to S3, and I have to say I like it a lot. I had some issues with version 1.0.1, but had a series of nearly real-time emails with the developer, was able to provide some tcpdump records of the problems, and version 1.0.2 came out soon thereafter that hasn't hiccupped once for me. I'm pushing MySQL database dumps up to S3 nightly with it, some of which are on the order of 30-50 Mb, and everything works great.

The other tool you don't mention is jets3t:

https://jets3t.dev.java.net/

It's a Java toolkit for using S3, but it also comes with two tools built atop the framework, one a command-line tool (Synchronize) that works flawlessly at moving directories of files up to S3 and back down to your computer (including metadata, and some version of checking to make sure files need to be pushed over the pipe before using the bandwidth). There's also a GUI, Cockpit, that lets you upload and download files as well as manage ACLs. I've now run enough tests with it that if I run into any problems with s3sync, I feel totally secure I could move over to it without a problem.

Oh, one more thing: I tried to get Backup Manager working on my Linux box, but couldn't get it to actually communicate with S3. Between it needing a bunch of Perl CPAN modules installed (without any documentation to that effect, meaning you have to watch for errors, decode which package is missing, go find it and hope it'll compile against your specific distro, etc.) and there not being any real way for me to debug what was going wrong with its attempt to talk to S3, it was an easy call for me to move on to other tools.

What Exactly Is Jeremy Zawodny's Job?
I have no idea. Neither does Jeremy.

you forgot this one.....
http://www.sync2s3.com/

Best,
E. David Zotter

Thom:

Huh? What does that have to do with any of this?

Does everything that I write on my web site have to be related to my job?

What are you getting at, anyway? Come on. Out with it!

(re: linkblog)

Well, what exactly is your official job title Jeremy? What exactly do you do everyday? I saw in a previous entry that you pretty much just answer emails. Come on. Out with it!

JungleDisk is exactly what I was looking for. I went ahead and just signed up for S3. Up until now S3 seemed very developer oriented and wouldn't hold up well for the person looking to backup various files/share them amongst computers. Now all I have to work on is getting a faster upload, as Comcast has something against us with a paltry 100kbit/s upload.

Interesting little tidbit on that Jeep on Water video (from the BBC's TopGear) - it's presenter, Richard Hammond recently crashed a jet car at 300mph and is currently in hospital. See http://news.bbc.co.uk/1/hi/england/north_yorkshire/5365676.stm
He suffered a serious brain injury and is probably only still with us due to the quick action of the Yorkshire Air Ambulance. Air Ambulances over here in the UK are completely supported by charity donations, and a page was set up after the accident to take donations for the Yorkshire Air Ambulance, which is here - http://www.justgiving.com/phrichardhammond

General opinion now is that Richard is going to be ok (http://news.bbc.co.uk/1/hi/uk/5390598.stm), but I think you can put that solely down to the Air Ambulance saving his life.

good job.

For those of you who are interested in trying out S3 without it costing you anything, or if you just want to test your homebrewed S3 backup suite, you might want to check out Park Place. It's an S3 clone that you can run on your own computer.

And for whatever reason, the URL didn't come through (m aybe I should't have tried putting it inside an <a> tag...
Anyway, here's where you can find out more about Park Place.

http://redhanded.hobix.com/inspect/parkPlaceTheS3CloneYouVeBeenAlwaysAlmostWantingToSaveFifteenCentsWith.html

It may not fit your requirements, but I'm evaluating Interarchy (Mac OS X) for use with S3. It treats S3 like any other remote server, and offers some nice automation features.

I have been thinking about offsite backups for awhile now. I posted a little bit about my thought process on my blog.

For now, I have decided to go with s3sync. I just finished a test yesterday backing up and restoring 2GB of photos. It worked great and I am now planning to document the process in detail when I implement it on my linux server to backup up 15GB-20GB of my important documents.

I actually upgraded my Comcast internet speed from 6 Mbps/384 Kbps up to 8 Mbps down/768 Kbps up, to double the upload speed ($10 more/month). In my test it took 3hr/GB upload, and about 17 minutes/GB download. Certainly it will take a long time for the initial upload, but it will only be the changed files after that.

I was quickly reminded I haven't enabled QoS on my router yet. My girlfriend called and asked why the "server was so slow" while she was trying to work with files on my ssh server. Of course I could barely hear her with my VoIP phone.

Comcast note: I overheard my barber talking about calling up comcast after seeing a TV special for TV pricing and they updated his existing account with the promotional pricing. I thought that can't be true, so I tried it last month with my internet connection and they gave me the latest special $33/month.

Here is a link to my thought process about different offsite backup solutions.

http://johneberly.blogspot.com/2006/10/cheap-reliable-secure-off-site-storage.html
not trying to spam, just thought it might help people.

By the way, is S3 backup really "open source"? I tried the Beta version which has an expiration date.

I am currently using a combination of BackupNinja (using the Duplicity module), Jungledisk and DAVFS2. A quite extensive list of programs, but it works perfectly.

BackupNinja is used to make dumps of the databases, make a backup using Duplicity, for the scheduling of the backups.

JungleDisk provides webdav access to the S3 space, and DAVFS mounts the webdav as a normal directory to which BackupNinja writes the backups.

Works like a charm, much more stable than using Duplicity's S3 access features. Moreover JungleDisk supports caching, so no problem if the internet connection is lost during the backup!

Thom, EC2+S3 would be a killer app. If only an S3 account could be mounted as a device. I know that there has been some work in doing that (see the S3 forums) but there's nothing stable yet.

Thank for the info, great post.

I use jungledisk with SuperSync http://supersync.com to access and backup multiple music collections. Running supersync on an ecc system would be sweet, but just with the jungledisk cache, its pretty sweet.

http://filicio.us

I couldn't tell if you are a developer or not, but if you are there is one more route: roll your own using the free library code amazon supplies. I did this as a "get to know ruby" project, and it isn't very difficult with the supplied code as a base.

something to keep in mind: s3 *isn't* a file system, it is a key/blob system where the keys can be almost anything and the blob can be anything as long as the size is less than 5G. want it encrypted? do it before you send it. compression? same. metadata? up to 4k anything you want using 'x-amz-meta-???' in the headers when you put the data. puts are theoretically all or nothing (there is no way to seek in a blob, so there is no way to provide true rsync-like functionality...if the blob changes you have to reput the whole thing). you can look at the md5 sum after a put so you can compare without resending all. there is no key rename and there is nothing equivalent to a hardlink (multiple keys pointing to one blob), so rolling .0, .1, etc style snapshot backups aren't convenient (because re-putting the data costs you both time and money). certain operations may require multiple gets, e.g. listing the contents of a bucket will at most give back 1000 keys per get, so you may need to iterate. there is no equivalent to 'rm -fr *' either, you must delete the contents of a bucket one at at time. the upload speed is limited by your ISP, so don't be shocked if it is 35kbytes/sec (100 hours for 12G).

last, the simple way to put doesn't stream the contents, meaning it's ok for 3M jpegs, but if you want to put a 3G binary you'd better have a *lot* of memory. I think I recall from the forums that the s3sync author has figured out how to make streaming work, but I haven't looked at the code. you can ask him in the forum. when you are evaling tools you'll definitely want to check for streaming.

given what you want I'd look at s3sync first. I wouldn't have written something myself if it had been available at the time. I doubt you'd be happy with the ones that make s3 look like a disk in explorer because you can't automate using them. ruby/perl/python etc solutions are nice because it is at least possible to make them portable.

gymbrall said:
"For those of you who are interested in trying out S3 without it costing you anything..."

Come on! Only if your own time is free. Installing an S3 sim is work. Whatever your hourly rate is, it's not pennies a month like S3.

S3 Backup got upgraded yet again: http://www.maluke.com/blog/s3-backup-beta5-gtd

I just signed up for S3 yesterday after doing some extensive research while I've been looking for an offsite backup solution. I looked at Carbonite and I do like it quite a lot, but S3 has much better prices.

Thank you for this post, this is literally EXACTLY what I was looking for.

jeremy, you say you store your data on three different hosting sites (ohio, san jose, sf). isn't this your "backup"? ohio goes kaput, copy from sf or sj, etc.

Most of these boxes are 3-4 years old and don't have the space I'd need to do that. The newest of them has ~120GB of RAID, so it can backup the others. But they cannot back it up.

I must say, s3 is very intriguing. I'm still not sure on it, mainly because cabonite and mozy exist. I'm really hoping a mac client comes out or else I'm sorta stuck w/ s3.

Pricing wise, carbonite wins because I'd back all my media up too.. which gets me into 40-60GB range. Mozy would be next because you can do block level changes, and have a private key. Mozy has a better interface I believe too IMO. If I were keeping my windows box, I'd go with them.

Jungledisk is interesting... I was almost sold until I read it stores your key in plain text on your computer. The jets3t client looks interesting as well, and s3 backup when it goes multiplatform.

I'm too drawn to carbonites "unlimited storage" but I know I run risks if I dump 200GB of media on there. I also cannot understand how they plan on staying in business.

I started using Duplicity recently - it lets you do incremental (using librsync) encrypted (using gpg) backups (using tar) that can be stored remotely (using ssh/scp, ftp or supposely S3 as well).

It is part of Fedora/Debian which I suppose gives it some non-abandonware-ness :)

someone in the previous post's comments mentioned Dreamhost offering 200GB / 2TB-per-month for $90 for 2 years... so that's what I've gone with. ;) An excellent deal.

Works like a dream. I've linked to you, written it up, and am spreading the word. Thanks.

This is a great blog and admittedly is aimed at those who's backup needs are stricter than mine.

All I want is a backup system for my key files and photos. Music not so important. I have a Mac and PC.

So, I'm planning to use iDisk of .mac to backup my photos on my Mac and the XDrive on AOL for my other things on the PC.

@Linh

I am in the same exact boat. I like the simplicity of S3, I really do, but I like the automation of Carbonite. Perhaps once the application front-ends for S3 have matured a bit I may make the full switch over, but for right now I think Carbonite is more what I am looking for.

As for the space consideration, I am looking to only backup around 10GB or less...unless I decide to backup my MP3's but then we're talking about needing 80GB of storage which I do not really trust any other company to hold on to.

I still wonder how Carbonite plans to stay in business. If I had to guess I'd say they are using S3 as their backend. There is no real other way they could offer the prices they do. I suppose I will stick with Carbonite for now, if (god forbid) they fail miserably then I will make the full switch over to S3, but I will also keep an eye on the development of some of the S3 backup programs.

This is a great blog, and it has a great group of commentors. Keep up the good work guys!

On the questions about how Carbonite will stay in business, just think about it a bit. They surely use S3 or can beat S3's prices internally.

They allow you to upload the first 50GB's unthrottled, but after that you are limited to 0.5 GB/day. ($7.50 /month at S3's prices) Plus they reserve the right to kick "abusers" of the network. (They define abuse.) So the maximum loss for power users is pretty limited.

Then consider that most of their users will only have a small amount of data that rarely changes and it does seem like on average they will make a profit. They just need to be so simple it is the solution you recommend to your mother. (A goal they meet pretty well.)

Just FYI Guys,

S3 Backup (beta 6) was released today.

http://www.maluke.com/s3man/

Whatever backup solution you use, it's vital that you try the following test:

1. Install it.
2. Do a backup.
3. Uninstall it.
4. Restore from the backup without it.

If you can't do this, then you will get instant bitrot when the backup program's manufacturer gets bored or goes bust. Or upgrades the format: all my Backup MyPC backups became suddenly unreadable when a new version was released without backwards compatibility!

assuming s3 itself survives, the keys to passing JB's test above are the format of the keys (I used the relative path as the key, e.g. "photos/2006/09.06.06/xyz.jpg", which is quite easy to figure out) and whether you can figure out the metadata (e.g. I keep the timestamp of the original file in the metadata as seconds since the epoch). if the key scheme isn't decipherable then you are dead meat without the tool. if you can't figure out the metadata and you need things like timestamp, links, etc, then you are not going to get what you want without the tool. if the tool is a script then you'll have the code, meaning in theory you can figure out what scheme the tools uses. even if you *can* dope it out don't forget applying a complicated scheme manually to possibly tens of thousands of files isn't going to happen.

I agree with Joseph Bruno, this is important.

I know JungleDisk uses some really weird naming scheme, I didn't figure it out in limited time I spent trying.

S3 Backup uses straight keys (the way supersaurus suggested), if the file is compressed it has a meta field, returned as a HTTP header X-amz-meta-compression: zlib (or bz2 depending on the algo used). If encrypted it has a similar header X-amz-meta-cipher: AES or Blowfish. Figuring out the key to decrypt is not that simple however. For maximum security the password used for encryption is being 'salted' with a file name and a random string, different every time. The salt is stored as X-amz-meta-key-digest-salt, the SHA hash of the salted string is stored as X-amz-meta-key-digest (we need this to verify the password before downloading file and trying to decrypt it). Also there's X-amz-meta-decrypted-size that tells where to truncate the decrypted data (block ciphers need data to be aligned to a block boundary, it's funny how the anonymous guy who develops jungledisk took the easy way out -- he just used the not-so-secure RC4 cipher just because it doesn't require aligning).

http://en.wikipedia.org/wiki/RC4

I'll make sure to release a console tool compatible with S3 Backup, so you can be safe that your data are your data. BTW, you can make it happen sooner by voting -- see the last post in the blog (too many links to S3 Backup already, so I wont spam with more).

Oh, btw, I already have introduced incopatible changes to S3 Backup (in hash database format), but I took precautions so both old and new formats can be read, but the app just uses the new, better format when writing. That's it -- fully transparent but still evolving.

seems the list of backup services are pretty popular these days. Another similar post:

http://davie.wordpress.com/2006/06/30/couchsurfingcom-goes-down-but-a-lesson-learned/

And here is the list of more than 150 similar services:

http://www.listible.com/list/online-file-storage

Some recomendation? Strongspace seems pretty good to me (but a little expencive for my taste):

http://www.strongspace.com/plans

zhesto, general online file storage is VERY different from backup. Not to say that list is not useful, but those are not "similar services" at all.

Sergey, like you see somebody already complained in the comments about mixing different kind of services. Still there are several backup solutions inside the list (s3 based like jungledisk or not, like carbonite).

Jeremy, I just found this link from Scoble.

http://www.cachefly.com/

Might be another one to investigate? They serve up content for Digg, TWiT and others.

Jeremy, thanks again for starting this topic. It finally prodded me to implement the offsite backups I had been wanting to for awhile.

I think the idea of using ECC with S3 to perform backups is very interesting, but I decided to go with a simple solution for now.

I posted how I automated my backups to S3 using s3sync here.. http://blog.eberly.org/

I have successfully uploaded and restored over 30000 objects (>4GB) with this method. I plan on backing up over 10GB starting this weekend. Once the initial upload is done, I estimate my monthly bill will be $2 or so.

Just another FYI guys,

For anyone who uses Firefox this is a nice extension,

http://www.rjonna.com/ext/s3fox.php

Basically looks/works just like a FTP client :)

Jeremy -- first, thanks for an excellent post! It's been keeping me busy. :) I had a quick comment: in the blog post your item for S3 Backup says that it's "an Open Source tool for backing up to S3". I didn't see any info on the S3 Backup site that it's Open Source.

Sergey, can you confirm or deny either way if the project is Open Source?

Thanks very much!

Thanks very much for

Omar, the app is "partially" open-source, not free in a FSF meaning for sure. I've been quite busy with the app itself, but I do plan to release a lot of underlying code with GPL or maybe MIT license. Another possibility is that I will release full source code with a restrictive license -- not for use in other projects, only for inspection. I don't mind making the application fully open-source if I was compensated for all the time I spent working on it, but I don't think it will happen any other way than that I will monetize it myself. BTW I wonder what Jungle Dave is going to do, JungleDisk has a massive install base already and as the app is closed source, I guess he has plans similar to mine.

In other news Mac OS X and Linux ports of S3 Backup are coming soon, maybe the next beta will span all platforms.

Thanks for the info, Sergey! I'm looking forward to using the next beta version. :-)

Quick update on state of S3 Backup on Mac: http://www.maluke.com/blog/s3-backup-mac

A new entry is Openfount's InfoMirror. It has a sophisticated Ajax GUI as well as a part that runs in the background.

Use SyncBackSE (love it for local backups) but event its backup to FTP did not cover my needs (unattended, encrypted, and secure).

Signed up for S3, tried most of the backup clients listed, stuck with JungleDisk, but was very disappointed with the upload speed. I know, it mostly depends on your up-stream, but it was painfully slow.

Came upon Mozy... and never looked back. Good interface, smart features, set it and forget it. Non-nonsense people behind the company which helps a lot. Check it out if you're on a PC.

(Shameful link/plug: i.e. I get free storage if you click this link as opposed to retyping it https://mozy.com/?ref=3F5CS8 )

Not really what you've been looking for, but saw this on CrunchGear today, and it's kinda cool..

http://aws.typepad.com/aws/2006/10/zettabyte_stora.html

Jeremy,

Thank you - and thanks to your commenters - for this list of tools. You pointed me in the direction of the s3sync Ruby script, which I now have running on my NSLU2 (a $100 NAS device from Linksys) - first successful sync today! - and this post helped a lot.

(Here’s the first of the blog posts I made as I went along, in case you or any of your readers are interested: http://www.gilesthomas.com/?p=5)

Cheers,

Giles

Just another solution: S3Drive

Basically looks/works just like a Windows network drive and therefore accessable in all Windows and DOS application and all programming languages that support file I/O.

http://www.suchwerk.net/sodcms_AMAZON_S3DRIVE.htm

I have been using iFolder.

www.ifolder.com

It is designed more for 'file sharing' than pure backup, but I use it for both.

What about s3backup ? Ruby-based executables ...

http://s3backup.blogspot.com/

Hi,

We are about a week away from launching a private beta of PutPlace 1.0. This will initially offer automated backup to S3 and an explorer style view of your content once its up there with the ability to retrieve at will. Registration for the beta is available at at http://putplace.com

Windows only at the moment with Mac in the works.

Hi Jeremy. Here is another tool to add to your list.
http://js3tream.sourceforge.net
It's a very simple bridge between TAR and S3.

Shane.

I tried jungle disk but it wouldn't do a scheduled backup - would do an on-demand one, but not scheduled - therefore pretty useless for me :-(

I have been using Super Flexible File Synchronizer for quite a while now to sync documents and photos between my laptop, desktop, and do ad-hoc backups to a second desktop drive and a USB HD.

It really is super flexible (well maybe too flexible, given the gui layout in some parts)

The latest version now supports S3 which I'm trying out. The program will run as a windows service, which is the sort of set-and-forget operation that I'd really like. So far so good.

Most tools I've looked at are simple and feel like a beta. SFFS goes beyond just a simple interface.

http://www.superflexible.com
30 day trial
windows only.

One little trick I've learnt is that Bucket names must be unique across all of S3 - so choose something obscure !

You can use a website to manage your Amazon S3 account: http://s3browse.com

Hello, a few months ago i started a similar list (inspired by this one). I'm trying to keep it up to date. If interested go to
http://elastic8.com/blog/tools_for_accessing_using_to_backup_your_data_to_and_from_s3.html

You can also check out S3-FTP at www.awsware.com. It implemnets an FTP server that stores to S3.

I run the s3fuse on a virtual server, http://www.redwoodvirtual.com/ However, I use dump/restore, and backup on an inode basis. I do a level 0 every 3-4 weeks and o tdhe tower of hanoi method of diffs. This keeps transfers very low, as well as disk usage. rsync is great, but it is just a copy of the data as opposed to incremental backups. Sometimes people (myself included) want to recover something that was intentionally deleted a day or week ago.

I tried duplicity but went back to dump/restore since almost everything duplicity tried to do was just a complicated way to do the similar thing dump/restore already does, but didn't even bother doing it on an inode level, but only on a file level. duplicity, at least the last time I tried, did not allow scripted gpg encryption. It required the gpg encryption to be "signed", and who in their right mind would leave their secret key around with no password, just so duplicity could use it to sign+encrypt. encrypt would have been good enough, but it was not possible.

gpg was no problem with dump/restore because I just gpg -e (ed) the stdout of dump before writing it do it's destination. Scriptable. Incremental.

Download using S3Fox doesn't seem to work on OS X. Looks like S3Fox is using backslashes where slashes are needed in destination paths for objects -- resulting in empty folders on the OS X side.

Thanks for the info, Sergey!i was looking forward for this..

Thanks for the info...

You might want to check out Quillen. It is a new project that uses a novel approach to minimize data transfer and storage on S3. It also strives to be simple with just a command line interface.

http://sourceforge.net/projects/quillen/

Backup Review:

I found the best solution to backup your computer online. Mozy offers great customer service and wonderful services. You can backup your data for FREE for up to 2GB or you can purchase a UNLIMITED account for only $4.95 a month Nate was correct you should all try: http://mozy.com/?ref=3f9a896b&kbid=27203&m=5

I personally recommend them AAA+++

Iliana
Rdealz.com

I used mozy for a while and liked it until I switched to ubuntu and realized they don't currently support Linux.

I'm going to investigate some S3 solutions. I'm guessing that if I setup encfs I can back up the encrypted folder(s) for an additional level of security.

Btw if you do utilize mozy, they do offer private key encryption but files are encrypted on a per-each basis. You have to manually decrypt each file after a restore.

We released a User Interface today, which is called Bucket Explorer. Technically, its probably the best solution out there to transfer files to S3 and when you want to be in control, as it has easy to use UI, which is built on top of the robust JetS3t API.

You can use Bucket Explorer as a simple FTP tool or a backup tool for Amazon S3 or you can use it to:

1. Browse buckets and the files stored at Amazon S3.
2. Upload and download files, to and from Amazon S3 buckets.
3. Set Server Access Logging (Bucket Logging) for Audits.
Synchronize data between multiple computers.
4. Upload files in HTML format for Web Hosting (even when the extension is not .html or .htm).
5. Create public URLs and signed URLs to share the files.
Access shared buckets and files from someone else's account.
6. Set Access Control on Buckets and Files for authorizing other Amazon users or non authenticated users with different access rights.

Bucket Explorer works on every OS where Java is supported. It uses Amazon's ETag and its own SHA-1 hash combination to make sure that a file is never transferred again to Amazon S3 if is it not changed, to save the bandwidth costs and time.
http://www.bucketexplorer.com/

just an FYI I've posted "s3fs" (no relation to the original projected named "s3fs"), a fuse based filesystem backed by amazon s3, on google code

http://code.google.com/p/s3fs

We use S3 at our startup for our embed tool. Thanks for pointing me to all these useful tools.

Hello
BEST. POST. EVER.
I had no idea how important it was to post content as HTML.I liked your blog, it was fun to read. I look for blogs all the time that can influence me to come up with unique ideas for my blog.

Bye

I've just come across S3, read most of the blog but didn't get the answer that I was looking for :)

Wonder if you know?

Are the any products that will run process on a linux box that will sync the data on the box with Amazon S3, based on a cron job?

WARNING. The JetS3t toolkit currently defaults to very weak (single 56-bit DES) encryption. Anybody considering using it needs to look into changing the algorithm it uses to PBEWithMD5AndTripleDES at least until they hopefully implement my bug report here:

https://jets3t.dev.java.net/issues/show_bug.cgi?id=42

Looked quickly at Backkup Manager and saw that there is no S3 restore capability. Meaning "they will help you check your bits in but not out".

Here are a few more tools for S3.

S3Drive plugs into the Windows file system and maps a S3 folder as local drive. It can be accessed like a network drive (\\servername\folder where "servername" is the name of your S3 account your specify in the S3Drive config for it). S3Drive is free. It creates a virtual file system structure in XML format. That means that the folder and file structure you see in Windows is not identical with the structure and names used for the resources on your S3 storage area. You can access the XML files via other tools, such as S3Fox etc.

http://www.s3drive.net/

S3 Plugin for Wordpress
http://tantannoodles.com/toolkit/wordpress-s3/

Flickr to S3 Backup Tool
http://www.postneo.com/2006/03/22/backing-up-flickr-photos-with-amazon-s3

Cheers!

check out 'Transmit' at http://www.panic.com/transmit/. so far so good, on the mac platform.

It might be interesting for you to consider this (Free) next generation backup product (Secobackup, S3SQL, www.secobackup.com). Its basically a "install and forget" type of product.

www.secobackup.com

Secobackup is a personal backup product - it provides continuous data protection - as you make changes and save them, they are automatically backed up into Amazon S3. One nice thing about this is that it uses deduplication internally. So if 3 of your PCs have the same JPEG picture stored on their disks, it will detect that they are duplicates and store only once.

S3SQL is a automated MySQL backup product utilizing Amazon S3 similarly. You set up scheduled backup of one or more of your MySQL databases, you point the backup to your Amazon account, and it will automatically do regular backups, compress them, encrypt them, do differential and de-duplication type optizations and store them on Amazon's S3 service. Its has a AJAX based browser UI. Install it on a Windows XP box and you can set up backup of MySQL databases from any number of hosts - Linux, Solaris, Windows etc.

Its free. Try it, our users typically are up and running in just a few minutes with the backup already going to Amazon S3. Its simple to setup, and gets you immediate peace of mind from disk crashes, leaky roof or whatever...

Enjoy!
GP

Hi,

Well our own little Windows and Mac OSX - Amazon S3 software is 3 months old today!

http://www.databucketpro.com/about

Regards

Marc Liron
Microsoft MVP

Jeremy,

thanks for the post.
I have a requirement that needs to back-up 1 Terabyte of Data FOREVER - basically, it's an archive of data. Once it's uploaded, it stays the same size indefinitely. Looking at your solutions, I think it would be better for me just to get 2 DreamHost accounts (2 500GB accounts) or maybe Carbonite or s3. I could be wrong. Any suggestions?

For business use or if you have multiple computers, servers or alternative operating systems you should look at www.filesafebackup.com. Also good if you need to back up several GBs instead of like... 5.

Cheers, very nice list. Needed some ideas for a couple of sites and found a gem to base on thanks to this list.

Pageman, please keep in mind that using a DreamHost account as backup is against their TOS. There was some controversy about this a while back when they explicitly said you shouldn't do it.

They probably won't do anything if you just host a couple of files, but they said that if the majority of an account was non-web accessible, that that would be considered abusing the service. I think two 500gb accounts just for storage would get you into trouble for sure!

I'm writing a implementation of S3 in PHP (for fun!). It's not exactly a backup, but if any of the tools above can backup to another S3 url then it might be useful. I'm writing it as I don't believe in a single source of failure and it might be useful for mocking / fail over source. It's not close to being finished yet, but given a few more weeks it'll be much closer to being useable.

WOW, thanks for this list, I was looking for Mac OS X S3 backup solution. Thank you!

I started some C# source awhile ago for accessing S3 from PowerShell.

S3Nas PowerShell Provider:
http://s3nas.com/Home/tabid/53/EntryID/10/Default.aspx

Cheers,
Timothy

I like S3Fox on Windows and Mac — it works reasonably well, but still, I wish there was an elegant S3 backup utility for Windows in the same vein as Transmit, which looks pretty spiffy! The Windows ones I've tried so far have disappointed me and been very clunky/awkward to use. :(