September 28, 2010

SpiderOak DIY: A space efficient key/value store for arbitrarily large values. Now in beta.

by with 4 comments

Update: SpiderOak DIY service has been discontinued, and is being replaced by the our new storage service which is a new work based on everything we learned from DIY and our previous internal storage projects. It is also open source, with a fancy new ZeroMQ based architecture. Please visit for more information and to request an invite to use that service. The information below is provided for historical purposes only.

We alpha launched DIY a few months ago to allow SpiderOak customers to directly store data on the SpiderOak storage network via https. It’s similar to Amazon S3, but tuned for large backup and archival class data, and thus much less expensive. It’s also open source, on both the server and client side.

Today DIY is now in beta, and we’ve been using it ourselves to implement new features for some time.

Basically, if you’re already using S3 as a backup storage, switching to DIY will save you a great deal. You could also use the DIY code to run your own space efficient, redundant storage clusters for large data.

One of the things we’re pleased with is how comprehensible the DIY implementation is. It turns out that focusing on space efficiency and high throughput (instead of low latency for each request) allows a number of design simplifications compared to other scalable storage systems.

This is a project you can easily jump in and make progress in quickly. It’s built using zfec for parity striping, Python, gevent, and RabbitMQ, with a framework we created for quickly building small message oriented processes.

Feed back from users and developers is much appreciated.

  1. Is it always cheaper than S3? S3 scales nicely as you are charged additional fees as storage increases in 1GB increments versus 100GB increments.

  2. SpiderOak DIY is positioned as a bulk, archival class storage service, for storing many GBs of data. As soon as you have 66 GB of data, the monthly cost for SpiderOak starts becoming much less (and that even disregards transfer costs.) We also offer the same discounts for DIY that we do for SpiderOak in general (education, etc.)

  3. Does anyone know who to get actual customer support from Solid Oak? I have been emailing but my specific questions are ignored and I just receive generic replies and what they say to do DOESN'T WORK. This has been going on for weeks. There is also no answer at their phone number.

  4. Kevin – I see Daniel already replied to you in another thread, but Just FYI, feel free to email me directly though and mention whatever ticket #s you had that aren't getting a good response. You can figure out my email address by guessing. :) Thanks in advance for your understanding, and I'm sure we can get you going!