April 23, 2011

Engineering Matters

by with 16 comments

If recent events in the cloud storage community have taught us anything, it’s that Engineering Matters. Features essential to you, to your life, cannot be treated as something to worry about later. We’re founded on principles that security and trust do not have to exclude convenience from modern computer use, and we are dismayed by the prevailing thought that it does.

What do we sell? We sell trust. Trust that a cup of coffee won’t wipe out the only copy of your child’s first steps that you captured on your digicam. Trust that you can back up your tax returns without anyone being able to read it. Trust that when it hits the fan, we will have your data safe and secure. That is what we sell, at the core. Trust is an integral, central part of the system. It’s engineered in, every line of Python, every system API call, every system architecture wiki page. Your data cannot be allowed to leave SpiderOak through an act of God or man without your explicit action to make it do so, and we sell you the trust that it won’t.

Placing trust as the central core of the system fundamentally changes how one architects it. The system is first and foremost engineered to be secure and trustworthy. All the fun features in the world will not change the core, fundamental design principles set into the architecture of the system, and given form through solid and tireless engineering. We engineer into the system the expectation that bad things will happen to good disk platters, and design around that. No amount of high-level policy and vague promises of security can replace solid, engineered-in trust. There may be stumbles and falls on the long journey to this combination of trust and ease of use, but that is testament to the fact that we care more about solving the hard problem – trust in your data that transcends hardware and humans – than quick cheap tick-boxes on the feature list.

This is a great new frontier, an amazing and wonderful time in history where mainframe style computing in the data center is meeting the smartphone in your hand somewhere in the middle, and everything Star Trek promised you is starting to come true. We’re moving into an age where instant-access, anywhere access to any information you want is no longer a Hollywood dream, but daily reality. How many of you will be reading this on a phone with smaller size and much higher capability than Kirk’s communicator? Now is when we can stop complaining that things just aren’t possible, and instead move to the wondering when it will be done!

As we move forward, however, how is this always-on, instant-access society impacting you? Do you not expect that the enablers of this magic to take you seriously, with your needs as an individual? Why does adding the phrase “on the internet” suddenly imply that it’s OK to be lax about trust? You have a file store at home: your computer’s hard drive. For those of us renting, we pay someone (the landlord) to house it. How would you react if your landlord or a maintenance man plugged your hard drive into his laptop and downloaded a copy of everything just because some man in a suit asked him to? Why does “file store on the internet” mean anything different? Why should we instantly relax our standards just because it’s online and shiny?

If this sounds like a manifesto, that’s because it is. At the core of this future is the bedrock we lay down today. What you will have tomorrow, the freedoms and limitations of tomorrow, are set in concrete form with the foundations of today. That is the point of this message, that engineering matters! Core design principles will outlive any set of bugs in an implementation, and that is what we do here. Our core is trust, our core is security, our core is safety. The engineering of the system now will have a direct impact for years to come.

In a world of talk, Engineering Matters.

  1. ROFL, are you trying to say that if the police or FBI wanted to raid your apartment, your landlord wouldn't be there with the key, muttering "I knew there was something not quite right about those pesky kids"?

  2. really nice write-up! With spideroak I make my first steps in slowly putting private dokuments somewhere I can't physically go or can't directly log into the server. I really like the possiblities you offer and hope you stay to your principles!

    @Rikki I think that's what the post was trying to say.

  3. The problem is spideroak bloated, slow, unresponsive client. Clicking View/Deleted items freezes the application. Support failed to answer… any clues how to recover space occupied by (locally) deleted files?

  4. @a user:
    using spideroak on lots of machines under Windows, OS X, Linux.
    The client never failed! Seriously!
    Why don't you give us more details what is happening. For me support was always there. But since I work also in an IT department, I know customers who are complaining about the product without giving enough information in order to work on the topic.
    Back to the topic, very nice blog post. Great to hear some words about security and privacy and especiallz from a cloud storage company :-)

  5. @a user: Hi, this is Laura from SpiderOak support. Did you receive an autoreply when you wrote in to Support? If so, what ticket number were you assigned? I can check up to make sure we received your message or find out what happened.

    If not, you might have had the wrong address or been marked as spam by our somewhat aggressive spam filter. Please try writing in again and I assure you we will answer you promptly.

  6. Hello, which mode of operation do you use for AES? I don't seem to be able to find this information on the website.

  7. Why trust a company with a website is copyrighted in 2010? Sorry people, it's 2011. Isn't it?

  8. Perhaps a chart on your website listing the last time SpiderOak did a full restore test of their backups? It would be like the chart in public restrooms detailing the last time they were cleaned. It might go a long way to ensuring that our data is safe.

  9. Great! But how can we trust that Spideroak isn't sold to a company a little less valiant, with a different set of principles and standards?

  10. Worst case if SpiderOak gets sold off to some company you don't trust: Stop using them. Seriously, they only have an encrypted version of your data, they can't do anything with it. And honestly, unless you're Al Qaeda and the purchasing compnay is the NSA, they're not going to invest the 20 gazillion computing-years to decrypt your stuff.

  11. You can (and should) test the message store at any time. Create a new directory on your machine, "back it up" in the client, and then sync all your other directories to it. If everything shows up intact in the new directory, you know they're good.

    You don't really have to worry about a corrupted backup overwriting good data on your machine. Because the backups are held in encrypted data blocks, if those blocks became corrupted, the directory structure inside the blocks would be bad, preventing the restore, not restoring bad data.

    Remember, SpiderOak is for backing up, syncing, and sharing data, not being the only place in the world that your data is stored.

  12. Jan, a copyright notice should show the date of the earliest copyrightable material on it. Assuming that the theme and such was developed in 2010, it's the right notice to apply.

    From the copyright office: "[The notice should have] The year of first publication of the work. In the case of compilations or derivative works incorporating previously published material, the year date of first publication of the compilation or derivative work is sufficient."

  13. I just installed spideroak on Vista 32 and I can also confirm that the Spideroak client is unrensponsive.
    Clicking on any of the actions on Backup takes forever (minutes) to react. The Window for the applciation says that it is not responding.
    It does not appear to be using much CPU or memory, but it is completely unusable

  14. I have just uninstalled the spider-oak application because of the unresponsiveness noted above.
    If this problem is resolved in the future I would install it again and give it a try.