May 14, 2010

Feeling disconnected? This is why.

by with 72 comments

If you’re having connection problems from the SpiderOak client, the solution is to upgrade to version 3.6.9658 or later.

… because 3 years ago when we launched SpiderOak I generated the SpiderOak SSL certificates that the SpiderOak client uses to verify the identify of the storage server. This is to protect against DNS poisoning attacks (i.e. otherwise an attacker that controlled DNS could attempt to convince your SpiderOak client to upload data to a different server.) These are not the same certificates as for the SpiderOak website.

I thought I generated certificates for 10 years, but they were only for the default of 3 years, and thus connections began expiring en masse about an hour ago. Most mistakes you should only make one time, and clearly this would fall under that category.

The verification for the cert is embedded along with the new SpiderOak client. We generated new certs, and fast tracked new builds through testing and release, so please visit the direct download link and all will be well again.

Please accept my deepest apologies; I’ve requested my flogging to be scheduled tomorrow at sunrise.

Comments
  1. Thank you guys for staying on top of issues like this. I appreciate the speedy resolution and your team actively working in keeping this a better service than anyone else!

  2. At least you had the guts to admit the mistake, rather than trying to cover it up (like most corporations would).
    I'm really impressed. Keep up the good work, I'll definitely keep using your product

  3. this is great customer service. And email heads up and a full explanation of the situation with no bs or sugar coating. Stuff happens and appreciate the honesty. Keep up the good work!

  4. No worries… as the others say, honesty, prompt resolution and a sense of humor go a long way.

    Believe it or not, I remember when Microsoft used to have this attitude/service :~{{

  5. After downloading the file and installing, I realized, that for security you shouldnt post any direct download links on your emails. You should just encourage your clients not to download from email links. Anyways, I really don't mind this, sh*t happens, what matter is how you respond to that, and you guys are A++.

  6. Agree with all the above (or below, depending where in the order this ends up). Thanks for the service

  7. Hey, $h!t happens. Keeping it honest and upfront wins huge bonus points. Thanks for the great service.

  8. Just moved across 2 days ago to you from SugarSync…. Its refreshing to know we made the RIGHT decision. They would have covered it up for days with no information. Even when we closed our account they just switched off the account with no so much as a thank you for your custom email or confirmation that the account was closed, so guess we wont EVER go back to them… But you guys rock, and we will recommend you to all our clients….

  9. A company that cares and backs it up with action. What more can anyone ask? You're miles ahead of everyone else, thank you for being there for us!

  10. These things happen sometimes, and always I've mostly seen people complain about lack of communications rather than the fault itself. I agree – as long as I'm kept informed about what's happened, why it's happened, what's being done about it and whether I will need to do anything, it's not that big of a deal.

    Maybe some deck scrubbing duty for a few days, sure, but the flogging can be cancelled as far as I'm concerned. :) Everything's still shiny!

  11. Agree with many comments up here. So from the positive, I really appreciate the honest explanation and the timely response, not a given in most Internet services. As someone say though, the link to download made me feel unsecure, I went to your website separately.

  12. Thanks for the e-mail and the quick solution. I have been recommending SpiderOak left and right, since it is such a perfect solution for back-up and sharing, and I really appreciate your positive attitude.

    Instead of having the flogging, could you update the yum and apt repositories, so most Linux users would automatically get an update?

  13. You rid the right thing and it gives us more confidence in the system than a cover-up.

    Perhaps you can arrange one of those, you know, *nice* floggings?

  14. Your client no longer supports Debian Etch. Looks like I'm looking for another provider. :(

  15. Alan, everytime I've made a serious mistake, I've asked to be flogged with a feather. Why don't you try that?

  16. Thanks for the good communication. Being honest and keeping us up to date with what is going on is very good customer service.

  17. @ Adam: Thank you for your comment. Please let me check on the Debian Etch version and either myself or someone else will post an update for your as soon as possible. Thank you in advance for your patience.

  18. Okay, the file I need to download is SpiderOak-9658-1.fc10.x86_64.rpm. Fedora has been Fedora 12 for about six months, and Fedora 13 is about to be released. Fedora 10 is no longer a maintained version. It seems like the numbering is not tied to the program's ability to run so please drop the "F10" from the naming to lessen confusion.

    Also, thanks. The one glitch that was keeping Firefox from launching when I clicked on the "HELP" button up top is fixed.

  19. Wow – such refreshing honesty! No worries – keep up the GREAT work. One thing that may help others – Norton prevented me from running the file because it was too "new" and not enough other Norton users had used it. I temporarily disabled antivirus, installed it, switched AV back on and it worked just fine. Again – thank you!

  20. One of the things that appealed to me about SpiderOak was the honesty with which the tech details are explained on the main site. It's not "magic", just a very nice service. So it's nice in this case to just hear what happened. I have personally had to get SSL certs reissued due to mistakes/errors many times.

    So as everyone has said, communication counts and this is really not a big deal. Great Service!

  21. Hmmm, big fan of SpiderOak but hate the idea of downloading content from an email post no matter how efficient. Would prefer a secure login. Oh well, will push the button and possibly be pwn'ed since I love SpiderOak :)

  22. There should be info on your home page about this, and the email should simply ask people to visit your website. Links in emails ain't the best way.

  23. Thanks for the post. What a bummer – as a developer I sympathize. Just a note though, this sort of notification should really go out on your Twitter account as well as your blog.

  24. @ kathy parker: You will have to download and install the new client on each of your machines running SpiderOak. Again – we do apologize for the inconvenience and thank you again for your understanding.

  25. @ Willy: If I can ask, are you trying it from the direct download link? If so, you can try to download the new version directly from the website here: https://spideroak.com/download.

    Please do keep us posted on your progress and thank you in advance for your patience.

  26. I just used the direct download link on my mac and had no issues.

    Thanks for your honestly and quick resolution.

  27. Working now. Sorry for the confusion. I had tried both links, but I realize now that both had cut off early. I have a bad connection at work, so that's probably the reason. It worked this time though. Thank you for the help.

  28. Ok thank you Alan. At least you have a solution and the way to solve all the problems. So, now I'm going to upgrade the version. I understand you and that's way I thank you are doing.

  29. This happened to me a couple times. The most "funny" was when I had to regenerate and install 25 certificates because the root CA that I used had expired.

    Thanks for the email, and keep up the good work.

  30. Thanks for the quick solution and transparency in communicating to users! The email with the direct link had made me suspicious as well, though.

  31. After downloading the file and saving the file, message box reports corrupt file, virus or other problem-also unknown publisher; ask for a new copy. This whole process sounds fishy, what is up?

  32. @Rene NL – There was a problem with the yum repository, which has been fixed. I've verified that the apt repository is working; you may have to check for new updates.

    @Tom – Try downloading the file again. Sometimes the file will only download partially and you'll get that problem. To be safe, download the latest version directly from our website (https://spideroak.com/).

  33. Only started using yesterday buy good to know we are using a company that does not bs! Accidents and mistakes happen it's how theyvare dealt with.

  34. I'm sure we all appreciate your honesty. Until now Spideroak has been rock solid. This update would have had to occur sooner or later anyway…

    Please keep up the good work. Your service is invaluable and is a stand-out offering on the web

  35. Thanks for the information. Though I'd like SpiderOak to be able to update itself–or at least download and launch the installer.

  36. @deozaan We surely can, the only problem with this particular update (that made it so annoying) is that the certification issue prevented us from communicating with the client, and hence we were unable to push an update.

  37. I've tried downloading the new software for Ubuntu Karmic from both Firefox and Chrome, on two different machines. It balks at about 1MB. Anyone else having problems?

  38. This updates Fails to install on Win7 Ultimate that I run. Something about "Error opening file for writing:"
    C:program filesSpiderOakMSVCR71.DLL

  39. I cannot seem to download this new setup. I get the Internet Explorer message: Internet Explorer cannot download SpiderOakSetup-9659.exe from SpiderOak.com. The server returned an invalid or unrecognized respone.

  40. @Matt – Sounds like another copy of SpiderOak is running at the same time the install is running. If you can find the other copy of SpiderOak and exit, it should be fine.

  41. @stephenbee, Mrs. M.

    Please try downloading again. We've noticed that some downloads are getting cut off prematurely, but most of them are succeeding. If you're still having trouble, please let us know.

  42. @Chip, the problem with partial downloads of the SO client has been an issue for quite some time. Please investigate.

  43. @T – I'm sorry to hear that. Would you be willing to help us test? The downloads have been working fine for me, and I'd like to get another perspective on the problem. Email us at support@spideroak.com if you're interested.

  44. My suspicion about the occasional aborted download we see, is that there's some little known firewall code or SSL library problem in the wild that causes 20+ meg downloads over SSL to fail more often than they would over plain http.

    I once spent a day just downloading SpiderOak over and over from a few different computers in different cities, running tcpdump on the server and the client to capture data. Maybe one out of 200 downloads would fail, but it was really random. They would succeed for hours and then 2 or 3 fail in a row. But, didn't seem to have any relationship to the traffic load on the webserver or anything like this.

    Normally all of spideroak.com is SSL only, and every request for http:// gets immediately redirected to https://, but we've disabled this only for the direct download link above. If you want to manually take the 's' out, you'll get a plain http download, which I'll wager will work just fine even if the SSL version has been failing for you. I sure wish we could track this down more specifically, though.

  45. First, thanks for the frank description of what happened – refreshing.

    I did the mandatory SpiderOak update on a couple of my computers, and now significant CPU usage on those computers is being consumed by a process where Spider Oak is monitoring for Directory changes. On my newer computer, it consumes about 50% of CPU time, and on my older computer, it sits at 99%CPU usage for that process. This continues even after I stop SpiderOak. I have to restart my computer to get rid of this process.

    I haven't upgraded SpoderOak for some time, so this may be due to changes some time ago. Have others of you experienced this same problem? Is there anything I can do about it?

  46. @Russ – Thanks for the report and apologies for the trouble. A few people have reported the same. Unfortunately that's caused by a rare Windows-only bug introduced in the 9657 beta release. We changed the Windows directory watcher to catch old short 8.3 style names that are occasionally reported by the OS and convert them to long names. It's an obscure case, and none of the beta testing exposed it, so we thought it was safe to include that change in 9658. It's fixed in 3.6.9659 (for Windows only.)

  47. Thanks for the news – Upgrade worked. I'm a new user to your storage solution. I'm still exploring. Wish you a blessed day.

  48. Elegant and exemplary explanation. These things happen, and you've shown unusual candidness and sincerity. Well done. You've enhanced your standing.

  49. I updated and the cpu usage for the Windows directory watcher jumped to 50 per cent. Please fix it. We all make mistakes. At least I do. "There should be no shame in your game."

    By the way, I think you have a wonderful, useful product. Thank you.

  50. Same here, CPU usage peaked with this last release. My fan goes crazy every time Spideroak starts looking for changes

  51. I have two desktops (one running XP, the other Windows 7) and have updated both to 9659 – but neither will now connect. I have restarted both systems after the install, and have re-installed the update, but to no avail. Any suggestions?

  52. I have been trying to download the update for a week now with no luck. What's up? other larger downloads have no trouble.

  53. Russell / Sean: If you don't find that your machines are connected by the time you read this, could you send an email to support with your IP address? (Go to whatismyip.com to find it.) We have a few scattered reports of people having trouble connecting after upgrading. I think it is an issue with our firewalls throttling reconnects from a given host after so many repeated failures (with the bad SSL cert.) It should reconnect on its own given a bit of time.