September 23, 2009

Regarding Recent Connection Issues (Updated Oct 13)

by with 49 comments

Final Update: To begin – I need to thank you all again for your patience during the last two weeks. We have worked tirelessly to bring all of your accounts active as quickly and as safely as possible.

At this time, all affected user accounts should be live and the upload process should be fully engaged. As mentioned previously, all of the data you have backed up will be present in the View tab although certain files may still not be available for download as we work to backfill the remaining data.

I would also like to apologize to the users who contacted us and did not get a timely response. Unfortunately, due to unforeseen circumstances with one of our employees, we had to adjust and we do appreciate greatly your understanding. Customer service has always been a top priority for us and we will do whatever we can to live up to the highest standards moving forward.

Lastly – for those interested – we will be posting a full disclosure of what occurred and the process taken for reactivation of accounts.

If you are still experiencing any connectivity related problems, please do email us at support@spideroak.com and we will address your issue immediately.


To the users affected by this recent connection issue: we apologize greatly for any inconvenience this has caused and thank you for your continued patience and understanding during this process.

By way of explanation, one of SpiderOak’s storage clusters (which contains your data) had an unexpected fault during a routine maintenance procedure earlier this week. We have corrected the fault and are working to restore normal operations.

Because this fault happened in a very unusual way and operating in an abundance of caution, we have completed a copy of all the data from this cluster to a new cluster. We believe it is most prudent for us to complete a full file system integrity check and then a full cluster storage audit before resuming operations. For very large file systems like the ones we use, these operations can take days to complete from start to finish.

It is important to note that this situation does not pose a risk of data loss. We have other measures of redundancy, including additional backups (of our backups). To reiterate, a very small portion of SpiderOak customers are affected and – quite unfortunately – if you have experienced connection issues over the last few days then you are among them.

Obviously we will stay in contact with you throughout this process and let you know when things are back online. In the meantime, if you have urgent backup requirements that need to proceed, we would like to offer you a second SpiderOak account with equal space at no charge and which can be brought online immediately. Please do let us know and we will start working with you to make this happen.

Thank you again for your patience and understanding during this outage; we are working day and night to bring the data back online in as safe and protected a way as possible. Please don’t hesitate to send further questions or thoughts at support@spideroak.com.

Our best wishes and deepest apologizes for the inconvenience,

The SpiderOak Team

Update: Again – to the users affected by this outage – your patience and understanding is greatly appreciated. We do apologize for not providing this update sooner but we have been working around the clock to bring the data back online as fast and as safe as possible. You will notice that your clients are now able to connect to the SpiderOak backend (a green light will appear in the Control Center). Unfortunately they are still unable to upload data but have been successfully migrated to a new cluster – the first and necessary step.

Over the next several hours and days, we will be slowly bringing accounts back online and to a fully operational status. The process of restoring from archival backups is slow and its speed not always predictable but we hope by the end of the weekend to have many of our users affected up and running.

In the meantime – if you do have additional questions – please don’t hesitate to contact us at support@spideroak.com and we will respond as soon as possible.

Update 2: We wanted to provide another update to inform you all of our progress. At this stage we have now started bringing users back online and to a fully functional status. This process will continue over the next several days until all of our users affected by this issue have been fully restored to operation.

If your account is still not connecting properly or you have more specific questions, please do get in touch with us at
support@spideroak.com and we will work to resolve any remaining or outstanding issues.

Thank you again for your continued patience and understanding throughout this process. We apologize greatly that it has taken longer than anticipated but the safety of restoring your data properly was and is always the priority.

Update 3: I am writing to provide another update on the recent connectivity problems we experienced.

To begin – I would like to thank you all for your patience during this outage. We have been and will continue to work diligently until the entire situation is resolved.

As it relates to the current status of bringing accounts online, we are continuing to make progress everyday but – unfortunately – the process has taken much longer than we had anticipated. To speed things up, we have devised a new method of activating user accounts that is faster. This will allow users to begin uploading their data to SpiderOak – ensuring changed and/or new data is being securely backed up. And whereas users will be able to see all of the folders and files already stored in their SpiderOak Network on the View tab, the contents will not be available for download at the moment as we will backfill the data in the days to follow.

Our sincere hope is to have everyone’s account online in the next couple of days. If you are still experiencing problems with your
account, please do send us an email to support@spideroak.com and we will be sure to provide you with additional information as to when your account is live. And just as a reminder, we have not lost or misplaced any of your data – it is all accounted for and will be available for you to access.

On a related note, I would also like to apologize for the delayed responses to many of your customer service inquires. Sadly, one of our team members suffered a terrible health problem and we are now getting caught up on your requests. Thank you in advance for your understanding.

Comments
  1. So why are the forums unavailable too? When I try access the forums using the tray icon it tells me 'SpiderOak cannot reach the server to retrieve a secure login URL for your account.' Is there a technical reason for this, or do you just not want people discussing this outage in the forums?

  2. Neville – It's because providing secure auto-login URLs for the forum is handled over the connection to the storage server. We did post the above to the forum also for anyone who's cookies keep them logged in.

  3. An estimate, even if it is simply hours/days/weeks would be helpful for planning purposes. Obviously downtime measured in days isn't great for a backup service, but I know you guys are working on it.

  4. An estimate, even if it is simply hours/days/weeks would be helpful for planning purposes. Obviously downtime measured in days isn't great for a backup service, but I know you guys are working on it.

  5. An estimate, even if it is simply hours/days/weeks would be helpful for planning purposes. Obviously downtime measured in days isn't great for a backup service, but I know you guys are working on it.

  6. An estimate, even if it is simply hours/days/weeks would be helpful for planning purposes. Obviously downtime measured in days isn't great for a backup service, but I know you guys are working on it.

  7. OK, now it's been more than 24 hours since our last status report and over 3 days since this problem began. I lost my hard drive earlier this week and am now waiting to restore my personal files from SpiderOak. However, I've started to re-create some of the files that I had stored there by using old copies, fragments, and other sources because, frankly, I'm getting a bad feeling about ever seeing my backup files again. You owe it to your paying customers to give us an honest assessment of the situation and your best estimate of when it will be repaired (if ever). So, what's the status of this outage? –George

  8. Still waiting here too. I check every 4 hours or so. I think we need a current factual update from SpiderOak staff since they've been silent since sometime Sunday. I suggest at least once a day updates from the people who know what's really happening. I also suggest they admit not knowing how long this is going to take if that is in fact the reality.

  9. @Richard, I agree with you entirely. I think there should really be a status page, dedicated to things like this.

    The status of an online backup network is probably one of the most important things people might want to check, so having a page specifically for it would be nice. … Know any other good syncing backup solutions, to sync between machines, by the way? Going to be in the market for one soon…

  10. I don't know if you were being honest about a request for a sync tool or if that was just a facetious comment about the length of this down time. However, if you work with Windows, I would recommend Microsoft LiveSync. I use it to sync between my home and office computers. It does a nice job and is free of charge. It permits you to select the folders you want to sync and will then keep them constantly updated. I work with both Windows and Linux and understand the concept of Open Source – please don't flame me for suggesting an MS solution. However, I've used LiveSync for a couple of years and really like it (I forget what it was called a couple of years ago, but I've been using it that long).

    By the way, my account is still down for more than a week. Losing hope of ever seeing my files again. Is SpiderOak still signing up new customers? A couple of days ago they did post a blog entry about the new features they are planning – how about a feature that permits customers to recover files?

  11. Update: I do apologize again about the delay in sending an update. We have been bringing users online steadily over the last few days and hope to have this process complete within the next two days meaning all of your data will be available.

    We will do a better job of informing you of our progress now – providing updates here every 6 hours until everything is back up and working.

    Again, I do sincerely apologize for the delay as it has taken us longer than anticipated but your data is safe and will be returned to active status within 48 hours at the longest.

  12. Update (@ George – Richard – John – Paul – Luke): If you would like, we can locate your account details and contact you directly when your account becomes active. If you are interested, please send an email to support@spideroak.com with the subject line 'Information' and we will contact you with updates such that you will not have to refer back to the blog. Furthermore, we will of course be providing several free months of service to our paid users and free GBs to our free users for the considerable inconvenience.

    Thank you again for your patience and understanding throughout this process.

  13. Update: We are continuing to bring more users to an active status with full access to their account. Again – if you are interested – please be sure to send an email to support@spideroak.com and we will alert you the moment your account becomes live.

    Thank you again for your ongoing patience and understanding. I will provide another update in the next several hours.

  14. Update: Without sounding too repetitious, we are continuing to bring users online. Again – if you are interested – please be sure to send an email to support@spideroak.com and we will alert you the moment your account becomes live.

    Thank you again for your ongoing patience and understanding. I will provide another update in the morning when surely many users will become active. Have a good evening.

  15. Nothing yet, still in progress. Can you please migrate my account? I've turned our NAS server off, as it had started to make a scary death rattle. A new account doesn't help, as it took several weeks to back up the 300GB or so. I just hope it starts up again and runs long enough for an incremental backup before it gives up.

  16. @ John: Please send an email to support@spideroak.com referencing this conversation and I will be able to get back to you a with more accurate status update. Thank you again for patience and we will do everything we can to get your account active immediately given your predicament.

  17. Update: To begin as I have in the past, thank you again for your continued patience and understanding during this process. We are still bringing more and more users online and will work diligently through the next 24 hours to make sure everyone has returned to fully active status.

    As a reminder and if you would like, we would be more than happy to email you directly when your account is live. Please send an email to support@spideroak.com with the subject line 'Information' and we will keep you closely informed.

  18. Update: As a point of clarification and to provide some updated information, during this outage we have not lost or misplaced ANY of our user's data. We are currently working on bringing some of our users with larger accounts online and it is taking a bit longer than we had projected. That said, the process is continuing and we hope to have many of these accounts up and running in the next several hours.

    Again, if you would like more specific information, please email us at support@spideroak.com and write 'Information' in the subject line and we will get back to you with additional progress as well as when you account becomes fully active.

    Thank you again for your continued patience and understanding and we apologize greatly for the delay some of the larger accounts continue to experience. We are working diligently to bring your accounts online as soon and as safely as possible.

  19. Still waiting as well, approaching two weeks down. Even with a usb cable instead of fiber it seems like an awfully long time. Can you provide some addition color on why it's taking so long and why it won't happen again? I love your service but I'd like a better understanding of what the deal is with the very extended outage. For a backup I was thinking annual downtime measured in hours or even minutes, not weeks… Some kind of detailed post-mortem once everyone is back up would be greatly appreciated and a good step towards restoring my confidence. Thanks!

  20. I echo the concern of John. Every time I read this blog or speak with support they say it will be back in 24 hours. Be honest with people. If it is going to take a week then it takes a week.

  21. Update:

    @ John: Thank you again for your patience and continued understanding. Your account is now fully active – I sent you an email to this effect but wanted to post on the blog in case you happened here first.

    Per the accounts not yet activated: We are still bringing accounts online and it is mainly the larger ones that are taking experiencing the long delays at this time. Again, please do email us at support@spideroak.com with the subject line 'Information' and we will keep you personally informed as to your progress.

    I do apologize again for the ongoing inconvenience as we are working as quickly as we can to get all all of our users active as soon as possible. Your patience continues to be amazingly appreciated.

  22. Yup, I'm up. Many thanks.

    FYI I had to restart the spideroak application before I could connect to the storage servers. It had been green, but then showed up brown and disconnected. Upon restart everything was normal.

  23. No service for over a two weeks, no response from support to two emails already. I'm very disappointed. I recommended spideroak to handful of people in my department at Univ of Rochester. And now I don't know what to tell them. You have to understand that being in the data protection business you can't afford to make such a mistakes. Your reputation is everything… If you were a bank, you would have been out of business already. Now I'm not really sure if I can ever trust you 100% with my data. If you didn't have sync option I would have switched to another company already!!!

    peace,

  24. I've been waiting patiently thus far, but at this point after not having access to the service I've paid for in two weeks, I have to echo others' requests for a greater deal of transparency and detailed explanation of the problem and delayed recovery. It is the standard now in the Web 2.0 world, and is a reasonable expectation for a failure of this magnitude.

    And to save you a few lines, yes, I've already emailed customer service and am awaiting notification regarding the restoration of my act.

  25. @ Mike: I do apologize that you have not received any additional information from us as we have sent you updates. We are thinking that the mail is either caught in a filter or did not make it to you for some other reason that we are examining. If you could, please send an email directly to ero@spideroak.com and I will make sure you get a response and an update immediately.

    In response to your comments below, we do understand your frustration and have been working tirelessly to return all accounts to active status. It is important to mention that all of your data is still safe despite the delay. We do greatly appreciate your continued patience and we will of course provide make good on this outage in the months to follow.

  26. @ Jim: Thank you as well for your continued patience during this process – it has been tremendously appreciated. I do further understand your frustration and point below and we do plan on relaying additional information on what occurred and the restore process. I can say that it was an extremely rare case and implementations had already been put in place to ensure against it happening in the future.

    As soon as everyone is live we will post additional information per your request (and as intended).

  27. Update: To begin, I do greatly apologize for the ongoing outage many of you are experiencing. And I am further sorry that I have been updating the Blog more frequently as I have been focused on sending out personal responses to those of you who have asked to receive them.

    As it relates to the current status, we are continuing to bring accounts online and will keep doing so daily. Unfortunately this process is taking a longer time than we expected; as such, we are now planning to bring accounts up sooner such that users can begin to upload their data. As part of this process, you will be able to view all of the data in your account by filename but some of the data within your account may still be unavailable as the data will be backfilled. This will enable you to upload the changes and/or additions to your dataset to ensure they are off-site. And of course the account wide de-duplication will still apply.

    I would like to close by saying that we still have all our user's data and there was NO data loss experienced during this outage. We would like to make sure you all understand this thoroughly.

  28. Yeah! I'm back up. Thank you for your diligence in this matter. I was pretty certain I'd never see my backup files again – but I was wrong. While I wish you hadn't been out for so long, I'm pretty happy knowing that you had, indeed, taken care of my files.

  29. You promised frequent updates, "We will do a better job of informing you of our progress now – providing updates here every 6 hours until everything is back up and working. " yet there has been no word from you in the past five days, as we approach the one month mark for this downtime.

  30. @ Jim: I do apologize – we have been working with individual requests that have come in over the last several days. That said, you are correct that we should have posted more often here as well.

    Please do let me know of you are still experiencing issues with you account as it should be free from connectivity issues at this time. So that we can look into your account specifically, please email me directly at ero@spideroak.com and do include your username.

  31. It is still not working. SpiderOak is doing nothing at all, although it shows perfect network health. From the logs:

    2009-10-13 23:18:18,531 INFO scheduler taking no action due to _globals["server_says_up2date"] being False
    2009-10-13 23:18:18,531 DEBUG spider status: set server_says_up2date to False
    2009-10-13 23:18:18,532 DEBUG spider status: signal: ss.server_says_up2date: (False,)
    2009-10-13 23:18:48,561 INFO scheduler taking no action due to _globals["server_says_up2date"] being False
    2009-10-13 23:18:48,562 DEBUG spider status: set server_says_up2date to False
    2009-10-13 23:18:48,562 DEBUG spider status: signal: ss.server_says_up2date:

  32. @ Nikolaus: Thank you for your message and I do apologize for the problems you are experiencing. As your account was actually not affected by this outage, we will look into the possible cause and get back to you as soon as possible via email.

  33. @ Nikolaus: Thank you for your message and I do apologize for the problems you are experiencing. As your account was actually not affected by this outage, we will look into the possible cause and get back to you as soon as possible via email.

  34. I'm still waiting for this "full disclosure of what occurred
    and the process taken for reactivation of accounts" that you promised.

  35. I think that if you have a blog, then it is what you need.
    Tell people about something, someone is advertising, to somebody else for something else.
    Personally, I think so.
    Here I have a blog about my travels, but I can not share them with people, because somebody would write that this is spam.
    A hoster will also kick :)