September 23, 2009
Regarding Recent Connection Issues (Updated Oct 13)
Final Update: To begin – I need to thank you all again for your patience during the last two weeks. We have worked tirelessly to bring all of your accounts active as quickly and as safely as possible.
At this time, all affected user accounts should be live and the upload process should be fully engaged. As mentioned previously, all of the data you have backed up will be present in the View tab although certain files may still not be available for download as we work to backfill the remaining data.
I would also like to apologize to the users who contacted us and did not get a timely response. Unfortunately, due to unforeseen circumstances with one of our employees, we had to adjust and we do appreciate greatly your understanding. Customer service has always been a top priority for us and we will do whatever we can to live up to the highest standards moving forward.
Lastly – for those interested – we will be posting a full disclosure of what occurred and the process taken for reactivation of accounts.
If you are still experiencing any connectivity related problems, please do email us at firstname.lastname@example.org and we will address your issue immediately.
To the users affected by this recent connection issue: we apologize greatly for any inconvenience this has caused and thank you for your continued patience and understanding during this process.
By way of explanation, one of SpiderOak’s storage clusters (which contains your data) had an unexpected fault during a routine maintenance procedure earlier this week. We have corrected the fault and are working to restore normal operations.
Because this fault happened in a very unusual way and operating in an abundance of caution, we have completed a copy of all the data from this cluster to a new cluster. We believe it is most prudent for us to complete a full file system integrity check and then a full cluster storage audit before resuming operations. For very large file systems like the ones we use, these operations can take days to complete from start to finish.
It is important to note that this situation does not pose a risk of data loss. We have other measures of redundancy, including additional backups (of our backups). To reiterate, a very small portion of SpiderOak customers are affected and – quite unfortunately – if you have experienced connection issues over the last few days then you are among them.
Obviously we will stay in contact with you throughout this process and let you know when things are back online. In the meantime, if you have urgent backup requirements that need to proceed, we would like to offer you a second SpiderOak account with equal space at no charge and which can be brought online immediately. Please do let us know and we will start working with you to make this happen.
Thank you again for your patience and understanding during this outage; we are working day and night to bring the data back online in as safe and protected a way as possible. Please don’t hesitate to send further questions or thoughts at email@example.com.
Our best wishes and deepest apologizes for the inconvenience,
The SpiderOak Team
Update: Again – to the users affected by this outage – your patience and understanding is greatly appreciated. We do apologize for not providing this update sooner but we have been working around the clock to bring the data back online as fast and as safe as possible. You will notice that your clients are now able to connect to the SpiderOak backend (a green light will appear in the Control Center). Unfortunately they are still unable to upload data but have been successfully migrated to a new cluster – the first and necessary step.
Over the next several hours and days, we will be slowly bringing accounts back online and to a fully operational status. The process of restoring from archival backups is slow and its speed not always predictable but we hope by the end of the weekend to have many of our users affected up and running.
In the meantime – if you do have additional questions – please don’t hesitate to contact us at firstname.lastname@example.org and we will respond as soon as possible.
Update 2: We wanted to provide another update to inform you all of our progress. At this stage we have now started bringing users back online and to a fully functional status. This process will continue over the next several days until all of our users affected by this issue have been fully restored to operation.
If your account is still not connecting properly or you have more specific questions, please do get in touch with us at
email@example.com and we will work to resolve any remaining or outstanding issues.
Thank you again for your continued patience and understanding throughout this process. We apologize greatly that it has taken longer than anticipated but the safety of restoring your data properly was and is always the priority.
Update 3: I am writing to provide another update on the recent connectivity problems we experienced.
To begin – I would like to thank you all for your patience during this outage. We have been and will continue to work diligently until the entire situation is resolved.
As it relates to the current status of bringing accounts online, we are continuing to make progress everyday but – unfortunately – the process has taken much longer than we had anticipated. To speed things up, we have devised a new method of activating user accounts that is faster. This will allow users to begin uploading their data to SpiderOak – ensuring changed and/or new data is being securely backed up. And whereas users will be able to see all of the folders and files already stored in their SpiderOak Network on the View tab, the contents will not be available for download at the moment as we will backfill the data in the days to follow.
Our sincere hope is to have everyone’s account online in the next couple of days. If you are still experiencing problems with your
account, please do send us an email to firstname.lastname@example.org and we will be sure to provide you with additional information as to when your account is live. And just as a reminder, we have not lost or misplaced any of your data – it is all accounted for and will be available for you to access.
On a related note, I would also like to apologize for the delayed responses to many of your customer service inquires. Sadly, one of our team members suffered a terrible health problem and we are now getting caught up on your requests. Thank you in advance for your understanding.