August 14, 2014

On status reports, transparency and overall safety

by with 33 comments

While running a company from the U.S. has its benefits, when the goal is to provide improvements on privacy, the “where” matters almost as much as the “how”.

One of the things we have been debating internally for some time is the implementation of a warrant canary. After much anticipation, the day is finally here!

What’s a canary?

When it comes to privacy, trust is a complex value to earn. We’ve been asked a lot of questions lately such as, “How can we know you haven’t inserted a backdoor and are now legally restricted as to what you can say?”.

If your conspiracy theories go far enough, then the answer to the above question is you can’t.

But, we are a widely distributed team, holding us all under some (possibly questionable) law is going to be hard, and the goal of what we are doing here is exactly that: making sure access to your data is as complicated as possible. It’s never going to be impossible though.

So a warrant canary is one more obstacle somebody or some entity with enough power would have to circumvent, and it works just like Homer Simpson’s Everything’s OK Alarm.

Everything's OK Alarm

Everything’s OK Alarm

The canary will be around as long as everything is going smoothly, otherwise it’s not going to be updated in the expected timeframe.

More details please?

The canary itself can take many forms, the one we’ve chosen is a specific plain text signed with multiple GPG keys.

The GPG keys belong to different SpiderOakers which we’ve selected based on geolocation. So besides doing all the legal (or illegal) things an adversary would need to do to get a backdoor somewhere in SpiderOak, they’ll also need to compel 3 people around the globe to sign a message at a specific moment in time.

How often will the OK alarm beep?

The timeframe between canaries varies depending on the aspects considered. We initially thought once a month was a reasonable timeframe, but then discussing it a bit longer and the reasons behind what other people have done, we realized it was too short.

The EFF has compiled a really helpful FAQ on this subject, and in there they raise an interesting point: even though you might want to publish a canary often, if you do get something like a NSL, you’ll have to go to court or do other legal stuff to find out more about it. If it turns out it’s just a bogus claim or something you can fight against, that’s great! But it might take a few months for you to find that out, and in the meantime you killed the canary.

In cases such as SpiderOak, killing a canary can quite possibly mean killing the business, so we switched to publishing the canary every 6 months. This means the first canary will be signed between August 10 and August 15, 2014. The next one will be signed between February 10 and February 15, 2015.

It indeed sounds like a lot of time, but it’s the best compromise we’ve found.

What if one of the people in the signing group goes rogue?

The canary signing group of people is bigger than 3, so if somebody is unavailable for any reason he or she can be replaced by a backup.

If somebody in the group goes rogue, or simply leaves SpiderOak, a canary-like message will be published, signed by the trusted parties explaining who is no longer in the group and who’s replacing that person.

Where will we find this canary thing?

It’ll be in the form of a blog post, with the same Category as this one: Status Reports. So people subscribed to the blog can be notified immediately of the canary update. And it will also be available through the following static URL:

https://spideroak.com/canary

Which will be what’s linked from SpiderOak’s homepage.

How do I verify that the canary is correct?

The way you can verify this:

  • First, create 4 files: one with the canary text (canary.txt), and one for each signature (sig1.asc, sig2.asc and sig3.asc).
  • Download gpg
  • Run the following commands:
 gpg --verify sig1.asc canary.txt
 gpg --verify sig2.asc canary.txt
 gpg --verify sig3.asc canary.txt

They should all output a message containing “Good signature from <person from canary group here>“.

The keys on the default canary group are:

Key ID: 0x1712D3E36F2F0403
Primary key fingerprint: 0DAB 1518 36A3 CBBA 0362  FC17 1712 D3E3 6F2F 0403

Key ID: 0xE435F15CA6B145F8
Primary key fingerprint: B411 3438 B56B D51C D208  E17E E435 F15C A6B1 45F8

Key ID: 0x132B9F2251131E5C
Primary key fingerprint: DC1E FB15 4444 E4B5 2726  8430 132B 9F22 5113 1E5C

I understand it’s not the easiest thing to do, but it’s something you can do twice a year if you choose.

Nothing is set in stone though, if we find an easier alternative to check while maintaining the same degree of security, we’ll migrate to that method.

Any change on how the canary works will be posted and signed in the same way as the regular canary.

Comments
  1. This ties in to my question on 22nd July. Thanks for taking the time to really consider this process and communicate it openly.

  2. Did you forget to tell your users to get your GPG keys from the server/ Doing exactly what you have posted above and nothing more yields:

    Thu Aug 14 19:49:37 Cipher: $ gpg –verify sig1.asc canary.txt
    gpg: Signature made Mon 11 Aug 2014 11:57:31 AM EDT using RSA key ID 6F2F0403
    gpg: Can’t check signature: public key not found

    • Well when you do import fetch the keys you find out there are still problems
      Every one of them fails:

      jsa@poulsbo:~> gpg –verify sig1.asc canary.txt
      gpg: enabled debug flags: memstat trust extprog assuan
      Version: GnuPG v1.4.13 (OpenBSD)
      gpg: armor header:
      gpg: Signature made Mon 11 Aug 2014 08:57:31 AM PDT using RSA key ID 6F2F0403
      gpg: using PGP trust model
      gpg: BAD signature from “Chip Black (SpiderOak Canary) ”
      gpg: binary signature, digest algorithm SHA512
      random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
      outmix=0 getlvl1=0/0 getlvl2=0/0
      secmem usage: 0/32768 bytes in 0 blocks
      jsa@poulsbo:~> gpg –verify sig2.asc canary.txt
      gpg: enabled debug flags: memstat trust extprog assuan
      Version: GnuPG v1
      gpg: armor header:
      gpg: Signature made Tue 12 Aug 2014 03:25:06 PM PDT using RSA key ID A6B145F8
      gpg: using PGP trust model
      gpg: BAD signature from “Frank Sievertsen ”
      gpg: binary signature, digest algorithm SHA512
      random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
      outmix=0 getlvl1=0/0 getlvl2=0/0
      secmem usage: 0/32768 bytes in 0 blocks
      jsa@poulsbo:~> gpg –verify sig3.asc canary.txt
      gpg: enabled debug flags: memstat trust extprog assuan
      Version: GnuPG v1.4.14 (Darwin)
      gpg: armor header:
      gpg: Signature made Wed 13 Aug 2014 06:20:18 AM PDT using RSA key ID 6A269F35
      gpg: using subkey 6A269F35 instead of primary key 51131E5C
      gpg: using PGP trust model
      gpg: BAD signature from “Tomas Touceda (SpiderOak canary) ”
      gpg: binary signature, digest algorithm SHA512
      random usage: poolsize=600 mixed=0 polls=0/0 added=0/0
      outmix=0 getlvl1=0/0 getlvl2=0/0
      secmem usage: 0/32768 bytes in 0 blocks

      • Make sure you’re reading the canary with utf8 encoding, otherwise the german line might be displayed incorrectly. In the case of Chrome, it’s in the View->Encoding->Unicode (UTF-8) menu.

        And copy just the text from en “en” at the very beginning to the “4” in “august 10, 2014″ at the end of the text.

        • Tomas Touceda:

          It still fails. you need to make the GPG work. then concentrate on making work easy. So far your keys are an epic fail…

          • Seems to work reasonably enough for me…

            andrew@hat ~/A/spideroak> gpg –keyserver keyserver.ubuntu.com –recv-key 0x1712D3E36F2F0403
            gpg: requesting key 6F2F0403 from hkp server keyserver.ubuntu.com
            gpg: key 6F2F0403: public key “Chip Black (SpiderOak Canary) ” imported
            gpg: no ultimately trusted keys found
            gpg: Total number processed: 1
            gpg: imported: 1 (RSA: 1)
            andrew@hat ~/A/spideroak> gpg –keyserver keyserver.ubuntu.com –recv-key 0xE435F15CA6B145F8
            gpg: requesting key A6B145F8 from hkp server keyserver.ubuntu.com
            gpg: key A6B145F8: public key “Frank Sievertsen ” imported
            gpg: no ultimately trusted keys found
            gpg: Total number processed: 1
            gpg: imported: 1 (RSA: 1)
            andrew@hat ~/A/spideroak> gpg –keyserver keyserver.ubuntu.com –recv-key 0x132B9F2251131E5C
            gpg: requesting key 51131E5C from hkp server keyserver.ubuntu.com
            gpg: key 51131E5C: public key “Tomas Touceda (SpiderOak canary) ” imported
            gpg: no ultimately trusted keys found
            gpg: Total number processed: 1
            gpg: imported: 1 (RSA: 1)

            andrew@hat ~/A/spideroak> gpg –verify sig1.asc canary.txt
            gpg: invalid armor header: iQGcBAABCgAGBQJT6OfrAAoJEBcS0+NvLwQDwFkMAI72ALXk0szqiEaRZbDfHCit\n
            gpg: Signature made Mon 11 Aug 2014 16:57:31 BST using RSA key ID 6F2F0403
            gpg: Good signature from “Chip Black (SpiderOak Canary) ”
            gpg: WARNING: This key is not certified with a trusted signature!
            gpg: There is no indication that the signature belongs to the owner.
            Primary key fingerprint: 0DAB 1518 36A3 CBBA 0362 FC17 1712 D3E3 6F2F 0403
            andrew@hat ~/A/spideroak> gpg –verify sig2.asc canary.txt
            gpg: invalid armor header: iQIcBAABCgAGBQJT6pRCAAoJEOQ18VymsUX4Mg4QAI6TQU8eTAm8TTk+8n8QdT50\n
            gpg: Signature made Tue 12 Aug 2014 23:25:06 BST using RSA key ID A6B145F8
            gpg: Good signature from “Frank Sievertsen ”
            gpg: WARNING: This key is not certified with a trusted signature!
            gpg: There is no indication that the signature belongs to the owner.
            Primary key fingerprint: B411 3438 B56B D51C D208 E17E E435 F15C A6B1 45F8
            andrew@hat ~/A/spideroak> gpg –verify sig3.asc canary.txt
            gpg: invalid armor header: iQGcBAABCgAGBQJT62YSAAoJEGQmtGZqJp81tpML/3tdQpB/1VMYFM6C4QKl/QSL\n
            gpg: Signature made Wed 13 Aug 2014 14:20:18 BST using RSA key ID 6A269F35
            gpg: Good signature from “Tomas Touceda (SpiderOak canary) ”
            gpg: WARNING: This key is not certified with a trusted signature!
            gpg: There is no indication that the signature belongs to the owner.
            Primary key fingerprint: DC1E FB15 4444 E4B5 2726 8430 132B 9F22 5113 1E5C
            Subkey fingerprint: D451 EDE1 788E C37C 0978 B92B 6426 B466 6A26 9F35

            My canary.txt file is saved as UTF-8, with Unix line endings (i.e. just LF aka “\n”, not the Windows-style CRLF or “\r\n”) *WITH* a trailing newline. Hexdump of it for people to compare to: http://pastebin.com/aFtHhh8c

        • Perhaps with English not being Tomas’s first language the page should be re-written to explain the the canary.txt file must contain ONLY the text that precedes the first signature. You must also fetch the keys FIRST (one time) and add them to your gpg keyring.

          This little bash script will to the validation once you have used the keyservers to fetch the keys

          –begin bash script next line–
          #!/bin/bash
          # script to download spideroak canry, split out three sigs then veerify the whole file
          wget https://spideroak.com/canary
          awk ‘/—–BEGIN PGP SIGNATURE—–/ { exit };{TN=”canary.txt”;p=1}p{print >TN}’ canary
          awk ‘/—–BEGIN PGP SIGNATURE—–/{if(FN)close(FN);FN=”sig”++i”.asc”;p=1}p{print >FN}/script/{p=0}’ canary
          gpg –verify sig1.asc canary.txt 2>&1 >/dev/null |grep -i “good”
          gpg –verify sig2.asc canary.txt 2>&1 >/dev/null |grep -i “good”
          gpg –verify sig3.asc canary.txt 2>&1 >/dev/null |grep -i “good”

    • 1) My GPG expects GPG signed messages to start with —–BEGIN PGP SIGNED MESSAGE—–
      2) Signatures successfully downloaded from MIT server, but they don’t work on signature verify

      gpg –verify canary.txt
      gpg: no signed data
      gpg: can’t hash datafile: No data

      Non-ASCII in the German text is going to complicate the canary.
      Recommend provide a link to a .TXT file, as a download link to solve TXT formatting questions

      If paste from website with headers added, gpg consumes, but signatures do not work
      gpg: WARNING: signature digest conflict in message
      gpg: Can’t check signature: General error
      gpg: Signature made 08/12/14 18:25:06 Eastern Daylight Time using RSA key ID A6B

      • You need to create one text file for the canary and another for each signature.

        The text is encoded in UTF8, which should not be a problem in any modern system.

            • This really makes the point recently in the news about how unfriendly pgp/gpg is. We’re technical and we’re still tripping up over delimiters and encodings.

              SpiderOak peeps, develop a really easy, powerful well thought out gpg interface into your client, or a good cross platform mail client with gpg baked in, linking into back-end services architected with your approach to security behind it, and you will solve a major problem and become rich.

  3. I still don’t get the logic for the 6-month period.

    Whatever the period, it is possible the ‘due date’ will fall in the middle of you being in court or “do[ing] other legal stuff”. So, if you are in the process of investigating and/or challenging a warrant or NSL and the due date falls, do you update the canary or do you ‘kill’ it?

    If you kill it then there is no difference between a short and long update cycle, so the justification breaks down.

    If you choose to update it, then the shorter, 1-month, period is actually superior because you can more quickly change the status from ‘Everything’s OK’ to, well, everything’s _not_ OK as the process develops.

    As a hypothetical, let’s say that the ‘other legal stuff’ takes 3 months and the due-date for the canary update falls after the second month. If you update it (because things are still ‘OK’) then what happens when, one month later, you find that things are most certainly _not_ OK? In the 6-month plan, that’s a 5-month gap where your customers believe everything is going along swimmingly.

    So the solution is a shorter period and you simply decide to publish the canary during any legal action/investigation until such time as it is clear that the canary needs to die, at which point you only have to wait MAXIMUM 1 month.

    And, by that logic, the shorter the period, the better, though the effort required should be factored in so down to a week or fortnight seems sensible.

    • I wonder if one possible solution would be to have a third ‘canary’ state for when there’s a possible problem they’re investigating? Though at that point, it might be smart to indicate just how concerned they are so everyone doesn’t panic at the first sign that the canary might not be feeling well.

    • I understand your concerns. Basically we are assuming 6 months is a reasonable period to find out whether we will be forced to do something we don’t want to and that may harm our users. If we suddenly realize we are in a complex situation, we can post a signed update saying that we will be updating the canary every week or month.

      On the other hand, a shorter period is also more prone to error, if we fail to update not because of a real problem, but because of an every regular life problem, then everything goes down for no reason.

      Bare in mind that this is not supposed to be THE solution to everything, it’s just one more barrier.

      • You can also revoke your keys if you suddenly find you need a shorter period but are forbidden to take down the page.

        Especially foreign signers could do this quickly.
        Its important for each user to verify the keys each time.

  4. Um… is there a published schedule as to when we should expect the next canary update? August 10 + six months is Feb 10 unless by “six months” you mean 182.525 days, which is, uh, sometime around Feb 10. So if there’s no update on Feb 11 we assume the canary’s dead? Feb 12? March 1?

    Either I’m missing something, or this should be clarified.

    • That is a good point. Given that we have to coordinate between several team members, we give it a time frame. So for instance, this one “started” on August 10 but was published on the 14th. So yes, you should expect the same timeframe but starting on February 10.

      I’ll update the post.

  5. Another example of why I recommend SpiderOak to people whenever I get the chance!

    In your instructions, you might modify the “get GPG” to “get GPG unless your operating system came with it.” (Just about every Linux distro does, I don’t know about OS X or Windows.)

    • This might be picking over a fine point, but linux distros do not “come” with anything – only what you select to be installed.

  6. That’s great, BUT… if you do not have the keys to our data, why so much hype – what can the government get other than a list of names of people who use your service? – which they already have based on traffic flow…

    • As I understand it, in theory this protects against the the government asking SpiderOak to update a targeted user with a backdoored client.

      • You mean something like what (probably) happened to TrueCrypt – they’d rather not release an “update” if that meant that the update would be compromised…

  7. “The canary will be around as long as everything is going smoothly, otherwise it’s not going to be updated in the expected timeframe.”

    Ok, if the US law / NSA / whatever can stipulate that a company have to implement a backdoor silently without telling anyone, what stops them from also forcing a company to publish the warrant canary normally, making it look like nothing bad has happened? I don’t understand why this canary thing is supposedly so safe.

    • So the improvement is basically that not only they will need to force SpiderOak to add a backdoor but they must also make people that might be out of their jurisdiction sign the canary.

      So, yes, it’s still possible, but it’s more difficult. The canary is not a one stop shop, it’s one more barrier they will have to circumvent.

      • There is also damn good legal theory that, while it may be tenuously constitutional to gag someone, there is no way in hell that it is constitutional to force someone to lie. And a forced canary update after an NSL would be a blatant lie.

  8. This works for me:

    gpg --recv-keys 0x1712D3E36F2F0403 0xE435F15CA6B145F8 0x132B9F2251131E5C --keyserver keyserver.ubuntu.com
    wget https://spideroak.com/canary
    sed -e "/BEGIN PGP SIGNATURE/Q" < canary > canary.txt
    gpg --verify canary canary.txt

  9. There is no denying that gpg is extremely hostile to any human. Given that, I finally got it to work, leveraging all the comments. I had to guess what keyserver to import the public keys from. I guessed pgp.mit.edu. It seemed to work.

    I still get “WARNING: This key is not certified with a trusted signature! There is no indication that the signature belongs to the owner.” Come on, gpg guys. Stop yanking my chain. Make this just work. I’m not an idiot; I got a BSEE and have over 45 years of programming, but I didn’t major in cryptography for gosh sake.

  10. Not to nitpick; this is definitely a great move, but I have some misgivings about the jurisdictionalgeography. Spain and Germany may not be blatantly in the USA’s pocket, but they definitely are subject to pressure from the USA “legal” (so-called) process and even less savory rub rosa processes. It would be nice to have one signer be in Russia or Cuba or Uruguay. I know that might be hard to implement, and might introduce its own problems.

  11. Can anyone spot the obvious deficiencies with this system?
    Edward wouldn’t because he’s a nerd, so what are the obvious ones!

    LOL

    by bye love