January 2, 2013

Git, clients, partners, and woes.

by with 5 comments

This post comes from the “hindsight is twenty-twenty department.” A
few years ago when we started our
White Label program, we were
wondering how to manage the different client branding, GUI
customization, etc. Our first thought was “I know! We’ll use

Then we had two problems.

We used to use one branch in git per white label partner. The
intention was that we would then effortlessly merge across updates and
ship updates and have everything happy. Reality, however, quickly set
in. Every partner branch needed to be kept track of and manually
merged with HEAD. Being a distributed company, keeping git branch
discipline is bad enough when there’s one production branch, and much
worse when someone may commit last-minute bug fixes under the gun on a
completely different production branch. Once that happens, you need to
round up the branches and start merging back, up, down, and heaven
help you if you wind up with a mutually exclusive merge
conflict. Things wound up in a state where our white label clients
would lag months and months behind our production SpiderOak client at

Our first attempt consolidated our generic white labels down to two
branches based on core GUI features included. This did a great job of
reducing complexity, but it still left many very interestingly
customized clients still in their own branches, and it left us needing
to make sure every branch was still lovingly merged and that fixes
accidentally committed to the wrong branch got brought everywhere
else. Something else was needed.

Our first step was to overhaul the builder, which we completed
recently. This gives us a flexible resource framework to drop in
everything from images to configuration files. The next step is to
boil down all our custom white label code into client code and builder
configuration files, which will again bring us back to one
production branch for everything we ship.

What does this mean for you, fair customer? The primary win is that
now, especially as we maintain
brands just under SpiderOak alone
these days, we will be able to work on features much more quickly and
deploy them to everyone. Bugs found by partners don’t get fixed only
for partners that report them, but for everyone. And finally, we get
only one, single place that we have to aim CI and testing tools. This
results in a far better SpiderOak experience.

And it sure results in a huge reduction in the amount of grey hairs
we accumulate with every release cycle. Our takeaway here at SpiderOak
is to really examine every new process we try to introduce for trying
to imagine even just a single year down the road. On the surface,
using git to manage different production releases of SpiderOak seemed
to be a splendid idea. After a couple of years? Worst. Idea. Ever.

  1. Funny, only a few days back I made the exact same transition with my app! :D

    Git branches are really only for features, releases and forks. NOT for configuration management!

    My life got so much easier as well :)

  2. it sounds like using git was not the worst idea ever, using git how you chose to use it was the problem.

    parameterized builds is a good solution, however you could have also gone with a submodule pattern, having one core product (ie, your new/current single prod branch) and having each of your partner(s) specific alterations maintained as their own repo, which references the core product as a sub-module. this would allow each of your partner(s) to choose when they want to accept new functionality from the core product and it allows you to keep partner specific(s) completely separate.

    you still need a parameterized build to allow you to perform a core build + partner specific builds, but you don't have to maintain all of your partner builds at once if you don't choose to.

    I love your product, thanks for all the hard work.

  3. Very interesting indeed. Did you share your experience with the Git users' community? Have there been other suggestions than Clint's above? I was unclear from you post whether you abandoned the use of Git or whether you modified how you use it. I have been interested in code reuse and thus versioning for number of decades and never found a truly good solution. I'd love to learn more details about your experience.

    Again referring to Clint's comments, I also very much like the idea of SpiderOak. However, its behavior is very puzzling and confusing to me. Particularly bizzare are contradictory messages and the great difficulty managing the files in my SpiderOak account as one set of files rather than the current presentation of many sets of different files located on different devices. Perhaps, like with Git and Clint's suggestions about it, you could suggest to me whether I am using SpiderOak in less than optimal manner. Perhaps the difficulties I experience are a result of the configuration woes you experienced?

    Thanks for all your work! Keep it going!

  4. I realize this is NOT a Git forum, but I do not know how else to reach Clint. Perhaps the moderator could forward the following comment to him with my contact information and then delete this post?
    Regarding Clint's closing suggestions, I wonder how would one manage the situation if clients A, B and C want feature X and Y, clients D and E want feature X and Z, clients F and G want feature Y and Z and so on? I just can't find a really good solution to kill the spaghetti monster!
    Thanks again!