June 28, 2010

An Erlang/OTP SSL proxy and load balancer for Python Twisted’s Perspective Broker

by with 3 comments

(If the above sounds like gibberish to you, you’re probably not a programmer
and this post won’t be very interesting.)

SpiderOak clients maintain a SSL connection to a Python Twisted Perspective
Broker service to coordinate their actions with the server and with each
other.

To load balance client connections across several Perspective Broker
processes per storage cluster, and route connections from a single public IP to
many storage nodes, we built a proxy server in Erlang. We’ve been running this
in production for several months now.

The design is simple. Erlang/OTP answers the socket, and speaks the
perspective broker protocol just long enough to learn the authentication
credentials the user is attempting to login with. The Erlang server looks up
the user’s assigned storage cluster and node. From there, it simply proxies
the connection (including replaying the authentication sequence) to a Python
Perspective Broker server. After that, it’s a byte-for-byte pass through proxy
server.

The proxy has some added logic to handle connection affinity — multiple
devices for the same SpiderOak user are passed to the same Perspective Broker
process.

This has allowed us to consume fewer public IP addresses (one per proxy
server, instead of one for each storage node) and take advantage of multiple
processors and greater concurrency per storage machine.

Another small benefit is offloading the cost of SSL from the Python
processes. Erlang has it’s own native implementation of SSL (not based on
OpenSSL) which seems to operate with more grace.

This is our first production Erlang/OTP service, and it hasn’t been without
its speed bumps, but these days it’s as stable as any of our other daemons
while handling much greater concurrency and traffic.

Today we’re publishing the code (AGPL3) in case it’s useful to anyone else
(and feedback from the Erlang community is certainly welcome!) It would be
useful to anyone wishing to be able to distribute a Perspective Broker service
across many backend nodes according to user assignment, or perhaps a starting
point for implementing a Perspective Broker server in Erlang. It will likely
require some minor massaging to with your database scheme. Here’s a link to
the tarball: href="https://spideroak.com/dist/spideroak_ssl_proxy.tar.bz2">spideroak_ssl_proxy.tar.bz2

Comments
  1. Looks interesting, although I'm not really into Erlang or functional programming (played quite a bit with Haskell, it's interesting and all, but I find it a bit too "academic").

    If I would need to use something else, not Python I would most likely choose NodeJS, although I'm pretty sure Python would work just as well in this case (would be interesting to see some benchmarks for this specific use case though).

  2. No one cares! Stop procrastinating with all of this "programming" and finish the Android app already — I hear those things write themselves ;)

  3. Awesome. I just so happen to have need for this, right at this moment. Thank you for preemptively reading my mind and publishing the source.