SXSW takes over Made by Many (2011 edition)

As I write this, ten twelve of the Many are packed into economy-class seating on their way to Austin, Texas for South by Southwest.

I’m not. I wasn’t invited. On the downside, I miss out on drinking my liver into a foie-gras-like state and eating burritos the size of Wales for breakfast¹. On the upside, I get to avoid the charmless shopping-mall ambiance of Heathrow airport², the legendarily unwelcoming demeanour of US immigration officials, and the scrotum-groping attentions of TSA goons, and I get to write snide blog posts about it. So it’s not all bad! And, in fact, despite the fact that I’m not going, I’ve ended up being involved with one rather fun aspect of the trip.

If you go to http://madebymany.com/ right now, you’ll see something a little different from usual.

For last year’s SXSW gimmick, we turned the home page of the Made by Many website into a Twitter dashboard that showed the latest tweets of everyone out in Austin. It went down well and did a good job of capturing a snapshot of what people were up to out there.

2010’s home page takeover was a bit of a last-minute rush, though. For 2011, we wanted to do something different, and we approached it in plenty of time. That turned into Holler Gram, which you can read about elsewhere.

But then, a week and a bit ago, Instagram made their API public³. We couldn’t resist playing with it. Foolishly, perhaps, I spent a few hours writing a little application to prove that we could aggregate Instagram updates (Instagrams?) into a similar kind of dashboard. And thus I unwittingly created some kind of monster.

We’ve squeezed in time here and there to put together an Instagram-based home page for the duration of SXSW 2011. It’s full of fresh, buzzword-compliant technologies, which I’ll attempt to explain below.

The server is a single Node.js process which uses

• Express to serve static files
• MongoDB to store and serve historical updates
• Socket.IO to provide a real-time transport mechanism between server and browser

It also runs a couple of periodic jobs to fetch new data:

• Poll each of the ten attendees’ feeds for updates
• Check the blog feed for new posts

The Instagram API does offer a real-time callback mechanism for receiving updates, but we’re just polling, Why?

• Callbacks are inconvenient when developing locally on a machine behind NAT
• They require that all users have authenticated with the application, unlike polling
• The API rate limits are generous enough to let us poll every person every minute

So callbacks failed from a cost-benefit point of view. Polling gets us close enough to real-time not to worry.

Updates from either source are turned into chunks of JSON data which are dumped into the database and broadcast to all currently connected clients. MongoDB makes it really easy to deal with this kind of arbitrary data, to store it, and to perform queries against it.

Here’s what happens when your browser connects to sxsw.madebymany.com:

• Your browser fetches an empty index page and some JavaScript
• Socket.IO sets up a connection with the server
• The server immediately sends the newest dozen items (these are cached in memory on the server)

When an update comes in:

• It’s added to the database
• It’s sent immediately to every connected browser

When you scroll to the bottom of the page:

• The browser sends a message asking for more, with the timestamp of the earliest update it knows about
• The server asks MongoDB for the last twelve items before that timestamp, and sends them back to the browser

Because Socket.IO liberates us from connection overheads, and MongoDB is really fast, the performance of this is very good. At least, it was very good …. This is going to be an interesting test of performance under load!

On the front end, we use text/html micro-templates and Mustache to transform updates into HTML that we then insert into the page.

It’s the first time that I’ve written any production code using Node.js, and it’s been an interesting experience. I don’t find JavaScript’s syntax too awful—in fact, I think its simplicity is a strength—but implementing callback-heavy code in a way that avoids excessive nesting and preserves readability has been an interesting challenge.

I’m still not sure whether Node.js is the future, but it’s certainly been a good fit for this project.

---

1. That didn’t stop me making burritos for dinner last night, but they were modestly sized, and, in any case, my digestion would probably rebel at being subjected to one for breakfast.

2. Now with added bomb scare!

3. I should confess to something. Until the API came out, I had no idea what Instagram was. I don’t have an iPhone, so my only exposure to Instagram was when I saw links posted on Twitter. Perhaps not unreasonably, I thus assumed that it was just another image hosting service with a bunch of filters. Burbn, please get on with releasing that long-awaited Android client!