As I write this, ten twelve of the Many are packed into economy-class seating on their way to Austin, Texas for South by Southwest. I’m not. I wasn’t inv...


For last year’s SXSW gimmick, we turned the home page of the Made by Many website into a Twitter dashboard that showed the latest tweets of everyone out in Austin. It went down well and did a good job of capturing a snapshot of what people were up to out there.



2010’s home page takeover was a bit of a last-minute rush, though. For 2011, we wanted to do something different, and we approached it in plenty of time. That turned into Holler Gram, which you can read about elsewhere.



But then, a week and a bit ago, Instagram made their API public3. We couldn’t resist playing with it. Foolishly, perhaps, I spent a few hours writing a little application to prove that we could aggregate Instagram updates (Instagrams?) into a similar kind of dashboard. And thus I unwittingly created some kind of monster.



We’ve squeezed in time here and there to put together an Instagram-based home page for the duration of SXSW 2011. It’s full of fresh, buzzword-compliant technologies, which I’ll attempt to explain below.



The server is a single Node.js process which uses




  • Express to serve static files


  • MongoDB to store and serve historical updates


  • Socket.IO to provide a real-time transport mechanism between server and browser



It also runs a couple of periodic jobs to fetch new data:




  • Poll each of the ten attendees’ feeds for updates


  • Check the blog feed for new posts



The Instagram API does offer a real-time callback mechanism for receiving updates, but we’re just polling, Why?




  • Callbacks are inconvenient when developing locally on a machine behind NAT


  • They require that all users have authenticated with the application, unlike polling


  • The API rate limits are generous enough to let us poll every person every minute



So callbacks failed from a cost-benefit point of view. Polling gets us close enough to real-time not to worry.



Updates from either source are turned into chunks of JSON data which are dumped into the database and broadcast to all currently connected clients. MongoDB makes it really easy to deal with this kind of arbitrary data, to store it, and to perform queries against it.



Here’s what happens when your browser connects to sxsw.madebymany.com:




  • Your browser fetches an empty index page and some JavaScript


  • Socket.IO sets up a connection with the server


  • The server immediately sends the newest dozen items (these are cached in memory on the server)



When an update comes in:




  • It’s added to the database


  • It’s sent immediately to every connected browser



When you scroll to the bottom of the page:




  • The browser sends a message asking for more, with the timestamp of the earliest update it knows about


  • The server asks MongoDB for the last twelve items before that timestamp, and sends them back to the browser



Because Socket.IO liberates us from connection overheads, and MongoDB is really fast, the performance of this is very good. At least, it was very good …. This is going to be an interesting test of performance under load!



On the front end, we use text/html micro-templates and Mustache to transform updates into HTML that we then insert into the page.



It’s the first time that I’ve written any production code using Node.js, and it’s been an interesting experience. I don’t find JavaScript’s syntax too awful—in fact, I think its simplicity is a strength—but implementing callback-heavy code in a way that avoids excessive nesting and preserves readability has been an interesting challenge.



I’m still not sure whether Node.js is the future, but it’s certainly been a good fit for this project.





1. That didn’t stop me making burritos for dinner last night, but they were modestly sized, and, in any case, my digestion would probably rebel at being subjected to one for breakfast.



2. Now with added bomb scare!



3. I should confess to something. Until the API came out, I had no idea what Instagram was. I don’t have an iPhone, so my only exposure to Instagram was when I saw links posted on Twitter. Perhaps not unreasonably, I thus assumed that it was just another image hosting service with a bunch of filters. Burbn, please get on with releasing that long-awaited Android client!

Paul Battley

Paul Battley

Paul has been called a traitor, threatened with death, and asked if what he's doing on the internet is actually legal. He's interested in open platforms and in reverse engineering closed ones. He writes software in glowing multicoloured letters on a black screen.

@threedaymonk