For last year’s SXSW gimmick, we turned the home page of the Made by Many website into a Twitter dashboard that showed the latest tweets of everyone out in Austin. It went down well and did a good job of capturing a snapshot of what people were up to out there.
2010’s home page takeover was a bit of a last-minute rush, though. For 2011, we wanted to do something different, and we approached it in plenty of time. That turned into Holler Gram, which you can read about elsewhere.
But then, a week and a bit ago, Instagram made their API public3. We couldn’t resist playing with it. Foolishly, perhaps, I spent a few hours writing a little application to prove that we could aggregate Instagram updates (Instagrams?) into a similar kind of dashboard. And thus I unwittingly created some kind of monster.
We’ve squeezed in time here and there to put together an Instagram-based home page for the duration of SXSW 2011. It’s full of fresh, buzzword-compliant technologies, which I’ll attempt to explain below.
The server is a single Node.js process which uses
Express to serve static files
MongoDB to store and serve historical updates
Socket.IO to provide a real-time transport mechanism between server and browser
It also runs a couple of periodic jobs to fetch new data:
Poll each of the ten attendees’ feeds for updates
Check the blog feed for new posts
The Instagram API does offer a real-time callback mechanism for receiving updates, but we’re just polling, Why?
Callbacks are inconvenient when developing locally on a machine behind NAT
They require that all users have authenticated with the application, unlike polling
The API rate limits are generous enough to let us poll every person every minute
So callbacks failed from a cost-benefit point of view. Polling gets us close enough to real-time not to worry.
Updates from either source are turned into chunks of JSON data which are dumped into the database and broadcast to all currently connected clients. MongoDB makes it really easy to deal with this kind of arbitrary data, to store it, and to perform queries against it.
Here’s what happens when your browser connects to sxsw.madebymany.com:
Socket.IO sets up a connection with the server
The server immediately sends the newest dozen items (these are cached in memory on the server)
When an update comes in:
It’s added to the database
It’s sent immediately to every connected browser
When you scroll to the bottom of the page:
The browser sends a message asking for more, with the timestamp of the earliest update it knows about
The server asks MongoDB for the last twelve items before that timestamp, and sends them back to the browser
Because Socket.IO liberates us from connection overheads, and MongoDB is really fast, the performance of this is very good. At least, it was very good …. This is going to be an interesting test of performance under load!
On the front end, we use text/html micro-templates and Mustache to transform updates into HTML that we then insert into the page.
I’m still not sure whether Node.js is the future, but it’s certainly been a good fit for this project.
1. That didn’t stop me making burritos for dinner last night, but they were modestly sized, and, in any case, my digestion would probably rebel at being subjected to one for breakfast.
3. I should confess to something. Until the API came out, I had no idea what Instagram was. I don’t have an iPhone, so my only exposure to Instagram was when I saw links posted on Twitter. Perhaps not unreasonably, I thus assumed that it was just another image hosting service with a bunch of filters. Burbn, please get on with releasing that long-awaited Android client!
Like this and want more?
We won’t share your email address with any organsations other than the platform we use to send the newsletter. There’s an unsubscribe link in every newsletter we send.