But the more I thought about it, those questions felt more and more important. Then, I had a chance to read about Meteor. I tried some of their tutorials and they were fantastic. I was so impressed at that point that I felt this framework had the power to change web development in a big way. I was even tempted to write an article similar to Why Meteor will kill Ruby on Rails for Django. But then, better sense prevailed and I dug deeper.
So I wrote this post to really understand the Real-time web application phenomena and pen my thoughts around it. It will look at how a traditional web framework like Django might tackle it or whether it really can. It is not a comprehensive analysis by any means but some of my thoughts regarding it.
What is a Real-time Application?
Real-time web applications are what used to be known as AJAX applications or Single-page applications. Except that, now we are not talking about just submitting a form without refreshing anymore. Upon an event, the server is supposed to push data into the browser enabling a two-way communication (thereby, making the terms Server and Client moot). To the Facebook crowd, this means that the application can notify them about events like your friends liking your post or starting a new chat conversation live - as and when it happens.
The application will give you the impression that it is “alive”, in the sense that traditional websites were pretty much stale after the page has loading. To use an analogy, traditional websites are like cooked food that get spoiled with time but real-time websites are like living, breathing organisms - constantly interacting with and responding to the environment. As we will see, both have their own advantages and purposes.
The word Real-time itself comes from the field of Real-time Computing. There are various forms of real-time computing based on how strict the systems perceives the response times. Realtime Web applications are probably the least strict and could be termed as ‘soft realtime’.
What kind of sites needs them?
Practically any kind of website can benefit from realtime updates. Off the top of my head, here are some examples:
- E-commerce Site: Offers against an item updates its discounted price realtime, user reviews
- News Site: Minute-by-minute update of an emerging story, errata of an article, reader reactions, polls
- Investment Site: Stock prices, exchange rates, any investment trigger set by user/customer advisor
- Utilities Site: Updates your utility usage in real-time, notifies of any upcoming downtimes
- Micro-blogging: Pretty much everything
Of course, there are several kinds of sites which would not be ideal for the real-time treatment. Sites with relatively static content like blogs or wikis. Imagine if a blog site was designed to constantly poll for post updates and it suddenly becomes viral. This stretches the limits of the server’s scalability for a largely unchanging content. The most optimal approach today would be to pre-compute and cache the article’s contents and serve the comments via Disqus to create an almost complete realtime experience. But, as we would soon see, this could change in the future as the fundamental nature of the web changes.
Websites are Turning into Multiplayer Games
Broadly, our examples have two kinds of real-time updates - from the server itself and from the peers. The former involves notifications like changes in the discounted price or external events like stock price change. But peer updates, from other users, are becoming extremely valuable as user-generated content becomes the basis for many modern sites. Especially for the constant inflow of information which keeps such sites fresh and interesting. In addition, the social factor adds to inherent human tendencies to form groups, to know and interact with each other.
In many ways, this is exactly how a multiplayer game works. There are global events like weather changes or a sudden disaster and there are player generated events like a melee attack or a chat conversation. Games, being some of first programs that I had written; I am quite fond of them and know quite a bit about how they work. Real-time web applications are designed like multiplayer games especially those that were designed to work with a central server rather than say over a LAN. Multiplayer games have been in existence for about three decades now so real-time web applications are able to leverage much of the work that has gone into creating them.
Technically, there are several similarities too. The event loop that every game programmer writes while beginning to write a game is at the heart of most event driven servers like Node.js or Tornado. Typically, the game client and server are written in two different languages for e.g. Eve Online using Stackless Python on the server and possibly C++ with Python scripting on the client side. This is because, like web applications, the server side needs to interact with a database for bookkeeping purposes and would be more IO bound rather than CPU/GPU-bound. Thus, the needs are different and games being extremely performance hungry creations, developers often use the best language or tool or framework for the client and server sides. They often end up being different.
How does Meteor Solve it?
These new frameworks also allow automatic synchronisation of data. In other words, part or whole of the server information is replicated between the server and all the connected clients. If you have ever programmed a multiplayer game then you would realise how hard it is to maintain a consistent state across all the clients. By automatically synchronising the database information, you have an extremely powerful abstraction for rapidly creating real-time web applications.
A high-level understanding of how real-time web frameworks work
However, like Drew (the creator of Dropbox) pointed out treating anything over the network as local accessible is a “leaky abstraction” due to network latencies. Programmers are very likely to under-engineer the various challenges that networks can bring up like a slower mobile connection or even a server crash. Users hate it when the real-time component stops working. In fact, once they start seeing ‘Connection Lost’ on a simple chat window the worse thing that could happen is that they lose their train of thought. But when an entire site becomes unresponsive, I believe they would start distrusting the site itself. Perhaps, it is critical not to entirely rely on the data synchronisation mechanism for all kinds of functionality.
Can Django Adapt?
Realtime web is a very real challenge that Django faces. There are some elegant solutions like running Django with Gevent. But these solutions look like hacks in the face of a comprehensive solution like Meteor.
Django community is quite aware of this. At last year’s DjangoCon US, the customary “Why I Hate Django” keynote was by Geoff Schmidt, a principal developer of Meteor. He starts off, in an ominous note, by comparing advent of real-time apps as possibly an extinction event for Django similar to the asteriod impact which nearly drove dinosaurs to extinction. Actually, Geoff is a big fan of Django and he tried to adapt it to his needs but was not quite happy with the results.
And he is not alone. Guido, in his Pycon keynote Q&A, mentioned how it would be difficult for traditional web frameworks like Django to completely adapt to an event driven model. He believes that newer frameworks might actually tackle that need better.
Before we answer the question whether Django can adapt, we need to find out what Django’s shortcomings are. Or rather, which parts of the framework are beginning to show its age in the new real-time paradigm.
Templates are no longer necessary
However, the previously mentioned Django solutions like Piston and Django REST does not encourage using serialised Python data structures directly for a good reason - Security. Data coming from the outside world cannot be trusted. You will need to define classes with appropriate data types to perform basic type validation. Essentially, you might end up repeating the model class definitions you already wrote for the front end.
HTTP and the WSGI interface
If you read the above examples closely, you will notice that real-time web works best for sites involving short bursts of data. For sites with long form content like blogs, wikis or even news sites, it is probably best to stick with traditional web frameworks or even static content. Even if you have a very high bandwidth connection, it would simply be too chatty to check for published information (unless you expect to have too many typos)
In fact, the web is specifically suited for the dissemination of long-form content. It works best for a request-reply mechanism for retrieval of documents (or hypertext if you like to be pedantic). It is stateless hence suited for caching, wherever possible. In fact, this explains why hacks like long-polling had to be created for the browser to support bidirectional communication until web sockets arrived.
This explains the design of WSGI, a synchronous protocol to handle request and response. It can handle only one request at a time and hence not ideally suited for creating realtime applications. Since Django is designed to be deployed on a server with a WSGI interface, this means that asynchronous code needs to bypass this mechanism.
This seems like a genuine shortcoming of Django. There might be hacks to work around it like frequent polling etc. But it would be much better if the framework can integrate with the asynchronous model.
Can you have the cake and eat it too?