django on Arunrocks

Home pages that morph

Tue, 13 Apr 2021 13:01:13 +0530

You are reading a post from a multi-part series of articles

Staying on the bleeding edge
Home pages that morph

Have you ever struggled to create a beautiful landing page for your Django site?

A common ask by users of the Edge project template was the support for home pages that had a completely different layout than the inner pages. For example, a first time visitor opening https://example.com should see a landing page that shows the benefits of signing up. But once signed-in the same url https://example.com should show the user interface of the web application.

This is a common pattern today. For example this is how different github.com looks to a first time visitor and a logged in user:

Github homepage before signing in and after signing in (two-frame animation)

What’s a Landing Page

A landing page is a standalone page created for a specific marketing conversion goal, like subscribing to a newsletter or ebook downloads. For the purpose of this article, we will focus on the conversion goal of signing up users to your website or online product.

A good landing page must be convincing enough for a visitor to share their personal details like their name and email. Even then, ask as few details as possible in the sign up form. If it is not relevant like the Company field, it is best to remove it. Expedia dropped the “Company” field from their booking form and saw an increase of $12 million a year in profit.

It must also be optimized for quick scanning. A widely quoted but often misunderstood study by Nielsen Norman Group showed how reading on the web is F-shaped. This is often bad for users and businesses since it skips important content. The recommended solution is to include important content in the first two paragraphs and have headings with the first two words bearing the most content.

F-shaped reading pattern in heat maps

A homepage can be distinct from a landing page. But for many visitors who come to know about your site say through a link shared on a social network or an ad, the home page is the gateway to your product. So many sites check if the user has not logged in and show a landing page instead.

Challenges in Django

The challenge for most Django sites is that this requires extensive changes to the base template. A base template defines the basic structure of the site. Django documentation explains it further - a base “skeleton” template contains all the common elements of your site and defines blocks that child templates can override.

The landing page differs considerably in content and layout from the inner pages. Here are some of the main differences:

Objectives: The primary purpose of the landing page is to typically get users to sign up. The entire page is focussed on convincing the reader to perform the Call to Action (CTA) of clicking the “Sign up” button. Once you have signed up, your objective changes to give a good user experience while using the product i.e. the web application.
Layout: Landing pages typically have a simplified and vertical layout where you are scrolling down to view engaging visuals and exciting animations. However a logged in user typically needs a more functional interface where various interface elements like sidebars are always within reach.
Navigation: Most modern landing pages remove the top menu entirely along with other links to internal and external pages to simplify the experience. It does not make sense to show the application menu or the user profile drop down to a visitor who has not yet signed up.
Assets: Landing pages, like other marketing pages, have scripts and other assets that track the effectiveness of pages like click through rates and conversion metrics. The inner pages would have different assets needed to create and maintain an effective user interface.
Optimizations: According to a study of the top 1000 websites, landing pages are 35% heavier but load faster by 56% than internal pages. This could be because the landing page needs to give a much more optimized user experience as it forms the crucial “first impression”.

In theory, a Django application could be crafted with a base template that can accommodate both a landing page and a web application layout. In practice it will be extremely cumbersome. As you have seen these two kinds of pages often have nothing in common. They might even be designed by two different teams. So, given a single base template you will often end up in overriding most of the blocks. There has to be a better way .

It’s morphin' time!

Before we go further, it will be useful to give a name to this common pattern. Despite a lot of research, I couldn’t find any existing literature that names this pattern. So I went ahead and called it a Dimorphic Home Page pattern. The word Dimorphic means a thing that exists in two different forms. In this case a home page that changes form depending on whether the user has been authenticated or not.

Dimorphic materials like Calcite (left) and Aragonite (right) is the same compound (Calcium carbonate) existing in two forms or crystal structures. Pic courtesy: Wikipedia

Here is a simple view (taken from Edge source ) that implements this solution:

from django.views.generic.base import TemplateView


class HomePageView(TemplateView):
	user_template_name = "users/home.html"
	anon_template_name = "anons/home.html"

	def get_template_names(self):
    	if self.request.user.is_authenticated:
        	return self.user_template_name
    	else:
        	return self.anon_template_name

We have used Class Based Generic Views instead of Function Based Views because they are easy to extend. For instance, if you need to pass a signup form in the context variable (and remove another page reload from conversion) then another view class can be derived easily.

In terms of organization, the templates for unauthenticated and authenticated users are kept in separate directories named say anon and users respectively. This makes it easier to prevent unintentional leakage of sensitive data in unauthenticated pages.

Designing a Beautiful Landing Page

The landing page of Edge has been designed based on modern landing page best practices. If you study several actual landing pages, you would observe many of them use common animation components like animate.css and wow.js. This is what powers the sliding and dancing images or text as you scroll down revealing new sections.

The dimorphic Edge homepage is a work in progress but it currently looks like this:

Django Edge homepage before signing in and after signing in (two-frame animation). Hero image courtesy Scale by Flexiple

Design sensibilities change over time. But the template should be a good starting point for solo Django developers who are not familiar with design. It would be less daunting than starting from a blank HTML document.

Try what is new

Check out the Edge Django 3.x branch with its dimorphic landing page. Follow the instructions in the README to create a quick Django project. We welcome your feedback and contributions.

Staying on the bleeding edge

Tue, 09 Mar 2021 11:28:29 +0530

You are reading a post from a multi-part series of articles

Staying on the bleeding edge
Home pages that morph

How fast can you go from getting an amazing product idea to bringing it in front of real users? It is a process that has several steps - some fun but mostly boring. It could also take a long time. In fact the longer it takes, the lower your motivation levels dip and competition starts looming large. This is why we use productivity enhancers like frameworks, libraries and templates not just to reach the users faster but also to ensure that we ship it.

Django Edge is a Django project starter template that I started in 2014 with the idea of making web apps faster using my go-to toolset - Django, Bootstrap and many of the most useful Django libraries. The project is quite popular with over 770 stars on Github. Unlike many other starter templates, it shipped with some essential pages like a homepage, register, login etc that were presentable and working.

An update to Edge has been long overdue. But the Django landscape has changed a bit. For a long time, the most common use of the framework was as a back-end for a front-end JavaScript framework like React or Vue (and any other possible front-ends like a mobile app). This typically uses the excellent Django Rest Framework (DRF) to build a REST API which the front-end consumes. This could be one possible direction that Edge could take.

However the use of state-heavy front-end frameworks with millions of dependencies has led to Javascript fatigue. The trend is somewhat coming back to back-end rendered pages. Choosing “Boring Technology” might counter-intuitively leave lots of room for innovation in the areas that you may find fun and interesting. There is really no need to throw away Django Templates if you need a single page application thanks to HTMX. So continuing to ship with beautifully designed templates is another direction that Edge could take as well.

I was tempted to try this “Double-edged” approach for a moment. But then I realized that it is already a lot of work to maintain just one open source project. Users who need Django for a REST API will probably find it incomplete unless it is also married with Rest, Vue, Angular or some other shiny new JavaScript framework out of the box. This is a truly fragmented base that keeps switching new frameworks or even abandoning all frameworks.

Hence I went back to the familiar Edge. Provide a fully working web application starter with the best of what Django can provide. Aim to be a good starting point for single developer projects. Focus on the long requested features like social authentication or Docker and improve on what it currently does.

Yay!...a working login page

So this is probably the first of many posts where I share what I am planning to do with Edge. I believe that open development is the best way to build open source projects. Try the beta here: https://github.com/arocks/edge/tree/django3

Pull requests are welcome! 😊

Fitting a Django Application in One File

Wed, 09 Dec 2020 22:54:50 +0530

Earlier this week, Anthony a french economics university student wanted to talk to me over Zoom about my ray tracer tutorials. He and his friend were new to Python but were excited to implement their own ray tracer after following my videos. One of the questions that popped up in the conversation was - “Can we put all the classes in one file instead of breaking it into individual files per class?”. I said, “Of course” and noticed a wave of relief in their faces. In fact, I explained, my earlier implementation was all in one file and later broken up for better pedagogy.

But the charm of an entire project in a single file is compelling. I remember seeing a Sinatra web application a few years ago containing the entire application and assets like HTML templates and CSS in a single file. Presenting all the components in the same file gave a complete high-level overview of the project by simply scrolling up and down.

Normally, at this point, someone would suggest a microframework. But it is not that easy.

Microframeworks

Microframeworks take a minimalistic approach by omitting certain components or directing you to a few recommended components. For instance, Bottle contains basic form handling capabilities but has no protection against CSRF or clickjacking.

So the approach is generally to use another library like bottle-utils-csrf. This leaves the task of integration to the developer. This is not to pooh-pooh tiny web frameworks. I love the idea (especially Bottle which I think is really cute). But for public facing sites, I prefer the safety and convenience of Django.

So I am tempted to try this one-file trick in Django. Let’s try to make a non-trivial web application with forms, templates and images. How does one go about doing something like that?

Note: if you prefer to watch the video version, click on the video below:

Django Applications in One File

Minimal

Let’s start small by creating a minimal Hello World application in Django. It might be amusing to some Django developers that we will not start with the startproject command. In fact, it is not necessary for Django to work at all. All that initial directory structure and files like settings.py are for your convenience.

First, create a simple file called app.py with the following:

import sys

from django.conf import settings
from django.urls import path
from django.http import HttpResponse

settings.configure(
	DEBUG=True,  # For debugging
	SECRET_KEY="a-bad-secret",  # Insecure! change this
	ROOT_URLCONF=__name__,
)


def home(request):
	return HttpResponse("Welcome!")


urlpatterns = [
	path("", home),
]

if __name__ == "__main__":
	from django.core.management import execute_from_command_line

	execute_from_command_line(sys.argv)

Yes, that’s all you need. There are some bad practices like hard coding the secret key (easily fixed). But the sheer elegance of everything being in hardly a screenful is quite rewarding.

The command to run this file is: python app.py runserver 8080

Now we will skip a couple of steps (they are in the video) and move on to a simple “Coming Soon” landing page.

Coming Soon Application

The idea of a coming-soon page is to gauge interest in a product before it is released. Such pages must have a clear call to action (CTA) like asking for your email. Ideally it should have minimum friction and yet collect all the relevant information.

Let’s look at my updated app.py:

import os
import sys

from django.conf import settings
from django.urls import path
from django.http import HttpResponse, HttpResponseRedirect
from django.core.wsgi import get_wsgi_application
from django.template import RequestContext, Template
from django import forms

CSV_LIST = "thelist.csv"

settings.configure(
	DEBUG=(os.environ.get("DEBUG", "") == "1"),
	ALLOWED_HOSTS=["*"],  # Disable host header validation
	ROOT_URLCONF=__name__,
	SECRET_KEY=os.environ.get("SECRET_KEY", "a-bad-secret"),
	TEMPLATES=[{"BACKEND": "django.template.backends.django.DjangoTemplates"}],
	MIDDLEWARE_CLASSES=(
    	"django.middleware.common.CommonMiddleware",
    	"django.middleware.csrf.CsrfViewMiddleware",
    	"django.middleware.clickjacking.XFrameOptionsMiddleware",
	),
)


class EnlistForm(forms.Form):
	email = forms.EmailField(
    	required=True,
    	label=False,
    	widget=forms.EmailInput(attrs={"placeholder": "Email"}),
	)
	referrer = forms.CharField(required=False, widget=forms.HiddenInput())


def home(request):
	if request.method == "POST":
    	form = EnlistForm(request.POST)
    	if form.is_valid():
        	email = form.cleaned_data["email"]
        	referrer = form.cleaned_data["referrer"]
        	ip = request.META.get("REMOTE_ADDR")
        	print(f"Got email of {email}")
        	with open(CSV_LIST, "a") as csv:
            	csv.write(f"{email},{referrer},{ip}\n")
        	return HttpResponseRedirect("/thanks/")
	else:
    	form = EnlistForm(initial={"referrer": request.META.get("HTTP_REFERER")})
	context = RequestContext(
    	request, {"content": "Sign up for early access", "form": form}
	)
	return HttpResponse(MAIN_HTML.render(context))


def thanks(request):
	context = RequestContext(
    	request,
    	{"content": "Thank you for signing up. We will contact you!", "form": None},
	)
	return HttpResponse(MAIN_HTML.render(context))


urlpatterns = [
	path("", home),
	path("thanks/", thanks),
]

app = get_wsgi_application()


MAIN_HTML = Template(
	"""
<html>
  <head>
	<title>Coming Soon | Flying Cars</title>
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<style>
 	@import url('https://fonts.googleapis.com/css2?family=Exo:wght@400;500;600;700;800;900&display=swap');
 	*{
   	margin: 0;
   	padding: 0;
   	box-sizing: border-box;
   	font-family: 'Exo', sans-serif;
 	}
 	html,body{
   	display: grid;
   	height: 100%;
   	width: 100%;
   	place-items: center;
   	background-color: #343434;
   	/* Thanks to Hero Patterns for the background */
   	background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 56 28' width='56' height='28'%3E%3Cpath fill='%23000000' fill-opacity='0.4' d='M56 26v2h-7.75c2.3-1.27 4.94-2 7.75-2zm-26 2a2 2 0 1 0-4 0h-4.09A25.98 25.98 0 0 0 0 16v-2c.67 0 1.34.02 2 .07V14a2 2 0 0 0-2-2v-2a4 4 0 0 1 3.98 3.6 28.09 28.09 0 0 1 2.8-3.86A8 8 0 0 0 0 6V4a9.99 9.99 0 0 1 8.17 4.23c.94-.95 1.96-1.83 3.03-2.63A13.98 13.98 0 0 0 0 0h7.75c2 1.1 3.73 2.63 5.1 4.45 1.12-.72 2.3-1.37 3.53-1.93A20.1 20.1 0 0 0 14.28 0h2.7c.45.56.88 1.14 1.29 1.74 1.3-.48 2.63-.87 4-1.15-.11-.2-.23-.4-.36-.59H26v.07a28.4 28.4 0 0 1 4 0V0h4.09l-.37.59c1.38.28 2.72.67 4.01 1.15.4-.6.84-1.18 1.3-1.74h2.69a20.1 20.1 0 0 0-2.1 2.52c1.23.56 2.41 1.2 3.54 1.93A16.08 16.08 0 0 1 48.25 0H56c-4.58 0-8.65 2.2-11.2 5.6 1.07.8 2.09 1.68 3.03 2.63A9.99 9.99 0 0 1 56 4v2a8 8 0 0 0-6.77 3.74c1.03 1.2 1.97 2.5 2.79 3.86A4 4 0 0 1 56 10v2a2 2 0 0 0-2 2.07 28.4 28.4 0 0 1 2-.07v2c-9.2 0-17.3 4.78-21.91 12H30zM7.75 28H0v-2c2.81 0 5.46.73 7.75 2zM56 20v2c-5.6 0-10.65 2.3-14.28 6h-2.7c4.04-4.89 10.15-8 16.98-8zm-39.03 8h-2.69C10.65 24.3 5.6 22 0 22v-2c6.83 0 12.94 3.11 16.97 8zm15.01-.4a28.09 28.09 0 0 1 2.8-3.86 8 8 0 0 0-13.55 0c1.03 1.2 1.97 2.5 2.79 3.86a4 4 0 0 1 7.96 0zm14.29-11.86c1.3-.48 2.63-.87 4-1.15a25.99 25.99 0 0 0-44.55 0c1.38.28 2.72.67 4.01 1.15a21.98 21.98 0 0 1 36.54 0zm-5.43 2.71c1.13-.72 2.3-1.37 3.54-1.93a19.98 19.98 0 0 0-32.76 0c1.23.56 2.41 1.2 3.54 1.93a15.98 15.98 0 0 1 25.68 0zm-4.67 3.78c.94-.95 1.96-1.83 3.03-2.63a13.98 13.98 0 0 0-22.4 0c1.07.8 2.09 1.68 3.03 2.63a9.99 9.99 0 0 1 16.34 0z'%3E%3C/path%3E%3C/svg%3E");
 	}
 	::selection{
   	color: #fff;
   	background: #FC4782;
 	}
 	.wrapper{
   	color: #eee;
   	max-width: 900px;
   	text-align: center;
   	padding: 0 50px;
 	}
 	.signup {
   	margin-top: 30px;
   	margin-bottom: 10px;
 	}
 	.content {
   	margin-top: 40px;
   	margin-bottom: 10px;
 	}
	</style>
  </head>
  <body>
	<div class="wrapper">
  	<svg width="600" height="300" version="1.1" viewBox="0 0 600 300" xmlns="http://www.w3.org/2000/svg"><g transform="translate(0,312)"><path d="m218.36-289.66h163.29c29.039 0 52.417 23.378 52.417 52.417v45.564c0 29.039-23.378 52.417-52.417 52.417h-163.29c-29.039 0-52.417-23.378-52.417-52.417v-45.564c0-29.039 23.378-52.417 52.417-52.417z" fill="#204a87" stop-color="#000000" stroke="#eeeeec" stroke-linecap="round" stroke-linejoin="round" stroke-width="4"/><g fill="#729fcf" stroke="#eeeeec" stroke-linejoin="round" stroke-width="4"><path d="m240.88-162.15c21.473-37.192 42.946-74.385 64.419-111.58" stop-color="#000000"/><g stroke-linecap="round"><path d="m276.15-249.32h-72.081" stop-color="#000000"/><path d="m259.9-221.32h-56.025" stop-color="#000000"/><path d="m267.69-235.32h-63.714" stop-color="#000000"/><path d="m370.37-162.15c-21.473-37.192-42.946-74.385-64.419-111.58" stop-color="#000000"/><path d="m332-249.93h72.081" stop-color="#000000"/><path d="m348.25-221.93h56.025" stop-color="#000000"/><path d="m340.46-235.93h63.714" stop-color="#000000"/><path d="m240.88-162.15c21.473-26.526 42.946-53.051 64.419-79.577" stop-color="#000000"/></g><path d="m370.37-162.15c-21.473-26.526-42.946-53.051-64.419-79.577" stop-color="#000000"/></g><g fill="#eeeeec"><path d="m183.74-116.06-6.3858 17.316h12.795zm-2.6568-4.6378h5.337l13.261 34.795h-4.8942l-3.1696-8.9261h-15.685l-3.1696 8.9261h-4.9641z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m222.66-120.7h4.7078v34.795h-4.7078z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m271.14-102.22q1.5149.51273 2.9365 2.1907 1.445 1.678 2.8899 4.6145l4.7777 9.5087h-5.0573l-4.4514-8.9261q-1.7246-3.4959-3.356-4.6378-1.6081-1.142-4.4048-1.142h-5.1273v14.706h-4.7078v-34.795h10.627q5.9663 0 8.9028 2.4937t2.9365 7.5278q0 3.2861-1.5382 5.4535-1.5149 2.1674-4.4281 3.0064zm-11.793-14.613v12.352h5.9196q3.4026 0 5.1273-1.5615 1.7479-1.5848 1.7479-4.6378t-1.7479-4.5912q-1.7246-1.5615-5.1273-1.5615z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m329.38-118.02v4.9641q-2.3772-2.214-5.0806-3.3094-2.6802-1.0954-5.7099-1.0954-5.9663 0-9.1358 3.659-3.1696 3.6357-3.1696 10.534 0 6.8752 3.1696 10.534 3.1696 3.6357 9.1358 3.6357 3.0297 0 5.7099-1.0954 2.7035-1.0954 5.0806-3.3094v4.9175q-2.4704 1.678-5.2438 2.517-2.7501.83901-5.8264.83901-7.9006 0-12.445-4.8243-4.5446-4.8476-4.5446-13.214 0-8.39 4.5446-13.214 4.5446-4.8476 12.445-4.8476 3.123 0 5.873.839 2.7734.8157 5.1972 2.4704z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m366.18-116.06-6.3858 17.316h12.795zm-2.6568-4.6378h5.337l13.261 34.795h-4.8942l-3.1696-8.9261h-15.685l-3.1696 8.9261h-4.9641z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m421.6-102.22q1.5149.51273 2.9365 2.1907 1.445 1.678 2.8899 4.6145l4.7777 9.5087h-5.0573l-4.4514-8.9261q-1.7246-3.4959-3.356-4.6378-1.6081-1.142-4.4048-1.142h-5.1272v14.706h-4.7078v-34.795h10.627q5.9663 0 8.9028 2.4937t2.9365 7.5278q0 3.2861-1.5382 5.4535-1.5149 2.1674-4.4281 3.0064zm-11.793-14.613v12.352h5.9196q3.4026 0 5.1272-1.5615 1.7479-1.5848 1.7479-4.6378t-1.7479-4.5912q-1.7246-1.5615-5.1272-1.5615z" style="text-decoration-color:#000000;text-decoration-line:none"/></g></g></svg>
  	<h1>All Your Traffic Problems Solved!</h1>
  	<h2>Feel the future with affordable levitating cars.</h2>
  	<div class="content">
    	{{ content }}
    	{% if form %}
      	<form action="." method="post" class="enlist_form">
        	{% csrf_token %}
        	{{ form.non_field_errors }}
        	{{ form.email.errors }}
        	{{ form.referrer }}
        	{{ form.referrer.errors }}
        	{{ form.email }}
        	<button type="submit">Add Me</button>
      	</form>
    	{% endif %}
  	</div>
	</div>
  </body>
</html>
"""
)


if __name__ == "__main__":
	from django.core.management import execute_from_command_line

	execute_from_command_line(sys.argv)

Except for the absence of individual files, most of the code should be familiar to a Django developer. There is a large HTML template (including two SVG images) embedded as a string.

Note that I do not use the ORM here. Django does seem to need a directory structure for that (Unless any reader could show me how to do it in a single file).

Hopefully this shows how minimal Django could be. You might be able to use your favourite framework in places which you didn’t think were possible.

3 Effective Examples of Django Async Views without Sleeping

Tue, 01 Dec 2020 13:33:19 +0530

In August this year, Django 3.1 arrived with support for Django async views. This was fantastic news but most people raised the obvious question – What can I do with it? There have been a few tutorials about Django asynchronous views that demonstrate asynchronous execution while calling asyncio.sleep. But that merely led to the refinement of the popular question – What can I do with it besides sleep-ing?

The short answer is – it is a very powerful technique to write efficient views. For a detailed overview of what asynchronous views are and how they can be used, keep on reading. If you are new to asynchronous support in Django and like to know more background, read my earlier article: A Guide to ASGI in Django 3.0 and its Performance.

Django Async Views

Django now allows you to write views which can run asynchronously. First let’s refresh your memory by looking at a simple and minimal synchronous view in Django:

def index(request):
    return HttpResponse("Made a pretty page")

It takes a request object and returns a response object. In a real world project, a view does many things like fetching records from a database, calling a service or rendering a template. But they work synchronously or one after the other.

In Django’s MTV (Model Template View) architecture, Views are disproportionately more powerful than others (I find it comparable to a controller in MVC architecture though these things are debatable). Once you enter a view you can perform almost any logic necessary to create a response. This is why Asynchronous Views are so important. It lets you do more things concurrently.

It is quite easy to write an asynchronous view. For example the asynchronous version of our minimal example above would be:

async def index_async(request):
    return HttpResponse("Made a pretty page asynchronously.")

This is a coroutine rather than a function. You cannot call it directly. An event loop needs to be created to execute it. But you do not have to worry about that difference since Django takes care of all that.

Note that this particular view is not invoking anything asynchronously. If Django is running in the classic WSGI mode, then a new event loop is created (automatically) to run this coroutine. So in this case, it might be slightly slower than the synchronous version. But that’s because you are not using it to run tasks concurrently.

So then why bother writing asynchronous views? The limitations of synchronous views become apparent only at a certain scale. When it comes to large scale web applications probably nothing beats FaceBook.

Views at Facebook

In August, Facebook released a static analysis tool to detect and prevent security issues in Python. But what caught my eye was how the views were written in the examples they had shared. They were all async!

# views/user.py
async def get_profile(request: HttpRequest) -> HttpResponse:
   profile = load_profile(request.GET['user_id'])
   ...
 
# controller/user.py
async def load_profile(user_id: str):
   user = load_user(user_id) # Loads a user safely; no SQL injection
   pictures = load_pictures(user.id)
   ...
 
# model/media.py
async def load_pictures(user_id: str):
   query = f"""
      SELECT *
      FROM pictures
      WHERE user_id = {user_id}
   """
   result = run_query(query)
   ...
 
# model/shared.py
async def run_query(query: str):
   connection = create_sql_connection()
   result = await connection.execute(query)
   ...

Note that this is not Django but something similar. Currently, Django runs the database code synchronously. But that may change sometime in the future.

If you think about it, it makes perfect sense. Synchronous code can be blocked while waiting for an I/O operation for several microseconds. However, its equivalent asynchronous code would not be tied up and can work on other tasks. Therefore it can handle more requests with lower latencies. More requests gives Facebook (or any other large site) the ability to handle more users on the same infrastructure.

Scalability Problems in the 1800s, I suppose

Even if you are not close to reaching Facebook scale, you could use Python’s asyncio as a more predictable threading mechanism to run many things concurrently. A thread scheduler could interrupt in between destructive updates of shared resources leading to difficult to debug race conditions. Compared to threads, coroutines can achieve a higher level of concurrency with very less overhead.

Misleading Sleep Examples

As I joked earlier, most of the Django async views tutorials show an example involving sleep. Even the official Django release notes had this example:

async def my_view(request):
    await asyncio.sleep(0.5)
    return HttpResponse('Hello, async world!')

To a Python async guru this code might indicate the possibilities that were not previously possible. But to the vast majority, this code is misleading in many ways.

Firstly, the sleep happening synchronously or asynchronously makes no difference to the end user. The poor chap who just opened the URL linked to that view will have to wait for 0.5 seconds before it returns a cheeky “Hello, async world!”. If you are a complete novice, you may have expected an immediate reply and somehow the “hello” greeting to appear asynchronously half a second later. Of course, that sounds silly but then what is this example trying to do compared to a synchronous time.sleep() inside a view?

The answer is, as with most things in the asyncio world, in the event loop. If the event loop had some other task waiting to be run then that half second window would give it an opportunity to run that. Note that it may take longer than that window to complete. Cooperative Multithreading assumes that everyone works quickly and hands over the control promptly back to the event loop.

Secondly, it does not seem to accomplish anything useful. Some command-line interfaces use sleep to give enough time for users to read a message before disappearing. But it is the opposite for web applications - a faster response from the web server is the key to a better user experience. So by slowing the response what are we trying to demonstrate in such examples?

Letting them Sleep Would Be Better Idea

The best explanation for such simplified examples I can give is convenience. It needs a bit more setup to show examples which really need asynchronous support. That’s what we are trying to explore here.

Better examples

A rule of thumb to remember before writing an asynchronous view is to check if it is I/O bound or CPU-bound. A view which spends most of the time in a CPU-bound activity for e.g. matrix multiplication or image manipulation would really not benefit from rewriting them to async views. You should be focussing on the I/O bound activities.

Invoking Microservices

Most large web applications are moving away from a monolithic architecture to one composed of many microservices. Rendering a view might require the results of many internal or external services.

In our example, an ecommerce site for books renders its front page - like most popular sites - tailored to the logged in user by displaying recommended books. The recommendation engine is typically implemented as a separate microservice that makes recommendations based on past buying history and perhaps a bit of machine learning by understanding how successful its past recommendations were.

In this case, we also need the results of another microservice that decides which promotional banners to display as a rotating banner or slideshow to the user. These banners are not tailored to the logged in user but change depending on the items currently on sale (active promotional campaign) or date.

Let’s look at how a synchronous version of such a page might look like:

def sync_home(request):
    """Display homepage by calling two services synchronously"""
    context = {}
    try:
        response = httpx.get(PROMO_SERVICE_URL)
        if response.status_code == httpx.codes.OK:
            context["promo"] = response.json()
        response = httpx.get(RECCO_SERVICE_URL)
        if response.status_code == httpx.codes.OK:
            context["recco"] = response.json()
    except httpx.RequestError as exc:
        print(f"An error occurred while requesting {exc.request.url!r}.")
    return render(request, "index.html", context)

Here instead of the popular Python requests library we are using the httpx library because it supports making synchronous and asynchronous web requests. The interface is almost identical.

The problem with this view is that the time taken to invoke these services add up since they happen sequentially. The Python process is suspended until the first service responds which could take a long time in a worst case scenario.

Let’s try to run them concurrently using a simplistic (and ineffective) await call:

async def async_home_inefficient(request):
    """Display homepage by calling two awaitables synchronously (does NOT run concurrently)"""
    context = {}
    try:
        async with httpx.AsyncClient() as client:
            response = await client.get(PROMO_SERVICE_URL)
            if response.status_code == httpx.codes.OK:
                context["promo"] = response.json()
            response = await client.get(RECCO_SERVICE_URL)
            if response.status_code == httpx.codes.OK:
                context["recco"] = response.json()
    except httpx.RequestError as exc:
        print(f"An error occurred while requesting {exc.request.url!r}.")
    return render(request, "index.html", context)

Notice that the view has changed from a function to a coroutine (due to async def keyword). Also note that there are two places where we await for a response from each of the services. You don’t have to try to understand every line here, as we will explain with a better example.

Interestingly, this view does not work concurrently and takes the same amount of time as the synchronous view. If you are familiar with asynchronous programming, you might have guessed that simply awaiting a coroutine does not make it run other things concurrently, you will just yield control back to the event loop. The view still gets suspended.

Let’s look at a proper way to run things concurrently:

async def async_home(request):
    """Display homepage by calling two services asynchronously (proper concurrency)"""
    context = {}
    try:
        async with httpx.AsyncClient() as client:
            response_p, response_r = await asyncio.gather(
                client.get(PROMO_SERVICE_URL), client.get(RECCO_SERVICE_URL)
            )

            if response_p.status_code == httpx.codes.OK:
                context["promo"] = response_p.json()
            if response_r.status_code == httpx.codes.OK:
                context["recco"] = response_r.json()
    except httpx.RequestError as exc:
        print(f"An error occurred while requesting {exc.request.url!r}.")
    return render(request, "index.html", context)

If the two services we are calling have similar response times, then this view should complete in _half _the time compared to the synchronous version. This is because the calls happen concurrently as we would want.

Let’s try to understand what is happening here. There is an outer try…except block to catch request errors while making either of the HTTP calls. Then there is an inner async…with block which gives a context having the client object.

The most important line is one with the asyncio.gather call taking the coroutines created by the two client.get calls. The gather call will execute them concurrently and return only when both of them are completed. The result would be a tuple of responses which we will unpack into two variables response_p and response_r. If there were no errors, these responses are populated in the context sent for template rendering.

Microservices are typically internal to the organization hence the response times are low and less variable. Yet, it is never a good idea to rely solely on synchronous calls for communicating between microservices. As the dependencies between services increases, it creates long chains of request and response calls. Such chains can slow down services.

Why Live Scraping is Bad

We need to address web scraping because so many asyncio examples use them. I am referring to cases where multiple external websites or pages within a website are concurrently fetched and scraped for information like live stock market (or bitcoin) prices. The implementation would be very similar to what we saw in the Microservices example.

But this is very risky since a view should return a response to the user as quickly as possible. So trying to fetch external sites which have variable response times or throttling mechanisms could be a poor user experience or even worse a browser timeout. Since microservice calls are typically internal, response times can be controlled with proper SLAs.

Ideally, scraping should be done in a separate process scheduled to run periodically (using celery or rq). The view should simply pick up the scraped values and present them to the users.

Serving Files

Django addresses the problem of serving files by trying hard not to do it itself. This makes sense from a “Do not reinvent the wheel” perspective. After all, there are several better solutions to serve static files like nginx.

'Serving simultaneously is not for everyone'

But often we need to serve files with dynamic content. Files often reside in a (slower) disk-based storage (we now have much faster SSDs). While this file operation is quite easy to accomplish with Python, it could be expensive in terms of performance for large files. Regardless of the file’s size, this is a potentially blocking I/O operation that could potentially be used for running another task concurrently.

Imagine we need to serve a PDF certificate in a Django view. However the date and time of downloading the certificate needs to be stored in the metadata of the PDF file, for some reason (possibly for identification and validation).

We will use the aiofiles library here for asynchronous file I/O. The API is almost the same as the familiar Python’s built-in file API. Here is how the asynchronous view could be written:

async def serve_certificate(request):
    timestamp = datetime.datetime.now().isoformat()

    response = HttpResponse(content_type="application/pdf")
    response["Content-Disposition"] = "attachment; filename=certificate.pdf"
    async with aiofiles.open("homepage/pdfs/certificate-template.pdf", mode="rb") as f:
        contents = await f.read()
        response.write(contents.replace(b"%timestamp%", bytes(timestamp, "utf-8")))
    return response

This example illustrates why we need asynchronous template rendering in Django. But until that gets implemented, you could use aiofiles library to pull local files without skipping a beat.

There are downsides to directly using local files instead of Django’s staticfiles. In the future, when you migrate to a different storage space like Amazon S3, make sure you adapt your code accordingly.

Handling Uploads

On the flip side, uploading a file is also a potentially long, blocking operation. For security and organizational reasons, Django stores all uploaded content into a separate ‘media’ directory.

If you have a form that allows uploading a file, then we need to anticipate that some pesky user would upload an impossibly large one. Thankfully Django passes the file to the view as chunks of a certain size. Combined with aiofile’s ability to write a file asynchronously, we could support highly concurrent uploads.

async def handle_uploaded_file(f):
    async with aiofiles.open(f"uploads/{f.name}", "wb+") as destination:
        for chunk in f.chunks():
            await destination.write(chunk)


async def async_uploader(request):
    if request.method == "POST":
        form = UploadFileForm(request.POST, request.FILES)
        if form.is_valid():
            await handle_uploaded_file(request.FILES["file"])
            return HttpResponseRedirect("/")
    else:
        form = UploadFileForm()
    return render(request, "upload.html", {"form": form})

Again this is circumventing Django’s default file upload mechanism, so you need to be careful about the security implications.

Where To Use

Django Async project has full backward compatibility as one of its main goals. So you can continue to use your old synchronous views without rewriting them into async. Asynchronous views are not a panacea for all performance issues, so most projects will still continue to use synchronous code since they are quite straightforward to reason about.

In fact, you can use both async and sync views in the same project. Django will take care of calling the view in the appropriate manner. However, if you are using async views it is recommended to deploy the application on ASGI servers.

This gives you the flexibility to try asynchronous views gradually especially for I/O intensive work. You need to be careful to pick only async libraries or mix them with sync carefully (use the async_to_sync and sync_to_async adaptors).

Hopefully this writeup gave you some ideas.

Thanks to Chillar Anand and Ritesh Agrawal for reviewing this post. All illustrations courtesy of Old Book Illustrations

A Guide to ASGI in Django 3.0 and its Performance

Tue, 10 Mar 2020 13:33:29 +0530

In Dec 2019, we saw the release of Django 3.0 with an interesting new feature — support for ASGI servers. I was intrigued by what this meant. When I checked the performance benchmarks of asynchronous Python web frameworks they were ridiculously faster than their synchronous counterparts often by a factor of 3x–5x.

So I set out to test Django 3.0 performance with a very simple Docker setup. The results — though not spectacular — were still impressive. But before that, you might need a little bit of background about ASGI.

Before ASGI there was WSGI

It was 2003 and various Python web frameworks like Zope, Quixote used to ship with their own web servers or had their own home grown interfaces to talk to popular web servers like Apache.

Being a Python web developer meant a devout commitment to learning an entire stack but relearning everything if you needed another framework. As you can imagine this led to fragmentation. A PEP 333 - “Python Web Server Gateway Interface v1.0” tried to solve this problem by defining a simple standard interface called WSGI (Web Server Gateway Interface). Its brilliance was in its simplicity.

In fact the entire WSGI specification can be simplified (conveniently leaving out some hairy details) as the server side invoking a callable object (i.e. anything from a Python function to a class with a call method) provided by the framework or the application. If you have a component that can play both roles, then you have created a “middleware” or an intermediate layer in this pipeline. Thus, WSGI components can be easily chained together to handle requests.

When connecting became easy, merriment followed

WSGI became so popular that it was adopted not just by the large web frameworks like Django and Pylons but also by microframeworks like Bottle. Your favourite framework could be plugged into any WSGI-compatible application server and it would work flawlessly. It was so easy and intuitive that there was really no excuse not to use it.

Road ‘Blocks’ to Scale

So if we were perfectly fine with WSGI, why did we have to come up with ASGI? The answer will be quite evident if you have followed the path of a webrequest. Check out my animation of how a webrequest flows into Django. Notice how the framework is waiting after querying the database before sending the response. This is the drawback of synchronous processing.

Frankly this drawback was not obvious or pressing until Node.js came into the scene in 2009. Ryan Dahl, the creator of Node.js, was bothered by the C10K problem i.e. why popular web servers like Apache cannot handle 10,000 or more concurrent connections (given a typical web server hardware it would run out of memory) . He asked “What is the software doing while it queries the database?”.

Looks like she has been waiting forever

The answer was, of course, nothing. It was waiting for the database to respond. Ryan argued that webservers should not be waiting on I/O activities at all. Instead it should switch to serving other requests and get notified when the slow activity is completed. Using this technique, Node.js could serve many orders of magnitude more users using less memory and on a single thread!

It was becoming increasingly clear that asynchronous event-based architectures are the right way to solve many kinds of concurrency problems. Probably that is why Python’s creator Guido himself worked towards a language level support with the Tulip project, which later became the asyncio module. Eventually Python 3.7 added the new keywords async and await to support asynchronous event loops. This has pretty significant consequences in not just how Python code is written but executed as well.

Two Worlds of Python

Though writing asynchronous code in Python might seem as easy as sliding an async keyword in front of a function definition, you have to be very careful not to break an important rule - Do not freely mix synchronous and asynchronous code.

This is because synchronous code can block an event loop in asynchronous code. Such situations can bring your application to a standstill. As Andrew Goodwin writes this splits your code into two worlds - “Synchronous” and “Asynchronous” with different libraries and calling styles.

When two worlds collide, results can be quite unexpected

Coming back to WSGI, this means we cannot simply write an asynchronous callable and plug it in. WSGI was written for a synchronous world. We will need a new mechanism to invoke asynchronous code. But if everyone writes their own mechanisms we would be back to the incompatibility hell we started with. So we need a new standard similar to WSGI for asynchronous code. Hence, ASGI was born.

ASGI had some other goals as well. But before that let’s look at two similar web applications greeting “Hello World” in WSGI and ASGI style.

In WSGI:

def application(environ, start_response):
    start_response("200 OK", [("Content-Type", "text/plain")])
    return b"Hello, World"

In ASGI:

async def application(scope, receive, send):
    await send({"type": "http.response.start", "status": 200, "headers": [(b"Content-Type", "text/plain")]})
    await send({"type": "http.response.body", "body": b"Hello World"})

Notice the change in the arguments passed into the callables. The scope argument is similar to the earlier environ argument. The send argument corresponds to start_response. But the receive argument is new. It allows clients to nonchalantly slip messages to the server in protocols like WebSockets that allow bidirectional communications.

Like WSGI, the ASGI callables can be chained one after the other to handle web requests (as well as other protocol requests). In fact, ASGI is a superset of WSGI and can call WSGI callables. ASGI also has support for long polling, slow streaming and other exciting response types without side-loading resulting in faster responses.

Thus, ASGI introduces new ways to build asynchronous web interfaces and handle bi-directional protocols. Neither the client or server needs to wait for each other to communicate - it can happen any time asynchronously. Existing WSGI-based web frameworks being written in synchronous code would not support this event-driven way of working.

Django Evolves

This also brings us to the crux of the problem with bringing all the async goodness to Django - all of Django was written in synchronous style code. If we need to write any asynchronous code then there needs to be a clone of the entire Django framework written in asynchronous style. In other words, create two worlds of Django.

Well, don’t panic — we might not have to write an entire clone as there are clever ways to reuse bits of code between the two worlds. But as Andrew Godwin who leads Django’s Async Project rightly remarks “it’s one of the biggest overhauls of Django in its history”. An ambitious project involving reimplementation of components like ORM, request handler, Template renderer etc in asynchronous style. This will be done in phases and in several releases. Here is how Andrew envisions it (not to be taken as a committed schedule):

Django 3.0 - ASGI Server
Django 3.1 - Async Views (see an example below)
Django 3.2/4.0 - Async ORM

You might be thinking what about the rest of the components like Template rendering, Forms, Cache etc. They may still remain synchronous or an asynchronous implementation be fitted somewhere in the future roadmap. But the above are the key milestones in evolving Django to work in an asynchronous world.

That brings us to the first phase.

Django talks ASGI

In 3.0, Django can work in a “async outside, sync inside” mode. This allows it to talk to all known ASGI servers such as:

Daphne - an ASGI reference server, written in Twisted
Uvicorn - a fast ASGI server based on uvloop and httptools
Hypercorn - an ASGI server based on the sans-io hyper, h11, h2, and wsproto libraries

It is important to reiterate that internally Django is still processing requests synchronously in a threadpool. But the underlying ASGI server would be handling requests asynchronously.

This means your existing Django projects require no changes. Think of this change as merely a new interface by which HTTP requests can enter your Django application.

But this is a significant first step in transforming Django from “outside-in”. You could also start using Django on the ASGI server which is usually faster.

How to use ASGI?

Every Django project (since version 1.4) ships with a wsgi.py file, which is a WSGI handler module. While deploying to production, you will point your WSGI server like gunicorn to this file. For instance, you might have seen this line in your Docker compose file

command: gunicorn mysite.wsgi:application

If you create a new Django project (for e.g. created by running the django-admin startproject command) then you will find a brand new file asgi.py alongside wsgi.py. You will need to point your ASGI server (like daphene) to this ASGI handler file. For example, the above line would be changed to:

command: daphene mysite.asgi:application

Note that this requires the presence of an asgi.py file.

Running Existing Django Projects under ASGI

None of the projects created before Django 3.0 have an asgi.py. So how do you go about creating one? It is quite easy.

Here is a side-by-side comparison (docstrings and comments omitted) of the wsgi.py and asgi.py for a Django project:

WSGI	ASGI
import os from django.core.wsgi import get_wsgi_application os.environ.setdefault(‘DJANGO_SETTINGS_MODULE’, ‘mysite.settings’) application = get_wsgi_application()	import os from django.core.asgi import get_asgi_application os.environ.setdefault(‘DJANGO_SETTINGS_MODULE’, ‘mysite.settings’) application = get_asgi_application()

WSGI

ASGI

import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault(‘DJANGO_SETTINGS_MODULE’, ‘mysite.settings’)
application = get_wsgi_application()

import os
from django.core.asgi import get_asgi_application
os.environ.setdefault(‘DJANGO_SETTINGS_MODULE’, ‘mysite.settings’)
application = get_asgi_application()

If you are squinting too hard to find the differences, let me help you - everywhere ‘wsgi’ is replaced by ‘asgi’. Yes, it is as straightforward as taking your existing wsgi.py and running a string replacement s/wsgi/asgi/g.

Gotchas

You should take care not to call any sync code in the ASGI Handler in your asgi.py. For example, if you make a call to some web API within your ASGI handler for some reason, then it must be an asyncio callable.

ASGI vs WSGI Performance

I did a very simple performance test trying out the Django polls project in ASGI and WSGI configurations. Like all performance tests, you should take my results with liberal doses of salt. My Docker setup includes Nginx and Postgresql. The actual load testing was done using the versatile Locust tool.

The test case was opening a poll form in the Django polls application and submitting a random vote. It makes n requests per second when there are n users. The wait time is between 1 and 2 seconds.

Being fast isn't quite enough, you need to avoid failures

The results shown below indicate around 50% increase in the number of simultaneous users when running in ASGI mode compared to WSGI mode.

Users	100	200	300	400	500	600	700
WSGI Failures	0%	0%	0%	5%	12%	35%	50%
ASGI Failures	0%	0%	0%	0%	0%	15%	20%

As the number of simultaneous request ramp up the WSGI or ASGI handler will not be able to cope up beyond a certain point resulting in errors or failures. The requests per second after the WSGI failures start varies wildly. The ASGI performance is much more stable even after failures.

As the table shows, the number of simultaneous users is around 300 for WSGI and 500 for ASGI on my machine. This is about 66% increase in the number of users the servers can handle without error. Your mileage might vary.

Frequent Questions

I did a talk about ASGI and Django at BangPypers recently and there were a lot of interesting questions the audiences raised (even after the event). So I thought I’ll address them here (in no particular order):

Q. Is Django Async the same as Channels?

Channels was created to support asynchronous protocols like Websockets and long polling HTTP. Django applications still run synchronously. Channels is an official Django project but not part of core Django.

Django Async project will support writing Django applications with asynchronous code in addition to synchronous code. Async is a part of Django core.

Both were led by Andrew Goodwin.

These are independent projects in most cases. You can have a project that uses either or both. For example if you need to support a chat application over web sockets, then you can use Channels without using Django’s ASGI interface. On the other hand if you want to make an async function in a Django view, then you will have to wait for Django’s Async support for views.

Q. Any new dependencies in Django 3.0?

Installing just Django 3.0 will install the following into your environment:

$ pip freeze
asgiref==3.2.3
Django==3.0.2
pytz==2019.3
sqlparse==0.3.0

The asgiref library is a new dependency. It contains sync-to-async and async-to-sync function wrappers so that you can call sync code from async and vice versa. It also contains a StatelessServer and a WSGI-to-ASGI adapter.

Q. Will upgrading to Django 3.0 break my project?

Version 3.0 might sound like a big change from its previous version Django 2.2. But that is slightly misleading. Django project does not follow semantic version exactly (where a major version number change may break the API) and the differences are explained in the Release Process page.

You will notice very few serious backward incompatible changes in the Django 3.0 release notes. If your project does not use any of them, then you can upgrade without any modifications.

Then why did the version number jump from 2.2 to 3.0? This is explained in the release cadence section:

Starting with Django 2.0, version numbers will use a loose form of semantic versioning such that each version following an LTS will bump to the next “dot zero” version. For example: 2.0, 2.1, 2.2 (LTS), 3.0, 3.1, 3.2 (LTS), etc.

Since the last release Django 2.2 was long-term support (LTS) release, the following release had to increase the major version number to 3.0. That’s pretty much it!

Q. Can I continue to use WSGI?

Yes. Asynchronous programming could be seen as an entirely optional way to write code in Django. The familiar synchronous way of using Django would continue to work and be supported.

Andrew writes:

Even if there is a fully asynchronous path through the handler, WSGI compatibility has to also be maintained; in order to do this, the WSGIHandler will coexist alongside a new ASGIHandler, and run the system inside a one-off eventloop - keeping it synchronous externally, and asynchronous internally.

This will allow async views to do multiple asynchronous requests and launch short-lived coroutines even inside of WSGI, if you choose to run that way. If you choose to run under ASGI, however, you will then also get the benefits of requests not blocking each other and using less threads

Q. When can I write async code in Django?

As explained earlier in the Async Project roadmap, it is expected that Django 3.1 will introduce async views which will support writing asynchronous code like:

async def view(request):
	await asyncio.sleep(0.5)
    return HttpResponse("Hello, async world!")

You are free to mix async and sync views, middleware, and tests as much as you want; Django will ensure that you always end up with the right execution context. Sounds pretty awesome, right?

At the time of writing, this the patch is almost certain to land in 3.1 now, awaiting a final review.

Wrapping Up

We covered a lot of background about what led to Django supporting asynchronous capabilities. We tried to understand ASGI and how it compares to WSGI. We also found some performance improvements in terms of increased number of simultaneous requests in ASGI mode. A number of frequently asked questions related to Django’s support of ASGI were also addressed.

I believe asynchronous support in Django could be a game changer. It will be one of the first large Python web frameworks to evolve into handling asynchronous requests. It is admirable that it is done with a lot of care not to break backward compatibility.

I usually do not make my tech predictions public (frankly many of them have been proven right over the years). So here goes my tech prediction — Most Django deployments will use async functionality in five years.

That should be enough motivation to check it out!

Thanks to Andrew Goodwin for his comments on an early draft. All illustrations courtesy of Old Book Illustrations

Understanding Django Channels

Tue, 03 Jul 2018 06:55:30 +0530

You are reading a post from a two-part tutorial series on Django Channels

Django Channels

Django Channels was originally created to solve the problem of handling asynchronous communication protocols like say WebSockets. More and more web applications were providing realtime capabilities like chat and push notifications. Various workarounds were created to make Django support such requirements like running separate socket servers or proxy servers.

Channels is an official Django project not just for handling WebSockets and other forms of bi-directional communication but also for running background tasks asynchronously. As of writing, Django Channels 2 is out which is a complete rewrite based on Python 3’s async/await based coroutines.

This article covers the concepts of Django Channels and leads you to a video tutorial implementing a notifications application in Django.

Here is a simplified block diagram of a typical Channels setup:

How a typical Django Channels infrastructure works

A client such as a web browser sends both HTTP/HTTPS and WebSocket traffic to an ASGI server like Daphene. Like WSGI, the ASGI (Asynchronous Server Gateway Interface) specification is a common way for applications servers and applications to interact with each other asynchronously.

Like a typical Django application, HTTP traffic is handled synchronously i.e. when the browser sends a request, it waits until it is routed to Django and a response is sent back. However, it gets a lot more interesting when WebSocket traffic happens because it can be triggered from either direction.

Once a WebSocket connection is established, a browser can send or receive messages. A sent message reaches the Protocol type router that determines the next routing handler based on its transport protocol. Hence you can define a router for HTTP and another for WebSocket messages.

These routers are very similar to Django’s URL mappers but map the incoming messages to a consumer (rather than a view). A consumer is like an event handler that reacts to events. It can also send messages back to the browser, thereby containing the logic for a fully bi-directional communication.

A consumer is a class whose methods you may choose to write either as normal Python functions (synchronous) or as awaitables (asynchronous). Asynchronous code should not mix with synchronous code. So there are conversion functions to convert from async to sync and back. Remember that the Django parts are synchronous. A consumer in fact a valid ASGI application.

So far, we have not used the channel layer. Ironically, you can write Channel applications without using Channels! But they are not particularly useful as there is no easy communication path between application instances, other than polling a database. Channels provides exactly that, a fast point-to-point and broadcast messaging between application instances.

A channel is like a pipe. A sender sends a message to this pipe from one end and it reaches a listener at the other end. A group defines a group of channels who are all listening to a topic. Every consumer listens to own auto-generated channel accessed by its self.channel_name attribute.

In addition to transports, you can trigger a consumer listening to a channel by sending a message, thereby starting a background task. This works as a very quick and simple background worker system.

Building a Channels Application Step-by-step

The following screencast covers the creation of a notification application using Django Channels. You can access the code on Github. The intermediate projects like the Echo Consumer can be accessed as branches of the git repository.

The show notes can be accessed on Github.

Check out the video tutorial and let me know if you found it useful! To know more, read the Django Channels documentation.

This article contains an excerpt from "Django Design Patterns and Best Practices" by Arun Ravindran

Get started with Async & Await

Wed, 14 Mar 2018 06:55:30 +0530

You are reading a post from a two-part tutorial series on Django Channels

Asyncio

Asyncio is a co-operative multitasking library available in Python since version 3.6. Celery is fantastic for running concurrent tasks out of process, but there are certain times you would need to run multiple tasks in a single thread inside a single process.

If you are not familiar with async/await concepts (say from JavaScript or C#) then it involves a bit of steep learning curve. However, it is well worth your time as it can speed up your code tremendously (unless it is completely CPU-bound). Moreover, it helps in understanding other libraries built on top of them like Django Channels.

This post is an attempt to explain the concepts in a simplified manner rather than try to be comprehensive. I want you to start using asynchronous programming and enjoy it. You can learn the nitty gritties later.

All asyncio programs are driven by an event loop, which is pretty much an indefinite loop that calls all registered coroutines in some order until they all terminate. Each coroutine operates cooperatively by yielding control to fellow coroutines at well-defined places. This is called awaiting.

A coroutine is like a special function which can suspend and resume execution. They work like lightweight threads. Native coroutines use the async and await keywords, as follows:

import asyncio


async def sleeper_coroutine():
    await asyncio.sleep(5)


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(sleeper_coroutine())

This is a minimal example of an event loop running one coroutine named sleeper_coroutine. When invoked this coroutine runs until the await statement and yields control back to the event loop. This is usually where an Input/Output activity occurs.

The control comes back to the coroutine at the same line when the activity being awaited is completed (after five seconds). Then then coroutine returns or is considered completed.

Explain async and await

[TLDR; Watch my screencast to understand this section with a lot more code examples.]

Initially, I was confused by the presence of the new keywords in Python: async and await. Asynchronous code seemed to be littered with these keywords yet it was not clear what they did or when to use them.

Let’s first look at the async keyword. Commonly used before a function definition as async def, it indicates that you are defining a (native) coroutine.

You should know two things about coroutines:

Don’t perform slow or blocking operations synchronously inside coroutines.
Don’t call a coroutine directly like a regular function call. Either schedule it in an event loop or await it from another coroutine.

Unlike a normal function call, if you invoke a coroutine its body will not get executed right away. Instead it will be suspended and returns a coroutine object. Invoking the send method of this coroutine will start the execution of the coroutine body.

>>> async def hi():
...     print("HOWDY!")
...
>>> o = hi()
>>> o
<coroutine object hi at 0x000001DAE26E2F68>
>>> o.send(None)
HOWDY!
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>

However, when the coroutine returns it will end in a StopIteration exception. Hence it is better to use the asyncio provided event loop to run a coroutine. The loop will handle exceptions in addition to all other machinery for running coroutines concurrently.

>>> import asyncio
>>> loop = asyncio.get_event_loop()
>>> o = hi()
>>> loop.run_until_complete(o)
HOWDY!

Next we have the await keyword which must be only used inside a coroutine. If you call another coroutine, chances are that it might get blocked at some point, say while waiting for I/O.

>>> async def sleepy():
...     await asyncio.sleep(3)
...
>>> o = sleepy()
>>> loop.run_until_complete(o)
# After three seconds
>>>

The sleep coroutine from asyncio module is different from its synchronous counterpart time.sleep. It is non-blocking. This means that other coroutines can be executed while this coroutine is awaiting the sleep to be completed.

When a coroutine uses the await keyword to call another coroutines, it acts like a bookmark. When a blocking operation happens, it suspends the coroutine (and all the coroutines who are await-ing it) and returns control back to the event loop. Later, when the event loop is notified of the completion of the blocking operation, then the execution is resumed from the await expression paused and continues onward.

Asyncio vs Threads

If you have worked on multi-threaded code, then you might wonder – Why not just use threads? There are several reasons why threads are not popular in Python.

Firstly, threads need to be synchronized while accessing shared resources or we will have race conditions. There are several types of synchronization primitives like locks but essentially, they involve waiting which degrades performance and could cause deadlocks or starvation.

A thread may be interrupted any time. Coroutines have well-defined places where execution is handed over i.e. co-operative multitasking. As a result, you may make changes to a shared state as long as you leave it in a known state. For instance you can retrieve a field from a database, perform calculations and overwrite the field without worrying that another coroutine might have interrupted you in between. All this is possible without locks.

Secondly, coroutines are lightweight. Each coroutine needs an order of magnitude less memory than a thread. If you can run a maximum of hundreds of threads, then you might be able to run tens of thousands of coroutines given the same memory. Thread switching also takes some time (few milliseconds). This means you might be able to run more tasks or serve more concurrent users (just like how Node.js works on a single thread without blocking).

The downsides of coroutines is that you cannot mix blocking and non-blocking code. So once you enter the event loop, rest of the code driven by it must be written in asynchronous style, even the standard or third-party libraries you use. This might make using some older libraries with synchronous code somewhat difficult.

If you really want to call asynchronous code from synchronous or vice versa, then do read this excellent overview of various cases and adaptors you can use by Andrew Godwin.

The Classic Web-scraper Example

Let’s look at an example of how we can rewrite synchronous code into asynchronous. We will look at a webscraper which downloads pages from a couple of URLs and measures its size. This is a common example because it is very I/O bound which shows a significant speedup when handled concurrently.

Synchronous web scraping

The synchronous scraper uses Python 3 standard libraries like urllib. It downloads the home page of three popular sites and the fourth is a large file to simulate a slow connection. It prints the respective page sizes and the total running time.

Here is the code for the synchronous scraper:

# sync.py
"""Synchronously download a list of webpages and time it"""
from urllib.request import Request, urlopen
from time import time

sites = [
    "https://news.ycombinator.com/",
    "https://www.yahoo.com/",
    "https://github.com/",
]


def find_size(url):
    req = Request(url)
    with urlopen(req) as response:
        page = response.read()
        return len(page)


def main():
    for site in sites:
        size = find_size(site)
        print("Read {:8d} chars from {}".format(size, site))


if __name__ == '__main__':
    start_time = time()
    main()
    print("Ran in {:6.3f} secs".format(time() - start_time))

On a test laptop, this code took 5.4 seconds to run. It is the cumulative loading time of each site. Let’s see how asynchronous code runs.

Asynchronous web scraping

This asyncio code requires installation of a few Python asynchronous network libraries such as aiohttp and aiodns. They are mentioned in the docstring.

Here is the code for the asynchronous scraper – it is structured to be as close as possible to the synchronous version so it is easier to compare:

# async.py
"""
Asynchronously download a list of webpages and time it

Dependencies: Make sure you install aiohttp using: pip install aiohttp aiodns
"""
import asyncio
import aiohttp
from time import time

# Configuring logging to show timestamps
import logging
logging.basicConfig(format='%(asctime)s %(message)s', datefmt='[%H:%M:%S]')
log = logging.getLogger()
log.setLevel(logging.INFO)

sites = [
    "https://news.ycombinator.com/",
    "https://www.yahoo.com/",
    "https://github.com/",
]


async def find_size(session, url):
    log.info("START {}".format(url))
    async with session.get(url) as response:
        log.info("RESPONSE {}".format(url))
        page = await response.read()
        log.info("PAGE {}".format(url))
        return url, len(page)


async def main():
    tasks = []
    async with aiohttp.ClientSession() as session:
        for site in sites:
            tasks.append(find_size(session, site))
        results = await asyncio.gather(*tasks)
    for site, size in results:
        print("Read {:8d} chars from {}".format(size, site))


if __name__ == '__main__':
    start_time = time()
    loop = asyncio.get_event_loop()
    loop.set_debug(True)
    loop.run_until_complete(main())
    print("Ran in {:6.3f} secs".format(time() - start_time))

The main function is a coroutine which triggers the creation of a separate coroutine for each website. Then it awaits until all these triggered coroutines are completed. As a best practice, the web session object is passed to avoid re-creating new sessions for each page.

The total running time of this program on the same test laptop is 1.5 s. This is a speedup of 3.6x on the same single core. This surprising result can be better understood if we can visualize how the time was spent, as shown below:

A simplistic representation comparing tasks in the synchronous and asynchronous scrapers

The synchronous scraper is easy to understand. Scraping activity needs very little CPU time and the majority of the time is spent waiting for the data to arrive from the network. Each task is waiting for the previous task to complete. As a result the tasks cascade sequentially like a waterfall.

On the other hand the asynchronous scraper starts the first task and as soon as it starts waiting for I/O, it switches to the next task. The CPU is hardly idle as the execution goes back to the event loop as soon as the waiting starts. Eventually the I/O completes in the same amount of time but due to the multiplexing of activity, the overall time taken is drastically reduced.

In fact, the asynchronous code can be speeded up further. The standard asyncio event loop is written in pure Python and provided as a reference implementation. You can consider faster implementations like uvloop for further speedup (my running time came down to 1.3 secs).

Concurrency is not Parallelism

Concurrency is the ability to perform other tasks while you are waiting on the current task. Imagine you are cooking a lot of dishes for some guests. While waiting for something to cook, you are free to do other things like peeling onions or cutting vegetables. Even when one person cooks, typically there will be several things happening concurrently.

Parallelism is when two or more execution engines are performing a task. Continuing on our analogy, this is when two or more cooks work on the same dish to (hopefully) save time.

It is very easy to confuse concurrency and parallelism because they can happen at the same time. You could be concurrently running tasks without parallelism or vice versa. But they refer to two different things. Concurrency is a way of structuring your programs while Parallelism refers to how it is executed.

Due to the Global Interpreter Lock, we cannot run more than one thread of the Python interpreter (to be specific, the standard CPython interpreter) at a time even in multicore systems. This limits the amount of parallelism which we can achieve with a single instance of the Python process.

Optimal usage of your computing resources require both concurrency and parallelism. Concurrency will help you avoid idling the processor core while waiting for say I/O events. While parallelism will help distribute work among all the available cores.

In both cases, you are not executing synchronously i.e. waiting for a task to finish before moving on to another task. Asynchronous systems might seem to be the most optimal. However, they are harder to build and reason about.

Why another Asynchronous Framework?

Asyncio is by no means the first cooperative multitasking or light-weight thread library. If you have used gevent or eventlet, you might find asyncio needs more explicit separation between synchronous and asynchronous code. This is usually a good thing.

Gevent, relies on monkey-patching to change blocking I/O calls to non-blocking ones. This can lead to hard to find performance issues due to an unpatched blocking call slowing the event loop. As the Zen says, ‘Explicit is better than Implicit’.

Another objective of asyncio was to provide a standardized concurrency framework for all implementations like gevent or Twisted. This not only reduces duplicated efforts by library authors but also ensures that code is portable for end users.

Personally, I think the asyncio module can be more streamlined. There are a lot of ideas which somewhat expose implementation details (e.g. native coroutines vs generator-based coroutines). But it is useful as a standard to write future-proof code.

Can we use asyncio in Django?

Strictly speaking, the answer is No. Django is a synchronous web framework. You might be able to run a seperate worker process, say in Celery, to run an embedded event loop. This can be used for I/O background tasks like web scraping.

However, Django Channels changes all that. Django might fit in the asynchronous world after all. But that’s the subject of another post.

This article contains an excerpt from "Django Design Patterns and Best Practices" by Arun Ravindran

Interview with Daniel Roy Greenfeld (PyDanny)

Fri, 09 Feb 2018 15:15:11 +0530

Daniel Roy Greenfeld needs no introduction to Djangonauts. Co-author of the book Two Scoops of Django which is probably on the shelves of any serious Django practitioner. But PyDanny, as he is fondly known as, is also a wonderful fiction author, fitness enthusiast and a lot more.

Having known Daniel for a while as a wonderful friend and a great inspiration, I am so excited that he agreed to my interview. Let’s get started…

How did the idea of writing an ice-cream themed book occur?

The first 50 pages that I wrote were angry and were for a book with an angry name. You see, I was tired of having to pick up after sloppy or unwise coding practices on rescue projects. I was furious and wanted to fix the world.

However, I was getting stuck in what I wanted to say, or didn’t know things. I kept asking my normal favorite resource for help, Audrey Roy Greenfeld. Eventually she started to write (or rewrite) whole sections and I realized that I wasn’t writing the book alone.

Therefore I asked Audrey to be my co-author. She’s a cheerful person and said that if she were to accept, the book had to be lightened. That meant changing the name. After a lot of different title names discussed over many ice cream sessions, we decided to use the subject matter at hand. Which worked out well as the ice cream theme made for a good example subject.

How do you and Audrey collaborate while writing a book?

We take turns writing original material that interests us. The other person follows them and acts as editor and proofreader. We go back and forth a few hundred times and there you go.

For tech writing we use Git as version control and LaTeX for formatting. For fiction we use Google docs followed by some Python scripts that merge and format the files.

What’s the most exciting recent development in Django? Where do you think it can improve?

I like the new URL system as it’s really nice for beginners and advanced coders alike. While I like writing regular expressions, I’m the exception in this case.

Where I think where Django can improve is having more non-US/Europe/Australian representation within the DSF and in the core membership. In short, most of Django core looks like me, and I think that’s wrong. Many of the best Django (and Python) people look nothing like me, and they deserve more recognition. While having Anna Makarudze on the DSF board is a wonderful development, as a community we can still do better in core.

In the case of Django’s core team, I believe this has happened because all the major Django conferences are in the US, Europe, and Australia, and from what I’ve seen over the years it’s through participation in those events is how most people get onto the Django core team. The DSF is aware of the problem, but I think more people should raise it as an issue. More people vocalizing this as a problem will get it resolved more quickly.

With the Ambria fantasy series, you have proven to be a prolific fiction author too. Reminds me Lewis Carroll who wrote children’s books and mathematical treatises. What is the difference in the writing process while writing fiction and non-fiction?

For us, the process is very similar. We both write and we both review our stuff. The difference is that if we make a mistake in our fiction, it’s not as critical. That means that the review process for fiction is a lot easier on us then it is to write technical books or articles. I can’t begin to tell you what a load that is off my shoulders.

Why fantasy? Any literary influences?

We like fantasy because we can just let our imaginations run away with us. For the Ambria series, our influences include Tolkien, Joseph Campbell, Glen Cook, Greek mythology, and various equine and religious studies.

Do you have a daily writing routine?

Like coding on a fun project, when we get to write, we get up early and just start working. When we get hungry or thirsty we stop. The day seems to fly by and we are very happy. We try not to mix writing days with coding days, as we like to focus on one thing at a time. Neither of us are big believers in multi-tasking, so sticking to one thing is important to us.

What’s your favorite part of the writing process?

Getting to write with my favorite person in the whole world, Audrey Roy Greenfeld. :-)

Also, having people read our stuff and comment on it, both positively and negatively.

Do you ever get writer’s block?

Not usually. Our delays are almost always because of other things getting in the way. We’re very fortunate that way!

When I do get writers block, I try to do something active. Be it exercise or fix something in the house that needs it.

Considering you can do cartwheels, I am assuming you are pretty fit. Do you think technology folks don’t give it enough importance?

I’m older than I look but even dealing with an unpleasant knee injury move faster and better than 90% of software developers. And when I look at other coders my age, I see people old before their years. I believe youth is fleeting unless you take a little bit of time every day to keep your strength and flexibility.

Anything else you would like to say?

To paraphrase Jurassic Park, “Just because you can do a thing doesn’t mean you should do a thing.”

As software developers, we have skills that let us do amazing things. With enough time and experience, we can do pretty much anything we are asked to do. That said, we should consider whether or not we should always do what we are asked to do.

For example, the combined power of image recognition, big data, and distributed systems is really fun to play with, but we need to be aware that these tools can be dangerous. In the past year we’ve seen it used to affect opinion and elections, and this is only the beginning. It’s our responsibility to the future to be aware that the tools we are playing with have a lot of power, and that the people who are paying us to use them might not have the best intentions.

Hence why I like to say, “Just because you can do a thing doesn’t mean you should do a thing.”

Do checkout “Two Scoops of Django 1.11: Best Practices for the Django Web Framework” by Two Scoops Press

Django Release Schedule and Python 3

Tue, 30 Jan 2018 06:59:49 +0530

Do long term releases confuse you? For the longest time I was not sure which version of Ubuntu to download - the latest release or the LTS? I see a number of Django developers confused about Django’s releases. So I prepared this handy guide to help you choose (or confuse?).

Which Version To Use?

Django has now standardized on a release schedule with three kinds of releases:

Feature Release: These releases will have new features or improvements to existing features. It will happen every 8 months and will have 16 months of extended support from release. They have version numbers like A.B (note there’s no minor version).

Long-Term Support (LTS) Release: These are special kind of feature releases, which have a longer extended support of three years from the release date. These releases will happen every two years. They have version numbers like A.2 (since every third feature release will be a LTS). LTS releases have few months of overlap to aid in a smoother migration.

Patch Release: These releases are bug fixes or security patches. It is recommended to deploy them as soon as possible. Since they have minimal breaking changes, these upgrades should be painless to apply. They have version numbers like A.B.C

Django roadmap visualized below should make the release approach clearer:

The dates are indicative and may change. This is not an official diagram but something that I created for my understanding.

The big takeaway is that Django 1.11 LTS will be the last release to support Python 2 and it is supported until April 2020. Subsequent versions will use only Python 3.

The right Django version for you will be based on how frequent you can upgrade your Django installation and what features you need. If your project is actively developed and the Django version can be upgraded at least once in 16 months, then you should install the latest feature release regardless of whether it is LTS or non-LTS.

Otherwise, if your project is only occasionally developed then you should pick the most recent LTS version. Upgrading your project’s Django dependency from one feature release to another can be a non-trivial effort. So, read the release notes and plan accordingly.

In any case, make sure you install Patch Releases as soon as they are released. Now, if you are still on Python 2 then read on.

Python 3 has crossed tipping point

When I decided to use Python 3 only while writing my book “Django Design Patterns and Best Practices” in 2015, it was a time when Python 2 versus Python 3 was hotly debated. However, to me Python 3 seemed much more cleaner without arcane syntax like class methods named __unicode__ and classes needing to derive from object parent class.

Now, it is quite a different picture. We just saw how Django no longer supports Python 2 except for the last LTS release. This is a big push for many Python shops to consider Python 3.

Many platforms have upgraded their default Python interpreter. Starting 1st March 2018, Python 3 is announced to be the default “python” in Homebrew installs. ArchLinux had completely switched to Python 3 since 2010.

Fedora has switched to Python 3 as its system default since version 23. Even though python command will launch python3, the symlink /usr/bin/python will still point to python2 for backward compatibility. So it is probably a good idea to use #!/usr/bin/env python idiom in your shell scripts.

On 26 April 2018, when Ubuntu 18.04 LTS (Bionic Beaver) will be released, it is planned to be have Python 3.6 as default. Further upstream, the next Debian release in testing - Debian 10 (Buster) is expected to transition to Python 3.6.

Moving to packages, the Python 3 Wall of Superpowers shows that with the backing of 190 out of 200 packages, at the time of writing, we have nearly all popular Python packages on Python 3. The only notable package remaining is supervisor, which is about to turn green in supervisor 4.0 (unreleased).

Common Python 3 Migration Blockers

You might be aware of atleast one project which is still on Python 2. It could be open source or an internal project, which may be stuck in Python 3 for a number of reasons. I’ve come across a number of such projects and here is my response to such reasons:

Reason 1: My Project is too complex

Some very large and complex projects like NumPy or Django have been migrated sucessfully. You can learn the migration strategies of projects like Django. Django maintained a common codebase for Python 2 and 3 using the six (2 × 3=6, get it?) library before switching to Python 3 only.

Reason 2: I still have time

It is closer than you think. Python clock shows there is a little more than 2 years and 2 months left for Python 2 support.

In fact, you have had a lot of time. It has been ten years since Python 3 was announced. That is a lot of overlap to transition from one version to another.

In today’s ‘move fast and break things’ world, a lot of projects decide to abruptly stop support and ask you to migrate as soon as a new release is out. This is a lot more realistic assumption for enterprises which need a lot more planning and testing.

Reason 3: I have to learn Python 3

But you already know most of it! You might need about 10 mins to learn the differences. In fact, I have written a post to guide Django coders to Python 3. Small Django/Python 2 projects need only trivial changes to work on Python 3.

You might see many old blog posts about Python 3 being buggy or slow. Well, that has not been true for a while. Not only it is extremely stable and bug-free, it is actually used in production by several companies. Performance-wise it has been getting faster in every release, so it is faster than Python 2 in most cases and slower in a few.

Of course, there are lot of awesome new features and libraries added to Python 3. You can learn them as and when you need them. I would recommend reading the release notes to understand them. I will mention my favourites soon.

Reason 4: Nobody is asking

Some people have the philosophy that if nobody is asking then nobody cares. Well, they do care if the application they run is on an unsupported technology. Better plan for the eventual transition than rush it on a higher budget.

Are you missing out?

To me, the biggest reason to switch was that all the newest and greatest features were coming to Python 3. My favourite top three exclusive features in Python 3 are:

asyncio: One of the coolest technologies I picked up recently. The learning process is sometimes mind-bending. But the performance boost in the right situations is incredible.
f-strings: They are so incredibly expressive that you would want to use them everywhere. Instant love!
dict: New compact dict implementation which uses less memory and are ordered!

Yes, FOMO is real.

Apart from my personal reasons, I would recommend everyone to migrate so that the community benefits from investing efforts into a common codebase. Plus we can all be consistent on which Python to recommend to beginners. Because…

There should be one– and preferably only one –obvious way to do it. Although that way may not be obvious at first unless you’re Dutch.

This article contains an excerpt from the upcoming second edition of book "Django Design Patterns and Best Practices" by Arun Ravindran

Python 3 Cheatsheet for Djangonauts

Wed, 20 May 2015 19:08:00 +0530

If you are already convinced to use Python 3 then you can directly jump to the section “Python 2 vs Python 3” for the cheatsheet

There is an intense debate whenever Python 3 is mentioned. Some think it is an unnecessary shift and Python 2 series should be continued. However, I chose Python 3 for my book because I strongly believe that it is the future of the language. Besides, the divide might not be as large you think.

Why Python 3?

Think of your favourite programming language and what you love about it. Now think of things that you don’t like about it or wish was improved. Sometimes these improvements might break older codebases but you badly want them.

Now imagine if the creator of the language takes upon himself/herself to revamp the language and rid of its warts. This is what actually led to Python 3 - Guido led the efforts to clean-up some fundamental design flaws with the 2.x series based on over fifteen years of experience developing a successful language.

While the development of Python 3 started in 2006, its first release, Python 3.0, was on December 3, 2008. The main reasons for a backwards incompatible version were – switching to Unicode for all strings, increased use of iterators, cleanup of deprecated features like old-style classes and some new syntactic additions like the nonlocal statement.

Python 2.7 development was supposed to end in 2015 but was extended for five more years through 2020. There will not be a Python 2.8. Soon all major Linux distributions would have completely switched to using Python 3 as a default. Many PaaS providers like Heroku already support Python 3.4.

Python 2 and Python 3 have been in parallel development by the core development team for a number of years. Eventually, Python 3 is expected to be the future of the language.

Porting Django

The reaction to Python 3 in the Django community was rather mixed. Even though the language changes between version 2 and 3 were small (and over time, reduced), porting the entire Django codebase was a significant migration effort.

Django has been supporting Python 3 since version 1.5. In fact, the strategy was to rewrite the code into Python 3 and deal with Python 2 as a backward compatibility requirement.

Most of the packages listed in the Python 3 Wall of Superpowers have turned green (indicating they have support for Python 3). Almost all the red ones have an actively developed Python 3 port. Also worth noting is the Django Wall of Superpowers with over 67% of the top 200 Django packages having a Python 3 port.

Python 3 Advantage

Many newcomers to Django wonder whether to start using Python 2 or 3. I recommend starting with Python 3 for the following reasons:

Better syntax: It fixes a lot of ugly syntax like izip, xrange and __unicode__ with the cleaner and more straightforward zip, range and __str__.
Sufficient third-party support: Of the top 200 third party libraries, more than 80% have Python 3 support.
No legacy code: Since they are creating a new project rather than dealing with legacy code which needs to support an older version.
Default in Modern Platforms: It is already the default Python interpreter in Arch Linux. Ubuntu and Fedora plan to complete the switch in a future release.
It is easy: From a Django development point of view, there are so few changes that it can be learnt in a few minutes.

The last point is important. Even if you know Python 2, you can pick up Python 3 in a short time.

Python 2 vs Python 3

This section covers the most important changes in Python 3 from a Django developer’s perspective. To understand the full list of changes, please refer to the recommended reading section in the end.

The examples are given in both Python 2 and Python 3. Depending on your installation, all Python 3 commands might need to be changed from python to python3 or python3.4 to invoke the interpreter.

Change all unicode methods into str

In Python 3, the __str__() method is called for string representation of your models rather than the awkward sounding __unicode__() method. This is one of the most evident ways to identify Python 3 ported code.

Python 2	Python 3
class Person(models.Model): name = models.TextField() def unicode(self): return self.name	class Person(models.Model): name = models.TextField() def str(self): return self.name

This reflects the difference in the way Python 3 treats strings. In Python 2, the human readable representation of a class could be returned by __str__() (bytes) or __unicode__() (text). However, in Python 3 the readable representation is simply returned by __str__() (text).

All Classes Inherit from object

Python 2 has two kinds of classes – old-style (classic) and new-style. New-style classes are classes which directly or indirectly inherit from object. Only new style classes can use Python’s advanced features like slots, descriptors and properties. Many of these are used by Django. However, classes were still old-style by default for compatibility reasons.

In Python 3, the old-style classes don’t exist anymore. As seen below, even if you don’t explicitly mention any parent classes the object class will be present as a base. So, all classes are new-style.

Python 2	Python 3
>>> class CoolMixin: … pass >>> CoolMixin.bases ()	>>> class CoolMixin: … pass >>> CoolMixin.bases (<class ‘object’>,)

Calling super() is Easier

The simpler call to super(), without any arguments, will save you some typing in Python 3.

Python 2	Python 3
class CoolMixin(object): def do_it(self): return super(CoolMixin, self).do_it()	class CoolMixin: def do_it(self): return super().do_it()

Specifying the class name and instance is optional, thereby making your code DRY and less prone to errors while refactoring.

Relative Imports Must be Explicit

Imagine the following directory structure for a package named app1:

/app1
  /__init__.py
  /models.py
  /tests.py

Now, in Python 3, let us run the following in the parent directory of app1:

$ echo "import models" > app1/tests.py
$ python -m app1.tests
Traceback (most recent call last):
   ... omitted ...
ImportError: No module named 'models'
$ echo "from . import models" > app1/tests.py
$ python -m app1.tests

Successfully import-ed

Within a package, you should use explicit relative imports while referring to a sibling module. You can omit __init__.py in Python 3, though it is commonly used to identify a package.

In Python 2, you could use import models to successfully import models.py module. But it was ambiguous and could accidentally import any other models.py in your Python path. Hence this is forbidden in Python 3 and discouraged in Python 2 as well. Use from . import models instead.

HttpRequest and HttpResponse have str and bytes Types

In Python 3, according to PEP 3333 (amendments to the WSGI standard), we are careful not to mix data coming from or leaving via HTTP, which will be in bytes; as opposed to text within the framework, which will be native (Unicode) strings.

Essentially, for HttpRequest and HttpResponse objects:

Headers will be always str objects
Input and output streams will be always byte objects

Unlike Python 2, the string and bytes are not implicitly converted while performing comparisons or concatenations with each other. Strings mean Unicode strings only.

Exception Syntax Changes and Improvements

Exception handling syntax and functionality has been significantly improved in Python 3.

In Python 3, you cannot use the comma separated syntax for the except clause. Use the as keyword instead:

Python 2	Python 3
try: pass except BaseException, e: pass	try: pass except BaseException as e: pass

The new syntax is recommended for Python 2 as well.

In Python 3, all exceptions must be derived (directly or indirectly) from BaseException. In practice, you would create your custom exceptions by deriving from the Exception class.

As a major improvement in error reporting, if an exception occurs while handling an exception then the entire chain of exceptions are reported:

Python 2	Python 3
>>> try: ... print(undefined) ... except Exception: ... print(oops) ... Traceback (most recent call last): File "", line 4, in <module> NameError: name 'oops' is not defined	>>> try: ... print(undefined) ... except Exception: ... print(oops) ... Traceback (most recent call last): File "<stdin>", line 2, in <module> NameError: name 'undefined' is not defined During the handling of the preceding exception, another exception occurred: Traceback (most recent call last): File “<stdin>”, line 4, in <module> NameError: name ‘oops’ is not defined

Python 2

Python 3

>>> try:
...   print(undefined)
... except Exception:
...   print(oops)
...
Traceback (most recent call last):
  File "", line 4, in <module>
  NameError: name 'oops' is not defined

>>> try:
...   print(undefined)
... except Exception:
...   print(oops)
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  NameError: name 'undefined' is not defined
During the handling of the preceding
exception, another exception occurred:
Traceback (most recent call last):
File “<stdin>”, line 4, in <module>
NameError: name ‘oops’ is not defined

Once you get used to this feature, you will definitely miss it in Python 2.

Standard Library Reorganized

The core developers have cleaned-up and better organized the Python standard library. For instance SimpleHTTPServer now lives in the http.server module:

Python 2	Python 3
$ python -m SimpleHTTPServer Serving HTTP on 0.0.0.0 port 8000 ...	$ python -m http.server Serving HTTP on 0.0.0.0 port 8000 ...

New Goodies

Python 3 is not just about language fixes. It is also where bleeding-edge Python development happens. This means improvements to the language in terms of syntax, performance and built-in functionality.

Some of the notable new modules added to Python 3 are:

asyncio : Asynchronous I/O, event loop, coroutines and tasks
unittest.mock : Mock object library for testing
pathlib : Object-oriented filesystem paths
statistics : Mathematical statistics functions

Even if some of these modules might have backports to Python 2, it is more appealing to migrate to Python 3 and leverage them as built-in modules.

Pyvenv and Pip are Built-in

Most serious Python developers prefer to use virtual environments. virtualenv is quite popular to isolate your project setup from the system-wide Python installation. Thankfully, Python 3.3 is integrated with a similar functionality using the venv module.

Since Python 3.4, a fresh virtual environment will be pre-installed with pip, a popular installer:

$ python -m venv djenv
[djenv] $ source djenv/bin/activate
[djenv] $ pip install django

Notice that the command prompt changes to indicate that your virtual environment has been activated.

Other Changes

We cannot possibly fit all the Python 3 changes and improvements in this post. But the other commonly cited changes are:

print is now a function : Previously it was a statement i.e. arguments were not in parenthesis.
Integers don’t overflow : sys.maxint is outdated, integers will have unlimited precision.
Inequality operator <> is removed : Use != instead.
True Integer Division : In Python 2, 3 / 2 would evaluate to 1. It will be correctly evaluated to 1.5 in Python 3.
Use range instead of xrange : range() will now return iterators as xrange() used to work before.
Dictionary keys are views : dict and dict-like classes (like QueryDict) will return iterators instead of lists for keys(), items() and values() method calls.

This article is an excerpt from the book "Django Design Patterns and Best Practices" by Arun Ravindran

Further Information

Read What’s New In Python 3.0 by Guido https://docs.python.org/3/whatsnew/3.0.html
To find what is new in each release of Python, read _What’s New in Python_https://docs.python.org/3/whatsnew/
For richly detailed answers about Python 3 read Python 3 Q & A by Nick Coghlan http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_answers.html