Home pages that morph

    You are reading a post from a multi-part series of articles

    1. Staying on the bleeding edge
    2. Home pages that morph

    Have you ever struggled to create a beautiful landing page for your Django site?

    A common ask by users of the Edge project template was the support for home pages that had a completely different layout than the inner pages. For example, a first time visitor opening https://example.com should see a landing page that shows the benefits of signing up. But once signed-in the same url https://example.com should show the user interface of the web application.

    This is a common pattern today. For example this is how different github.com looks to a first time visitor and a logged in user:

    Github-width50
    Github homepage before signing in and after signing in (two-frame animation)

    What’s a Landing Page

    A landing page is a standalone page created for a specific marketing conversion goal, like subscribing to a newsletter or ebook downloads. For the purpose of this article, we will focus on the conversion goal of signing up users to your website or online product.

    A good landing page must be convincing enough for a visitor to share their personal details like their name and email. Even then, ask as few details as possible in the sign up form. If it is not relevant like the Company field, it is best to remove it. Expedia dropped the “Company” field from their booking form and saw an increase of $12 million a year in profit.

    It must also be optimized for quick scanning. A widely quoted but often misunderstood study by Nielsen Norman Group showed how reading on the web is F-shaped. This is often bad for users and businesses since it skips important content. The recommended solution is to include important content in the first two paragraphs and have headings with the first two words bearing the most content.

    F-shaped
    F-shaped reading pattern in heat maps

    A homepage can be distinct from a landing page. But for many visitors who come to know about your site say through a link shared on a social network or an ad, the home page is the gateway to your product. So many sites check if the user has not logged in and show a landing page instead.

    Challenges in Django

    The challenge for most Django sites is that this requires extensive changes to the base template. A base template defines the basic structure of the site. Django documentation explains it further - a base “skeleton” template contains all the common elements of your site and defines blocks that child templates can override.

    The landing page differs considerably in content and layout from the inner pages. Here are some of the main differences:

    • Objectives: The primary purpose of the landing page is to typically get users to sign up. The entire page is focussed on convincing the reader to perform the Call to Action (CTA) of clicking the “Sign up” button. Once you have signed up, your objective changes to give a good user experience while using the product i.e. the web application.
    • Layout: Landing pages typically have a simplified and vertical layout where you are scrolling down to view engaging visuals and exciting animations. However a logged in user typically needs a more functional interface where various interface elements like sidebars are always within reach.
    • Navigation: Most modern landing pages remove the top menu entirely along with other links to internal and external pages to simplify the experience. It does not make sense to show the application menu or the user profile drop down to a visitor who has not yet signed up.
    • Assets: Landing pages, like other marketing pages, have scripts and other assets that track the effectiveness of pages like click through rates and conversion metrics. The inner pages would have different assets needed to create and maintain an effective user interface.
    • Optimizations: According to a study of the top 1000 websites, landing pages are 35% heavier but load faster by 56% than internal pages. This could be because the landing page needs to give a much more optimized user experience as it forms the crucial “first impression”.

    In theory, a Django application could be crafted with a base template that can accommodate both a landing page and a web application layout. In practice it will be extremely cumbersome. As you have seen these two kinds of pages often have nothing in common. They might even be designed by two different teams. So, given a single base template you will often end up in overriding most of the blocks. There has to be a better way .

    It’s morphin' time!

    Before we go further, it will be useful to give a name to this common pattern. Despite a lot of research, I couldn’t find any existing literature that names this pattern. So I went ahead and called it a Dimorphic Home Page pattern. The word Dimorphic means a thing that exists in two different forms. In this case a home page that changes form depending on whether the user has been authenticated or not.

    Dimorphic materials
    Dimorphic materials like Calcite (left) and Aragonite (right) is the same compound (Calcium carbonate) existing in two forms or crystal structures. Pic courtesy: Wikipedia

    Here is a simple view (taken from Edge source ) that implements this solution:

    from django.views.generic.base import TemplateView
    
    
    class HomePageView(TemplateView):
    	user_template_name = "users/home.html"
    	anon_template_name = "anons/home.html"
    
    	def get_template_names(self):
        	if self.request.user.is_authenticated:
            	return self.user_template_name
        	else:
            	return self.anon_template_name
    

    We have used Class Based Generic Views instead of Function Based Views because they are easy to extend. For instance, if you need to pass a signup form in the context variable (and remove another page reload from conversion) then another view class can be derived easily.

    In terms of organization, the templates for unauthenticated and authenticated users are kept in separate directories named say anon and users respectively. This makes it easier to prevent unintentional leakage of sensitive data in unauthenticated pages.

    Designing a Beautiful Landing Page

    The landing page of Edge has been designed based on modern landing page best practices. If you study several actual landing pages, you would observe many of them use common animation components like animate.css and wow.js. This is what powers the sliding and dancing images or text as you scroll down revealing new sections.

    The dimorphic Edge homepage is a work in progress but it currently looks like this:

    Edge-width50
    Django Edge homepage before signing in and after signing in (two-frame animation). Hero image courtesy Scale by Flexiple

    Design sensibilities change over time. But the template should be a good starting point for solo Django developers who are not familiar with design. It would be less daunting than starting from a blank HTML document.

    Try what is new

    Check out the Edge Django 3.x branch with its dimorphic landing page. Follow the instructions in the README to create a quick Django project. We welcome your feedback and contributions.

    Comments →

    Print from 1 to 100 without loops or numbers in Python

    A new programming problem has been trending recently - write a program to print numbers from one to hundred without using any loops or mentioning numbers in the source. Naturally, I was interested in a solution in Python.

    I got many brilliant answers on Twitter. But first let me show you how I approached it. I was looking for a general and readable solution than the shortest one. My thought process was – even if we cannot mention numbers we can convert non-numeric types to numbers. So I tried with Booleans and Strings:

    zero = int(False)
    one = int(True)
    hundred = int(f"{one}{zero}{zero}")
    
    
    def shownum(i):
        if i <= hundred:
            print(i)
            shownum(i + one)
    
    
    shownum(one)
    

    Starting from scratch, we get 0 and 1 from the False and True values. Then to create 100, the upper limit of the loop, we use the new favourite string interpolation method – f-strings.

    Overcoming the limitation of not using loops was quite straightforward - just use Recursion. We have to be careful though, Python does not have Tail Call Optimization (since Guido prefers to have proper tracebacks), so if you keep increasing the upper limit of the loop you will end up in a stack overflow.

    Whiz kids on Twitter

    To my poser tweet yesterday, there were many wonderful solutions that were way shorter than mine. Here are some of them:

    The Long Scream by Abhiram

    What I like here is the clever use of the * operator to convert the range object into a list tersely. And of course the liberal use of the “a”-s that just screams “I cannot be unseen”.

    Short and Succinct by Rohan

    Cleverly taking advantage of Python’s built-in ord function to get the ASCII code of lower case ‘e’ (it is 101 not 100), we sort of have a code-golf winner. Sort of because it starts count from 0 not 1 as the original question posed. But this is Twitter, a correction soon emerged.

    Note that the map and int functions are not really required here, making it even more shorter! In a direct message Rohan shared an improved solution that starts counting from one:

    list(filter(lambda x: x,range(ord(b"e"))))
    

    Evolved by Anirudh

    With a minor correction, this code fixes the start count to 1. As a nice touch the strings form ‘bae’ which must be the cutest term for a friend.

    Can This Get Shorter by Arun

    With the benefit of having seen all the ideas, I can finally put forth my shortest solution (with 23 characters):

    [*range(True,ord("e"))]
    

    So that’s the end of this code golf for me. That is, until someone else comes up with something shorter!

    Comments →

    Staying on the bleeding edge

    You are reading a post from a multi-part series of articles

    1. Staying on the bleeding edge
    2. Home pages that morph

    How fast can you go from getting an amazing product idea to bringing it in front of real users? It is a process that has several steps - some fun but mostly boring. It could also take a long time. In fact the longer it takes, the lower your motivation levels dip and competition starts looming large. This is why we use productivity enhancers like frameworks, libraries and templates not just to reach the users faster but also to ensure that we ship it.

    Django Edge is a Django project starter template that I started in 2014 with the idea of making web apps faster using my go-to toolset - Django, Bootstrap and many of the most useful Django libraries. The project is quite popular with over 770 stars on Github. Unlike many other starter templates, it shipped with some essential pages like a homepage, register, login etc that were presentable and working.

    An update to Edge has been long overdue. But the Django landscape has changed a bit. For a long time, the most common use of the framework was as a back-end for a front-end JavaScript framework like React or Vue (and any other possible front-ends like a mobile app). This typically uses the excellent Django Rest Framework (DRF) to build a REST API which the front-end consumes. This could be one possible direction that Edge could take.

    However the use of state-heavy front-end frameworks with millions of dependencies has led to Javascript fatigue. The trend is somewhat coming back to back-end rendered pages. Choosing “Boring Technology” might counter-intuitively leave lots of room for innovation in the areas that you may find fun and interesting. There is really no need to throw away Django Templates if you need a single page application thanks to HTMX. So continuing to ship with beautifully designed templates is another direction that Edge could take as well.

    I was tempted to try this “Double-edged” approach for a moment. But then I realized that it is already a lot of work to maintain just one open source project. Users who need Django for a REST API will probably find it incomplete unless it is also married with Rest, Vue, Angular or some other shiny new JavaScript framework out of the box. This is a truly fragmented base that keeps switching new frameworks or even abandoning all frameworks.

    Hence I went back to the familiar Edge. Provide a fully working web application starter with the best of what Django can provide. Aim to be a good starting point for single developer projects. Focus on the long requested features like social authentication or Docker and improve on what it currently does.

    Django Edge Beta
    Yay!...a working login page

    So this is probably the first of many posts where I share what I am planning to do with Edge. I believe that open development is the best way to build open source projects. Try the beta here: https://github.com/arocks/edge/tree/django3

    Pull requests are welcome! 😊

    Comments →

    Fitting a Django Application in One File

    Earlier this week, Anthony a french economics university student wanted to talk to me over Zoom about my ray tracer tutorials. He and his friend were new to Python but were excited to implement their own ray tracer after following my videos. One of the questions that popped up in the conversation was - “Can we put all the classes in one file instead of breaking it into individual files per class?”. I said, “Of course” and noticed a wave of relief in their faces. In fact, I explained, my earlier implementation was all in one file and later broken up for better pedagogy.

    But the charm of an entire project in a single file is compelling. I remember seeing a Sinatra web application a few years ago containing the entire application and assets like HTML templates and CSS in a single file. Presenting all the components in the same file gave a complete high-level overview of the project by simply scrolling up and down.

    Normally, at this point, someone would suggest a microframework. But it is not that easy.

    Microframeworks

    Microframeworks take a minimalistic approach by omitting certain components or directing you to a few recommended components. For instance, Bottle contains basic form handling capabilities but has no protection against CSRF or clickjacking.

    So the approach is generally to use another library like bottle-utils-csrf. This leaves the task of integration to the developer. This is not to pooh-pooh tiny web frameworks. I love the idea (especially Bottle which I think is really cute). But for public facing sites, I prefer the safety and convenience of Django.

    So I am tempted to try this one-file trick in Django. Let’s try to make a non-trivial web application with forms, templates and images. How does one go about doing something like that?

    Note: if you prefer to watch the video version, click on the video below:

    Django Applications in One File

    Minimal

    Let’s start small by creating a minimal Hello World application in Django. It might be amusing to some Django developers that we will not start with the startproject command. In fact, it is not necessary for Django to work at all. All that initial directory structure and files like settings.py are for your convenience.

    First, create a simple file called app.py with the following:

    import sys
    
    from django.conf import settings
    from django.urls import path
    from django.http import HttpResponse
    
    settings.configure(
    	DEBUG=True,  # For debugging
    	SECRET_KEY="a-bad-secret",  # Insecure! change this
    	ROOT_URLCONF=__name__,
    )
    
    
    def home(request):
    	return HttpResponse("Welcome!")
    
    
    urlpatterns = [
    	path("", home),
    ]
    
    if __name__ == "__main__":
    	from django.core.management import execute_from_command_line
    
    	execute_from_command_line(sys.argv)
    

    Yes, that’s all you need. There are some bad practices like hard coding the secret key (easily fixed). But the sheer elegance of everything being in hardly a screenful is quite rewarding.

    The command to run this file is: python app.py runserver 8080

    Now we will skip a couple of steps (they are in the video) and move on to a simple “Coming Soon” landing page.

    Coming Soon Application

    The idea of a coming-soon page is to gauge interest in a product before it is released. Such pages must have a clear call to action (CTA) like asking for your email. Ideally it should have minimum friction and yet collect all the relevant information.

    Let’s look at my updated app.py:

    import os
    import sys
    
    from django.conf import settings
    from django.urls import path
    from django.http import HttpResponse, HttpResponseRedirect
    from django.core.wsgi import get_wsgi_application
    from django.template import RequestContext, Template
    from django import forms
    
    CSV_LIST = "thelist.csv"
    
    settings.configure(
    	DEBUG=(os.environ.get("DEBUG", "") == "1"),
    	ALLOWED_HOSTS=["*"],  # Disable host header validation
    	ROOT_URLCONF=__name__,
    	SECRET_KEY=os.environ.get("SECRET_KEY", "a-bad-secret"),
    	TEMPLATES=[{"BACKEND": "django.template.backends.django.DjangoTemplates"}],
    	MIDDLEWARE_CLASSES=(
        	"django.middleware.common.CommonMiddleware",
        	"django.middleware.csrf.CsrfViewMiddleware",
        	"django.middleware.clickjacking.XFrameOptionsMiddleware",
    	),
    )
    
    
    class EnlistForm(forms.Form):
    	email = forms.EmailField(
        	required=True,
        	label=False,
        	widget=forms.EmailInput(attrs={"placeholder": "Email"}),
    	)
    	referrer = forms.CharField(required=False, widget=forms.HiddenInput())
    
    
    def home(request):
    	if request.method == "POST":
        	form = EnlistForm(request.POST)
        	if form.is_valid():
            	email = form.cleaned_data["email"]
            	referrer = form.cleaned_data["referrer"]
            	ip = request.META.get("REMOTE_ADDR")
            	print(f"Got email of {email}")
            	with open(CSV_LIST, "a") as csv:
                	csv.write(f"{email},{referrer},{ip}\n")
            	return HttpResponseRedirect("/thanks/")
    	else:
        	form = EnlistForm(initial={"referrer": request.META.get("HTTP_REFERER")})
    	context = RequestContext(
        	request, {"content": "Sign up for early access", "form": form}
    	)
    	return HttpResponse(MAIN_HTML.render(context))
    
    
    def thanks(request):
    	context = RequestContext(
        	request,
        	{"content": "Thank you for signing up. We will contact you!", "form": None},
    	)
    	return HttpResponse(MAIN_HTML.render(context))
    
    
    urlpatterns = [
    	path("", home),
    	path("thanks/", thanks),
    ]
    
    app = get_wsgi_application()
    
    
    MAIN_HTML = Template(
    	"""
    <html>
      <head>
    	<title>Coming Soon | Flying Cars</title>
    	<meta name="viewport" content="width=device-width, initial-scale=1.0">
    	<style>
     	@import url('https://fonts.googleapis.com/css2?family=Exo:wght@400;500;600;700;800;900&display=swap');
     	*{
       	margin: 0;
       	padding: 0;
       	box-sizing: border-box;
       	font-family: 'Exo', sans-serif;
     	}
     	html,body{
       	display: grid;
       	height: 100%;
       	width: 100%;
       	place-items: center;
       	background-color: #343434;
       	/* Thanks to Hero Patterns for the background */
       	background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 56 28' width='56' height='28'%3E%3Cpath fill='%23000000' fill-opacity='0.4' d='M56 26v2h-7.75c2.3-1.27 4.94-2 7.75-2zm-26 2a2 2 0 1 0-4 0h-4.09A25.98 25.98 0 0 0 0 16v-2c.67 0 1.34.02 2 .07V14a2 2 0 0 0-2-2v-2a4 4 0 0 1 3.98 3.6 28.09 28.09 0 0 1 2.8-3.86A8 8 0 0 0 0 6V4a9.99 9.99 0 0 1 8.17 4.23c.94-.95 1.96-1.83 3.03-2.63A13.98 13.98 0 0 0 0 0h7.75c2 1.1 3.73 2.63 5.1 4.45 1.12-.72 2.3-1.37 3.53-1.93A20.1 20.1 0 0 0 14.28 0h2.7c.45.56.88 1.14 1.29 1.74 1.3-.48 2.63-.87 4-1.15-.11-.2-.23-.4-.36-.59H26v.07a28.4 28.4 0 0 1 4 0V0h4.09l-.37.59c1.38.28 2.72.67 4.01 1.15.4-.6.84-1.18 1.3-1.74h2.69a20.1 20.1 0 0 0-2.1 2.52c1.23.56 2.41 1.2 3.54 1.93A16.08 16.08 0 0 1 48.25 0H56c-4.58 0-8.65 2.2-11.2 5.6 1.07.8 2.09 1.68 3.03 2.63A9.99 9.99 0 0 1 56 4v2a8 8 0 0 0-6.77 3.74c1.03 1.2 1.97 2.5 2.79 3.86A4 4 0 0 1 56 10v2a2 2 0 0 0-2 2.07 28.4 28.4 0 0 1 2-.07v2c-9.2 0-17.3 4.78-21.91 12H30zM7.75 28H0v-2c2.81 0 5.46.73 7.75 2zM56 20v2c-5.6 0-10.65 2.3-14.28 6h-2.7c4.04-4.89 10.15-8 16.98-8zm-39.03 8h-2.69C10.65 24.3 5.6 22 0 22v-2c6.83 0 12.94 3.11 16.97 8zm15.01-.4a28.09 28.09 0 0 1 2.8-3.86 8 8 0 0 0-13.55 0c1.03 1.2 1.97 2.5 2.79 3.86a4 4 0 0 1 7.96 0zm14.29-11.86c1.3-.48 2.63-.87 4-1.15a25.99 25.99 0 0 0-44.55 0c1.38.28 2.72.67 4.01 1.15a21.98 21.98 0 0 1 36.54 0zm-5.43 2.71c1.13-.72 2.3-1.37 3.54-1.93a19.98 19.98 0 0 0-32.76 0c1.23.56 2.41 1.2 3.54 1.93a15.98 15.98 0 0 1 25.68 0zm-4.67 3.78c.94-.95 1.96-1.83 3.03-2.63a13.98 13.98 0 0 0-22.4 0c1.07.8 2.09 1.68 3.03 2.63a9.99 9.99 0 0 1 16.34 0z'%3E%3C/path%3E%3C/svg%3E");
     	}
     	::selection{
       	color: #fff;
       	background: #FC4782;
     	}
     	.wrapper{
       	color: #eee;
       	max-width: 900px;
       	text-align: center;
       	padding: 0 50px;
     	}
     	.signup {
       	margin-top: 30px;
       	margin-bottom: 10px;
     	}
     	.content {
       	margin-top: 40px;
       	margin-bottom: 10px;
     	}
    	</style>
      </head>
      <body>
    	<div class="wrapper">
      	<svg width="600" height="300" version="1.1" viewBox="0 0 600 300" xmlns="http://www.w3.org/2000/svg"><g transform="translate(0,312)"><path d="m218.36-289.66h163.29c29.039 0 52.417 23.378 52.417 52.417v45.564c0 29.039-23.378 52.417-52.417 52.417h-163.29c-29.039 0-52.417-23.378-52.417-52.417v-45.564c0-29.039 23.378-52.417 52.417-52.417z" fill="#204a87" stop-color="#000000" stroke="#eeeeec" stroke-linecap="round" stroke-linejoin="round" stroke-width="4"/><g fill="#729fcf" stroke="#eeeeec" stroke-linejoin="round" stroke-width="4"><path d="m240.88-162.15c21.473-37.192 42.946-74.385 64.419-111.58" stop-color="#000000"/><g stroke-linecap="round"><path d="m276.15-249.32h-72.081" stop-color="#000000"/><path d="m259.9-221.32h-56.025" stop-color="#000000"/><path d="m267.69-235.32h-63.714" stop-color="#000000"/><path d="m370.37-162.15c-21.473-37.192-42.946-74.385-64.419-111.58" stop-color="#000000"/><path d="m332-249.93h72.081" stop-color="#000000"/><path d="m348.25-221.93h56.025" stop-color="#000000"/><path d="m340.46-235.93h63.714" stop-color="#000000"/><path d="m240.88-162.15c21.473-26.526 42.946-53.051 64.419-79.577" stop-color="#000000"/></g><path d="m370.37-162.15c-21.473-26.526-42.946-53.051-64.419-79.577" stop-color="#000000"/></g><g fill="#eeeeec"><path d="m183.74-116.06-6.3858 17.316h12.795zm-2.6568-4.6378h5.337l13.261 34.795h-4.8942l-3.1696-8.9261h-15.685l-3.1696 8.9261h-4.9641z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m222.66-120.7h4.7078v34.795h-4.7078z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m271.14-102.22q1.5149.51273 2.9365 2.1907 1.445 1.678 2.8899 4.6145l4.7777 9.5087h-5.0573l-4.4514-8.9261q-1.7246-3.4959-3.356-4.6378-1.6081-1.142-4.4048-1.142h-5.1273v14.706h-4.7078v-34.795h10.627q5.9663 0 8.9028 2.4937t2.9365 7.5278q0 3.2861-1.5382 5.4535-1.5149 2.1674-4.4281 3.0064zm-11.793-14.613v12.352h5.9196q3.4026 0 5.1273-1.5615 1.7479-1.5848 1.7479-4.6378t-1.7479-4.5912q-1.7246-1.5615-5.1273-1.5615z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m329.38-118.02v4.9641q-2.3772-2.214-5.0806-3.3094-2.6802-1.0954-5.7099-1.0954-5.9663 0-9.1358 3.659-3.1696 3.6357-3.1696 10.534 0 6.8752 3.1696 10.534 3.1696 3.6357 9.1358 3.6357 3.0297 0 5.7099-1.0954 2.7035-1.0954 5.0806-3.3094v4.9175q-2.4704 1.678-5.2438 2.517-2.7501.83901-5.8264.83901-7.9006 0-12.445-4.8243-4.5446-4.8476-4.5446-13.214 0-8.39 4.5446-13.214 4.5446-4.8476 12.445-4.8476 3.123 0 5.873.839 2.7734.8157 5.1972 2.4704z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m366.18-116.06-6.3858 17.316h12.795zm-2.6568-4.6378h5.337l13.261 34.795h-4.8942l-3.1696-8.9261h-15.685l-3.1696 8.9261h-4.9641z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m421.6-102.22q1.5149.51273 2.9365 2.1907 1.445 1.678 2.8899 4.6145l4.7777 9.5087h-5.0573l-4.4514-8.9261q-1.7246-3.4959-3.356-4.6378-1.6081-1.142-4.4048-1.142h-5.1272v14.706h-4.7078v-34.795h10.627q5.9663 0 8.9028 2.4937t2.9365 7.5278q0 3.2861-1.5382 5.4535-1.5149 2.1674-4.4281 3.0064zm-11.793-14.613v12.352h5.9196q3.4026 0 5.1272-1.5615 1.7479-1.5848 1.7479-4.6378t-1.7479-4.5912q-1.7246-1.5615-5.1272-1.5615z" style="text-decoration-color:#000000;text-decoration-line:none"/></g></g></svg>
      	<h1>All Your Traffic Problems Solved!</h1>
      	<h2>Feel the future with affordable levitating cars.</h2>
      	<div class="content">
        	{{ content }}
        	{% if form %}
          	<form action="." method="post" class="enlist_form">
            	{% csrf_token %}
            	{{ form.non_field_errors }}
            	{{ form.email.errors }}
            	{{ form.referrer }}
            	{{ form.referrer.errors }}
            	{{ form.email }}
            	<button type="submit">Add Me</button>
          	</form>
        	{% endif %}
      	</div>
    	</div>
      </body>
    </html>
    """
    )
    
    
    if __name__ == "__main__":
    	from django.core.management import execute_from_command_line
    
    	execute_from_command_line(sys.argv)
    

    Except for the absence of individual files, most of the code should be familiar to a Django developer. There is a large HTML template (including two SVG images) embedded as a string.

    Note that I do not use the ORM here. Django does seem to need a directory structure for that (Unless any reader could show me how to do it in a single file).

    Hopefully this shows how minimal Django could be. You might be able to use your favourite framework in places which you didn’t think were possible.

    Comments →

    3 Effective Examples of Django Async Views without Sleeping

    In August this year, Django 3.1 arrived with support for Django async views. This was fantastic news but most people raised the obvious question – What can I do with it? There have been a few tutorials about Django asynchronous views that demonstrate asynchronous execution while calling asyncio.sleep. But that merely led to the refinement of the popular question – What can I do with it besides sleep-ing?

    The short answer is – it is a very powerful technique to write efficient views. For a detailed overview of what asynchronous views are and how they can be used, keep on reading. If you are new to asynchronous support in Django and like to know more background, read my earlier article: A Guide to ASGI in Django 3.0 and its Performance.

    Django Async Views

    Django now allows you to write views which can run asynchronously. First let’s refresh your memory by looking at a simple and minimal synchronous view in Django:

    def index(request):
        return HttpResponse("Made a pretty page")
    

    It takes a request object and returns a response object. In a real world project, a view does many things like fetching records from a database, calling a service or rendering a template. But they work synchronously or one after the other.

    In Django’s MTV (Model Template View) architecture, Views are disproportionately more powerful than others (I find it comparable to a controller in MVC architecture though these things are debatable). Once you enter a view you can perform almost any logic necessary to create a response. This is why Asynchronous Views are so important. It lets you do more things concurrently.

    It is quite easy to write an asynchronous view. For example the asynchronous version of our minimal example above would be:

    async def index_async(request):
        return HttpResponse("Made a pretty page asynchronously.")
    

    This is a coroutine rather than a function. You cannot call it directly. An event loop needs to be created to execute it. But you do not have to worry about that difference since Django takes care of all that.

    Note that this particular view is not invoking anything asynchronously. If Django is running in the classic WSGI mode, then a new event loop is created (automatically) to run this coroutine. So in this case, it might be slightly slower than the synchronous version. But that’s because you are not using it to run tasks concurrently.

    So then why bother writing asynchronous views? The limitations of synchronous views become apparent only at a certain scale. When it comes to large scale web applications probably nothing beats FaceBook.

    Views at Facebook

    In August, Facebook released a static analysis tool to detect and prevent security issues in Python. But what caught my eye was how the views were written in the examples they had shared. They were all async!

    # views/user.py
    async def get_profile(request: HttpRequest) -> HttpResponse:
       profile = load_profile(request.GET['user_id'])
       ...
     
    # controller/user.py
    async def load_profile(user_id: str):
       user = load_user(user_id) # Loads a user safely; no SQL injection
       pictures = load_pictures(user.id)
       ...
     
    # model/media.py
    async def load_pictures(user_id: str):
       query = f"""
          SELECT *
          FROM pictures
          WHERE user_id = {user_id}
       """
       result = run_query(query)
       ...
     
    # model/shared.py
    async def run_query(query: str):
       connection = create_sql_connection()
       result = await connection.execute(query)
       ...
    

    Note that this is not Django but something similar. Currently, Django runs the database code synchronously. But that may change sometime in the future.

    If you think about it, it makes perfect sense. Synchronous code can be blocked while waiting for an I/O operation for several microseconds. However, its equivalent asynchronous code would not be tied up and can work on other tasks. Therefore it can handle more requests with lower latencies. More requests gives Facebook (or any other large site) the ability to handle more users on the same infrastructure.

    Illustration
    Scalability Problems in the 1800s, I suppose

    Even if you are not close to reaching Facebook scale, you could use Python’s asyncio as a more predictable threading mechanism to run many things concurrently. A thread scheduler could interrupt in between destructive updates of shared resources leading to difficult to debug race conditions. Compared to threads, coroutines can achieve a higher level of concurrency with very less overhead.

    Misleading Sleep Examples

    As I joked earlier, most of the Django async views tutorials show an example involving sleep. Even the official Django release notes had this example:

    async def my_view(request):
        await asyncio.sleep(0.5)
        return HttpResponse('Hello, async world!')
    

    To a Python async guru this code might indicate the possibilities that were not previously possible. But to the vast majority, this code is misleading in many ways.

    Firstly, the sleep happening synchronously or asynchronously makes no difference to the end user. The poor chap who just opened the URL linked to that view will have to wait for 0.5 seconds before it returns a cheeky “Hello, async world!”. If you are a complete novice, you may have expected an immediate reply and somehow the “hello” greeting to appear asynchronously half a second later. Of course, that sounds silly but then what is this example trying to do compared to a synchronous time.sleep() inside a view?

    The answer is, as with most things in the asyncio world, in the event loop. If the event loop had some other task waiting to be run then that half second window would give it an opportunity to run that. Note that it may take longer than that window to complete. Cooperative Multithreading assumes that everyone works quickly and hands over the control promptly back to the event loop.

    Secondly, it does not seem to accomplish anything useful. Some command-line interfaces use sleep to give enough time for users to read a message before disappearing. But it is the opposite for web applications - a faster response from the web server is the key to a better user experience. So by slowing the response what are we trying to demonstrate in such examples?

    Illustration
    Letting them Sleep Would Be Better Idea

    The best explanation for such simplified examples I can give is convenience. It needs a bit more setup to show examples which really need asynchronous support. That’s what we are trying to explore here.

    Better examples

    A rule of thumb to remember before writing an asynchronous view is to check if it is I/O bound or CPU-bound. A view which spends most of the time in a CPU-bound activity for e.g. matrix multiplication or image manipulation would really not benefit from rewriting them to async views. You should be focussing on the I/O bound activities.

    Invoking Microservices

    Most large web applications are moving away from a monolithic architecture to one composed of many microservices. Rendering a view might require the results of many internal or external services.

    In our example, an ecommerce site for books renders its front page - like most popular sites - tailored to the logged in user by displaying recommended books. The recommendation engine is typically implemented as a separate microservice that makes recommendations based on past buying history and perhaps a bit of machine learning by understanding how successful its past recommendations were.

    In this case, we also need the results of another microservice that decides which promotional banners to display as a rotating banner or slideshow to the user. These banners are not tailored to the logged in user but change depending on the items currently on sale (active promotional campaign) or date.

    Let’s look at how a synchronous version of such a page might look like:

    def sync_home(request):
        """Display homepage by calling two services synchronously"""
        context = {}
        try:
            response = httpx.get(PROMO_SERVICE_URL)
            if response.status_code == httpx.codes.OK:
                context["promo"] = response.json()
            response = httpx.get(RECCO_SERVICE_URL)
            if response.status_code == httpx.codes.OK:
                context["recco"] = response.json()
        except httpx.RequestError as exc:
            print(f"An error occurred while requesting {exc.request.url!r}.")
        return render(request, "index.html", context)
    

    Here instead of the popular Python requests library we are using the httpx library because it supports making synchronous and asynchronous web requests. The interface is almost identical.

    The problem with this view is that the time taken to invoke these services add up since they happen sequentially. The Python process is suspended until the first service responds which could take a long time in a worst case scenario.

    Let’s try to run them concurrently using a simplistic (and ineffective) await call:

    async def async_home_inefficient(request):
        """Display homepage by calling two awaitables synchronously (does NOT run concurrently)"""
        context = {}
        try:
            async with httpx.AsyncClient() as client:
                response = await client.get(PROMO_SERVICE_URL)
                if response.status_code == httpx.codes.OK:
                    context["promo"] = response.json()
                response = await client.get(RECCO_SERVICE_URL)
                if response.status_code == httpx.codes.OK:
                    context["recco"] = response.json()
        except httpx.RequestError as exc:
            print(f"An error occurred while requesting {exc.request.url!r}.")
        return render(request, "index.html", context)
    

    Notice that the view has changed from a function to a coroutine (due to async def keyword). Also note that there are two places where we await for a response from each of the services. You don’t have to try to understand every line here, as we will explain with a better example.

    Interestingly, this view does not work concurrently and takes the same amount of time as the synchronous view. If you are familiar with asynchronous programming, you might have guessed that simply awaiting a coroutine does not make it run other things concurrently, you will just yield control back to the event loop. The view still gets suspended.

    Let’s look at a proper way to run things concurrently:

    async def async_home(request):
        """Display homepage by calling two services asynchronously (proper concurrency)"""
        context = {}
        try:
            async with httpx.AsyncClient() as client:
                response_p, response_r = await asyncio.gather(
                    client.get(PROMO_SERVICE_URL), client.get(RECCO_SERVICE_URL)
                )
    
                if response_p.status_code == httpx.codes.OK:
                    context["promo"] = response_p.json()
                if response_r.status_code == httpx.codes.OK:
                    context["recco"] = response_r.json()
        except httpx.RequestError as exc:
            print(f"An error occurred while requesting {exc.request.url!r}.")
        return render(request, "index.html", context)
    

    If the two services we are calling have similar response times, then this view should complete in _half _the time compared to the synchronous version. This is because the calls happen concurrently as we would want.

    Let’s try to understand what is happening here. There is an outer try…except block to catch request errors while making either of the HTTP calls. Then there is an inner async…with block which gives a context having the client object.

    The most important line is one with the asyncio.gather call taking the coroutines created by the two client.get calls. The gather call will execute them concurrently and return only when both of them are completed. The result would be a tuple of responses which we will unpack into two variables response_p and response_r. If there were no errors, these responses are populated in the context sent for template rendering.

    Microservices are typically internal to the organization hence the response times are low and less variable. Yet, it is never a good idea to rely solely on synchronous calls for communicating between microservices. As the dependencies between services increases, it creates long chains of request and response calls. Such chains can slow down services.

    Why Live Scraping is Bad

    We need to address web scraping because so many asyncio examples use them. I am referring to cases where multiple external websites or pages within a website are concurrently fetched and scraped for information like live stock market (or bitcoin) prices. The implementation would be very similar to what we saw in the Microservices example.

    But this is very risky since a view should return a response to the user as quickly as possible. So trying to fetch external sites which have variable response times or throttling mechanisms could be a poor user experience or even worse a browser timeout. Since microservice calls are typically internal, response times can be controlled with proper SLAs.

    Ideally, scraping should be done in a separate process scheduled to run periodically (using celery or rq). The view should simply pick up the scraped values and present them to the users.

    Serving Files

    Django addresses the problem of serving files by trying hard not to do it itself. This makes sense from a “Do not reinvent the wheel” perspective. After all, there are several better solutions to serve static files like nginx.

    Illustration
    'Serving simultaneously is not for everyone'

    But often we need to serve files with dynamic content. Files often reside in a (slower) disk-based storage (we now have much faster SSDs). While this file operation is quite easy to accomplish with Python, it could be expensive in terms of performance for large files. Regardless of the file’s size, this is a potentially blocking I/O operation that could potentially be used for running another task concurrently.

    Imagine we need to serve a PDF certificate in a Django view. However the date and time of downloading the certificate needs to be stored in the metadata of the PDF file, for some reason (possibly for identification and validation).

    We will use the aiofiles library here for asynchronous file I/O. The API is almost the same as the familiar Python’s built-in file API. Here is how the asynchronous view could be written:

    async def serve_certificate(request):
        timestamp = datetime.datetime.now().isoformat()
    
        response = HttpResponse(content_type="application/pdf")
        response["Content-Disposition"] = "attachment; filename=certificate.pdf"
        async with aiofiles.open("homepage/pdfs/certificate-template.pdf", mode="rb") as f:
            contents = await f.read()
            response.write(contents.replace(b"%timestamp%", bytes(timestamp, "utf-8")))
        return response
    

    This example illustrates why we need asynchronous template rendering in Django. But until that gets implemented, you could use aiofiles library to pull local files without skipping a beat.

    There are downsides to directly using local files instead of Django’s staticfiles. In the future, when you migrate to a different storage space like Amazon S3, make sure you adapt your code accordingly.

    Handling Uploads

    On the flip side, uploading a file is also a potentially long, blocking operation. For security and organizational reasons, Django stores all uploaded content into a separate ‘media’ directory.

    If you have a form that allows uploading a file, then we need to anticipate that some pesky user would upload an impossibly large one. Thankfully Django passes the file to the view as chunks of a certain size. Combined with aiofile’s ability to write a file asynchronously, we could support highly concurrent uploads.

    async def handle_uploaded_file(f):
        async with aiofiles.open(f"uploads/{f.name}", "wb+") as destination:
            for chunk in f.chunks():
                await destination.write(chunk)
    
    
    async def async_uploader(request):
        if request.method == "POST":
            form = UploadFileForm(request.POST, request.FILES)
            if form.is_valid():
                await handle_uploaded_file(request.FILES["file"])
                return HttpResponseRedirect("/")
        else:
            form = UploadFileForm()
        return render(request, "upload.html", {"form": form})
    

    Again this is circumventing Django’s default file upload mechanism, so you need to be careful about the security implications.

    Where To Use

    Django Async project has full backward compatibility as one of its main goals. So you can continue to use your old synchronous views without rewriting them into async. Asynchronous views are not a panacea for all performance issues, so most projects will still continue to use synchronous code since they are quite straightforward to reason about.

    In fact, you can use both async and sync views in the same project. Django will take care of calling the view in the appropriate manner. However, if you are using async views it is recommended to deploy the application on ASGI servers.

    This gives you the flexibility to try asynchronous views gradually especially for I/O intensive work. You need to be careful to pick only async libraries or mix them with sync carefully (use the async_to_sync and sync_to_async adaptors).

    Hopefully this writeup gave you some ideas.

    Thanks to Chillar Anand and Ritesh Agrawal for reviewing this post. All illustrations courtesy of Old Book Illustrations

    Comments →

    Page 1 of 39 Older »