python on Arunrocks

Print from 1 to 100 without loops or numbers in Python

Tue, 23 Mar 2021 12:22:02 +0530

A new programming problem has been trending recently - write a program to print numbers from one to hundred without using any loops or mentioning numbers in the source. Naturally, I was interested in a solution in Python.

I got many brilliant answers on Twitter. But first let me show you how I approached it. I was looking for a general and readable solution than the shortest one. My thought process was – even if we cannot mention numbers we can convert non-numeric types to numbers. So I tried with Booleans and Strings:

zero = int(False)
one = int(True)
hundred = int(f"{one}{zero}{zero}")


def shownum(i):
    if i <= hundred:
        print(i)
        shownum(i + one)


shownum(one)

Starting from scratch, we get 0 and 1 from the False and True values. Then to create 100, the upper limit of the loop, we use the new favourite string interpolation method – f-strings.

Overcoming the limitation of not using loops was quite straightforward - just use Recursion. We have to be careful though, Python does not have Tail Call Optimization (since Guido prefers to have proper tracebacks), so if you keep increasing the upper limit of the loop you will end up in a stack overflow.

Whiz kids on Twitter

To my poser tweet yesterday, there were many wonderful solutions that were way shorter than mine. Here are some of them:

The Long Scream by Abhiram

print([*range(len('a'), len('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'))])
😂
— Abhiram R (@abhicantdraw) March 22, 2021

What I like here is the clever use of the * operator to convert the range object into a list tersely. And of course the liberal use of the “a”-s that just screams “I cannot be unseen”.

Short and Succinct by Rohan

list(map(int,range(ord(b"e"))))
— Rohan Verma (@rhnvrm) March 22, 2021

Cleverly taking advantage of Python’s built-in ord function to get the ASCII code of lower case ‘e’ (it is 101 not 100), we sort of have a code-golf winner. Sort of because it starts count from 0 not 1 as the original question posed. But this is Twitter, a correction soon emerged.

Note that the map and int functions are not really required here, making it even more shorter! In a direct message Rohan shared an improved solution that starts counting from one:

list(filter(lambda x: x,range(ord(b"e"))))

Evolved by Anirudh

list(range(ord('a')-ord('a'), ord('e')))
— Anirudh Menon (@anikmenon) March 23, 2021

With a minor correction, this code fixes the start count to 1. As a nice touch the strings form ‘bae’ which must be the cutest term for a friend.

Can This Get Shorter by Arun

With the benefit of having seen all the ideas, I can finally put forth my shortest solution (with 23 characters):

[*range(True,ord("e"))]

So that’s the end of this code golf for me. That is, until someone else comes up with something shorter!

Fitting a Django Application in One File

Wed, 09 Dec 2020 22:54:50 +0530

Earlier this week, Anthony a french economics university student wanted to talk to me over Zoom about my ray tracer tutorials. He and his friend were new to Python but were excited to implement their own ray tracer after following my videos. One of the questions that popped up in the conversation was - “Can we put all the classes in one file instead of breaking it into individual files per class?”. I said, “Of course” and noticed a wave of relief in their faces. In fact, I explained, my earlier implementation was all in one file and later broken up for better pedagogy.

But the charm of an entire project in a single file is compelling. I remember seeing a Sinatra web application a few years ago containing the entire application and assets like HTML templates and CSS in a single file. Presenting all the components in the same file gave a complete high-level overview of the project by simply scrolling up and down.

Normally, at this point, someone would suggest a microframework. But it is not that easy.

Microframeworks

Microframeworks take a minimalistic approach by omitting certain components or directing you to a few recommended components. For instance, Bottle contains basic form handling capabilities but has no protection against CSRF or clickjacking.

So the approach is generally to use another library like bottle-utils-csrf. This leaves the task of integration to the developer. This is not to pooh-pooh tiny web frameworks. I love the idea (especially Bottle which I think is really cute). But for public facing sites, I prefer the safety and convenience of Django.

So I am tempted to try this one-file trick in Django. Let’s try to make a non-trivial web application with forms, templates and images. How does one go about doing something like that?

Note: if you prefer to watch the video version, click on the video below:

Django Applications in One File

Minimal

Let’s start small by creating a minimal Hello World application in Django. It might be amusing to some Django developers that we will not start with the startproject command. In fact, it is not necessary for Django to work at all. All that initial directory structure and files like settings.py are for your convenience.

First, create a simple file called app.py with the following:

import sys

from django.conf import settings
from django.urls import path
from django.http import HttpResponse

settings.configure(
	DEBUG=True,  # For debugging
	SECRET_KEY="a-bad-secret",  # Insecure! change this
	ROOT_URLCONF=__name__,
)


def home(request):
	return HttpResponse("Welcome!")


urlpatterns = [
	path("", home),
]

if __name__ == "__main__":
	from django.core.management import execute_from_command_line

	execute_from_command_line(sys.argv)

Yes, that’s all you need. There are some bad practices like hard coding the secret key (easily fixed). But the sheer elegance of everything being in hardly a screenful is quite rewarding.

The command to run this file is: python app.py runserver 8080

Now we will skip a couple of steps (they are in the video) and move on to a simple “Coming Soon” landing page.

Coming Soon Application

The idea of a coming-soon page is to gauge interest in a product before it is released. Such pages must have a clear call to action (CTA) like asking for your email. Ideally it should have minimum friction and yet collect all the relevant information.

Let’s look at my updated app.py:

import os
import sys

from django.conf import settings
from django.urls import path
from django.http import HttpResponse, HttpResponseRedirect
from django.core.wsgi import get_wsgi_application
from django.template import RequestContext, Template
from django import forms

CSV_LIST = "thelist.csv"

settings.configure(
	DEBUG=(os.environ.get("DEBUG", "") == "1"),
	ALLOWED_HOSTS=["*"],  # Disable host header validation
	ROOT_URLCONF=__name__,
	SECRET_KEY=os.environ.get("SECRET_KEY", "a-bad-secret"),
	TEMPLATES=[{"BACKEND": "django.template.backends.django.DjangoTemplates"}],
	MIDDLEWARE_CLASSES=(
    	"django.middleware.common.CommonMiddleware",
    	"django.middleware.csrf.CsrfViewMiddleware",
    	"django.middleware.clickjacking.XFrameOptionsMiddleware",
	),
)


class EnlistForm(forms.Form):
	email = forms.EmailField(
    	required=True,
    	label=False,
    	widget=forms.EmailInput(attrs={"placeholder": "Email"}),
	)
	referrer = forms.CharField(required=False, widget=forms.HiddenInput())


def home(request):
	if request.method == "POST":
    	form = EnlistForm(request.POST)
    	if form.is_valid():
        	email = form.cleaned_data["email"]
        	referrer = form.cleaned_data["referrer"]
        	ip = request.META.get("REMOTE_ADDR")
        	print(f"Got email of {email}")
        	with open(CSV_LIST, "a") as csv:
            	csv.write(f"{email},{referrer},{ip}\n")
        	return HttpResponseRedirect("/thanks/")
	else:
    	form = EnlistForm(initial={"referrer": request.META.get("HTTP_REFERER")})
	context = RequestContext(
    	request, {"content": "Sign up for early access", "form": form}
	)
	return HttpResponse(MAIN_HTML.render(context))


def thanks(request):
	context = RequestContext(
    	request,
    	{"content": "Thank you for signing up. We will contact you!", "form": None},
	)
	return HttpResponse(MAIN_HTML.render(context))


urlpatterns = [
	path("", home),
	path("thanks/", thanks),
]

app = get_wsgi_application()


MAIN_HTML = Template(
	"""
<html>
  <head>
	<title>Coming Soon | Flying Cars</title>
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<style>
 	@import url('https://fonts.googleapis.com/css2?family=Exo:wght@400;500;600;700;800;900&display=swap');
 	*{
   	margin: 0;
   	padding: 0;
   	box-sizing: border-box;
   	font-family: 'Exo', sans-serif;
 	}
 	html,body{
   	display: grid;
   	height: 100%;
   	width: 100%;
   	place-items: center;
   	background-color: #343434;
   	/* Thanks to Hero Patterns for the background */
   	background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 56 28' width='56' height='28'%3E%3Cpath fill='%23000000' fill-opacity='0.4' d='M56 26v2h-7.75c2.3-1.27 4.94-2 7.75-2zm-26 2a2 2 0 1 0-4 0h-4.09A25.98 25.98 0 0 0 0 16v-2c.67 0 1.34.02 2 .07V14a2 2 0 0 0-2-2v-2a4 4 0 0 1 3.98 3.6 28.09 28.09 0 0 1 2.8-3.86A8 8 0 0 0 0 6V4a9.99 9.99 0 0 1 8.17 4.23c.94-.95 1.96-1.83 3.03-2.63A13.98 13.98 0 0 0 0 0h7.75c2 1.1 3.73 2.63 5.1 4.45 1.12-.72 2.3-1.37 3.53-1.93A20.1 20.1 0 0 0 14.28 0h2.7c.45.56.88 1.14 1.29 1.74 1.3-.48 2.63-.87 4-1.15-.11-.2-.23-.4-.36-.59H26v.07a28.4 28.4 0 0 1 4 0V0h4.09l-.37.59c1.38.28 2.72.67 4.01 1.15.4-.6.84-1.18 1.3-1.74h2.69a20.1 20.1 0 0 0-2.1 2.52c1.23.56 2.41 1.2 3.54 1.93A16.08 16.08 0 0 1 48.25 0H56c-4.58 0-8.65 2.2-11.2 5.6 1.07.8 2.09 1.68 3.03 2.63A9.99 9.99 0 0 1 56 4v2a8 8 0 0 0-6.77 3.74c1.03 1.2 1.97 2.5 2.79 3.86A4 4 0 0 1 56 10v2a2 2 0 0 0-2 2.07 28.4 28.4 0 0 1 2-.07v2c-9.2 0-17.3 4.78-21.91 12H30zM7.75 28H0v-2c2.81 0 5.46.73 7.75 2zM56 20v2c-5.6 0-10.65 2.3-14.28 6h-2.7c4.04-4.89 10.15-8 16.98-8zm-39.03 8h-2.69C10.65 24.3 5.6 22 0 22v-2c6.83 0 12.94 3.11 16.97 8zm15.01-.4a28.09 28.09 0 0 1 2.8-3.86 8 8 0 0 0-13.55 0c1.03 1.2 1.97 2.5 2.79 3.86a4 4 0 0 1 7.96 0zm14.29-11.86c1.3-.48 2.63-.87 4-1.15a25.99 25.99 0 0 0-44.55 0c1.38.28 2.72.67 4.01 1.15a21.98 21.98 0 0 1 36.54 0zm-5.43 2.71c1.13-.72 2.3-1.37 3.54-1.93a19.98 19.98 0 0 0-32.76 0c1.23.56 2.41 1.2 3.54 1.93a15.98 15.98 0 0 1 25.68 0zm-4.67 3.78c.94-.95 1.96-1.83 3.03-2.63a13.98 13.98 0 0 0-22.4 0c1.07.8 2.09 1.68 3.03 2.63a9.99 9.99 0 0 1 16.34 0z'%3E%3C/path%3E%3C/svg%3E");
 	}
 	::selection{
   	color: #fff;
   	background: #FC4782;
 	}
 	.wrapper{
   	color: #eee;
   	max-width: 900px;
   	text-align: center;
   	padding: 0 50px;
 	}
 	.signup {
   	margin-top: 30px;
   	margin-bottom: 10px;
 	}
 	.content {
   	margin-top: 40px;
   	margin-bottom: 10px;
 	}
	</style>
  </head>
  <body>
	<div class="wrapper">
  	<svg width="600" height="300" version="1.1" viewBox="0 0 600 300" xmlns="http://www.w3.org/2000/svg"><g transform="translate(0,312)"><path d="m218.36-289.66h163.29c29.039 0 52.417 23.378 52.417 52.417v45.564c0 29.039-23.378 52.417-52.417 52.417h-163.29c-29.039 0-52.417-23.378-52.417-52.417v-45.564c0-29.039 23.378-52.417 52.417-52.417z" fill="#204a87" stop-color="#000000" stroke="#eeeeec" stroke-linecap="round" stroke-linejoin="round" stroke-width="4"/><g fill="#729fcf" stroke="#eeeeec" stroke-linejoin="round" stroke-width="4"><path d="m240.88-162.15c21.473-37.192 42.946-74.385 64.419-111.58" stop-color="#000000"/><g stroke-linecap="round"><path d="m276.15-249.32h-72.081" stop-color="#000000"/><path d="m259.9-221.32h-56.025" stop-color="#000000"/><path d="m267.69-235.32h-63.714" stop-color="#000000"/><path d="m370.37-162.15c-21.473-37.192-42.946-74.385-64.419-111.58" stop-color="#000000"/><path d="m332-249.93h72.081" stop-color="#000000"/><path d="m348.25-221.93h56.025" stop-color="#000000"/><path d="m340.46-235.93h63.714" stop-color="#000000"/><path d="m240.88-162.15c21.473-26.526 42.946-53.051 64.419-79.577" stop-color="#000000"/></g><path d="m370.37-162.15c-21.473-26.526-42.946-53.051-64.419-79.577" stop-color="#000000"/></g><g fill="#eeeeec"><path d="m183.74-116.06-6.3858 17.316h12.795zm-2.6568-4.6378h5.337l13.261 34.795h-4.8942l-3.1696-8.9261h-15.685l-3.1696 8.9261h-4.9641z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m222.66-120.7h4.7078v34.795h-4.7078z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m271.14-102.22q1.5149.51273 2.9365 2.1907 1.445 1.678 2.8899 4.6145l4.7777 9.5087h-5.0573l-4.4514-8.9261q-1.7246-3.4959-3.356-4.6378-1.6081-1.142-4.4048-1.142h-5.1273v14.706h-4.7078v-34.795h10.627q5.9663 0 8.9028 2.4937t2.9365 7.5278q0 3.2861-1.5382 5.4535-1.5149 2.1674-4.4281 3.0064zm-11.793-14.613v12.352h5.9196q3.4026 0 5.1273-1.5615 1.7479-1.5848 1.7479-4.6378t-1.7479-4.5912q-1.7246-1.5615-5.1273-1.5615z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m329.38-118.02v4.9641q-2.3772-2.214-5.0806-3.3094-2.6802-1.0954-5.7099-1.0954-5.9663 0-9.1358 3.659-3.1696 3.6357-3.1696 10.534 0 6.8752 3.1696 10.534 3.1696 3.6357 9.1358 3.6357 3.0297 0 5.7099-1.0954 2.7035-1.0954 5.0806-3.3094v4.9175q-2.4704 1.678-5.2438 2.517-2.7501.83901-5.8264.83901-7.9006 0-12.445-4.8243-4.5446-4.8476-4.5446-13.214 0-8.39 4.5446-13.214 4.5446-4.8476 12.445-4.8476 3.123 0 5.873.839 2.7734.8157 5.1972 2.4704z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m366.18-116.06-6.3858 17.316h12.795zm-2.6568-4.6378h5.337l13.261 34.795h-4.8942l-3.1696-8.9261h-15.685l-3.1696 8.9261h-4.9641z" style="text-decoration-color:#000000;text-decoration-line:none"/><path d="m421.6-102.22q1.5149.51273 2.9365 2.1907 1.445 1.678 2.8899 4.6145l4.7777 9.5087h-5.0573l-4.4514-8.9261q-1.7246-3.4959-3.356-4.6378-1.6081-1.142-4.4048-1.142h-5.1272v14.706h-4.7078v-34.795h10.627q5.9663 0 8.9028 2.4937t2.9365 7.5278q0 3.2861-1.5382 5.4535-1.5149 2.1674-4.4281 3.0064zm-11.793-14.613v12.352h5.9196q3.4026 0 5.1272-1.5615 1.7479-1.5848 1.7479-4.6378t-1.7479-4.5912q-1.7246-1.5615-5.1272-1.5615z" style="text-decoration-color:#000000;text-decoration-line:none"/></g></g></svg>
  	<h1>All Your Traffic Problems Solved!</h1>
  	<h2>Feel the future with affordable levitating cars.</h2>
  	<div class="content">
    	{{ content }}
    	{% if form %}
      	<form action="." method="post" class="enlist_form">
        	{% csrf_token %}
        	{{ form.non_field_errors }}
        	{{ form.email.errors }}
        	{{ form.referrer }}
        	{{ form.referrer.errors }}
        	{{ form.email }}
        	<button type="submit">Add Me</button>
      	</form>
    	{% endif %}
  	</div>
	</div>
  </body>
</html>
"""
)


if __name__ == "__main__":
	from django.core.management import execute_from_command_line

	execute_from_command_line(sys.argv)

Except for the absence of individual files, most of the code should be familiar to a Django developer. There is a large HTML template (including two SVG images) embedded as a string.

Note that I do not use the ORM here. Django does seem to need a directory structure for that (Unless any reader could show me how to do it in a single file).

Hopefully this shows how minimal Django could be. You might be able to use your favourite framework in places which you didn’t think were possible.

Black Holes and Python

Wed, 17 Apr 2019 10:14:36 +0530

Black Hole in M87 - Image Courtesy: BBC

The first picture of a black hole is probably one of the most exciting developments in the world of science. The blurry ring of fiery orange might not seem difficult to produce. In fact, it involved years of effort by an international team of scientists, including computer scientists.

Reading the account, I am excited about the role Python played in this endeavour. This is interesting because when we talk about a scientific discovery we usually talk about the people - the scientists who made leaps of intuitions and found correlations that no one else had. But increasingly technology is playing a significant role in discoveries by sifting through enormous amounts of data and extracting valuable insights.

Python’s popularity in the scientific computing would not be a surprise to most Python programmers today. But back in 2009 when I attended the first PyCon India in IISc Bangalore, I was surprised to see talks on experimental Physics and fluid simulations. When I asked Prof Prabhu on why Python is so popular in scientific computing, he said “it is very accessible to us – non-programmers”.

Casually browsing through the software used by the astronomers, you will find mentions of Python libraries like Numpy, Scipy, Matplotlib, Pandas and Jupyter. Remarkably, entire projects such as eht-python are written only in Python. Python is not just the language of choice, it is the lingua franca of scientific computing.

Yet, if you think about it, there are better programming languages be it in terms of - speed, type safety or brevity. But Python overcomes these limitations and sometimes succeeds due to some pragmatic language design decisions.

Speed Can Be Delegated

Ironically, plain Python code can perform very poorly for computation intensive tasks. But libraries like NumPy are de facto when it comes to any form of number crunching. It provides an N-dimensional array object with several high level operations like cross product or transpose. The C engine of the library accelerates these operations close to raw machine speed.

In the early days of Python, it was expected that performance intensive parts would be written in other languages like C or FORTRAN and a wrapper interface would be used to invoke them. Over time, wealth of libraries like NumPy made it unnecessary to write any custom C code. Why reinvent the wheel when you can just “import” and use it?

Libraries that Play Well

import gravity - Obligatory XKCD 353

Working with third party C libraries is not for the weak hearted. In 2002, when I was adapting the algorithm in a paper for my project on wavelet-based image compression, I learnt this the hard way. We needed to use an existing Fast Fourier transform library written in C.

The library worked when you used it as is. But if you tried to extend a data structure, you might end up with a null pointer exception. Manual memory management by working out all the code paths turned out to be very stressful. The library was well documented but we practically needed to understand every line before tweaking it.

Eventually, we gave up and started implementing most of the project in Python. It was much easier to work with higher level data structures like dictionaries and lists without going through the dance of malloc and free. Even better, the Python code was pretty much a direct translation of the mathematics in the paper to code.

Python libraries tend to compose quite well (while C libraries don’t). This is partly due to its dynamic typing and automatic memory management, but I personally feel it is mainly due to good conventions. Most of the Python idioms are well documented and this leads to minimum surprises. For instance, a deeply nested class hierarchy is frowned upon because “flat is better”.

Interactive Exploration

Research is explorative. We not know what we may find. Even if we do, we cannot wait for ages to find out because we might be chasing a dead end. An interactive interface is a key tool for a researcher or scientist. A Jupyter notebook is close to the ideal with its live code and embedded visualization abilities.

If you need to try a computation with a different set of parameters, you can invoke it and view the results. Even plot it to visualize it better. Then you could take the results and feed it to another computation. This recorded transcript is a valuable data pipeline that can be replayed by a different user for verification or with a different set of observations.

If you think about it, a conversational interface could be more approachable to a non-programmer. Alan Kay was very impressed with an early interactive programming environment called JOSS developed in RAND that appealed to economists. I find it endearing that it replied to any command it did not understand with a “Eh?” or “SORRY”.

Imagine using today’s voice recognition technology to build such a conversational virtual assistant for scientists. Considering much of science (especially physics) involves mathematics – spelling out complex equations can quickly get tedious. Listening to tables of numbers is no fun either. So unless the conversation steps up with an amazing level of artificial intelligence (imagine a reply like “I have run simulations on every known element, and none can serve as a viable replacement for the palladium core."), we are probably stuck with current interfaces.

Future of Python

Katie Bouman, one of the key contributors, with the amazing amount of data

M87 EHT project involved processing petabytes of information (which is publicly available). They plan to add new telescopes in the future, increasing the volume of data by orders of magnitude. In general, the computational demands of science will keep growing and even enter new domains. The question is - will Python keep up or get replaced?

Python has a strong ecosystem with hundreds of libraries. It will be hard for another language to reproduce that. It is a very easy language to pick up. The readability is so good that Python code is often compared to pseudocode. I believe, it has changed the expectation of how code should look like. Any new language should have equal or better readability to inspire a switch.

While there are several other promising languages like Julia or Rust, I am confident that Python will remain the scientist’s favourite programming language for a long while. Despite its limitations, Python has found a sweet spot between ease and power.

Every year technological progresses keeps accelerating. This can translate into progress for humanity if we can make technology more accessible. We need physicists, mathematicians, biologists, economists, farmers and so on to use cheap computing power to build better things.

Python does play a significant role by making coding less intimidating and more collaborative. That’s why I believe you will see it in bringing more people to computers and being a part of more future breakthroughs.

Understanding Django Channels

Tue, 03 Jul 2018 06:55:30 +0530

You are reading a post from a two-part tutorial series on Django Channels

Django Channels

Django Channels was originally created to solve the problem of handling asynchronous communication protocols like say WebSockets. More and more web applications were providing realtime capabilities like chat and push notifications. Various workarounds were created to make Django support such requirements like running separate socket servers or proxy servers.

Channels is an official Django project not just for handling WebSockets and other forms of bi-directional communication but also for running background tasks asynchronously. As of writing, Django Channels 2 is out which is a complete rewrite based on Python 3’s async/await based coroutines.

This article covers the concepts of Django Channels and leads you to a video tutorial implementing a notifications application in Django.

Here is a simplified block diagram of a typical Channels setup:

How a typical Django Channels infrastructure works

A client such as a web browser sends both HTTP/HTTPS and WebSocket traffic to an ASGI server like Daphene. Like WSGI, the ASGI (Asynchronous Server Gateway Interface) specification is a common way for applications servers and applications to interact with each other asynchronously.

Like a typical Django application, HTTP traffic is handled synchronously i.e. when the browser sends a request, it waits until it is routed to Django and a response is sent back. However, it gets a lot more interesting when WebSocket traffic happens because it can be triggered from either direction.

Once a WebSocket connection is established, a browser can send or receive messages. A sent message reaches the Protocol type router that determines the next routing handler based on its transport protocol. Hence you can define a router for HTTP and another for WebSocket messages.

These routers are very similar to Django’s URL mappers but map the incoming messages to a consumer (rather than a view). A consumer is like an event handler that reacts to events. It can also send messages back to the browser, thereby containing the logic for a fully bi-directional communication.

A consumer is a class whose methods you may choose to write either as normal Python functions (synchronous) or as awaitables (asynchronous). Asynchronous code should not mix with synchronous code. So there are conversion functions to convert from async to sync and back. Remember that the Django parts are synchronous. A consumer in fact a valid ASGI application.

So far, we have not used the channel layer. Ironically, you can write Channel applications without using Channels! But they are not particularly useful as there is no easy communication path between application instances, other than polling a database. Channels provides exactly that, a fast point-to-point and broadcast messaging between application instances.

A channel is like a pipe. A sender sends a message to this pipe from one end and it reaches a listener at the other end. A group defines a group of channels who are all listening to a topic. Every consumer listens to own auto-generated channel accessed by its self.channel_name attribute.

In addition to transports, you can trigger a consumer listening to a channel by sending a message, thereby starting a background task. This works as a very quick and simple background worker system.

Building a Channels Application Step-by-step

The following screencast covers the creation of a notification application using Django Channels. You can access the code on Github. The intermediate projects like the Echo Consumer can be accessed as branches of the git repository.

The show notes can be accessed on Github.

Check out the video tutorial and let me know if you found it useful! To know more, read the Django Channels documentation.

This article contains an excerpt from "Django Design Patterns and Best Practices" by Arun Ravindran

Get started with Async & Await

Wed, 14 Mar 2018 06:55:30 +0530

You are reading a post from a two-part tutorial series on Django Channels

Asyncio

Asyncio is a co-operative multitasking library available in Python since version 3.6. Celery is fantastic for running concurrent tasks out of process, but there are certain times you would need to run multiple tasks in a single thread inside a single process.

If you are not familiar with async/await concepts (say from JavaScript or C#) then it involves a bit of steep learning curve. However, it is well worth your time as it can speed up your code tremendously (unless it is completely CPU-bound). Moreover, it helps in understanding other libraries built on top of them like Django Channels.

This post is an attempt to explain the concepts in a simplified manner rather than try to be comprehensive. I want you to start using asynchronous programming and enjoy it. You can learn the nitty gritties later.

All asyncio programs are driven by an event loop, which is pretty much an indefinite loop that calls all registered coroutines in some order until they all terminate. Each coroutine operates cooperatively by yielding control to fellow coroutines at well-defined places. This is called awaiting.

A coroutine is like a special function which can suspend and resume execution. They work like lightweight threads. Native coroutines use the async and await keywords, as follows:

import asyncio


async def sleeper_coroutine():
    await asyncio.sleep(5)


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(sleeper_coroutine())

This is a minimal example of an event loop running one coroutine named sleeper_coroutine. When invoked this coroutine runs until the await statement and yields control back to the event loop. This is usually where an Input/Output activity occurs.

The control comes back to the coroutine at the same line when the activity being awaited is completed (after five seconds). Then then coroutine returns or is considered completed.

Explain async and await

[TLDR; Watch my screencast to understand this section with a lot more code examples.]

Initially, I was confused by the presence of the new keywords in Python: async and await. Asynchronous code seemed to be littered with these keywords yet it was not clear what they did or when to use them.

Let’s first look at the async keyword. Commonly used before a function definition as async def, it indicates that you are defining a (native) coroutine.

You should know two things about coroutines:

Don’t perform slow or blocking operations synchronously inside coroutines.
Don’t call a coroutine directly like a regular function call. Either schedule it in an event loop or await it from another coroutine.

Unlike a normal function call, if you invoke a coroutine its body will not get executed right away. Instead it will be suspended and returns a coroutine object. Invoking the send method of this coroutine will start the execution of the coroutine body.

>>> async def hi():
...     print("HOWDY!")
...
>>> o = hi()
>>> o
<coroutine object hi at 0x000001DAE26E2F68>
>>> o.send(None)
HOWDY!
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>

However, when the coroutine returns it will end in a StopIteration exception. Hence it is better to use the asyncio provided event loop to run a coroutine. The loop will handle exceptions in addition to all other machinery for running coroutines concurrently.

>>> import asyncio
>>> loop = asyncio.get_event_loop()
>>> o = hi()
>>> loop.run_until_complete(o)
HOWDY!

Next we have the await keyword which must be only used inside a coroutine. If you call another coroutine, chances are that it might get blocked at some point, say while waiting for I/O.

>>> async def sleepy():
...     await asyncio.sleep(3)
...
>>> o = sleepy()
>>> loop.run_until_complete(o)
# After three seconds
>>>

The sleep coroutine from asyncio module is different from its synchronous counterpart time.sleep. It is non-blocking. This means that other coroutines can be executed while this coroutine is awaiting the sleep to be completed.

When a coroutine uses the await keyword to call another coroutines, it acts like a bookmark. When a blocking operation happens, it suspends the coroutine (and all the coroutines who are await-ing it) and returns control back to the event loop. Later, when the event loop is notified of the completion of the blocking operation, then the execution is resumed from the await expression paused and continues onward.

Asyncio vs Threads

If you have worked on multi-threaded code, then you might wonder – Why not just use threads? There are several reasons why threads are not popular in Python.

Firstly, threads need to be synchronized while accessing shared resources or we will have race conditions. There are several types of synchronization primitives like locks but essentially, they involve waiting which degrades performance and could cause deadlocks or starvation.

A thread may be interrupted any time. Coroutines have well-defined places where execution is handed over i.e. co-operative multitasking. As a result, you may make changes to a shared state as long as you leave it in a known state. For instance you can retrieve a field from a database, perform calculations and overwrite the field without worrying that another coroutine might have interrupted you in between. All this is possible without locks.

Secondly, coroutines are lightweight. Each coroutine needs an order of magnitude less memory than a thread. If you can run a maximum of hundreds of threads, then you might be able to run tens of thousands of coroutines given the same memory. Thread switching also takes some time (few milliseconds). This means you might be able to run more tasks or serve more concurrent users (just like how Node.js works on a single thread without blocking).

The downsides of coroutines is that you cannot mix blocking and non-blocking code. So once you enter the event loop, rest of the code driven by it must be written in asynchronous style, even the standard or third-party libraries you use. This might make using some older libraries with synchronous code somewhat difficult.

If you really want to call asynchronous code from synchronous or vice versa, then do read this excellent overview of various cases and adaptors you can use by Andrew Godwin.

The Classic Web-scraper Example

Let’s look at an example of how we can rewrite synchronous code into asynchronous. We will look at a webscraper which downloads pages from a couple of URLs and measures its size. This is a common example because it is very I/O bound which shows a significant speedup when handled concurrently.

Synchronous web scraping

The synchronous scraper uses Python 3 standard libraries like urllib. It downloads the home page of three popular sites and the fourth is a large file to simulate a slow connection. It prints the respective page sizes and the total running time.

Here is the code for the synchronous scraper:

# sync.py
"""Synchronously download a list of webpages and time it"""
from urllib.request import Request, urlopen
from time import time

sites = [
    "https://news.ycombinator.com/",
    "https://www.yahoo.com/",
    "https://github.com/",
]


def find_size(url):
    req = Request(url)
    with urlopen(req) as response:
        page = response.read()
        return len(page)


def main():
    for site in sites:
        size = find_size(site)
        print("Read {:8d} chars from {}".format(size, site))


if __name__ == '__main__':
    start_time = time()
    main()
    print("Ran in {:6.3f} secs".format(time() - start_time))

On a test laptop, this code took 5.4 seconds to run. It is the cumulative loading time of each site. Let’s see how asynchronous code runs.

Asynchronous web scraping

This asyncio code requires installation of a few Python asynchronous network libraries such as aiohttp and aiodns. They are mentioned in the docstring.

Here is the code for the asynchronous scraper – it is structured to be as close as possible to the synchronous version so it is easier to compare:

# async.py
"""
Asynchronously download a list of webpages and time it

Dependencies: Make sure you install aiohttp using: pip install aiohttp aiodns
"""
import asyncio
import aiohttp
from time import time

# Configuring logging to show timestamps
import logging
logging.basicConfig(format='%(asctime)s %(message)s', datefmt='[%H:%M:%S]')
log = logging.getLogger()
log.setLevel(logging.INFO)

sites = [
    "https://news.ycombinator.com/",
    "https://www.yahoo.com/",
    "https://github.com/",
]


async def find_size(session, url):
    log.info("START {}".format(url))
    async with session.get(url) as response:
        log.info("RESPONSE {}".format(url))
        page = await response.read()
        log.info("PAGE {}".format(url))
        return url, len(page)


async def main():
    tasks = []
    async with aiohttp.ClientSession() as session:
        for site in sites:
            tasks.append(find_size(session, site))
        results = await asyncio.gather(*tasks)
    for site, size in results:
        print("Read {:8d} chars from {}".format(size, site))


if __name__ == '__main__':
    start_time = time()
    loop = asyncio.get_event_loop()
    loop.set_debug(True)
    loop.run_until_complete(main())
    print("Ran in {:6.3f} secs".format(time() - start_time))

The main function is a coroutine which triggers the creation of a separate coroutine for each website. Then it awaits until all these triggered coroutines are completed. As a best practice, the web session object is passed to avoid re-creating new sessions for each page.

The total running time of this program on the same test laptop is 1.5 s. This is a speedup of 3.6x on the same single core. This surprising result can be better understood if we can visualize how the time was spent, as shown below:

A simplistic representation comparing tasks in the synchronous and asynchronous scrapers

The synchronous scraper is easy to understand. Scraping activity needs very little CPU time and the majority of the time is spent waiting for the data to arrive from the network. Each task is waiting for the previous task to complete. As a result the tasks cascade sequentially like a waterfall.

On the other hand the asynchronous scraper starts the first task and as soon as it starts waiting for I/O, it switches to the next task. The CPU is hardly idle as the execution goes back to the event loop as soon as the waiting starts. Eventually the I/O completes in the same amount of time but due to the multiplexing of activity, the overall time taken is drastically reduced.

In fact, the asynchronous code can be speeded up further. The standard asyncio event loop is written in pure Python and provided as a reference implementation. You can consider faster implementations like uvloop for further speedup (my running time came down to 1.3 secs).

Concurrency is not Parallelism

Concurrency is the ability to perform other tasks while you are waiting on the current task. Imagine you are cooking a lot of dishes for some guests. While waiting for something to cook, you are free to do other things like peeling onions or cutting vegetables. Even when one person cooks, typically there will be several things happening concurrently.

Parallelism is when two or more execution engines are performing a task. Continuing on our analogy, this is when two or more cooks work on the same dish to (hopefully) save time.

It is very easy to confuse concurrency and parallelism because they can happen at the same time. You could be concurrently running tasks without parallelism or vice versa. But they refer to two different things. Concurrency is a way of structuring your programs while Parallelism refers to how it is executed.

Due to the Global Interpreter Lock, we cannot run more than one thread of the Python interpreter (to be specific, the standard CPython interpreter) at a time even in multicore systems. This limits the amount of parallelism which we can achieve with a single instance of the Python process.

Optimal usage of your computing resources require both concurrency and parallelism. Concurrency will help you avoid idling the processor core while waiting for say I/O events. While parallelism will help distribute work among all the available cores.

In both cases, you are not executing synchronously i.e. waiting for a task to finish before moving on to another task. Asynchronous systems might seem to be the most optimal. However, they are harder to build and reason about.

Why another Asynchronous Framework?

Asyncio is by no means the first cooperative multitasking or light-weight thread library. If you have used gevent or eventlet, you might find asyncio needs more explicit separation between synchronous and asynchronous code. This is usually a good thing.

Gevent, relies on monkey-patching to change blocking I/O calls to non-blocking ones. This can lead to hard to find performance issues due to an unpatched blocking call slowing the event loop. As the Zen says, ‘Explicit is better than Implicit’.

Another objective of asyncio was to provide a standardized concurrency framework for all implementations like gevent or Twisted. This not only reduces duplicated efforts by library authors but also ensures that code is portable for end users.

Personally, I think the asyncio module can be more streamlined. There are a lot of ideas which somewhat expose implementation details (e.g. native coroutines vs generator-based coroutines). But it is useful as a standard to write future-proof code.

Can we use asyncio in Django?

Strictly speaking, the answer is No. Django is a synchronous web framework. You might be able to run a seperate worker process, say in Celery, to run an embedded event loop. This can be used for I/O background tasks like web scraping.

However, Django Channels changes all that. Django might fit in the asynchronous world after all. But that’s the subject of another post.

This article contains an excerpt from "Django Design Patterns and Best Practices" by Arun Ravindran

Interview with Daniel Roy Greenfeld (PyDanny)

Fri, 09 Feb 2018 15:15:11 +0530

Daniel Roy Greenfeld needs no introduction to Djangonauts. Co-author of the book Two Scoops of Django which is probably on the shelves of any serious Django practitioner. But PyDanny, as he is fondly known as, is also a wonderful fiction author, fitness enthusiast and a lot more.

Having known Daniel for a while as a wonderful friend and a great inspiration, I am so excited that he agreed to my interview. Let’s get started…

How did the idea of writing an ice-cream themed book occur?

The first 50 pages that I wrote were angry and were for a book with an angry name. You see, I was tired of having to pick up after sloppy or unwise coding practices on rescue projects. I was furious and wanted to fix the world.

However, I was getting stuck in what I wanted to say, or didn’t know things. I kept asking my normal favorite resource for help, Audrey Roy Greenfeld. Eventually she started to write (or rewrite) whole sections and I realized that I wasn’t writing the book alone.

Therefore I asked Audrey to be my co-author. She’s a cheerful person and said that if she were to accept, the book had to be lightened. That meant changing the name. After a lot of different title names discussed over many ice cream sessions, we decided to use the subject matter at hand. Which worked out well as the ice cream theme made for a good example subject.

How do you and Audrey collaborate while writing a book?

We take turns writing original material that interests us. The other person follows them and acts as editor and proofreader. We go back and forth a few hundred times and there you go.

For tech writing we use Git as version control and LaTeX for formatting. For fiction we use Google docs followed by some Python scripts that merge and format the files.

What’s the most exciting recent development in Django? Where do you think it can improve?

I like the new URL system as it’s really nice for beginners and advanced coders alike. While I like writing regular expressions, I’m the exception in this case.

Where I think where Django can improve is having more non-US/Europe/Australian representation within the DSF and in the core membership. In short, most of Django core looks like me, and I think that’s wrong. Many of the best Django (and Python) people look nothing like me, and they deserve more recognition. While having Anna Makarudze on the DSF board is a wonderful development, as a community we can still do better in core.

In the case of Django’s core team, I believe this has happened because all the major Django conferences are in the US, Europe, and Australia, and from what I’ve seen over the years it’s through participation in those events is how most people get onto the Django core team. The DSF is aware of the problem, but I think more people should raise it as an issue. More people vocalizing this as a problem will get it resolved more quickly.

With the Ambria fantasy series, you have proven to be a prolific fiction author too. Reminds me Lewis Carroll who wrote children’s books and mathematical treatises. What is the difference in the writing process while writing fiction and non-fiction?

For us, the process is very similar. We both write and we both review our stuff. The difference is that if we make a mistake in our fiction, it’s not as critical. That means that the review process for fiction is a lot easier on us then it is to write technical books or articles. I can’t begin to tell you what a load that is off my shoulders.

Why fantasy? Any literary influences?

We like fantasy because we can just let our imaginations run away with us. For the Ambria series, our influences include Tolkien, Joseph Campbell, Glen Cook, Greek mythology, and various equine and religious studies.

Do you have a daily writing routine?

Like coding on a fun project, when we get to write, we get up early and just start working. When we get hungry or thirsty we stop. The day seems to fly by and we are very happy. We try not to mix writing days with coding days, as we like to focus on one thing at a time. Neither of us are big believers in multi-tasking, so sticking to one thing is important to us.

What’s your favorite part of the writing process?

Getting to write with my favorite person in the whole world, Audrey Roy Greenfeld. :-)

Also, having people read our stuff and comment on it, both positively and negatively.

Do you ever get writer’s block?

Not usually. Our delays are almost always because of other things getting in the way. We’re very fortunate that way!

When I do get writers block, I try to do something active. Be it exercise or fix something in the house that needs it.

Considering you can do cartwheels, I am assuming you are pretty fit. Do you think technology folks don’t give it enough importance?

I’m older than I look but even dealing with an unpleasant knee injury move faster and better than 90% of software developers. And when I look at other coders my age, I see people old before their years. I believe youth is fleeting unless you take a little bit of time every day to keep your strength and flexibility.

Anything else you would like to say?

To paraphrase Jurassic Park, “Just because you can do a thing doesn’t mean you should do a thing.”

As software developers, we have skills that let us do amazing things. With enough time and experience, we can do pretty much anything we are asked to do. That said, we should consider whether or not we should always do what we are asked to do.

For example, the combined power of image recognition, big data, and distributed systems is really fun to play with, but we need to be aware that these tools can be dangerous. In the past year we’ve seen it used to affect opinion and elections, and this is only the beginning. It’s our responsibility to the future to be aware that the tools we are playing with have a lot of power, and that the people who are paying us to use them might not have the best intentions.

Hence why I like to say, “Just because you can do a thing doesn’t mean you should do a thing.”

Do checkout “Two Scoops of Django 1.11: Best Practices for the Django Web Framework” by Two Scoops Press

Django Release Schedule and Python 3

Tue, 30 Jan 2018 06:59:49 +0530

Do long term releases confuse you? For the longest time I was not sure which version of Ubuntu to download - the latest release or the LTS? I see a number of Django developers confused about Django’s releases. So I prepared this handy guide to help you choose (or confuse?).

Which Version To Use?

Django has now standardized on a release schedule with three kinds of releases:

Feature Release: These releases will have new features or improvements to existing features. It will happen every 8 months and will have 16 months of extended support from release. They have version numbers like A.B (note there’s no minor version).

Long-Term Support (LTS) Release: These are special kind of feature releases, which have a longer extended support of three years from the release date. These releases will happen every two years. They have version numbers like A.2 (since every third feature release will be a LTS). LTS releases have few months of overlap to aid in a smoother migration.

Patch Release: These releases are bug fixes or security patches. It is recommended to deploy them as soon as possible. Since they have minimal breaking changes, these upgrades should be painless to apply. They have version numbers like A.B.C

Django roadmap visualized below should make the release approach clearer:

The dates are indicative and may change. This is not an official diagram but something that I created for my understanding.

The big takeaway is that Django 1.11 LTS will be the last release to support Python 2 and it is supported until April 2020. Subsequent versions will use only Python 3.

The right Django version for you will be based on how frequent you can upgrade your Django installation and what features you need. If your project is actively developed and the Django version can be upgraded at least once in 16 months, then you should install the latest feature release regardless of whether it is LTS or non-LTS.

Otherwise, if your project is only occasionally developed then you should pick the most recent LTS version. Upgrading your project’s Django dependency from one feature release to another can be a non-trivial effort. So, read the release notes and plan accordingly.

In any case, make sure you install Patch Releases as soon as they are released. Now, if you are still on Python 2 then read on.

Python 3 has crossed tipping point

When I decided to use Python 3 only while writing my book “Django Design Patterns and Best Practices” in 2015, it was a time when Python 2 versus Python 3 was hotly debated. However, to me Python 3 seemed much more cleaner without arcane syntax like class methods named __unicode__ and classes needing to derive from object parent class.

Now, it is quite a different picture. We just saw how Django no longer supports Python 2 except for the last LTS release. This is a big push for many Python shops to consider Python 3.

Many platforms have upgraded their default Python interpreter. Starting 1st March 2018, Python 3 is announced to be the default “python” in Homebrew installs. ArchLinux had completely switched to Python 3 since 2010.

Fedora has switched to Python 3 as its system default since version 23. Even though python command will launch python3, the symlink /usr/bin/python will still point to python2 for backward compatibility. So it is probably a good idea to use #!/usr/bin/env python idiom in your shell scripts.

On 26 April 2018, when Ubuntu 18.04 LTS (Bionic Beaver) will be released, it is planned to be have Python 3.6 as default. Further upstream, the next Debian release in testing - Debian 10 (Buster) is expected to transition to Python 3.6.

Moving to packages, the Python 3 Wall of Superpowers shows that with the backing of 190 out of 200 packages, at the time of writing, we have nearly all popular Python packages on Python 3. The only notable package remaining is supervisor, which is about to turn green in supervisor 4.0 (unreleased).

Common Python 3 Migration Blockers

You might be aware of atleast one project which is still on Python 2. It could be open source or an internal project, which may be stuck in Python 3 for a number of reasons. I’ve come across a number of such projects and here is my response to such reasons:

Reason 1: My Project is too complex

Some very large and complex projects like NumPy or Django have been migrated sucessfully. You can learn the migration strategies of projects like Django. Django maintained a common codebase for Python 2 and 3 using the six (2 × 3=6, get it?) library before switching to Python 3 only.

Reason 2: I still have time

It is closer than you think. Python clock shows there is a little more than 2 years and 2 months left for Python 2 support.

In fact, you have had a lot of time. It has been ten years since Python 3 was announced. That is a lot of overlap to transition from one version to another.

In today’s ‘move fast and break things’ world, a lot of projects decide to abruptly stop support and ask you to migrate as soon as a new release is out. This is a lot more realistic assumption for enterprises which need a lot more planning and testing.

Reason 3: I have to learn Python 3

But you already know most of it! You might need about 10 mins to learn the differences. In fact, I have written a post to guide Django coders to Python 3. Small Django/Python 2 projects need only trivial changes to work on Python 3.

You might see many old blog posts about Python 3 being buggy or slow. Well, that has not been true for a while. Not only it is extremely stable and bug-free, it is actually used in production by several companies. Performance-wise it has been getting faster in every release, so it is faster than Python 2 in most cases and slower in a few.

Of course, there are lot of awesome new features and libraries added to Python 3. You can learn them as and when you need them. I would recommend reading the release notes to understand them. I will mention my favourites soon.

Reason 4: Nobody is asking

Some people have the philosophy that if nobody is asking then nobody cares. Well, they do care if the application they run is on an unsupported technology. Better plan for the eventual transition than rush it on a higher budget.

Are you missing out?

To me, the biggest reason to switch was that all the newest and greatest features were coming to Python 3. My favourite top three exclusive features in Python 3 are:

asyncio: One of the coolest technologies I picked up recently. The learning process is sometimes mind-bending. But the performance boost in the right situations is incredible.
f-strings: They are so incredibly expressive that you would want to use them everywhere. Instant love!
dict: New compact dict implementation which uses less memory and are ordered!

Yes, FOMO is real.

Apart from my personal reasons, I would recommend everyone to migrate so that the community benefits from investing efforts into a common codebase. Plus we can all be consistent on which Python to recommend to beginners. Because…

There should be one– and preferably only one –obvious way to do it. Although that way may not be obvious at first unless you’re Dutch.

This article contains an excerpt from the upcoming second edition of book "Django Design Patterns and Best Practices" by Arun Ravindran

A Gentle Introduction to Creating a Minimal Hugo Site

Mon, 10 Jul 2017 10:51:31 +0530

When I started using Hugo, I was very impressed by its speed. But I was daunted by the directory structure it creates for a new project. With directory names like archetypes and static, a Hugo site felt unfamiliar and confusing. Fortunately, not every site needs them.

This post tells you how to start small with just the bare minimum files and directories to build a Hugo site without errors. Being minimal, this site will have only one page (essentially, the home page).

You can see the finished project on Github. Let’s start looking at only the top-level files of the project:

.
├── config.toml
├── content/
├── .git/
├── .gitignore
└── themes/

There are only two directories: content which contains your site’s content like posts or articles and themes which are contain various themes - the non-content part of your site like its design and page layouts.

For Hugo, config.toml contains all the configuration settings of your site like the name of the site, author name, theme etc. For this minimal site, we will only mention two lines:

baseURL = "http://example.org/"
theme = "bare"

This is a TOML file. It has a very simple syntax. Each line is written like this:

key = "value"

The baseURL value mentions the URL where the site will be published. Strictly speaking, we don’t need it for a minimal site. But Hugo throws an error if baseURL is not specified.

Next, we mention that we are using the “bare” theme. Essentially, “bare” is a directory inside “themes” directory. We will look at it closely soon.

The Git files are worth mentioning. I prefer to exclude the generated site (having rendered HTML pages) from Git. Assuming that the generated files will go into a directory named “public”, my .gitignore file is simply this:

public

Content files

A content file is usually a text file containing your blog post or article. Typically content files are written in Markdown syntax. But Hugo supports other text formats like Asciidoc, reStructuredText or Org-Mode, which gets converted to HTML. This is easier than directly editing HTML files.

The only content file in this minimal site is _index.md. This filename is special to Hugo and used to specify a page leading to a list of pages. Typically, an index file is used for a home page, a section, a taxonomy or a taxonomy terms listing.

Our _index.md looks like this:

+++
title = "Home sweet home"
+++

This page has **bold** and *italics* formatting.

The two +++ separators divides the document into two parts - the front matter (first three lines) in TOML format and the rest of the document in Markdown format. Front matter specifies the metadata of the file like its title, date or category. Most text editors will treat this file as a Markdown file (due to the .md extension) and ignore the front matter.

Since the _index.md is located inside content directory at the top-level, it will be the first page seen when the user opens the baseURL location i.e. the home page. However, for this page to be rendered, there must be a corresponding template within the theme.

Theme files

So far, we have not specified the look and feel of the site. This is typically mentioned in a separate theme directory (it can also be mentioned inside a layouts directory within the site but its more cleaner this way).

Unlike say Wordpress themes, you might not be able to download an arbitrary Hugo theme and apply it to your site. This is because a theme makes certain assumptions like that your site is a blog and the posts are one-level deep etc, which might not be true in your case. So, I prefer making my own theme while creating new kind of sites. Besides you would eventually want to customize your theme anyway.

The bare theme is located inside the themes directory. Here is directory structure of that theme:

themes/
└── bare/
    ├── layouts/
    │   └── index.html
    └── theme.toml

Note that this is literally a “bare” theme in that it has no stylesheets or images. It can just render a single home page in plain HTML.

The theme.toml like config.toml contains some metadata about this theme. As seen below, it is fairly self-explanatory:

name = "Bare"
license = "MIT"

The layouts directory contains templates which specify how your content files should be rendered into HTML. As you might have guessed, index.html at the top-level is used for rendering a top-level _index.md. This file contains:

<html>
  <body>
    <h1>Welcome!</h1>

    <h2>{{ .Title }}</h2>
    {{ .Content }}

  </body>
</html>

This looks like an HTML page except for parts enclosed in double curly braces ( {{ something }} ). These parts are in Go Template language and will be replaced by their values.

To understand what .Title means you have to understand that the dot in the beginning refers to the current context, which is the current page. In other programming languages this might be written as ThisPage.Title, but here ThisPage is implied and hence omitted.

The value for .Title comes from the front matter. While rendering “_index.md”, it will be replaced by “Home sweet home” (refer _index.md mentioned earlier). The value for .Content will be the rendered HTML from the Markdown file.

Most themes will have certain common elements across a site like a header and footer. Just because we only need a template for one page, the “bare” theme omits all those flourishes.

You can checkout the finished project at Github. You can use this as a landing page or could be a starting point to a larger Hugo project.

Rendering and Publishing

While developing a Hugo site or previewing a post that is being composed, you might want to use the built-in test server. The files are not generated (in the project folder) but you can browse your site locally in your browser

For local development, execute the following command (in the project directory):

$ hugo server -w

This is what you should see in your browser:

Once you are ready to publish the site, you’ll need the generate the files into an output directory. The following command will create generate your site into the “public” directory:

$ hugo --destination public

Now you can copy the files from public directory into the web root of “example.com” or whichever domain that you mentioned as your baseURL. You can use GitHub Pages to host this site for free. I would recommend reading up instructions in their GitHub Pages documentation.

Next, I would recommend looking at other Hugo sites to understand how various features of Hugo have been used. Personally, I wrote this post so that I have a minimal starting point when I start a new project. Enjoy exploring Hugo!

Migrating Blogs (Again) from Pelican to Hugo

Tue, 02 May 2017 10:51:31 +0530

Our universe is in a continuous cycle of creation and destruction. How can this humble site escape that cosmic dance? It is time to change the blogging platform of arunrocks.com again.

You might remember my earlier lengthy justification of using Pelican. I have done numerous hacks and plugins to twist and bend it to my liking. While it has served me well for the past four years, it is time to move on.

The reason, of course, is a much better static site generator - Hugo. A fairly young but very actively developed project, Hugo has some extremely compelling set of qualities, which made the choice to migrate my site fairly obvious. Some of them are:

Content Organization: Hugo believes in the philosophy that your content should be arranged in the same way they are intended for the rendered website. This brings tremendous clarity in identifying file locations and reduces the need for hacks.

By default, some blogging engines (including Pelican) assume that all your “pages” will be finally rendered into the pages folder. A configuration setting can change it to use any URL scheme. But why doesn’t it leave it where it was? The source structure is nested and organized. Why flatten it and lose that information?
Extremely Fast: Even if your site contains hundreds of files, the build time is usually under a second, without cached compilation,… inside a virtual machine, … without an SSD. Yes, it is unbelievable.

This makes the edit-preview development cycle a pleasure to use. Tweak an SCSS variable and the browser will live reload almost instantaneously. I make a hundred tweaks after a page gets rendered, so a short feedback loop is ideal for my workflow.
No Stack: This is usually marketed as the “single executable” advantage of Go applications. But it matters much more profoundly in the case of Hugo.

Many new languages like Javascript and Python need a pile of third party libraries to make their tools run (even if they are “batteries included”). While it might be easy to setup initially with a single command like pip install Foo, future invocations might need you to update everything to their latest versions first and then fixing whatever breaks.

Even though dynamic languages do not require a lengthy compilation step, a constant stream of updates can get tiring. “Updating my packages”" has become the new “I am compiling”.

Ok, end of my rant. Hugo is a single self contained executable (in all platforms). This not only reduces the number of moving pieces that you need to keep track of (my requirements.txt for the site listed 20 packages), it simplifies the ongoing maintenance of your site.

I would rather not have the headache of managing code environments just for writing a blog. In Arch Linux, where I develop my site, it is even easier considering it is a rolling distribution. Hugo automatically updates along with the rest of the system. The updates are usually backward compatible and nothing breaks.
Actively Developed: Many static blog generator projects have this problem. The early days would have the community engaged and actively involved. Then the interest dwindles, code gets stale and the backlog of issues grows. Hugo is still in its early days and has a energetic community. Hopefully the interest will continue to grow.
Not too Blog-oriented: My site, like most personal sites, is predominantly a blog. But it is not just a blog, it is a full-fledged site. Hugo has very minimal assumptions about what kind of site you have. For instance, I can convert a JSON file with all my talks into a Talks page. I just need to define a template for a new type of page in my layouts.

It is not all Peaches and Cream

Hugo does have some rough edges. But considering the rate of its development, you might find that all these problems have been solved by the time you read it. But, for the record, I did come across these issues:

Lack of Static Assets Pipeline: I use SCSS to create my CSS files. In general, Hugo is unaware of any static assets pipeline which may involve compiling SCSS, minification or compression. I use a Makefile which invokes the compiler as a dependency for build. Sometimes I need to stop live reloading and explicitly call Make. Since everything is fast, it doesn’t matter much.
Menu Generation Flaky: I found the “active section in menu” pattern used in most website menus a bit hard to implement. There are others who have faced similar issues. I resorted to not using this pattern for now.
Outdated Blog posts Hugo documentation is extensive and covers a lot of material. But good documentation is very hard to get right. Some sections are confusing and I often look for better explanations say from blogs.

But one needs to watch out for outdated content. For instance, I needed to change my RSS feed location and one blog recommended changing RSSUri in the config file. This is deprecated. Now, you need to use output formats say like this:
```
outputFormats.RSS:
  BaseName: "index"
  Path: "feed/index.xml"
```
Python Tooling: While using Pelican, I was happy that if I needed any additional functionality, I could always write a plugin in Python. This doesn’t worry me for two reasons. First, Hugo’s template system is very expressive and handles many of such needs. Second, ShortCodes offer a much cleaner alternative to plugins.
Inconsistent Casing Sometimes a variable is mentioned in title case in one place and lower case in another. I would end up wondering whether it is copyright or Copyright or CopyRight. Probably, there is a naming convention. But I haven’t found it yet.

As you can see, most of these issues are solvable. Hugo is a long way from version 1.0 (currently we have v20.7). So this might all be fixed by then.

Changing a Jet Engine Mid-flight

Once your site achieves a certain amount of traffic, then making any change is fraught with risk. Every time I change this site’s underlying platform I make a short TODO list. It looks something like this:

Setup Redirects: Map the old link structure to the new ones
Fix Renders: Markdown is not quite a standard. Each Markdown engine has its own quirk. I manually check the rendered HTML is some cases.
Check Missing Images: Things have to move around to adjust for various source layouts. So images, scripts or download links can return 404s.

Even after all these checks, I still need to watch the error log for omissions. As this happens in my spare time, the whole migration “project” takes months. It is pretty much like changing a jet’s engine mid-flight.

Some Takeaways

In short, I would say you can never underestimate the sheer amount of work involved in migrating a site. There is a very good post on site migration by folks at Mozilla that covers many aspects that people tend to miss. Unless you have very good reasons to move, I would suggest that you stick with your blogging platform of choice.

As for me, time to start thinking about migration to HTTPS 😉

Hangman in more than three lines of Python

Tue, 16 Feb 2016 20:03:00 +0530

Recently, Danver wrote about a brilliant implementation of the Hangman game in just three lines of Python. But the shortened code might be hard to understand. In this post we will rewrite his code in a more idiomatic Python 3 style.

Spoiler Alert: If you prefer a fun exercise, try doing it yourself first and then compare with my version.

Danver’s implementation cleverly uses the system dictionary (pre-installed in Linux and Mac, but we’ll see how it can work in Windows too) for picking a random word. It also displays a neat ASCII-art hangman at every step. Except for a few minor improvements, I have tried to retain the essence of Danver’s original in my rewrite below.

# -*- coding: utf-8 -*-
"""A simple text-based game of hangman

A re-implementation of Hangman 3-liner in more idiomatic Python 3
Original: http://danverbraganza.com/writings/hangman-in-3-lines-of-python

Requirements:
  A dictionary file at /usr/share/dict/words

Usage:
  $ python hangman.py

Released under the MIT License. (Re)written by Arun Ravindran http://arunrocks.com

"""

import random

DICT = '/usr/share/dict/words'
chosen_word = random.choice(open(DICT).readlines()).upper().strip()
guesses = set()
scaffold = """
|======
|   |
| {3} {0} {5}
|  {2}{1}{4}
|  {6} {7}
|  {8} {9}
|
"""
man = list('OT-\\-//\\||')
guesses_left = len(man)

while not guesses.issuperset(chosen_word) and guesses_left:
    print("{} ({} guesses left)".format(','.join(sorted(guesses)), guesses_left))
    print(scaffold.format(*(man[:-guesses_left] + [' '] * guesses_left)))
    print(' '.join(letter if letter in guesses else '_' for letter in chosen_word))
    guesses.update(c.upper() for c in input() if str.isalpha(c))
    guesses_left = max(len(man) - len(guesses - set(chosen_word)), 0)

if guesses_left > 0:
    print("You win!")
else:
    print("You lose!\n{}\nHANGED!".format(scaffold.format(*man)))
print("Word was", chosen_word)

You can view or download the gist on Github as well.

HO_ IT _ORKS

The code makes excellent use of Python’s built-in set data structure and string format function.

Right after a boring module docstring and the import statement, we initialize the following variables:

DICT: Path to a dictionary file
chosen_word: Randomly picked line (aka. a word) from DICT
guesses: Set of letters guessed by the user, so far
scaffold: Hangman drawn in ASCII art as a format string
man: Missing ASCII characters for scaffold
guesses_left: Remaining letter guesses

The while loop, whose exit condition will be explained soon, is our main game loop. The first print function reminds you the guesses you have entered so far (in alphabetic order) and the number of guesses left. The next print function shows the ASCII hangman with enough missing parts from man based on the number of wrong guesses.

Finally, the third print function shows the chosen_word by revealing only the characters from your guesses and replacing the rest with underscores. In the following line, we read a character or even several characters from the user, uppercase them, filter out the non-alphabets and update the guesses set, all in a single line:

guesses.update(c.upper() for c in input() if str.isalpha(c))

Next, we calculate the number of letters we got wrong. As the diagram below shows, this is the difference of the sets guesses and chosen_words (In the diagram, the letters - C,M,T). We find how many guesses are left by subtracting the set of letters of man from the letters we got wrong. The max function ensures that we don’t go below zero.

Eventually, you could win if your guesses cover all the right letters. In the language of sets, this is when guesses set grows large enough to become a super set of chosen_word set. This is indicated by the dashed purple circle. Hence, the while loop will continue until this happens or we run out of remaining guesses.

Finally, we show a win or lose message based on whether we ran out of guesses. In either case we reveal the initial chosen_word.

IMPRO_EMEN_S

For comparison, here is Danver’s original three-liner:

license, chosen_word, guesses, scaffold, man, guesses_left = 'https://opensource.org/licenses/MIT', ''.join(filter(str.isalpha, __import__('random').choice(open('/usr/share/dict/words').readlines()).upper())), set(), '|======\n|   |\n| {3} {0} {5}\n|  {2}{1}{4}\n|  {6} {7}\n|  {8} {9}\n|', list('OT-\\-//\\||'), 10
while not all(letter in guesses for letter in chosen_word) and guesses_left: _, guesses_left = map(guesses.add, filter(str.isalpha, raw_input('%s(%s guesses left)\n%s\n%s:' % (','.join(sorted(guesses)), guesses_left, scaffold.format(*(man[:10-guesses_left] + [' '] * guesses_left)), ' '.join(letter if letter in guesses else '_' for letter in chosen_word))).upper())), max((10 - len(guesses - set(chosen_word))), 0)
print 'You', ['lose!\n' + scaffold.format(*man), 'win!'][bool(guesses_left)], '\nWord was', chosen_word

Got it? Okay, maybe it needs a bit of explanation. Python is a whitespace-aware language designed for maximum readability. So it takes a lot of questionable tricks to write one-liners, some of which I have used myself in my post “Python One-liner Games”. This time we are going in the opposite direction.

Probably, the most visible changes are:

License: Mentioned in a doc string rather than as a string
Import: No longer need to use the internal __import__ function
Scaffold: Multi-line string rather than a one-liner
Print: Changed from a statement to a function, as in Python 3
Raw_input: Changed to an input function, as in Python 3
C-style %s formats: Replaced with str.format everywhere

But, there are also some less evident code changes:

Loop condition: Replaced list comprehension in while condition with a superset check
Negative index: No need of man[:10-guesses_left], just man[:-guesses_left]
Set update: Replace map/filter with a set.update i.e. from this:

    map(guesses.add, filter(str.isalpha, raw_input...

with this:

    guesses.update(c.upper() for c in input() if str.isalpha(c))

Magic numbers: The number 10 is no longer hard-coded and replaced with len(man)

As you can see, I have preferred set functions over list comprehensions. The choice of set functions like my_set.update over set operators like my_set += was intentional as the former allows any iterable as an argument and not just sets. In most cases, I have used generator expressions for efficiency.

W_Y PYT_ON 3?

I must admit that the actual reason I rewrote the Hangman was because it didn’t work on my Arch Linux box, which runs on Python 3. Over the last few months, I have completely switched over all my projects (and my book) to Python 3. So far I have had no reason to complain.

Apart from the syntactic differences, you will find some solid advantages of using Python 3 versus Python 2:

Unicode files: Since the dictionary file path can be changed, I created a test file with a few Malayalam words in separate lines. The Python 2 version just exits with a win message. This is probably because str.isalpha filters out all the characters. Whereas, in Python 3, everything just works.
Removes arcane forms: No more raw_input (who uses the old input?) or print statements

If you are on Windows, you can download pretty much any of the dictionaries online (even the offensive ones) and change the path in DICT to something like dict.txt. Or, you can create a simple text file with one word on each line. Pro Tip: Hangman is great for learning foreign languages so try creating a Unicode file with some foreign language words.

Before any of the Python 2 fans lash out on me, let me note in the passing that this code will work perfectly on Python 2.7+ if you replace input with raw_input. Cheers!

FI_AL COMME_T

This post is by no means a critique of Danver’s code. In fact, I loved his implementation and totally didn’t waste my weekend playing Hangman (27 wins out of 30 games!!!). It had a lot of fun mapping the problem to sets and making small improvements.

Hope you found some new Python best practices or just an awesome word game to kill time!