ArunRocks - All articles

Easy and Practical Web scraping in Python
This post is inspired by an excellent post called Web Scraping 101 with Python. It is a great intro to web scraping to Python, but I noticed two problems with it:
1. It was slightly cumbersome to select elements
2. It could be done easier
If you ask me, I would write such scraping scripts using an interactive interpreter like IPython and by using the simpler CSS selector syntax.

Let’s see how to create such throwaway scripts. For serious web scraping, Scrapy is a more complete solution when you need to perform repeated scraping or something more complex.

The Problem

We are going to solve the same problem mentioned in the first link. We are interested in knowing the winners of Chicago Reader’s Best of 2011. Unfortunately the Chicago Reader page shows only the five sections. Each of these sections contain award categories e.g. ‘Best vintage store’ in ‘Goods & Services’. Within each of these award category pages you will find the winner and runner up. Our mission is to collect the names of winners and runner ups for every award and present them as one simple list.

The Setup

Start python, IPython, bpython or any other interactive python interpreter ...
Continue Reading →
Moving Blogs to Pelican

Redesigning a site takes a lot of planning. I usually take it as an opportunity to study what is the state of the art and implement many long standing items from my wish list. So this post will be a lot more than explaining why I moved to Pelican from Jekyll.

Background

I had stopped using Wordpress about two years back. I was a long time user of Wordpress and had even written some plugins. But I was afraid that whatever I had written was turning into a binary blob in some database.

Don’t believe me? Just open your Wordpress database tables in phpmyadmin or any other tool. If you use Microsoft Word, be prepared to get shocked seeing a lot of unnecessary tags like <SPAN>, class names like class=MsoNormal or converted punctuation marks like ‘. Even web-based composers add a lot of horrible markup. I even tried using the plain text editor with Markdown markup. But something would still get mangled somehow.

Another major annoyance was that Wordpress had a huge attack surface. Everytime someone finds out a Wordpress exploit, your site is at risk. Only way to secure your content was to take frequent backups and ...

Continue Reading →
Treeify - Converting Tree Data Structures
Some of the most interesting algorithms in Computer Science are involving trees. They are simple and often leverage recursion. For instance, pre-order traversal of a tree, of any complexity, can be written as follows:
```
void preorder(tree t)
{
        if(t == NULL)
                return;
        printf("%d ", t->val);
        preorder(t->left);
        preorder(t->right);
}
```
For a hobby project, I was faced with an interesting problem of converting a flat representation of a tree into a nested data structure. A flat representation of a tree looks like this:
```
0
0
1
1
2
3
2
1
```
Each number refers to the nesting level within a tree. After conversion to a nested structure, it should look as follows (square brackets is the Python syntax for a list):
```
[ 0,
  0,
  [ 1,
    1,
    [ 2,
      [ 3 ],
    2],
1]]
```
I expected this algorithm to be fairly easy to find, but I didn’t have much success with Google. So, as any self respecting programmer would, I rolled up my sleeves and wrote a Python implementation:
```
def treeify(cs):
    cur = 0
    tree = []
    stack = [tree]
    for c in cs:
        if c['level'] > cur:
            l = [c]
            stack[-1].append(l)
            stack.append(l)
        elif c['level'] < cur:
            while 1:
                stack.pop ...
```
Continue Reading →
Buying Kesar in Kashmir

One of the most adventurous things we did after marriage was to go to Kashmir for Honeymoon. Right opposite the Mangalore STP office we have Swastika Tours, a friendly operator whom we have trusted for many tours in the past. In 2008, when we approached him we had all sorts of exotic places in our mind like Maldives, Mauritius or Europe.

But with the memory of the Tsunami fresh in our minds, we quickly ruled out all islands and sea facing locations. The hill-stations and snow covered locales in the northern hemisphere remained. But it was Santosh from Swastika who planted the idea of Kashmir in our minds. In 2007, Kashmir was considered to be a dangerous place to visit, thanks to the media reports of fresh gun fires or terrorist attacks. We were hesitant. “Why not?”, he asked, “My sister just visited the place and she liked it better than her Switzerland trip. It is completely safe these days.” We looked at each other silently. We both knew that we were up for the challenge.

The one week Kashmir trip indeed turned out to be one of the best trips we have ever had. It was a land completely ...

Continue Reading →
The Last Drop on the Planet

“Chaos often breeds life, when order breeds habit.” — Henry B. Adams

It was a breezy, luminescent and quiet evening in planet Terra. Perhaps too quiet for an eventful day such as this. Perhaps that’s why emperor Kilter ordered a Pan-D911. Soon random bright spots popped all over the planet like an attack of pimples. Soon it smeared itself onto millions of other bright dots turning into something like an attack of rash.

Like a huge tidal wave the dots arose and swiftly spiralled upward into the cloudy magenta skies. With militaristic precision, the dots arranged themselves into concentric rings in space. From Terra, it was a magnificent spectacle. However, it didn’t impress the Queen, Triara. Made of sixty billion nanobots, Triara was by far the most exquisitely complex being in their planet. That meant she was female and extremely attractive, not to mention that she was undoubtedly their Queen. Even though Kilter was called the mighty Emperor and launched vain crusades that displayed his prowess, it was obvious that it was the Queen upon whom the future course of events rested upon. She scoffed at this ostentatious waste of Nino. Nino, being the hydrocarbon fuel, which powered everything ...

Continue Reading →
Bonus Screencast - HTML5 makeover for the Django blog

Thank you for the great feedback on my first screencast about building a blog in Django 1.3. A lot of people told me that they found the video helpful in understanding what Django is capable of.

As promised earlier, I have recorded a short video (7 mins) that explains how you can use an HTML5 template to improve the appearance of your blog. You can use any free template or create your own.

Django’s (version 1.3) static files app is being used here. Unlike prior versions, there are no changes to urls.py required here. Linking to the admin site is also demonstrated.

It is best to watch it directly at Youtube in HD.

UPDATE: The blog with the updated template has been uploaded to github

Continue Reading →
Building a blog in 30 mins with Django (Screencast)
Just uploaded a quick screencast showing how to build a blog in Django in just 30 minutes (plus a couple of seconds :) ). It shows off a lot of new features that we have in Django since 1.2.

It basically covers:
- Using django.contrib.admin to create fully-featured admin pages
- Adding tags to posts easily using taggit app
- Class-based generic views for rapidly building pages (New in Django 1.3)
- Using template inheritance and filters
- Leveraging django.contrib.syndication for a simple feed
It is best to watch it directly at Youtube in HD. This was my first screencast so I apologize if something is not properly done.

I plan to add a short bonus video soon demonstrating how this bare-bones blog can be restyled into a modern-looking HTML5 site (UPDATE: Bonus video is up now).

UPDATE: The source code for the created project has been uploaded to github
Continue Reading →
Sports Day

This is a short story that was written in three small installments

It was a blazing summer afternoon. The dusty maidaan seemed rippling under the intense summer sun. Yet, at the northern end there was a wide white stall decorated with multi-coloured festoons and balloons. Hanging from the front of the roof was a cloth banner that read ‘Annual Sports Day, Vilvattam Senior Secondary School’.

Uniformed children were sitting in big groups outside the oval perimeter drawn with chalk powder. When they were not cheering for their favourites, they talked, sang songs and picked fights over sharing the precious glucose powder given by the teachers. They were having a hell of a good time. The static on the loudspeaker indicated that the microphone was turned on. Nobody paid attention to the announcer, who was in fact Malathi teacher seated behind a covered metal table inside the stall. Her naturally loud voice ensured that she was the announcer by choice, every year.

“… and ask Mary teacher to come soon”, a broken piece of an earlier conversation inadvertently screamed through the loudspeakers. “Sorry. Next, we have the boys’ 100 meters dash”. Suddenly, silence prevailed among the noisy crowd. This was one event ...

Continue Reading →
Atari and Gaming History

Imagine your 8 year old cousin, who is learning to use computers, asking you about the Save icon on Microsoft Word. The image of a floppy disk is almost universally used to denote ‘saving a file’. But with floppy disks gone, would the coming generation understand the symbolism?

~~~~~~~~~~~~~~~

I had the same problem with the joystick icon commonly used to denote Games. Typically it is drawn as a stick pointing up, protruding from a box. Most of the consoles that I was familiar with had much more elaborate game controllers or gamepads. However this simple black-coloured gaming device remained a mystery to me.

Until I read the book ‘Racing the Beam: The Atari Video Computer System’ that is. I started reading this book out of curiosity and due to a certain interest in gaming history. But soon realised that it is a must-read for anyone interested in Game design or even the roots of Gaming culture. It covers not just the historical context of the Atari VCS and its popular games but some of its nearly impossible technical constraints.

One of the most amazing thing about Atari games were that they could use only 128 bytes of RAM. Yes you ...

Continue Reading →
This is a Game. There are rules

I have been recently reading a book on game design that primarily focuses on non-digital games. It is called Challenges for Game Designers. It distills the art of game design to using easily available materials like paper, dice or playing cards. They call it a non-digital approach.

We have all played such games at some point in time. Before video games, as kids, we used to play snakes and ladders or chess. In Kerala, we have several indoor games using easily available materials like pebbles, cowries (kavadi in Malayalam), circassian seeds (manjaadi in Malayalam) and irkili sticks (stiff mid-ribs of coconut leaves). The rules were simple - say, remove the smallest stick without disturbing others or align the seeds in a row. These ‘eco-friendly’ and ‘low-cost’ games used to provide hours of fun to children and adults.

Compared to modern video games these games lack high production values marked by hundreds of detailed 3d modelled characters and unfolding of several hours worth of intricate plots and storylines. These traditional games don’t have any stories, they are mostly tests of skill or chance. They were fun interactive experiences fuelled by imagination.

What’s wrong with my digital game?

The biggest advantage ...

Continue Reading →

Page 1 / 16 »

The Problem

The Setup

Background

What’s wrong with my digital game?