Performance awareness when coding

Image from https://www.pexels.com/@cesar-galeao-1673528/

 

Python is slow compared to many popular programming languages. It is because of its dynamic nature and versatility. Moreover, it is not statically typed which does not allow the compiler to optimize the code. This blog is about two cases where we can write better Python code (w.r.t. performance). The concept is the same when we are writing in other programming languages too.

I am running the code posted below on MacbookPro 2.6 GHz 6-Core Intel Core i7.

Setup

We created a decorator for measuring the time taken to execute a function.

import time

def timer(message: str):
    def decorator(fn):
        start = time.process_time()
        fn()
        print(f"{message}: time taken = {time.process_time() - start} secs")
    return decorator

Finding the minimum value in a list

We have seen this many times where software engineers sort a list just to get the minimum value. It can be because they are tired or do not pay much attention.

list_integers = [random.randint(1, 10000001) for item in range(1, 10000001)]

@timer("sort on list of integers")
def sort_list_int():
    list_integers.sort()
    print(list_integers[0])

and we got

sort on list of integers: time taken = 4.519529 secs

Some engineers may attempt to implement a for loop like this

@timer("for-loop on list of integers")
def min_fn():
    min_val = sys.maxsize
    for i in list_integers:
        min_val = min(min_val, i)
    print(min_val)

and the time taken is

for-loop on list of integers: time taken = 1.5227250000000012 secs

We can use the min function directly on the list to get better performance

@timer("min on list of integers")
def min_list_int():
    print(min(list_integers))

min is a Python built-in function, and we always lean towards using built-in functions. This function also shows the beauty of Python, just a single line of human-readable code.

min on list of integers: time taken = 0.15220400000000112 secs

We see a significant improvement in execution time in these three examples (from 4.5 to 1.5 to 0.15 seconds). Following is the code on the list of objects.

import random
import sys
from dataclasses import dataclass

@dataclass
class Sample:
    x: int
    y: int
    
list_objects = [Sample(random.randint(1, 10000001), random.randint(1, 10000001))
               for item in range(1, 10000001)]

@timer("sort list of objects")
def sort_objects():
    list_objects.sort(key=lambda item : item.x)
    print(list_objects[0])

@timer("for-loop on list of objects")
def min_fn_list_objects():
    min_val = sys.maxsize
    for i in list_integers:
        min_val = min(min_val, i)
    print(min_val)

@timer("min on list of objects")
def min_objects():
    print(min(list_objects, key=lambda item : item.x))

and we got

Sample(x=1, y=9125320)
sort list of objects: time taken = 7.828590999999996 secs

1
for-loop on list of objects: time taken = 6.7428839999999965 secs

Sample(x=1, y=9125320)
min on list of objects: time taken = 4.963247000000003 secs

From these results, we see that differences on a list of objects are not as good as on a list of integers.

Finding the first match

There are 2 ways to find the first match result.

@timer("comprehension list on list of integers")
def comp_list_on_list_int():
    print([x for x in list_integers if x < 1000][0])

@timer("generator on list of integers")
def generator_on_list_int():
    print(next(filter(lambda x: x < 1000, list_integers)))

for simplicity, we do not handle the Index out of bound and Stop Iteration error.

The former builds the entire list just to get the first item on the list. The latter creates a generator. And here are the result.

4
comprehension list on list of integers: time taken = 1.3737079999999935 secs
4
generator on list of integers: time taken = 2.3999999996249244e-05 secs

It is evident that the latter wins by a huge margin.

We also did the same tests on a list of objects

@timer("comprehension list on list of objects")
def comp_list_on_list_objects():
    print([x for x in list_objects if x.x < 1000][0])

@timer("generator on list of objects")
def generator_on_list_objects():
    print(next(filter(lambda x: x.x < 1000, list_objects)))

and we got

Sample(x=1, y=9125320)
comprehension list on list of objects: time taken = 4.486391999999995 secs

Sample(x=1, y=9125320)
generator on list of objects: time taken = 1.6000000002236447e-05 secs

For the list of objects, we are also seeing a huge performance boost.

Conclusion

These concepts do not limit to Python. I am using Python as the programming language here. And, these are also things that software developers need to know. Being aware of little things like this helps us to write better code.

I am active on Stackoverflow (https://stackoverflow.com/users/3808826/d-seah), and I see rooms for improvement in this area when I am looking at the questions posted.

gist for the code above