Py Wrangler

Using _ (underscore) as a placeholder or temporary variable name

2017-01-17T12:05:00.000-08:00

The underscore character is often been used as a placeholder or throwaway variable in python. Like the following snippet:

for _ in range(10):
    print("Hello World")

Here, as we do not use the variable needed for iterating range,it's replaced with the `_` (underscore) character.

In python there are three main usage of `_` (underscore) character. (Taking reference from a well viewed Stack Overflow answer)

To hold the result of the last executed statement in an interactive interpreter session. In the following snippet we did not assign the return value of the split()method. But we can access the value using the `_` character.

In [1]: "Hello World".split()
Out[1]: ['Hello', 'World']

In [2]: _
Out[2]: ['Hello', 'World']

For translation lookup in i18n. Like the following often seen python web based projects.

forms.ValidationError(_("Please enter a correct username"))

As a general purpose "throwaway" variable name to indicate that part of a function result is being deliberately ignored. Here get_product will returns 3 value but we are only interested in the first two.

product_name, product_code, _ = get_product(product_id)

If we need a throwaway or placeholder variable, it's always a good practice to use the underscore character. Using a named variable in such case can create confusion. Using the `_` (underscore) characters in these circumstance will tell the reader that the data is unnecessary.

Avoiding direct comparison with None or True, False

2017-01-02T20:24:00.000-08:00

Comparing Bool type objects or None

Consider the following snippets:
snippet 1

if doc is None:
    doc = {"key": "value"}

snippet 2

if not doc:
    doc = {"key": "value"}

I have seen both types of code several times. Now the obvious question is which one is more "pythonic" and why.

Here the second approach is the "pythonic" way to do it. We know that the is keyword determines if the comparing objects are actually same objects or not. So if the doc is any other objects that are considered False in python then the code will not work as it was supposed to. But the second code will perform for any thing that is considered False in python.

Things that are considered False in python are:

False
None
0 (Numeric)
[] (empty list)
{} (empty dictionary)
if __len__() or __nonzero__() return False or 0

So unless it is necessary to compare directly with any of these objects, snippet 2 type of approach should be followed.

GNU Screen how to

2016-01-28T02:12:00.000-08:00

Overview

screen (GNU screen) is a software which is used to create virtual terminals. It can run on it's own when not under direct user control. This is especially useful while working with a remote system, when a disconnect can occur at any moment. In this case, the screen session will continue to execute and you may reattach a running screen later.

I found it very useful in the cases where I need to run a script in a remote server which takes long time time to execute.

Installation

In a Debian based operating system it can be installed by this command:

sudo apt-get install screen

Basic Usage

Starting a new screen session:

screen -S <screen_name>

This command will start a new screen session. After running the process, the screen can be detached by pressing:

ctrl + a + d

All available screen can be viewed using this command:

screen -ls

Reattaching a screen can be performed by typing:

screen -r <screen_name>

REST Api with Flask-Restful and MongoDB

2015-05-01T09:13:00.000-07:00

What is REST

REST (Representational State Transfer) is an architecture style for designing networked applications. The idea is using simple HTTP methods for making calls between machines rather than complex mechanisms such as CORBA, RPC or SOAP.

So a RESTful system typically communicates over the Hypertext Transfer Protocol with the same HTTP verbs (GET, POST, PUT, DELETE, etc.)

Flask and Flask-RESTful

Flask is a microframework for Python based on Werkzeug, Jinja2. Flask-RESTful provides an extension to Flask for building REST APIs.

Installing packages

For the api, we need to install some packages like Flask-Restful, pymongo etc. It's good practice to install packages and modules for a project in a virtual environment. So i am going to create a virtual environment and activate it.

$ virtualenv env

$ source env/bin/activate

Now the required packages can be installed using pip. (Link: requirements.txt)

$ pip install -r requirements.txt

Api

Lets Code the API first.

This is a basic api with only one Resource Student. First we created an instance of Flask,
app = Flask(__name__)
after that we configured app with MongoDB.

The resource Student has HTTP methods defined in it as it's own method. By calling different HTTP methods we can perform basic CRUD operations. Like post for inserting/creating, put for updating, get for reading and delete for deleting a document.

In the code post, put, delete methods are basic. These methods perform create, update and delete operation on the collection student.

But in the get method, by calling different url we can perform different operations. If no parameter passed with url, the method will return all the documents in the collection. If a registration number passed with the url then, it will check the parameter name and will return one document. If request is made in /api/department/<string:department> url, it will check for parameter name department in the get method and returns all the documents for a particular department.

There are several urls for same resource so we added different endpoints for them.

Testing the Api

We will use python requests module (version > 2.4.2) for testing the api. But also curl and other mechanism can be used. For get request we can view the data in browsers also.

requests module gives us convenient way to test our api. Enough REST. Lets take some rest ;)

PYTHONPATH and importing modules in python

2015-03-10T11:23:00.000-07:00

What is PYTHONPATH

PYTHONPATH is an environment variable that defines additional locations of Python modules. This is the default search path for module files. This means that python interpreter will search for modules in the location of the PYTHONPATH.

How importing works in python

When a module named mymodule is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named mymodule.py in a list of directories given by the variable sys.path. sys.path is initialized from these locations:

The directory containing the input script (or the current directory).
PYTHONPATH (a list of directory names, with the same syntax as the shell variable PATH).
the installation-dependent default.

problem scenario and solve:

|--__init__.py

|--B

|--|--__init__.py

|--|--greetings.py

|--C

|--|--__init_.py

|--|--mymodule.py

Here A is a python package. Inside A, there are two packages B and C.

greetings.py

def my_function():
    print "Congrats Bangladesh Cricket Team!! :D"

mymodule.py

from B.greetings import my_function
my_function()

Now, there is a problem. Package A is not in python interpreter's search path. So it can not import greetings function from B within C. If mymodule.py is run, it will raise an ImportError as expected.

$ python mymodule.py

Traceback (most recent call last):

File "mymodule.py", line 3, in <module>

from B.greetings import my_function

ImportError: No module named A.B.greetings

To solve it, we need to add the absolute path of package A in sys.path inside mymodule.py before importing greetings from B. Path will be appended in sys.path. This will not affect other python program because the path will not be modified for any other Python processes.

mymodule.py

import os
import sys

PATH = os.path.abspath(os.path.join(os.pardir))
sys.path.append(PATH)

from B.greetings import my_function
my_function()

Now as the path of A is added, Python interpreter can find the package B inside A. Let's run again this function

$ python mymodule.py

Congrats Bangladesh Cricket Team!! :D

MongoDB model with PyMongo

2015-01-29T11:26:00.002-08:00

MongoDB

MongoDB is a popular open-source NoSQL database. NoSQL is a database technology to store and retrieve data which are not stored in database like the traditional tabular form of RDBMS. Scalability and superior performance are main beneifts of NoSQL databases over RDBMS. Mongodb uses JSON-style documents with dynamic schema to store data. Installation instructions of MongoDB will be found in it's documentation - Install MongoDB.

PyMongo

PyMongo is a Python distribution containing tools for working with MongoDB, and is the recommended way to work with MongoDB from Python. To install PyMongo execute the command in terminal:

$ sudo pip install pymongo

Creating a model

I am going to create student model which will be used to get, create, update, delete objects form database.

models.py

# -*- coding: utf-8 -*-
import pymongo


class Student(object):
    def __init__(self, registration, **kwargs):
        self._id = registration
        self.first_name = kwargs.get('first_name', '')
        self.last_name = kwargs.get('last_name', '')
        self.department = kwargs.get('department', '')
        self.session = kwargs.get('session', '')
        self.mobile = kwargs.get('mobile', '')

    def __str__(self):
        return self._id

    @staticmethod
    def get_db(dbname='student_db'):
        """
        make database connection and return a database instance
        """
        client = pymongo.MongoClient(host='localhost', port=27017)
        return client[dbname]

    @classmethod
    def get(cls, registration):
        """
        get student by registration number
        """
        db = Student.get_db()
        student_info = db.students.find_one({'_id': registration}, {'_id': 0})
        if student_info:
            student = Student(registration=registration, **student_info)
        else:
            student = Student(registration=registration)
        db.connection.close()
        return student

    @classmethod
    def get_all(cls):
        """
        get all students and return a list. for many student return a cursor instead.
        """
        db = Student.get_db()
        students = list(db.students.find())
        db.connection.close()
        return students

    def save(self):
        """
        save() method will create or update a document based on the _id attribute.
        """
        db = Student.get_db()
        db.students.save(self.__dict__)
        db.connection.close()

    @classmethod
    def remove_all(cls):
        """
        remove all documents for the collection.
        """
        db = Student.get_db()
        db.students.remove()
        students = list(db.students.find())
        db.connection.close()
        return students

    def remove(self):
        """
        remove a object from the collection.
        """
        db = Student.get_db()
        db.students.remove(self._id)
        db.connection.close()
        return self._id

Now this model can be used for CRUD operations.

>>>from models import Student
>>>student = Student.get('2008331041')
>>>student.first_name = "Salman"
>>>student.last_name = "Wahed"

>>>student.save()
>>>Student.get_all()

[{u'_id': u'2008331041',
u'department': u'',
u'first_name': u'Salman',
u'last_name': u'Wahed',
u'mobile': u'',
u'session': u''}]

Unit testing in python using unittest framework

2015-01-23T02:13:00.002-08:00

What is Unit test

Unit test is a process of testing an unit of testable code like functions, classes, procedures etc of an application using test code. Mainly programmers write unit tests for their codes while developing software. Unit testing reflects the programmers view on how things should work. This is the main theme of TDD (Test-driven development).

Unit test in python

In python there are some frameworks to do unit testing like unittest, nose, pytest etc. Among these, unittest module comes with default python installation.

I have written a function which will take an integer value as argument and return if that integer is even or odd.

pytesting/even_odd.py

# -*-coding: utf-8-*-
def my_even_odd(number):
    if isinstance(number, int):
        if number % 2 == 0:
            return "Even"
        return "Odd"    
    return "Integer required"

This function will return "Integer required" if it gets an argument other than int. And for each argument it should return a string. Here, I am going to test two things: one is if I pass an integer or a non integer argument what it will return and two, if it returns anything other than string.

pytesting/unittest_even_odd.py

# -*-coding: utf-8-*-
import unittest
from even_odd import my_even_odd

test_values = [
    (1, "Odd"),    (0, "Even"),    (2147483648, "Integer required"),    (2147483647, "Odd"),    (-2147483649, "Integer required"),    (-2147483648, "Even"),    ('', "Integer required"),    ({'a': 1}, "Integer required"),    ([2, 3, 5], "Integer required"),]


class TestMyEvenOdd(unittest.TestCase):
    def test_my_even_odd(self):
        for tpl in test_values:
            self.assertEqual(my_even_odd(tpl[0]), tpl[1], msg="Failed for {}".format(tpl[0]))
            self.assertTrue(isinstance(my_even_odd(tpl[0]), str))


if __name__ == '__main__':
    unittest.main()

In unittest framework to create a test we need to create a subclass of unittest.TestCase class and add or override appropriate methods. In this case I added a method test_my_even_odd. There are two main parts. One is the test class we have written and another is test fixture. A test fixture represents the preparation needed to perform one or more tests, and any associate cleanup actions. This may involve, for example, creating temporary or proxy databases, directories, or starting a server process etc. In our case we keep it simple by using a list test_values. assertEqual and and assertTrue methods of a TestCase class.

assertEqual(self, first, second, msg=None)

This method will raise an AssertionError and test will fail if first and second objects are unequal as determined by `==`

assertTrue(self, expr, msg=None)

This method will check if the expression expr is True and raises AssertionError if False.

Now if we run unittest_even_odd.py we will get output like this,

$ python unittest_even_odd.py

.
--------------------------------------------------------------------
Ran 1 test in 0.000s

OK

That means my little function passed the test.

Vagrant up and running on ubuntu trusty

2015-01-20T12:32:00.001-08:00

What is vagrant?

Vagrant is a software for creating and configuring complete development environment. The environment is sandboxed in a virtual machine. The virtual machine is create by a virtualization software such as VirtualBox, VMware.

Vagrant works as a wrapper around a virtualization software and around configuration management software such as Ansible, Chef, Salt or Puppet.

Why and when use vagrant?

In software development process it's a common scenario that some software works on one developers machine and simply does not work in some other programmers machine for some dependency or some other issue. Vagrant is the answer here. It will create an identical development environment for everyone who intend to run/develop the software. Also running provisioning software like puppet, chef or ansible is a nice feature to have.

Installation

Vagrant needs virtualbox. Virtualbox is a virtualization software by oracle and its free. It can be downloaded from here. After downloading .deb file for specific ubuntu version, it can be installed by this command in the shell.

$ sudo dpkg -i <location/of/the/file>

Next step is installing vagrant. .deb file of it can be downloaded from here. Installation process is same as the virtualbox. Once vagrant get installed it will seem like noting has happened but we get some command line options for vagrant.

Up and running

I am going to create a virtual environment of a ubuntu trusty 32 bit system in my ubuntu trusty computer. Reason of choosing 32 bit system is, it takes less RAM than its 64 bit counterpart in the host machine. The host machine is my personal computer and the guest machine is the virtual machine I am going to create.

Create a directory trusty32 and cd to that directory from terminal. Directory name can be anything.

$ mkdir trusty32

$ cd trusty32

Now we add vagrant box. A box is a base image to quickly clone a virtual machine. The easiest way to download boxes is downloading it from HashiCorp's Atlas using this command,

$ vagrant box add ubuntu/trusty32

Another way is to add a box from vagrantbox.es site. In this case we follow the site's instruction and add the box like this,

$ vagrant box add trusty32 https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-i386-vagrant-disk1.box

vagrant box add command will download the specified box. This will take some time. Once downloaded it can be reused in multiple projects.

The next command is,

$ vagrant init trusty32

The command above will place a Vagrantfile in trusty32 directory. Vagrantfile describes type of machine required for a project, and how to configure and provision these machines. For now it will be not changed.

To boot up our virtual machine,

$ vagrant up

Above command creates and configures guest machines according to our Vagrantfile. To check if a guest machine is running,

$ vagrant status

We can ssh to the machine and gain access to the guest machine shell,

$ vagrant ssh

To shut down a running machine we have to exit from the ssh session and initiate the command,

$ vagrant halt

If we want to destroy the virtual machine and it's resource,

$ vagrant destroy

Python map, filter, lambda explained with example

2015-01-18T06:40:00.000-08:00

map(function, sequence[, sequence, ...]) -> list

map(function, iterable, ...) is a python builtin function.

It returns a list of results applying the function to the items of the argument sequence(iterable) or sequences.
If more than one iterable arguments are passed, the function must take that many arguments. The function is called with an argument list consisting of the corresponding item of each sequence.
If one iterable is shorter than another it is assumed to be extended with None items.

We can examine these rule with some codes.

def square(a):
return a ** 2

Here square is a function that makes a number square. We want the square of number 0 to 5. Using map(),

>>> map(square, range(6))

[0, 1, 4, 9, 16, 25]

So map() takes a function square and a list as arguments and returns a list of square(item). Here item is elements in the list range(6).

If we pass two iterable in map(), the function must take two arguments.

def my_pow(a, n):
return a ** n

Now my_pow function takes two arguments a, n and returns aⁿ.

>>> map(my_pow, [2, 3], [3, 2])
[8, 9]

But what if the two iterables are not of same length? In this particular case it will generate a TypeError.

filter(function or None, sequence) -> list, tuple, or string

filter(function, iterable) is a python builtin function.

It returns those items of sequence for which function(item) is True.
If function is None, return the items that are True.
If sequence is a tuple or string, return the same type, else return a list.

We want to filter out the even integer from a list and we have written a function for this.

def is_odd(n):
if n % 2 == 1:
return True
return False

is_odd function takes an integer and returns True if it's odd number otherwise returns False.

>>> filter(is_odd, range(10))
[1, 3, 5, 7, 9]

What happens here is, filter() applies the is_odd function on each item of the list range(10) and returns a list of those items for which is_odd returned True. It filters out the other items.

>>> filter(None, k = [True, False, True, False])
[True, True]

If the function is None it return the items that are True

lmabda function:

Now, fun begins with lambda functions. in Python,

lambda operator is a way to create small anonymous function. An anonymous function is a function without a name.
lambda function can be used wherever a normal function is used but it's restricted to a single expression.
function require multiple expression can not be expressed using lambda.

lambda functions are written as:

lambda arguments: expression (operation to perform on arguments)

lambda function returns the result of the operation.

What we have done in map() and filter() examples is, separately wrote a function and pass those in map(), filter()

We can do the same using lambda function.

>>> map(lambda n: n**2, range(6))
[0, 1, 4, 9, 16, 25]

what happens here is same in the map() example but in this case a lambda function is used.

lambda n: n**2

On the left of : are the arguments. Here n is the argument and on right side is the expression that will be applied to the argument. The return value is the result.

>>> map(lambda a, n: a**n, [2, 3], [3, 2])
[8, 9]

This is equivalent to the my_pow() example.

Suppose we want to get all the digits from a string. According to our filter() and lambda knowledge,

>>> filter(lambda s: s.isdigit(), "adf9df215")
'9215'

We can write a little complex expression in lambda.

>>> filter(lambda s: False if s.isspace() else True, "This is SPARTAAAAAAA")
'ThisisSPARTAAAAAAA'

This filters out all the space from the string. Hope king Leonidas don't mind.

Python built-in: setattr(), getattr()

2014-12-26T08:53:00.002-08:00

setattr(object, name, value)

setattr() sets a named attribute on an object. The arguments are an object, a string and an arbitrary value.

setattr(x, 'foobar', 123) is equivalent to x.foobar = 123.

Example Usage:

Lets assume Peter's mother has to write a class for preparing tiffin box for her son Peter who does not want to eat very regularly. Everyday in a week she puts random sandwichs and fruits in the tiffin box and checks if peter has eaten that or not. Let help her write the class using setattr()

class TiffinBox(object):
def __init__(self):
setattr(self, random.choice(['bacon', 'egg', 'jam']),

random.choice(range(1, 4)))
setattr(self, random.choice(['bacon', 'egg', 'jam']),

random.choice(range(1, 4)))

Here we are setting a random food with a random number from 1 to 3. The number defines the amount of that food in Peter's tiffin box.
Now we also want Peter to drink something and check if he has eaten the food. Let's write some methods in this class to do those.

class TiffinBox(object):
    def __init__(self):
        setattr(self, random.choice(['apple', 'banana', 'oranges']),
             random.choice(range(1, 4)))
        setattr(self, random.choice(['bacon', 'egg', 'jam']),
             random.choice(range(1, 4)))

    def take_drink(self):
        drinks = random.choice(['coffee', 'tea'])
        setattr(self, 'drink', drinks)

    def eat_food(self):
        for food in self.__dict__:
            setattr(self, food, 0)
        setattr(self, 'eaten', True)

Peter can eat using eat_food method and if he eats that food we will add a new attribute to the object named eaten set it to Boolean True and make the all the food count 0 assuming Peter ate all the foods.

We can also add a weekday attribute here. Now the class looks like this.

# -*- coding: utf-8 -*-
import random
from datetime import datetime

class TiffinBox(object):
    def __init__(self):
        setattr(self, random.choice(['apple', 'banana', 'oranges']), random.choice(range(1, 4)))
        setattr(self, random.choice(['bacon', 'egg', 'jam']), random.choice(range(1, 4)))
        self.tiffin_day = self.get_weekday(datetime.now().weekday()) + '_tiffin'
    def __str__(self):
        return self.tiffin_day

    def take_drink(self):
        drinks = random.choice(['coffee', 'tea'])
        setattr(self, 'drink', drinks)

    def get_weekday(self, weekday):
        py_week = {
            0: 'monday', 1: 'tuesday', 2: 'wednesday', 3: 'thursday', 4: 'friday', 5: 'saturday',6: 'sunday' }
        return py_week.get(weekday)

    def eat_food(self):
        for food in self.__dict__:
            setattr(self, food, 0)
        setattr(self, 'eaten', True)

Now Peter's mother can prepare tiffin for him.

>>>tiffin =TiffinBox()

>>>tiffin.__dict__

>>>{'apple': 3, 'jam': 3, 'tiffin_day': 'friday_tiffin'}
>>>tiffin.eat_food()
>>>tiffin.__dict__
>>>{'eaten': True, 'apple': 0, 'jam': 0, 'tiffin_day': 'friday_tiffin'}

Neat !!

getattr(object, name[, default]) -> value

getattr() is a python built-in that gets a named attribute from an object and return the value of that attribute.

The attribute name must be a string. getattr(x, 'y') is equivalent to x.y. default is an optional argument.

If the named attribute does not exist, default is returned if provided, otherwise AttributeError is raised.

>>>la = range(5)

>>>getattr(la, 'pop')

<built-in method pop of list object at 0xb7470eac>

>>>la.pop

<built-in method pop of list object at 0xb7470eac>

so getattr(la, 'pop') is equivalent to la.pop. It can be used like a function also.

>>>getattr(la, 'pop')()

>>>getattr(la, 'abc')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute 'abc'

A list object has no attribute named 'abc'. So a AttributeError is raised here. But if we use the optional default argument, then the default value is returned.

>>> getattr(la, 'abc', 'no such attribute')
'no such attribute'

Example Usage:

Let's help Peter to find out what his mother has put in the box and in how many amount. He does not know if it's a apple or banana, bacon or jam! Time to use getattr()

mybox = TiffinBox()

for atr in mybox.__dict__:
print atr, ': ', getattr(mybox, atr)

tiffin_day : friday_tiffin
bacon : 2
oranges : 3

Hope Peter likes bacons and oranges!

Python built-in: any(), all()

2014-12-24T11:32:00.000-08:00

any(iterable) -> bool

any(iterable) is a built-in function in python's __builtin__ module. It returns True if bool(x) is True for any x in the iterable. If the iterable is empty, return False.
Now let's take a list.

>>>la = [True, False, False]
>>>any(ls)
>>>True

This returns True because at least one of the value in la was True. Let's take another list,

>>>lb = [False, False, False, False]
>>>any(lb)
>>>False

So any() will return True if at least one of the value in the iterable is True else it will return False.

Example Usage:

>>>str = "Wikipedia is the seventh-most popular website \

...and constitutes the internet's largest and most popular \

...general reference work"

>>>keywords = ['Google', 'Wikipedia', 'Facebook', 'Microsoft']

We want to check if there is keyword from keywords list is present in the string str. With the use of any() function it can be done in a one-liner!

>>>any(keyword in str for keyword in keywords)

>>>True

That's awesome!! isn't it? But what sorcery is this? Breaking it down:

>>>[keyword in str for keyword in keywords]

>>>[False, True, False, False]

So it's like the earlier la, lb type list. It's returning True for the keyword 'Wikipedia'. Now what is happening in the list

compression is:

for keyword in keywords:

if keyword in str:

# append True in a_list

return <a_list>

all(iterable) -> bool

This function takes an iterable as argument and returns True if bool(x) is True for all values x in the iterable. If the iterable is empty, return True.

It will return if all the values in the iterable are True.

>>>all(la)
>>>False
>>>all(lb)
>>>False
>>>lc = [True]*3
>>>all(lc)
>>>True

Example Usage:

We are going to check if a list is a odd number list.

>>>all(map(lambda x:x%2==1, range(1, 10, 2)))

>>>True

Breaking it down:

>>> range(1, 10, 2)

>>>[1, 3, 5, 7, 9]

map() is a built-in which will return a list of result applying the lambda function to the item of the [1, 3, 5, 7, 9] list.

>>>map(lambda x:x%2==1, range(1, 10, 2))

>>>[True, True, True, True, True]

>>> all([True, True, True, True, True])

>>>True

Nice!!