Python 2.x provides a data type called a Unicode string for working with Unicode data using string encoding and decoding methods.

In order to figure out what “encoding” and “decoding” is all about, let’s look at an example string:

>>>s = "Flügel"

We can see our string s has a non-ASCII character in it, namely “ü” or “umlaut-u.” Assuming we’re in the standard Python 2.x IDLE GUI, let’s see what happens when we reference the string, and when it’s printed:

>>>s
'Fl\xfcgel'
>>>print s
Flügel

Printing gave us the value that we assigned to the variable, but something obviously happened along the way that turned it from what we typed into the interpreter to something seemingly incomprehensible. The non-ASCII character “ü” was translated into a code phrase, i.e. “\xfc,“ by a set of rules behind-the-scenes. In other words, it was encoded.

At this point, s is an 8-bit string, which to us basically means it isn’t a Unicode string. Let’s examine how to make a Unicode string with the same data. The simplest way is with a “u” prefix in front of the literal string marking it as a Unicode string:

u = u"Flügel"

If we reference and print u like we did with s, we’ll find something similar:

>>>u
u'Fl\xfcgel'
>>>print u
Flügel

We can see that the code phrase for our “umlaut-u” is still “\xfc“ and it prints the same—so does that mean our Unicode string is encoded the same way as our 8-bit string s? To figure that out let’s look at what the encode() method does when we try it on u and s:

>>>u.encode('latin_1')
'Fl\xfcgel'
>>>s.encode('latin_1')
Traceback (most recent call last):
   File "<pyshell#35>", line 1, in <module>
   s.encode('latin_1')
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfc in position 2: ordinal not in range(128)

Now it seems encoding the Unicode string (with the ‘latin-1’ encoding) retuned the same value as string s, but the encode() method didn’t work on string s. Since we couldn’t encode s, what about decoding it? Will it give us the same value as u? Let’s find out:

>>>s.decode('latin-1')
u'Fl\xfcgel'

That’s exactly what it does after all. So then, what difference does it make that s is an 8-bit string and u is a Unicode string? They behave the same way, don’t they?  In our “umlaut-u” example, there seemed to be little difference, aside from the “u” hanging out in front of the Unicode string.

Well, the difference is that the Unicode string u was using a code phrase that the Unicode standard defines for the character “umlaut-u,” and the 8-bit string s was using a code phrase that the ‘latin-1’ codec (rule-set) defines for “umlaut-u.”

OK, well…that’s great, but…they were still the same, right? So why does that matter?

To illustrate the difference and why it matters, let’s consider a new 8-bit string:

new_s = '\xe5\xad\x97'

Unlike the first one, our new 8-bit string is only code phrases—completely incomprehensible.

Why didn’t we just type in (or copy-and-paste) the characters like the last 8-bit string? Well, assuming we’re still using the standard Python 2x IDLE, we couldn’t type or paste this value into the interpreter—it would not accept the value if we did. Why? Because our new_s is an encoded string for an Asian script character (yes, only one character), and IDLE is averse to such input (if you have the appropriate input keyboard installed for your system you can try this and find out).

The question now is how can we turn these code phrases into the character they’re supposed to display? In the first example using a print statement on s worked fine, so it should be the same with new_s, right? Let’s see what our unknown Asian script character is:

>>>print new_s
å­—

Uh-oh…that isn’t right. First of all, that is not an Asian script character. Secondly, it’s more than one character.  Simply referencing new_s would give us the string we assigned to it, and print didn’t seem to work. Let’s see if a Unicode string will help us out.

To create our new Unicode string new_u, we can’t follow the method in our first example—to do that we’d have to input the literal character of our string with a “u” prefix (we haven’t seen our character yet, and anyway IDLE wouldn’t accept it as input).

However, we did get the value of u by decoding s, so in the same way we should be able to get the value for our new_u by decoding new_s. Let’s try decoding as we did in the first example:

>>>new_u = new_s.decode('latin_1')
>>>new_u
u'\xe5\xad\x97'

Great, now that we’ve stored the decoded new_s string value using the same method as in our first example, let’s print our Unicode string and see what our script character is:

>>>print new_u
å

­—Uh…Isn’t that the same thing we got when we tried to print the ­­new_s string?? So then using the Unicode string really isn’t any different?

Not so fast—there’s one detail that was purposefully glossed-over to prove a point: the encoding we used to decode the string is the same as the first example, the ‘latin-1’ codec. However, the 8-bit string new_s was not encoded in ‘latin-1,’ it was encoded in ‘utf-8.’

OK, so there was no way for you really to know that unless explicitly told, but this still illustrates the point: the encoding/codec/rule-set makes all the difference when encoding/decoding strings.

With the right encoding, let’s see what happens:

>>>new_u = new_s.decode('utf_8')
>>>new_u
u'\u5b57'
>>>print new_u
字

FINALLY! Our long-lost script character has been found, and it looks pretty good. Now, try copying-and-pasting the character as input—you’ll find it doesn’t work (we’re still talking about Python 2x IDLE).

You might also notice something different about the value of new_u, i.e. it appears to consist of only one code phrase (this time in the form ‘\uXXXX’). This Unicode standard has a unique code phrase for every character or script character that can possibly be displayed on your screen. With that in mind you can also tell that the first time we tried to decode new_s, the value was wrong (“u’\xe5\xad\x97’” has 3 code phrases, and for the Unicode standard, that means 3 unique characters).

Well, now that those annoying examples are finished, let’s recap the main points of all this hoopla:

1. Strings are one of the most common data types in Python, and sometimes they’ll include non-ASCII characters.
2. When strings contain non-ASCII characters, they can either be 8-bit strings (encoded strings), or they can be Unicode strings (decoded strings).
3. To print or display some strings properly, they need to be decoded (Unicode strings).
4. THE ENCODING/CODEC MAKES ALL THE DIFFERENCE WHEN ENCODING/DECODING STRINGS.

The encoding/codec is like the DNA of your string—even with the same nutrients (input), using the wrong DNA (codec) will give you an orange when you should have had an apple.

Lists and tuples are part of the group of sequence data types—in other words, lists and tuples store one or more objects or values in a specific order. The objects stored in a list or tuple can be of any type, including the “nothing” type defined by the None keyword.

The big difference between lists and tuples is that lists are mutable, but tuples are immutable, meaning once tuples are created, objects can’t be added or removed, and the order cannot be changed. However, certain objects in tuples can be changed, as we’ll see later.

Creating a list or tuple is easy, here are some empty ones:

>>>emptylst = [] # lists must be surrounded by brackets
>>>emptytup = () # tuples may or may not be surrounded by parenthesis

To create non-empty lists or tuples, values are separated by commas:

>>>lst = [1, 2, 3] # lists must be surrounded by brackets
>>>lst
[1, 2, 3]

>>>tup = 1, 2, 3 # tuples may or may not be surrounded by parenthesis
>>>tup
(1, 2, 3)

>>>tup = ('one', 'two', 'three')
>>>tup
('one', 'two', 'three')

Note: To create a tuple with only one value, add a trailing comma to the value.

>>>tup2 = 0, # the length of this tuple is 1
>>>tup2
(0,)

The values in lists and tuples, as with other sequence types, are referenced by index (beginning with 0 for  the first value):

>>>lst[0]
1
>>>tup[1]
'two'

Using an index below zero gets the values starting from the end of the list or tuple:

>>>lst[-1]
3
>>>tup[-2]
'two'

Multiple values can be referenced by “slicing” the list or tuple (referencing a range [start : stop] ):

>>>lst[0:2]
[1, 2]
>>>tup[1:5] # note an out-of-range stopindex translates to the end
('two', 'three')

Values for lists can be assigned with indexes (as long as the index already exists), but not for tuples:

>>>lst[2] = 'three'
>>>lst
[1, 2, 'three']
>>>tup[2] = 3
Traceback (most recent call last):
   File "<pyshell#68>", line 1, in <module>
   tup[2] = 3
TypeError: 'tuple' object does not support item assignment

Values can also be added to lists (and lists can be combined) with the ‘+’ operator, or with the standard append() method:

>>>lst += [None] # concatenation, the same as lst = lst + [None]
>>>lst
[1, 2, 'three', None]
>>>lst.append(5)
>>>lst
[1, 2, 'three', None, 5]

Values can be removed from lists with the del keyword:

>>>del lst[3]
>>>lst
[1, 2, 'three', 5]
>>>del lst[2:] # slicing deletion; no stopindex means through the end
>>>lst
[1, 2]

Assignment and deletion methods for lists don’t work for tuples, but concatenation works:

>>>tup += (4,) # note only tuples can be added to tuples
>>>tup
('one', 'two', 'three', 4)

Even though tuples are not mutable like lists, with the slicing technique and a little creativity tuples can be manipulated like lists:

>>>tup = tup[0:2] # almost like slice deletion
>>>tup
('one', 'two')
>>>tup2 += tup
>>>tup2
>>>tup2 = tup2[0:1] + (1,) + tup2[2:] # almost like index assignment
>>>tup2
(0, 1, 'two')

In addition, if a tuple contains a list, the mutability of the list in that tuple is preserved:

>>>tup3 = 0, 'one', None, []
>>>tup3[3].append('three')
>>>tup3
(0, 'one', None, ['three'])

Aside from the methods described here more exist for lists and tuples, but these cover  the most common operations. And remember, lists and tuples can be nested (can contain lists and tuples, and other data types that store sets of values, such as dictionaries), and can store different types simultaneously, making them very useful.

The objects or values stored in a dictionary can basically be anything (even the “nothing” type defined as None), but keys can only be immutable type-objects–e.g., strings, tuples, integers, etc.

Note: Immutable types that store objects like tuples can only contain other immutable types to be dictionary keys.

Creating a dictionary is easy, here is an empty one:

emptydict = {}

Here is one with a few values (note the { key1: value1, key2: value2, …} structure) :

smalldict = { 'dlist': [], 'dstring': 'pystring', 'dtuple': (1, 2, 3) }

You can also use the dictionary constructor (note with this method keys can only be keyword strings):

smalldict = dict(dlist=[], dstring='pystring', dtuple=(1, 2, 3))

To get one of the values, just use its key:

>>>smalldict = {'dlist': [], 'dstring': 'pystring', 'dtuple': (1, 2, 3)}
>>>smalldict['dstring']
'pystring'

>>>smalldict['dtuple']
(1, 2, 3)

To remove a key: valuepair, simply delete it with del

>>>del smalldict['dlist']
>>>smalldict
{'dstring': 'pystring', 'dtuple': (1, 2, 3)}

Or to assign a new key, just use the new key:

>>>smalldict['newkey'] = None
>>>smalldict
{'dstring': 'pystring', 'newkey': None, 'dtuple': (1, 2, 3)}

Note: All the keys in a dictionary are unique, so if you assign a value to an existing key, it will be overwritten! For example:

>>>smalldict['dstring'] = 'new pystring'
>>>smalldict
{'dstring': 'new pystring', 'newkey': None, 'dtuple': (1, 2, 3)}

If you want to change the key for a value, you can create the new key with the value, then simply delete the old key:

>>>smalldict['changedkey'] = smalldict['newkey']
>>>del smalldict['newkey']
>>>smalldict
{'dstring': 'new pystring', 'changedkey': None, 'dtuple': (1, 2, 3)}

To check if a key exists in the dictionary, just use key in smalldict:

>>>'dstring' in smalldict
True

Note: This does not check whether a value exists in the dictionary, only a key:

>>>'new pystring' in smalldict
False

To check for a value, you can use one of the dictionary’s standard methods, viewvalues():

>>>'new pystring' in smalldict.viewvalues()
True

There are many more useful methods and recipes for dictionary objects, but those are the basics.

Remember, keys don’t have to be strings, but can be any immutable object you can create—and values can be just about anything, including other dictionaries. Don’t be afraid to be creative and discover what you can do with them!

IPython offers lots of simple hooks for you to customise and extend the way it works. Because it’s so easy to modify what IPython does, it can be quite difficult to define exactly what it is. When you first install it, you get an app that essentially boils down to an enhanced, Python, interactive interpreter. You can do all the stuff you can do in a regular, interactive session, and you can do quite a bit more.

Magic Commands

The very first feature you’ll notice is the coloured prompt, a green In [1]: . Colour’s always cool, and it’s put to good use, but IPython’s not just a pretty stack trace. A more powerful enhancement to the standard Python session is the use of Magic Commands – Python functions that you can call using a shell-like syntax. You get a bunch built in, some really clever ones, and you can easily define your own. To use a magic, you just type its name, then any arguments, separated by a space:

magicname arg0 arg1

Some of the built-in magics are there to make the Python session operate more like a conventional shell. You have cd, ls, cp and so on. If you like to use these commands, you can just type them in as you normally would:

cd ~/scripts

If you want to use shell commands that you don’t have a magic set up for, you can just start the line with a bang and write any command that would normally work:

!echo IPython

Editing the Namespace

One of the most useful magics, I think, is edit. If you want to write any moderately complex Python, you don’t want to do it at the command line. In IPython, rather than starting with a text editor, then running the file, as you normally would, the process is reversed: We start at the prompt and simply invoke an editor from within the session. You can use any editor you like for this, but Vim rules.

You can use edit by just entering ‘edit’ at the prompt. IPython will open your editor with an empty tempfile. When you save and exit the editor, any code you wrote is injected into the namespace.

For example, you could enter ‘edit’ at the prompt, then define a function named spam in your editor, then save the file and exit the editor, then call spam at the prompt, passing whatever arguments you like. What’s especially cool, if you later enter ‘edit spam’ at the prompt, IPython reopens whichever tempfile contains spam, so you can edit it some more.

There’s also ways to inject the contents of regular files into the same namespace. And, of course, you can save what you’ve built, using your editor’s save feature, or just writing stuff to files from the command line. There’s also a magic named store that you can use to make objects persist across sessions.

The ability to easily build a namespace interactively, developing it as you go, gives you much more freedom to explore and interact with the objects you’re creating. This approach is a natural fit for writing Python, which you’d normally develop rapidly and in very small increments.

Configuring the Interpreter

IPython has a powerful configuration system, which is pure Python itself. You can use it to create and manage any number of configurations, customising each of IPython’s features, as well as specifying code to be automatically injected into new sessions, further enhancing your ability to fully customise the way your interpreter works.

Pretty much everything is configurable, and, if you know a little Python, it’s easy to do.

Developing your Environment

Once you’re comfortable using IPython, you should check out other articles on Python Central that cover the main features in more detail, including defining your own magic commands and configuring sessions.

Over time, as you use IPython more, and add more to it, you develop a heavily customised, console IDE, conveniently wrapping everything else you’re doing in Python.

Grab a copy from ipython.org, and let us know where you end up going with it.

These features make threads lightweight and handy in situations like network programming, when you try to ping (send network packets or requests) hundreds of workstations and you don’t want to ping them one after another! Since network replies may come after a significant amount of delay, the program will be extremely slow if you don’t ping many workstations concurrently.

Threading in Python is easy. First thing you need to do is to import Thread using the following code:

from threading import Thread

Next, you will want to make your class work as a thread. For this, you should subclass your class from the Thread class:

class MyThread(Thread):

Now, our MyThread class is a child class of Thread class. We then define a run() method in our class. This function will be executed when we call the start() method of any object of our MyThread class.

The code of the complete class is shown below. We use the sleep() function to make the thread “sleep” i.e. prevent it from executing for a random amount of time. If we don’t do this, the code will be executed so quickly that we will not be able to notice any worthwhile changes.

class MyThread(Thread):
   def __init__(self, val):
      """
         Constructor
      """
      Thread.__init__(self)
      self.val = val

   def run(self):
      for i in range(1, self.val):
         print "Value %d in thread %s" % (i, self.getName())
         # Sleep for random time between 1 ~ 3 second
         secondsToSleep = randint(1, 5)
         print "%s sleeping fo %d seconds..." % (self.getName(), secondsToSleep)
         time.sleep(secondsToSleep)

The next step is to create some objects (two in this example) of our thread-supported class. We call the start() method of each object – this in turn executes the run() method of each object.

# Run following code when the program starts
if __name__ == "__main__":
   # Declare objects of MyThread class
   myThreadOb1 = MyThread(4)
   myThreadOb1.setName("Thread 1")

   myThreadOb2 = MyThread(4)
   myThreadOb2.setName("Thread 2")

   # Start running the threads!
   myThreadOb1.start()
   myThreadOb2.start()
   # Wait for the threads to finish...
   myThreadOb1.join()
   myThreadOb2.join()

   print "Main Terminating..."

That’s it! Note that we need to call the join() method of each object – otherwise, the program will terminate before the threads complete their execution.

The complete version of the program looks like this:

#!/usr/bin/env python
from threading import Thread
from random import Random
import time
from Crypto.Random.random import randint

class MyThread(Thread):

   def __init__(self, val):
      """
         Constructor
      """
      Thread.__init__(self)
      self.val = val

   def run(self):
      for i in range(1, self.val):
         print "Value %d in thread %s" % (i, self.getName())
         # Sleep for random time between 1 ~ 3 second
         secondsToSleep = randint(1, 5)
         print "%s sleeping fo %d seconds..." % (self.getName(), secondsToSleep)
         time.sleep(secondsToSleep)

# Run following code when the program starts
if __name__ == "__main__":
   # Declare objects of MyThread class
   myThreadOb1 = MyThread(4)
   myThreadOb1.setName("Thread 1")

   myThreadOb2 = MyThread(4)
   myThreadOb2.setName("Thread 2")

   # Start running the threads!
   myThreadOb1.start()
   myThreadOb2.start()
   # Wait for the threads to finish...
   myThreadOb1.join()
   myThreadOb2.join()

   print "Main Terminating..."

Working with packages is really simple. All you need to do is:

• Create a directory & give it your package’s name
• Put your classes in it
• Create an __init__.py file in the directory

That’s all! The __init__.py file is necessary because with this file, python will know that this directory is a python package directory other than ordinary directories (or folders – whatever you call it). Anyway, it is in this file where we’ll write some import statements to import classes from our brand new package.

Step 1: Create the package directory
In this tutorial, we will create an “Animals” package – which simply contains two module files named “Mammals” & “Birds” containing Mammals & Birds classes, respectively.

So, first we create a directory and rename it to “Animals”.

Step 2 – Create some classes for our package
Now, we create the two classes for our package. First, create a file named “Mammals.py” inside the Animals directory and put the following code in it:

class Mammals:
   def __init__(self):
      """
      Constructor for this class
      """
      # Create some member animals
      self.members = ['Tiger', 'Elephant', 'Wild Cat']
   def printMembers(self):
      print "Printing members of Mammals Class:"
      for x in self.members:
         print "%s " % x

The code is pretty much self-explanatory! The class has a property named members – which is a list of some mammals we might be interested in. It also has a method named printMembers which simply prints the list of mammals of this class!

Next we create another class named Birds. Create a file named “Birds.py” inside
the Animals directory and put the following code in it:

class Birds:
   def __init__(self):
      """
      Constructor for this class
      """
      # Create some member animals
      self.members = ['Sparrow', 'Robin', 'Duck']
   def printMembers(self):
      print "Printing members of Birds Class:"
      for x in self.members:
         print "%s " % x 

This code is similar to the code we presented for the Mammals class.

Step 3 – Create the “__init__.py” file
Finally, we create a file named “__init__.py” inside the Animals directory and put the following code in it:

from Mammals import Mammals
from Birds import Birds

That’s it! We are now ready to test our code! For testing, we create a simple file named “test.py” in the same directory where “Animals” directory is located. We place the following code in the test.py file:

#!/usr/bin/env python
# Import classes from your brand new package
from Animals import Mammals
from Animals import Birds
# Create an object of Mammals class & call a method of it
myMammal = Mammals()
myMammal.printMembers()
# Create an object of Birds class & call a method of it
myBird = Birds()
myBird.printMembers()

The code is well-documented, and the complete project is also available for you to download! Check out the source code & directory organization there if anything confuses you.

Below is a simple python script that uses the PIL image module. It adds a visible text watermark to the image using the system font.

from PIL import Image, ImageDraw, ImageFont, ImageEnhance
import sys

FONT = "Arial.ttf"

We start with importing the PIL modules and load the Truetype font ‘Arial.ttf’.  By default, it searches for the font in the same folder and then it looks into your font directory (eg: C:/Windows/Fonts)

def add_watermark(in_file, text, out_file="watermark.jpg", angle=23, opacity=0.25): 

We define a function ‘add_watermark’ and declare some default parameters.

• in_file – input file
• text – the watermark text
• out_file – output file (default: watermark.jpg)
• angle – angle of the watermark (default: 23 degrees)
• opacity – opaque level for watermark (default: 0.25)

img = Image.open(in_file).convert("RGB")
watermark = Image.new('RGBA', img.size, (0,0,0,0)) 

First, we open the input file and create a watermark image of similar dimensions. Both files need to be in RGB mode since we are working with the alpha channel.

size = 2
n_font = ImageFont.truetype(FONT, size)
n_width, n_height = n_font.getsize(text)

Starting with font size 2, we create the text and obtain the width and height of the text.

while (n_width+n_height < watermark.size[0]):
   size += 2
   n_font = ImageFont.truetype(FONT, size)
   n_width, n_height = n_font.getsize(text)

By incrementing the font size, we search for a text length that doesn’t exceed the dimension (width) of the image.

draw = ImageDraw.Draw(watermark, "RGBA")
 draw.text(((watermark.size[0]-n_width)/2,(watermark.size[1]-n_height)/2),\text, font=n_font)

With the correct font size, we draw the text onto the center of the watermark image, using the system font declared in the header section.

watermark = watermark.rotate(angle, Image.BICUBIC)

We then rotate the image (default is 23 degrees), using Image.BICUBIC approximation.

alpha = watermark.split()[3]
 alpha = ImageEnhance.Brightness(alpha).enhance(opacity)
 watermark.putalpha(alpha) 

On the alpha channel, we reduce the opacity of the watermark (ie: reduce brightness & contrast) by the default value of 0.25. (Note: value 1 returns the original image).

Image.composite(watermark, img, watermark).save(out_file, "JPEG")

Lastly, we merge the watermark back into the original image and save it as a new JPEG file.

The entire code is as follows:

from PIL import Image, ImageDraw, ImageFont, ImageEnhance
 import sys
 FONT = "Arial.ttf"
 def add_watermark(in_file, text, out_file="watermark.jpg", angle=23, opacity=0.25):
    img = Image.open(in_file).convert("RGB")
    watermark = Image.new('RGBA', img.size, (0,0,0,0))
    size = 2
    n_font = ImageFont.truetype(FONT, size)
    n_width, n_height = n_font.getsize(text)
    while (n_width+n_height < watermark.size[0]):
       size += 2
       n_font = ImageFont.truetype(FONT, size)
       n_width, n_height = n_font.getsize(text)
    draw = ImageDraw.Draw(watermark, "RGBA")
    draw.text(((watermark.size[0]-n_width)/2,(watermark.size[1]-n_height)/2), text, font=n_font)
    watermark = watermark.rotate(angle,Image.BICUBIC)
    alpha = watermark.split()[3]
    alpha = ImageEnhance.Brightness(alpha).enhance(opacity)
    watermark.putalpha(alpha)
    Image.composite(watermark, img, watermark).save(out_file, "JPEG")
 if __name__ == "__main__":
    if len(sys.argv) < 3
       sys.exit("Usage: %s <input-image> <text> <output-image> <angle> <opacity> "
          % os.path.basename(sys.argv[0]))
    add_watermark(*sys.argv[1:])

Virtualenv lets you create virtual Python environments. Everything you install/remove in that environment stays there and other environments are not affected. Most importantly, you don’t pollute the global package directory of your system. For example, if you want to test an unstable package, virtualenv is the best way to go. If the unstable module you’re testing have some errors in the installation/uninstallation process, your system will not be affected by it. When you are ready to remove the new unstable module, what you need to do is just removing the virtualenv environment you created.

In this article, we will be using pip as the Python Package Manager on a Unix-based Operating System. You can also use easy_install if you like. First, let’s setup up pip (You may need to become root or use sudo):

$ easy_install pip

The next step is to install the virtualenv package :

$ pip install virtualenv

The next step is to create the environment with virtualenv:

$ virtualenv --no-site-packages my_blog_env

With the previous command, we created an isolated Python environment under the directory my_blog_env. The option –no-site-packages is telling virtualenv to create for us an empty Python environment. If we didn’t supply that option, our virtual environment would contain all of the packages in our global Python directory (eg /usr/lib/python2.7). If you need all of your global packages and just want to add a few packages to them, you can ignore that argument and create your environment that way. The next step is to activate your virtual environment:

$ cd my_blog_env/
$ source bin/activate

You should now be seeing that your prompt changed a little bit (refer to parenthesis at the beginning):

(my_blog_env)makkalot@makkalot-linux:~/my_blog_env$

Now you’re in your virtual environment. If you list the contents of the current directory(virtualenv) you will see 3 directories :

$ ls
bin  include  lib

The “bin” directory includes the executables for our virtualenv environment. That is the place where the Python interpreter is. Also the executables that are installed by some packages will be included in that directory. The “include” directory contains the header files of the environment. Finally, the “lib” directory includes the python files of the installed modules of our virtualenv system.

The next step is to install some packages and use our environment. As we stated in our example let’s install an old version of Django version 1.0.

$ pip install Django==1.0

Now, we can check if Django is installed in our virtual environment by checking in Python shell.

$ python
>>> import django
>>> print django.VERSION
(1, 0, 'final')
>>>
$ deactivate

Another interesting feature of virtualenv is the ability to create sandboxes for different versions of Python interpreter. With the -p flag you can create environments that use different versions of Python interpreter. Let’s say for example that you want to create a project with Python’s latest version 3. To be able to do that, you first need to install the version of Python you want to try on your system. After that, you need to find the path of the executable of the interpreter. You can typically find it under /usr/bin (eg: /usr/bin/python3):

$ virtualenv --no-site-packages -p /usr/bin/python3 p3_test
 $ cd p3_test
 $ source bin/activate
 $ python
Python 3.2 (r32:88445, Dec  8 2011, 15:26:51)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print("help")
help

Now, you are ready to experiment with your new Python 3 environment.

Usage

After writing the code, save the file with a .py extension, for example movie_list_maker.py. Then put the file in a directory where you’ve got movie files (and other files, as well). The code in the script will recursively traverse all other folders within it, and check for video files. If you’re using Windows and have Python IDLE installed, you can just double-click on the file and check the output. If you’re on Linux, open a terminal and cd to the directory where your script is, then open the script typing ./movie_list_maker.py in your terminal (make sure the file is executable first).

Let’s Code!

Let’s write the code of traversal within a function, which looks like:

def processFile(currentDir):
    '''
    Process video files within this directory
    '''
    # Get absolute path of currentDir parameter
    currentDir = os.path.abspath(currentDir)

    # Get a list of files in currentDir
    filesInCurDir = os.listdir(currentDir)

    # Traverse through all files
    for file in filesInCurDir:
        curFile = os.path.join(currentDir, file)

        # Check if normal file/directory
        if os.path.isfile(curFile):

        # Get extention
        curFileExtention = curFile[-3:]

        # Check if file has an extention of typical video files
        if curFileExtention in ['avi', 'dat', 'mp4', 'mkv', 'vob']:

            # We got a video file! Increment counter
            processFile.counter += 1

            # Print it's name
            print '\n%s' % curFile
        else:
            # We got a directory, enter into it for further processing
            processFile(curFile)

The code is pretty much self-explanatory along with the comments. This function takes one argument: the name of a directory to work on. Then it gets a list of all files & folders in this directory using the os.listdir method. We use a for loop to work on the list, check whether the file variable is a normal file or directory using os.path.isfile method. If it’s a normal file, we get the extension of the file using: curfile[-3:] – we assume that last three characters of the filename is it’s extension, so we’ve used a negative sign while slicing the curfile variable.

If the filename is a directory other than a regular file, we recursively call the function itself to further process it.

Call the function…

Now, we call this function within the __main__ scope:

if __name__ == '__main__':
    # Get the current working directory
    currentDir = os.getcwd()

    print('Starting processing in %s' % currentDir)

    # Set the number of processed files equal to zero
    processFile.counter = 0

    # Start Processing
    processFile(currentDir)

    # We are done. Exit now.
    print( '\n -- %s Movie File(s) found in directory %s --' % (processFile.counter, currentDir))
    print( '\n Press ENTER to exit!')

    # Wait untile user presses an <enter>
    raw_input()

Once again, os.getcwd method helps us to get the current working directory, i.e. the directory where the script resides. It calls our just-written function and also has a counter, it counts how many video files it got. Finally, we print all information we have, and wait for a prompt from user to terminate the program using raw_input method.

First we need to get the html source for the page . We can do that with urllib, one of the libraries that comes standard with Python.

import urllib

optionsUrl = "http://finance.yahoo.com/q/op?s=AAPL+Options"

optionsPage = urllib.urlopen(optionsUrl)<strong></strong>

This code retrieves the html and returns a file-like object.

If you go to the page we opened with Python and use your browser’s “get source” command you’ll see that it’s a large, complicated html file. It will be Python’s job to simplify and extract the useful data using the Beautiful Soup module.  Beautiful Soup is an external module so you’ll have to install it. If you haven’t installed Beautiful Soup you can get it here.

The following code will load the page into BeautifulSoup:

from BeautifulSoup import BeautifulSoup

soup = BeautifulSoup(optionsPage)

Now we can start trying to extract information. We can see that the options have pretty unique looking names in the “symbol” column something like AAPL120127C00505000.  The symbols might be slightly different by the time you read this but we can solve the problem by using Beautiful Soup to search the document for this unique string.

Let’s search the soup for this particular option (you may have to substitute a different symbol, just get one from the webpage):

>>> soup.findAll( text="AAPL120127C00505000")

[u'AAPL120127C00505000']

This result isn’t very useful yet.  It’s just a unicode string (that’s what the ‘u’ means) of what we searched for.  But Beautiful Soup returns things in a tree format so we can find the context in which this text occurs by asking for it’s ‘parent’ node like so:

>>> soup.findAll( text="AAPL120127C00505000")[0].parent

<a href="/q?s=AAPL120127C00505000">AAPL120127C00505000</a>

We don’t see all the information from the table.  Let’s try the next higher level.

>>> soup.findAll( text="AAPL120127C00505000")[0].parent.parent

<td><a href="/q?s=AAPL120127C00505000">AAPL120127C00505000</a></td>

And again.

>>> soup.findAll( text="AAPL120127C00505000")[0].parent.parent.parent
<tr><td nowrap="nowrap"><a href="/q/op?s=AAPL&amp;k=110.000000"><strong>110.00</strong></a></td><td><a href="/q?s=AAPL120127C00505000">AAPL120127C00505000</a></td><td align="right"><b>1.25</b></td><td align="right"><span id="yfs_c63_aapl120127c00505000"> <b style="color:#000000;">0.00</b></span></td><td align="right">0.90</td><td align="right">1.05</td><td align="right">10</td><td align="right">10</td></tr>

Bingo.  It’s still a little messy, but you can see all of the data that we need is there.  If you ignore all the stuff in brackets, you can see that this is just the data from one row.

optionsTable = [[x.text for x <strong>in</strong> y.parent.contents] <strong>for</strong> y <strong>in</strong> soup.findAll('td', attrs={"class" : "yfnc_h", "nowrap" : "nowrap"})]

This code is a little dense, so let’s take it apart piece by piece.  The code is a list comprehension within a list comprehension.  Let’s look at the inner one first:

for y in soup.findAll('td', attrs={"class" : "yfnc_h", "nowrap" : "nowrap"})

This uses Beautiful Soup’s findAll function to get all of the html elements with a ‘td’ tag and a class of yfnc_h and a nowrap of nowrap.  We chose this because it’s a unique element in every table entry.  If we had just gotten ‘td’s with the class ‘yfnc_h’ we would have gotten seven elements per table entry.  Another thing to note is that we have to wrap the attributes in a dictionary because ‘class’ is one of python’s reserved words.  From the table above it would return this:

<td nowrap="nowrap"><a href="/q/op?s=AAPL&amp;k=110.000000"><strong>110.00</strong></a></td>

We need to get one level higher and then get the text from all of the child nodes of this node’s parent.  That’s what this code does:

[x.text for x in y.parent.contents]

This works, but you should be careful if this is code you plan to frequently reuse.  If Yahoo changed the way they format their html, this could stop working.  If you plan to use code like this in an automated way it would be best to wrap it in a try/catchblock and validate the output.

This should only give you an idea of what you can do with HTML and XML parsing in Python.   You can find the documentation for Beautiful Soup here.  You’ll find a lot more tools for searching and validating html documents.