How To Examples for Python

How to get today's date in YYYYMMDD format

2009-12-25T21:44:00.000-08:00

Code


import time                           #1
print time.strftime("%Y%m%d")         #2  
print time.strftime("%Y%m%d%H%M%S")   #3

How this works - Line by Line

import time                           #1

Imports the time module which is standard

print time.strftime("%Y%m%d")         #2

Prints the date in "YYYYMMDD" format. Note the upper case Y and lower case m and d.

print time.strftime("%Y%m%d%H%M%S")   #3

Prints the time in "YYYYMMDDHHMMSS" format.

How to fill out the search form on craigslist and display the results

2009-12-25T20:10:00.000-08:00

Code


from urllib2 import urlopen                           #1
from ClientForm import ParseResponse                  #2
from BeautifulSoup import BeautifulSoup               #3

html = urlopen("http://sfbay.craigslist.org/cta")     #4
forms = ParseResponse(html, backwards_compat = False) #5
form = forms[0]                                       #6
form['query'] = "toyota"                              #7  
form['srchType'] = ["T"]                              #8

response = urlopen(form.click())                      #9
soup = BeautifulSoup(response)                        #10

rows = soup('p')                                      #11

for row in rows:                                      #12
    print row('a')[0].contents[0]                     #13

How this works - Line by Line

from urllib2 import urlopen                           #1

This imports the urlopen method from the standard urllib2 module

from ClientForm import ParseResponse                  #2

This imports the ParseResponse method from the ClientForm module which you can get HERE

from BeautifulSoup import BeautifulSoup               #3

This imports the BeautifulSoup method from the BeautifulSoup module which you can get HERE

html = urlopen("http://sfbay.craigslist.org/cta")     #4

This defines the variable html as the html of the cars and trucks category of craigslist San Francisco Bay Area.

forms = ParseResponse(html, backwards_compat = False) #5

This defines the variable forms as the result set of the ParseResponse method which has the html variable passed into it. The result set is a list of forms on the website. Since some websites have more than one form, a result set is returned from the ParseResponse method.

form = forms[0]                                       #6

This defines the variable form as the first form in the result set variable forms. In plain english, since there is only one form on the craigslist website, this sets the variable form to that one form. If there was more than one form, you'd have to figure out which form you are trying to fill out and set the variable form to the appropriate forms[x]. You can do this by doing the following:

for form in forms:
    print form
    print

form['query'] = "toyota"                              #7

This fills out the query of the form to "toyota". You can view the fields that are in the form by printing the form (as seen in the explanation for line #6).

form['srchType'] = ["T"]                              #8

This sets the field on the form to search only titles

response = urlopen(form.click())                      #9

This submits the query and sets the html response as the variable response


soup = BeautifulSoup(response)                        #10
rows = soup('p')                                      #11
for row in rows:                                      #12
    print row('a')[0].contents[0]                     #13

See my posting on "How to screen scrape craigslist using python and BeautifulSoup" for how this works.

How to redirect sys.stdout from a file back to idle

2009-12-25T19:35:00.000-08:00

CODE


import sys                          #1
idlestdout = sys.stdout             #2
outfile = open('outfile.txt','w')   #3
sys.stdout = outfile                #4
print 'hello world'                 #5
outfile.flush()                     #6
outfile.close()                     #7
sys.stdout = idlestdout             #8

How this works - Line by Line
Note that this code redirects the standard output from a file back to any IDE, not just idle.

import sys                          #1

This imports the sys module which comes standard

idlestdout = sys.stdout             #2

This defines the variable idlestdout as the current output. You need to do this so that Python later knows what to redirect its output back to.

outfile = open('outfile.txt','w')   #3

This defines the variable outfile as a file called "outfile.txt" that we will be writing to.

sys.stdout = outfile                #4

This redirects the standard output to the file defined in line #3

print 'hello world'                 #5

This prints "hello world" to the standard output memory, or in this case, the outfile

outfile.flush()                     #6

This writes all of the pending text (line #5) to the outfile

outfile.close()                     #7

This closes the outfile file

sys.stdout = idlestdout             #8

This redirects the sys.stdout back to the IDE.

How to get stock quotes and other data from yahoo finance

2009-12-25T19:10:00.000-08:00

Code


import ystockquote                                     #1
for item in ystockquote.get_all('goog').iteritems():   #2
    print item                                         #3

How this works - Line by Line

import ystockquote                                     #1

This imports the ystockquote module which you can get HERE

for item in ystockquote.get_all('goog').iteritems():   #2

You can change the stock ticker from 'goog' to the ticker of interest.
for example, to get yahoo's stock quote, change 'goog' to 'yhoo'
This loops through each item in the ystockquote fields

    print item                                         #3

This prints each item

How to get the last item of a loop

2009-12-25T18:53:00.000-08:00

Code


items = ["hello","world"]                   #1
for item in items:                          #2
    if item == items[-1]:                   #3
        print "%s is the last item" %item   #4

How this works - line by line

items = ["hello","world"]                   #1

I'm setting a list called items for demonstration purposes.
The contents of this list are two strings - "hello" and "world"

for item in items:                          #2

looping through the two strings in the list

    if item == items[-1]:                   #3

if the current item in the loop is the last item in the list..

        print "%s is the last item" %item   #4

then print "%s is the last item"

Note that if you wanted the first object in the loop, you'd change line #3 to

    if item == items[0]:

How to screen scrape craigslist using Python and BeautifulSoup

2009-12-25T17:50:00.000-08:00

Code


from BeautifulSoup import BeautifulSoup   #1
from urllib2 import urlopen               #2

site = "http://sfbay.craigslist.org/rea/" #3
html = urlopen(site)                      #4
soup = BeautifulSoup(html)                #5
postings = soup('p')                      #6

for post in postings:                     #7
    print post('a')[0].contents[0]        #8
    print post('a')[0]['href']            #9

How this works - line by line:

from BeautifulSoup import BeautifulSoup   #1

this imports the BeautifulSoup method from the BeautifulSoup module
you can get this module from here

from urllib2 import urlopen               #2

this imports the urlopen method from the urllib2 module
this module comes standard with the most recent versions of python

site = "http://sfbay.craigslist.org/rea/" #3

this defines the variable site as a craigslist site.
In this case, the san francisco bay area real estate.
you can change the site value to match the site you want to scrape
for example, san francisco bay area cars+trucks would be
"http://sfbay.craigslist.org/cta/"

html = urlopen(site)                      #4

this defines the variable html as the html source code of the site variable

soup = BeautifulSoup(html)                #5

this defines the variable soup as html that has been parsed by BeautifulSoup

postings = soup('p')                      #6

this defines the variable postings as the 'p' tags in the variable soup. postings is a resultset, i.e. a set of objects

for post in postings:                     #7

since the postings is a resultset, we need to loop through the set to get the individual post header and link

    print post('a')[0].contents[0]        #8

this returns the header of the post. this works as follows -- the header is the contents of the first 'a' tag within each 'p' tag (i.e. the postings).

Thus, post('a') returns a resultset of 'a' items in 'p' and post('a')[0] returns the first 'a' item in the set.

To understand this more clearly, i've provided the following example: If the html looks like this:


<p>
    <a>hello</a>
    <a>world<a/>
</p>
<p>
    <a>my</a>
    <a>name</a>
    <a>is</a>
</p>


then after this has been passed through soup = BeautifulSoup(html),
soup('p')[0]('a')[0] = hello
soup('p')[0]('a')[1] = world
soup('p')[1]('a')[0] = my
soup('p')[1]('a')[1] = name
soup('p')[1]('a')[2] = is

    print post('a')[0]['href']            #9

to get the link, you need to get the ['href'] of each post.  Note that the link provided needs to be appended to the appropriate prefix link