My tutorial was intended to encourage women in computer science by serving two purposes: discussing the source code and functionality of a particular project, and openly discussing some of the issues they faced in their current programs and surroundings. I was pleasantly surprised to discover that these women needed no technical or social encouragement. They are already enthused, technically and socially well prepared, and on their way to a very bright future in engineering or computer science. I was pleasantly surprised that they were very comfortable in the bash shell, and as comfortable in their current college curriculum. Like children at play, they picked up the moderate-levelled tutorial code quickly, made great strides in such a short time, and had a lot of fun doing it.

It was profoundly encouraging to see such a small computer science program achieve diversity as well as such a high level of skill. It made me wonder why larger colleges and universities cannot accomplish the same on bigger budgets, with larger staff, and a more diverse mix of students. It touched me to hear and see the enthusiasm, eagerness, and skill of the students in this program. Their learning experience under the Department Chair, Dr. Burch, comes as close to perfect as I have ever seen.

The next day’s events at PyArkansas were held at the also-very-nice campus of University of Central Arkansas, where an entire day’s worth of tutorials took place. Two Python 101 tracks were held: one for programmers and one for non programmers (a great concept). An all-day Django Track was given, where the advanced course was taught by Jacob himself. I held an afternoon tutorial addressing advanced Python concepts, with downloadable example code, where we compared and contrasted build and deployment tools, played with regex, and showed examples of some internal Python oddities involving static variables. I unfortunately missed the Python Blender tutorial, held at the same time as mine, and I heard it went quite well.

The campus facilities were very accomodating. Everything was well organized,and up and running for us when we arrived. This is a very welcome surprise to anyone who has travelled a bit to do tutorials. I was specifically told by Dr. Chenyi Hu, the Department Chair of UCA, that he really does care about diversity, and it is something they strive to achieve. This was truly touching, quite impressive, and a pleasant surprise from such a small town.

Kudos to Greg Lindstrom, Dr. Carl Burch of Hendrix College, Dr. Chenyi Hu of UCA, and everyone else involved. You induced a big technical “tremor” through your small town, which echoed far and wide. It is yet another example of the great people drawn to the Python community, and the amount of quality effort they are willing to give back. I feel honoured to have been part of this event, and I hope to be involved in many more to come.

Gloria

(1) Find related web sites and drop relevant strategic links on those sites without “spamming”.
(2) Create an online social app/blog presence for this company, announcing new products, philanthropic events, etc. for this company.
(3) Follow accounts at Facebook, MySpace, product blogs, etc. Help set up the look and feel of these sites (working alongside a writer, a designer, and a software developer).
(4) Help us build this presence in creative ways: event photo slideshows, podcasting interviews of the owners, the technical aspects of the company, etc.

Please submit a resume and some examples of your work online. Also give me an example of how you would promote such a situation. Creativity is definitely encouraged:

I am handling the system admin and software design/development. Over the next few months, I’ll be taking it in new directions, by bringing in a free Python apprenticeship training group, and using this site as training ground for people new to Python. This will bring it some social exposure from the tecchie direction. How and where would you discuss this unique approach of growing a business while providing a public service?

Please send email to gloriajw_66, at the address yahoo dot com

Thank you, and good luck!
Gloria

PS: Working alongside me means this job _always_ has the opportunity to morph in a more technical direction, if you so choose. This straddles many disciplines, and can turn out to be quite fun if you wish to expand your current knowledge.

But if you’re looking for a small, easily manageable and extremely dynamic ReST dispatching solution without the heft of an entire web framework, I’m about to show you how CherryPy can help you in three different ways, depending on your model.

Assuming this mapping:

HTTP GET or HEAD = DB Read
HTTP POST = DB update
HTTP PUT = DB insert
HTTP DELETE = DB delete

Let’s also standardize on one common method across all examples, for determining the HTTP request type, and matching it to the function of the same name. Here is the full code snippet for accomplishing this task:

methods = ('OPTIONS','GET','HEAD','POST',
'PUT','DELETE','TRACE','CONNECT')

if cherrypy.request.method not in self.methods:
    raise cherrypy.HTTPError(400,'Bad Request')

# If request method is HEAD, return the page handler
# for GET, and let CherryPy take care of dropping
# the response body
method = cherrypy.request.method

if cherrypy.request.method == "HEAD":
    method = "GET"

http_method = getattr(self,method)

#print "HTTP Method: %s" % method

result=(http_method)(args,kwargs)

In our examples, we’re going to shorten this to:

http_method = getattr(self.m,cherrypy.request.method)
return (http_method)(args,kwargs)

All of this essentially determines how HTTP was called (GET/PUT/POST/DELETE), and calls the method in a class which exactly matches this name (self.GET(), self.PUT(), etc)
When you see this code, know that it’s just the HTTP method resolving code.

Now for the fun. Let’s look at the dispatcher options we have.

Way 1: A hard-coded URL pointing to fixed resources:

CherryPy can be used in a manner similar to this to establish a fixed URL, and corresponding resources, driven from predefined classes instantiated in the ‘root’ hierarchy:

import cherrypy

class ReSTPaths1:
	@cherrypy.expose
	def index(self):
		http_method = getattr(self,cherrypy.request.method)
		return (http_method)()

	def GET(self):
		return "In GET 1.."

class ReSTPaths2:
	@cherrypy.expose
	def index(self):
		http_method = getattr(self,cherrypy.request.method)
		return (http_method)()

	def GET(self):
		return "In GET 2.."

class ReSTPaths3:
	@cherrypy.expose
	def index(self,client_id=None):
		http_method = getattr(self,cherrypy.request.method)
		return (http_method)(client_id)

	def GET(self,client_id=None):
		return "IN Get 3, your client_id is %s\n" % (client_id)

cherrypy.server.socket_port=8081

root=ReSTPaths1()
root.client = ReSTPaths2()
root.client.address = ReSTPaths3()
cherrypy.quickstart(root)

Once this is running, the URL to invoke it looks like this:
http://localhost:8081/
http://localhost:8081/client/
http://localhost:8081/client/address/
http://localhost:8081/client/address/?client_id=34567
http://localhost:8081/client/address/34567

Output looks something like this:

In GET 1..
In GET 2..
IN Get 3, your client_id is None

If you’re new to CherryPy or Python in general, I’ll reiterate for you how we are calling the GET method in our class.

When we issue this request, we’re issuing what HTTP calls a GET request:

http://localhost:8081/

The CherryPy service above, listening on port 8081, calls the index() method on the root class. The root class was set to:

root=ReSTPaths1()

at the bottom of that file. The index() method from the ReSTPaths1 Class looks like this, at the top of that file:

	def index(self):
		http_method = getattr(self,cherrypy.request.method)
		return (http_method)()

If we were to insert a print cherrypy.request.method statement before the return, we would see it set to “GET”.

getattr simply says: “get me the function name in self, matching the string “GET”.
it returns a reference to self.GET(), which is set directly below the index:

	def GET(self):
		return "In GET 1.."

Notice that the index() method has a @cherrrypy.expose decorator above it. This makes the index method callable by the public. The GET method does not have it, which means we could never invoke the GET method by typing:

http://localhost:8081/GET

If you try this, you’ll get a 404 Not Found error, because it’s not visible through the CherryPy interface.

GET() has to be invoked through index(), which means GET can only be called if an HTTP GET request is issued. If we posted form data to this same URL from, say, a form entry asking people for data input, we would need to add a POST method to this ReSTPaths1() class, to receive the POST data entered in the form fields.

Now back to our example:

In this example, no part of the URL or associated resources are dynamic, in either initialization or run time. This is fine, and suits the needs of most ReSTful CRUD interfaces.

Way 2: URL paths and associated components dynamically set once, upon dispatcher init/startup:

Now let’s say we want to determine the contents of the root, and therefore the URLs and associated resources for our ReSTful interface, dynamically during initialization/startup.

We can assign the root setting by using a Python metaclass to generate classes in our CherryPy startup code, and set the root components to each generated class. This goes beyond the average needs for CRUD access, but it’s such a nice implementation that I must show it off:

import cherrypy

class MetaCRUD(type):
	@cherrypy.expose
	def index(cls):
		http_method = getattr(cls,cherrypy.request.method)
		return (http_method)()

	def GET(cls): return "In class ", cls.__name__, ', received a GET request.'

	def PUT(cls): return "In class ", cls.__name__, ', received a PUT request.'

	def POST(cls): return "In class ", cls.__name__, ', received a POST request.'

	def DELETE(cls): return "In class ", cls.__name__, ', received a DELETE request.'

baseCRUD = MetaCRUD('baseCRUD',(),{})
root = baseCRUD

dynamic_class = {}

for d in ['legacy_dbi','new_dbi','some_other_dbi']:
	dynamic_class[d] = MetaCRUD(d,(),{})
	setattr(root,d,dynamic_class[d])

cherrypy.server.socket_port=8081
cherrypy.quickstart(root)

Here we’re using a metaclass, with CherryPy exposed methods, to generate a dictionary of dynamic classes. We set the root.classname = the_new_class by using the setattr() method.

After initialization, URL components and resources are fixed in this model. But wow, the awesome power we have during initialization, in 28 lines really rocks. I wrote this in 30 minutes, and realized again why I am so head-over-heels in love with this language.

When we hit these URLs:

http://localhost:8081/
http://localhost:8081/legacy_dbi/
http://localhost:8081/new_dbi/
http://localhost:8081/some_other_dbi/

We see this output:

In class baseCRUD, received a GET request.
In class legacy_dbi, received a GET request.
In class new_dbi, received a GET request.
In class some_other_dbi, received a GET request.

Let’s issue a POST request via curl, on the command line. The response is returned:

[gloriajw@g-monster ~]$ curl http://localhost:8081/some_other_dbi/ -d ""
In class some_other_dbi, received a POST request.

This model could be used for, say, reading the contents of the Postgres template1 databases list or the mysql ’show databases’ command, and auto-generating a ReSTful CRUD interface for each. Access of each resources can be controlled via HTTP Auth methods. This is a great solution to providing, and restricting, legacy database access for new processes through a standard interface.

Way 3: Live, ever-dynamic determination of URL and associated component:

Some ReSTful URL models may need to be ‘run-time dynamic’, especially in the case where databases are dynamically created, and the associated resources per new database could vary. There is a simple example of a dynamic URL and resource model:

import cherrypy
import pprint

class ReSTPaths:
	@cherrypy.expose
	def __init__(self):
		pass

	@cherrypy.expose
	def client(self,*args,**kwargs):
		return "Your HTTP method was %s. Your args are: %s and your kwargs are: %s\n" \
		% (cherrypy.request.method, pprint.pformat(args), pprint.pformat(kwargs))

	@cherrypy.expose
	def address(self,*args,**kwargs):
		return "Your HTTP method was %s. Your args are: %s and your kwargs are: %s\n" \
		% (cherrypy.request.method, pprint.pformat(args), pprint.pformat(kwargs))

cherrypy.quickstart(ReSTPaths())

This allows for dynamic URLs such as:
http://localhost:8080/client/address/34567
http://localhost:8080/client/address?client_id=34567
http://localhost:8080/address/client?client_id=34567
http://localhost:8080/address/client/34567
http://localhost:8080/address/anything/anything_else

The output from this code looks like this:

Your HTTP method was GET. Your args are: ('address', '34567') and your kwargs are: {}
Your HTTP method was GET. Your args are: ('address',) and your kwargs are: {'client_id': '34567'}
Your HTTP method was GET. Your args are: ('client',) and your kwargs are: {'client_id': '34567'}
Your HTTP method was GET. Your args are: ('client', '34567') and your kwargs are: {}
Your HTTP method was GET. Your args are: ('anything', 'anything_else') and your kwargs are: {}

Notice that we only have keyword args (kwargs) when we pass a named parameter, such as client_id=34567

Let’s try a POST request from curl, on the command line:

[gloriajw@g-monster ~]$ curl -d "something_else=whatever_i_want" http://localhost:8080/address/anything/anything_else
Your HTTP method was POST. Your args are: ('anything', 'anything_else') and your kwargs are: {'something_else': 'whatever_i_want'}

In this code, the sky is the limit. You can place whatever code you like in these methods, dynamically creating classes and resources as needed, letting them only persist until the result is returned. This may add some inefficiency, but in exchange offer more secure network resources.

Code is attached, Enjoy!

Gloria

http://www.devchix.com/wp-content/uploads/2009/04/restfixedargs.py

http://www.devchix.com/wp-content/uploads/2009/04/restmeta.py

http://www.devchix.com/wp-content/uploads/2009/04/restvarargs.py

import urllib2, base64
import sys
import feedparser
#import configuration

class Page:

    def __init__(self):
        self.data = {}
        self.data["url"] = 'http://www.twitter.com/gloriajw'
        self.data["username"] = 'gloriajw'
        self.data["password"] = 'XXXXXXXXXX'

        self.data["urlrss"] = 'http://twitter.com/statuses/user_timeline/18107956.rss'

    def __getitem__(self, key):
        return self.data[key]

    def __setitem__(self,key, value):
        self.data[key] = value

    def getContent(self):
        base64string = base64.encodestring('%s:%s' % (self.data['username'], self.data['password']))[:-1]
        authheader =  "Basic %s" % base64string

        req = urllib2.Request(self.data["url"])
        req.add_header("Authorization", authheader)
        try:
            handle = urllib2.urlopen(req)
        except IOError, e:                  # here we shouldn't fail if the username/password is right
            print "It looks like the username or password is wrong."
            sys.exit(1)

        return handle.read()

    def getRSS(self):
        base64string = base64.encodestring('%s:%s' % (self.data['username'], self.data['password']))[:-1]
        authheader =  "Basic %s" % base64string

        req = urllib2.Request(self.data["urlrss"])
        req.add_header("Authorization", authheader)
        try:
            handle = urllib2.urlopen(req)
        except IOError, e:                  # here we shouldn't fail if the username/password is right
            print "It looks like the username or password is wrong."
            sys.exit(1)

        return handle.read()

    def getData(self):
        """auth = urllib2.HTTPBasicAuthHandler()
        auth.add_password('BasicTest', 'twitter.com', self.data['username'], self.data['password'])

        return feedparser.parse('http://www.twitter.com/statuses/user_timeline/18107956.rss', handlers=[auth])
        """

        return feedparser.parse('http://%s:%s@twitter.com/statuses/user_timeline/18107956.rss' % (self.data['username'], self.data['password']))

To invoke it:

class Data:
    def __init__(self, entries):
        self.entries = entries

    def save(self):
        pass

    def parse(self):
        pass

    def imprimir(self):
        for item in self.entries:
            print item.title

And this:

def main():
    page = Page()    

    statuses = page.getData().entries

    data = Data(statuses)
    data.save()

    data.imprimir()    

if __name__ == "__main__":
    main()

This code will also be attached, in case of copy/paste mangling.

In both getContent() and getData(), Gaba constructs the HTTP response header so that the encoded username and password are passed in the Authorization section of the header. This is easier and more secure than making two requests, and maintaining session cookies. Very nice indeed. This can be used to sign into any web site which accepts HTTP Basic authentication headers (there are different types of HTTP authentication (BASIC, DIGEST, FORM, and CLIENT-CERT).

It is left as an exercise for you to get the content (not the feed) and use BeautifulSoup to extract the data portions. If you want to try this, and need help, post questions here.

Enjoy,
Gloria

http://us.pycon.org/2009/tutorials/schedule/1AM7/

Alex McFerron is my assistant.

The fee which I would normally collect to teach this tutorial at PyCon (something like $750) will be donated to Alex’s Math Circle Outreach for kids.

All tutorials will be videotaped and put somewhere on the net. Exciting, eh? Watch for us. We’re going to rock this out.

If you’re at PyCon, feel free to say hi and take me out for a beer, to which I’ll gladly reciprocate. I’m staying for all of the tutorials and code sprints. Woot! I can’t wait!

My email addy: gloriajw_66 at yahoo dot com

Gloria

As hardware costs plummet and greater bandwidth becomes more affordable, cloud computing becomes even more alluring and feasible for the hobbyist. Working with large data sets is an interesting problem set, now attainable by the average developer with a free LINUX hard drive partition and some spare time.

Hadoop is essentially a framework, written in Java, for accessing large data sets. For the average developer, it’s usually used in conjunction with HDFS which is a custom file system designed for stroing large data sets on cheaper hardware. HDFS data is usually stored in units of entire web pages written to disk “as-is”. Once stored, this large data set is accessible for parsing and analysis.

Indexing of this data is done via what is called a map-reduce algorithm. The easiest map-reduce algorithm to envision is the word count algorithm. Each word on a page is assigned the value “1″ in the map portion, then all words are counted by sorting the words and adding their values. The result is a frequency count of each word in a page. This is the simplest of ten well known map-reduce algorithms, found in this white paper:

http://www.cs.stanford.edu/people/ang//papers/nips06-mapreducemulticore.pdf

The magic of Michael’s article is in how to prevent the stringent, often painful Jython interface to Hadoop. This trick is by streaming this data on the command line from his simpler, more Pythonic map-reduce algorithm directly to Hadoop, thereby avoiding the complex Jython interface. Very clever indeed! It would be wonderful to see all of the relatively standard map-reduce algorithms written in Python.

He has a couple of other great articles on exactly how to install and configure Hadoop on Ubuntu here:
http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)

and here:

http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_(Multi-Node_Cluster)

On top of Hadoop is usually a metdata store, holding high level information about which page sets or data “chunks” can be found on which volumes, for efficiency. After attending a cloud computing conference in November, I realized that many companies have chosen to write their own metadata layer using HiveDB, or custom metadata tools, and many distributed MySQL databases, rather than settling for the expensive commercial solutions. This great work is the primary cause of the proliferation of easily accessible Open Source cloud computing tools.

Another project to watch is called Mahout, which is an attempt to make all of the well known map-reduce algorithms generally accessible across multicore Hadoop systems.

I run Fedora 10, and have a bit more work to do, but it is outstanding to see such an easily accessible installation for such a relatively complex set of tools. So many software tools and toys, so little time!

Gloria

http://change.gov/newsroom/entry/towards_a_21st_century_government/

This copyright license affects most content on the website change.gov, guaranteeing transparency and the feedom to redistribute and discuss it’s content.

All I can say is, wow. This president not only believes in transparency, but understands how to implement it using existing tools and conventions. He understands the spirit of Open Source software development, and everything that has come about since this movement began. Imagine an “Open Government”, genuinely implemented and influenced by the people, and it’s hard not to want to get involved.

The change.gov web site has a very nice, simple form, requesting your ideas and suggestions. They already have proposals for nationwide fiber optic broadband, automation and transparency in the patent process, and automation of health insurance data to bring healthcare costs down. There are many more places where technology can automate away inefficient and error prone processes.

We here on DevChix all know how technology has changed our lives. Let them know how it can change a nation. Submit your ideas and suggestions, and let’s help educate and influence this new administration.

Gloria

Please give the sample code a try, and feel free to ask any questions, etc.

Gloria

1: A nonexistent PHP array key generates no error or warning. When trying to iterate over a nonexistent array key, a warning occurs. In other languages, both of these conditions throw an exception.

Try this code for example:

<?
$dictionary=array('one'=>'got one','two'=>'have two','four'=>'missing three?');
foreach (array_keys($dictionary) as $key)
{
	print "Key is:".$key.", value is:".$dictionary[$key]."\n";
}
print "Try undefined key three, no warning occurs:".$dictionary['three']."\n";
foreach ($dictionary['three'] as $value)
{
	print "Now we're iterating over a nonexistent key:";
	print "Key is: three, value is:".$dictionary['three']."\n";
}
?>

Running it results in this output:

php test.php
Key is:one, value is:got one
Key is:two, value is:have two
Key is:four, value is:missing three?
Try undefined key three, no warning occurs:

Warning: Invalid argument supplied for foreach() in /root/test.php on line 8

If it is vital to me to make sure I am aware of missing keys, I only have two choices. If I need a proactive solution, I have to use the array_key_exists() function to do existence checking before use. If I want a reactive solution, I write a log scanner, to pick up on these warnings. In every other OO language I have used, an exception was thrown for this condition, and my exception handling determined if the error was vital enough to have to exit immediately or not. This seems like a more efficient way to handle this condition. I would imaging PHP5 does not do this because of it’s need to be backward compatible with PHP4, but this is a guess.

It would be wonderful to have a -OO flag for PHP, which gives you the option to run PHP and expect more standard, stricter OO behavior in these instances.

2: Warnings cannot be “caught” like exceptions. Exceptions and warnings are distinctly separate beasts, and never the twain shall meet. Fine, I thought, maybe I could detect warnings similar to how we detect errors. But it seems like warnings cannot be detected when they happen. There is no PHP code I know of which can check if a warning had occurred in runtime. I tried to detect it using array error_get_last() but to no avail. if you know how, post your trick here.

3: In PHP, ‘true’ evaluates to an integer ‘1′. To get the boolean ‘true’ value from a ‘true’ statement, one needs to var_export() a true statement. Similarly, or maybe not, ‘false’ evaluates to no output. Here is an example:

<?
print "\nThe raw value of a true statement in PHP:".true;
print "\nThe raw value of a false statement in PHP:".false;
print "\nThe exported value of a true statement in PHP:".var_export(true,true);
print "\nThe exported value of a false statement in PHP:".var_export(false,true);
print "\n";
?>

And the output:

The raw value of a true statement in PHP:1
The raw value of a false statement in PHP:
The exported value of a true statement in PHP:true
The exported value of a false statement in PHP:false

This may not be noticeable to you in a standard expression. But if you’re doing funky stuff, like using the evaluated expression values as key references into the dictionary of a decision tree, for example, 1 does not equal ‘true’, and the difference matters quite a bit.

4: Long running processes with recursive circular references (such as Doctrine code) run out of memory. This is documented in many places, and the free() function works sometimes. A fix is coming in PHP 5.3. The foolproof solution for my code in production today (youch!) is to periodically restart the daemon. If you’re cringing right now, know that you’re not cringing alone.

There may be a part II to this article. Feel free to add your own PHP5 observations.

Gloria

I have a bit of a breather between contracts, so I’m back from a writing hiatus to show you more nifty Python tools and juicy goodness.

This is yet another great example of Python’s elegance and ease of use:

http://exogen.case.edu/projects/geopy/

It doesn’t just geocode. It gives you a common API to just about all of the publicly accessible URL-based geocode services. The web page labels which ones require a key for access, and which ones do not. Beautiful!

The library is also able to calculate distances using several different distance approximation formulae, as well as parse out and reformat geographical data. Now this is some nifty py-candy, eh?

Nice Job Brian, you rocked this out.

My only gripes are not with this tool at all, but with geocoding in general:

1: Where to get accurate, free IP address geocoding?

2: Why isn’t this data free for download, damn it? What if I want to use a cross section of it, and store it in my own database, without paying a fortune for it? One day, will Open Street Map be the source for Open Source geocode data which is freely downloadable, thus eliminating the 1000 queries per day limit?

Maybe we should all join forces, and store the results from 1000 queries per day per person, and build this database ourselves? Hmmm…

Gloria

PS: The geopy project has a code sprint scheduled for this Sunday, Nov. 16. Join in!