It is what I would consider a second generation framework. It has a mature level of feature and plug-in support, and a naturally extensible model, two things which are either rare in most frameworks, or too difficult to be practical. Audio and video plug-in support is amazingly clean in the Kamaelia examples. The Kamaelia kernel, named Axon, is the core component for module execution. Module intercommunication happens via pipes, so the issue of the context switch slowness for threaded apps is not an issue here.

Kamaelia also supports OS level threading, but this support seems to be nested down into the TCP protocol module. Nevertheless, the snap-together pipe model is tempting for any app, since it’s quite easy to grok. The code examples are exactly what Python enthusiasts have come to expect: clean, short, and almost intuitive.

Like many great tools, it comfortably straddles the line between tool and toy, with clean integration of pygame, as well as the support of many common protcols (HTTP, Torrent, etc.) and audio and video codecs (Vorbis, Wav, etc.) supplemented by a solid engine.

The component list for Kamaelia is called the Component Toybox, setting the tone for this project. We are encouraged to play, and novice contributors are encouraged to join and contribute. This is what I love the most about this project. It is unpretentious, approachable by anyone who wants to learn and contribute, and it is well organized and well designed. The documentation flows smoothly, although I’d love to see more detail on helping newbies find, install and configure all of the dependencies for the dependencies for each architecture. Pygame, for example, needs SDL development libs to run the provided examples. But newbies are stuck calling friends like me to explain this to them and help them past the installation hump. This is a difficult problem to resolve in any toolset dependent on many external toolsets, having their own development paths and practices. Maybe the Python buildout tool, plus some additional scripting could resolve this issue?

This framework is exciting. It opens the possibilities of faster web and app based integration of tools and tricks. It makes you ponder the infinite possibilities of nested protocol support, not just encapsulation of protocols within HTTP. Python development just keeps getting better and more fun, and this is most certainly a project to watch for ideas and possibilities of things to come.

Gloria

The encoding requirements took some digging through obscure docs, but the rest wasn’t so bad. If you want to know how to do unicode language support to file in Python, this is for you.

import urllib2
import codecs
import BeautifulSoup
import re
import pdb
import os

class GetBBC:
	def __init__(self):
		print "In constructor"
		self.language_links = []
		self.dir = ‘BBC_Language_pages’
		try:
			os.makedirs(self.dir)
		except OSError:
			pass

	def getLanguageChoices(self):
		lang_page = urllib2.urlopen(”http://www.bbc.co.uk/worldservice/languages/”).read()
		self.soup = BeautifulSoup.BeautifulSoup(lang_page)
		# match langtexttop too
		links = self.soup.findAll(attrs={’class’:re.compile(’^langtext*’)})
		for x in links:
			self.language_links.append(x)
			print “Appending %s with link %s ” % (x.a.string,x.a['href'])

		print “There are %d language choices for the BBC news page!” % len(self.language_links)

	def archiveLanguagePages(self):
		os.chdir(self.dir)
		for x in self.language_links:
			lang_page = urllib2.urlopen(’http://www.bbc.co.uk’ + x.a['href']).read()
			clean_page = BeautifulSoup.BeautifulSoup(lang_page).prettify()
			rawfile = codecs.open(x.a.string,’wb+’,'ISO8859-1′)
			rawfile.write(unicode(clean_page,’ISO8859-1′))
			rawfile.close()
			print “Saved the %s page.” % x.a.string
		os.chdir(’..’)

	def readLanguagePage(self,language):
		os.chdir(self.dir)
		rawfile = codecs.open(language,’rb’,'ISO8859-1′)
		file = rawfile.read()
		rawfile.close()
		os.chdir(’..’)
		return rawfile

if __name__ == “__main__”:
	x=GetBBC()
	x.getLanguageChoices()
	x.archiveLanguagePages()
	y = x.readLanguagePage(’Portuguese’)

There are languages for which ISO8859-1 encoding may not work, so you may need to experiment with encoding codecs for languages not supported by the BBC.

I wrote this in May 2007, as a language support test for GrrlCamp, which is an online Open Source development group for women. We will be recruiting again in late June. If you are female, interested in volunteering development effort in exchange for learning, and have at least 6 hours free each week to do cutting edge fun Python design and development in a supportive and great online community, please post your email address and we will get back to you.

Gloria

The unmodified code

If you only need to check what region and state a particular zip code belongs, this is for you. If anyone can point me to a free longitude/latitude/full zip code site, please post that info to a reply, and I’ll rewrite this module.

import urllib2
import codecs
import re
import pdb
import os
import BeautifulSoup
import sqlalchemy

class GetZips:
	def __init__(self):
		self.zip_info = []

	def getZipPrefixes(self):
		zip_page = urllib2.urlopen(”http://pe.usps.gov/text/dmm300/L002.htm”).read()
		self.soup = BeautifulSoup.BeautifulSoup(zip_page)
		# match zip columns
		zips = self.soup.findAll(attrs={’class’:re.compile(’^trBodyRow*’)})
		for i in zips:
			y = i.find(attrs={’class’:re.compile(’^pTblBodyLL pAlignLeft*’)})
			”’
			X is the symbol for an unused 3 digit zip prefix.
			”’
			if y.span and y.span.string == ‘X’:
				continue

			# last 3 digits
			zip_prefix_3 = y.a.next.next
			zip_prefix_3 = re.sub(’[\n\r]+’,”,zip_prefix_3)

			# finding the first column will suffice.
			y = i.find(attrs={’class’:re.compile(’^pTblBodyLL pAlignRight*’)}) 

			region_state = y.a.next.next.split()

			region = region_state[-3]
			state_abbrev = region_state[-2]

			if region_state[-1] != zip_prefix_3:
				print “There is a problem here: %s” % i

			self.zip_info.append((region,state_abbrev,zip_prefix_3))
			print “Found %s %s %s” % (region,state_abbrev,zip_prefix_3)

	def injectIntoDB(self):
		engine = sqlalchemy.create_engine(’postgres://%s:%s@%s/%s’ % (’postgresql’,’something’,'127.0.0.1:5432′,’zip_db’),strategy=’threadlocal’)
		”’
		The sqlalchemy explicit scope is done for clarity. Of course
		you can “from sqlalchemy import *” instead, and change the scope
		of these calls.
		”’
		metadata = sqlalchemy.MetaData()
		metadata.bind = engine
		zip_table = sqlalchemy.Table(’zip_abbrevs’, metadata,
			sqlalchemy.Column(’zip_abbrevs_id’, sqlalchemy.Integer, primary_key=True),
			sqlalchemy.Column(’three_digit_abbrev’, sqlalchemy.String(4)),
			sqlalchemy.Column(’region’, sqlalchemy.VARCHAR(50)),
			sqlalchemy.Column(’state_abbrev’, sqlalchemy.String(3)))

		metadata.create_all(engine)

		for (region, state, zip) in self.zip_info:
			print “Injecting %s %s %s\n” % (region, state,zip)
			zip_table.insert(values={’region’:region,’state_abbrev’:state,’three_digit_abbrev’:zip}).execute()

if __name__ == “__main__”:
	x=GetZips()
	x.getZipPrefixes()
	x.injectIntoDB()

# vim:ts=4: noet:

I am writing this code for the nonprofit called The Freelancer’s Union in NYC, which currently has a nationwide member drive: http://www.freelancersunion.org/advocacy/index.html.
I will shamelessly plug them in exchange for sharing this code with the world.

The more members they get in each US state, the better nationwide insurance plans they can offer. They offer E&O insurance for IT freelancers as well, so even if you freelance part-time, this could be for you. This organization rocks. I’ve been a member for three years, and now I proudly write code for them.

Gloria

The unmodified code

The database is such an essential part of the design, and being able to automatically create, archive, purge and delete databases is tremendously handy in so many of these designs.

I wrote this example to generate dynamic databases for short courses, training sessions, and demos. I use the example itself to demonstrate the functionality of CherryPy and SqlAlchemy.

First of all, I start my CherryPy process on my local port 7001, allowing me to do this:

http://grrlcamp.org:7001/dbi/create_my_db?db_type=postgres

or this:

http://grrlcamp.org:7001/dbi/create_my_db?db_type=mysql

First, the small bits of quirkiness in Postgresql and MySql

Starting a Postgresql session on the command line, we see what happens before the URL is referenced:

[cwc2008@localhost user1]$ psql -d template1 -U my_super_user
Welcome to psql 8.2.6, the PostgreSQL interactive terminal.

Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help with psql commands
       \g or terminate with semicolon to execute query
       \q to quit

template1=# SELECT datname FROM pg_database ;
  datname
———–
 postgres
 template1
 template0
(3 rows)

template1=#

and after (Note: I had to exit the command line session, run the URL, then re-enter the command line session):

template1=# SELECT datname FROM pg_database ;
   datname
--------------
 postgres
 template1
 template0
 cwcuser_7001
(4 rows)

template1=#

The cwcuser_7001 name is auto-generated from the user’s ID and port number, making it useful for training sessions with many users.

my_super_user was created using the Postgresql createuser command, and, of course, must be a super user to do this operation. The template1 database is the Postgresql master database, from which all other databases in that instance are visible.

Caveat: In Postgresql, only one user can access template1 at any given time. Violating this rule gives you this error:

template1=# create database blahblah;
ERROR:  source database "template1" is being accessed by other users
template1=#

Be sure to close your database descriptor when finished with template1. If this is not a production app, and/or you feel comfortable bouncing the instance, this lock can also be cleared by running:

-bash-3.2$ /usr/local/postgresql/8.2.6/bin/pg_ctl restart -m immediate -D/usr/local/postgresql/8.2.6/data

waiting for server to shut down... done
server stopped
server starting
-bash-3.2$ LOG:  database system was interrupted at 2008-02-14 15:07:47 EST
LOG:  checkpoint record is at 0/4B9054
LOG:  redo record is at 0/4B9054; undo record is at 0/0; shutdown FALSE
LOG:  next transaction ID: 0/2046; next OID: 57344
LOG:  next MultiXactId: 1; next MultiXactOffset: 0
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  record with zero length at 0/4B909C
LOG:  redo is not required
LOG:  database system is ready

This is nasty. In live production environments, do this with great care.

MySql is a bit forgiving when adding, removing or accessing new databases. It does not require the isolation level to be set, as in Postgresql.

We see what databases exist before:

/usr/local/mysql6/bin/mysql --socket=/tmp/mysql10.sock  -u my_super_user -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 170
Server version: 6.0.3-alpha-log Source distribution

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| test               |
+--------------------+
3 rows in set (0.00 sec)

mysql>

and after:

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| cwcuser_7001       |
| mysql              |
| test               |
+--------------------+
4 rows in set (0.00 sec)

mysql>

MySql does not require my hopping in and out of the command line like Postgresql. It handles user scope much better, in my opinion. It does, however require each new non-super user to be granted access to each database they need, but this is (1) trivial, and (2) outside of the scope of this article.

The SqlAlchemy code to handle the quirky bits

This is invoked by CherryPy, but can be run in any other manner of Python execution. I have a Class called DBInterface, method is called create_engine:

class DBInterface:
    #~~~~~~~~~~~~~~~~~~~~~~~
    def create_engine(self,choice,local_dbname=None):

        if choice == 'super_mysql':

            '''
            Super user MySql commands do not require connection to a database.
            '''
            db = sqlalchemy.create_engine('mysql://%s:%s@%s' % ('my_super_user','some_password','127.0.0.1:3333'),strategy='thr
eadlocal')
            # 'self' is not threadsafe! Don't use it!
            #self.show_databases = "show databases ;"
            cherrypy.session['show_databases'] = “show databases ;”

        elif choice == ‘mysql’:
            db = sqlalchemy.create_engine(’mysql://%s:%s@%s/%s’ % (’my_super_user’,’some_password’,'127.0.0.1:3333′,local_dbnam
e),strategy=’threadlocal’)

        elif choice == ’super_postgres’:
            ”’
            Super user Postgresql commands require that you connect to the
            built-in template1 database, and set the isolation level to 0.
            This can’t be done in SqlAlchemy, so we do it in Psycopg.
            ”’
            DSN_string = “dbname=template1 port=2222 user=my_super_user  host=127.0.0.1 password=some_password”
            def connection():
                self.local_conn = psycopg.connect(DSN_string)
                self.local_conn.set_isolation_level(0)
                return self.local_conn

            db = sqlalchemy.create_engine(’postgres://’,creator=connection, strategy=’threadlocal’)
            # NO!
            #self.show_databases = “SELECT datname FROM pg_database ;”
            cherrypy.session['show_databases'] = “SELECT datname FROM pg_database ;”

        elif choice == ‘postgres’:
            db = sqlalchemy.create_engine(’postgres://%s:%s@%s/%s’ % (’my_super_user’,’some_password’,'127.0.0.1:2222′,local_db
name),strategy=’threadlocal’)

        else:
            db = None

        return db

The exposed method in the same class which invokes the create_engine method, is called create_my_db, which we saw in the URL above (I will address the CherryPy URL functionality after this point):

    @cherrypy.expose
    def create_my_db(self,db_type):

        '''
        Creating a db is a super user act.
        '''
        cherrypy.session['db_type'] = ’super_’ + db_type
        cherrypy.session['my_dbname'] = “cwcuser_%s” % (cherrypy.request.config['server.socket_port'])

        db = self.create_engine(cherrypy.session['db_type'])
        db.execute(”create database %s ;” % cherrypy.session['my_dbname'])
        databases = db.execute(cherrypy.session['show_databases'])

        dblist = []
        for d in databases:
             dblist.append(d[0])

        print ‘, ‘.join(dblist)
        self.dbclose(cherrypy.session['db_type'],databases)
        return ‘, ‘.join(dblist)

Because I clump together super user and non-super user database engine connection functionality, I distinguish the difference by modifying the users db type to a “super_” setting in a CherryPy session variable. This way, only one method is called in various internal modes of operation. When in non-super user mode, the SqlAlchemy connect string is the same for both Postgresql and MySql.

No more db quirks in this app: The rest is clean SqlAlchemy

All db operations after this point use the exact same SqlAlchemy code for both MySql and Postgresql. That is a profoundly beautiful thing.

The destroy_my_db method is generic:

    @cherrypy.expose
    def destroy_my_db(self,db_type):

        '''
        Destroying a db is a super user act.
        '''
        cherrypy.session['db_type'] = ’super_’ + db_type

        if not cherrypy.session.has_key(’my_dbname’):
            cherrypy.session['my_dbname'] = “cwcuser_%s” % (cherrypy.request.config['server.socket_port'])

        db = self.create_engine(cherrypy.session['db_type'])
        db.execute(”drop database %s ;” % cherrypy.session['my_dbname'])
        databases = db.execute(cherrypy.session['show_databases'])

        dblist = []
        for d in databases:
             dblist.append(d[0])

        print ‘, ‘.join(dblist)
        self.dbclose(cherrypy.session['db_type'],databases)
        return ‘, ‘.join(dblist)

As above, all other database operations from this point forward are generic SqlAlchemy code. Yaaay!

The CherryPy layer

My CherryPy main module, which sets up the needed objects and assoiated URLs, looks like this:

import cherrypy
import pdb
import sys,os,re,traceback
import user_data
sys.path.append('../')

#hostname = os.environ.get('HOSTNAME')
#hostname = re.sub('\..*$','',hostname)

# Add your app here.
import db_wrapper

# Only generic headers and footers go here.
# To use generic templates in your app, you may need to symbolicly link
# them into your Templates dir, or references them by absolute path.
TemplatePath = os.path.join(os.path.dirname(__file__), '/cherrypy/dev/templates')

# Add your class here. Note that the 'root' should be the authentication module.

# To test, go to http://grrlcamp.org:1112/run_test
root = db_wrapper.WrapperTest()

# http://grrlcamp.org:port/dbi/create_my_db, destroy_my_db, etc.
root.dbi = db_wrapper.DBInterface()

# http://grrlcamp.org:port/user/get_user_data
root.user = user_data.UserData()

try:
        cherrypy.tree.mount(root)
        cherrypy.config.update(os.path.join(os.path.dirname(__file__),'cp.conf'))
        cherrypy.server.quickstart()
        cherrypy.engine.start()

except SystemExit:
        #smtpInterface.sendSysAdminEmail('cherrypy@xyz.com','sysadmin@xyz.com',"From %s: CherryPy is going down due to shutdown." % hostname,"")
        sys.exit()

except:
        print "ERROR: CherryPy crashed, error:"
        raise

finally:
        #smtpInterface.sendSysAdminEmail('cherrypy@xyz.com','sysadmin@xyz.com',"From %s: CherryPy is going down (from finally clause, check for another error message after this one)." % hostname,"")
        pass

The CherryPy “root” determines what objects are accessible via URL. I mounted a simple test under the root, and nested the above database code under root.dbi, hence the /dbi/ reference in the URL.

root.dbi contains an instance of the DBInterface class we saw earlier. All exposed methods in the DBInterface class become accessible via the /dbi/method_name URL when CherryPy is started.

I’ve attached the complete code used for training, configured to run on localhost, and connect to both MySql and Postgresql daemons. It contains some explicit redundancy and commented code for learning purposes. Your configuration variables will vary slightly. Enjoy.

auto-create-any-db.tar

Qualifications:
- Female (willing to voice verify. You’d be surprised that this is necessary, but at times, it is.)
- Programmer, or programmer wannabe.
- Able to meet online weekly for one hour.
- Able to dedicate a few hours a week to this project.
- Willing to read, try, do, roll up your sleeves and design and develop, even if you’ve never done this before.
- Able to travel to NYC in May for the software wrap-up and release (optional). Your expenses = your round-trip flight/ride, and minimal donations for food. Lodging is free, food will be prepared and cooked by participants.
- Willing to work in Python (If we get a Rails following, we may use both).
- Willing to learn Dojo, CSS and HTML for the user-facing interface.
- Willing to release the software to the public, free of charge.
- Willing to provide small bits of support after development is complete.

This is a large project, touching on everything from basic Linux admin, svn setup, LINUX shell, Apache configuration, threading, database schema design and access, process/content management, front end design and development, inter-language data transfer, and the list goes on. If you are willing, this is an outstanding learning opportunity.

If interested, reply to this post with a valid email address, and we will send you an invitation.

Please, no observers/reporters. This is strictly a development group, and all members must be willing to write code.

My first step was to get to know the front end Dojo guru who was working on the user interface. We were separated by thousands of miles and many time zones. We had never met in person (and still have not to this day). We were going to be spending long and occasionally stressful nights together in virtual space, so we spent time chatting to get to know each other. After the first month, we could both almost intuit when the other was tired, frustrated, or both. After a short time we worked so well together that communication was fast and efficient, and the work was not hindered by our physical distance. There were occasions where it would have been great to draw together on a real whiteboard. But we compensated for this loss fairly well. It was the first entirely virtual project I had worked on, and the social aspects of how we worked together were just as interesting as the technical aspects of the project.

We spent time evaluating tools which would achieve our goals of rapid design and development, and excellent performance both in the browser and on the server. For the server development I chose CherryPy and SqlAlchemy, using Postgresql. For the front end we chose Dojo and Javascript, using JSON to communicate between the browser and the server.

I started with the database schemas in Postgresql, then the wrappers which automatically generate one database per set of client data. The basics on how I do this in Postgresql are here:

http://www.devchix.com/2007/05/27/dynamic-web-content-what-you-can-do-in-postgresql-that-you-cannot-do-in-oracle/

This model of one database per data set per client worked so well with the client’s particular data that it was a natural fit. Scripting dynamic archiving and recovery took under one day in Python and Psycopg, and worked like a charm.

Here is a peek at how Postgresql can be used in CherryPy (or any other web framework, for that matter) to snapshot to backup tables, and subsequently to disk:

         def take_snapshot(self):
                try:
                        '''
                        All timestamps are UTC.
                        '''
                        snaptime = datetime.datetime.now()
                        snaptime = pytz.timezone(config.server_timezone).localize(snaptime)
                        snaptime = snaptime.astimezone(pytz.timezone('UTC'))
                        snaptime_str = snaptime.strftime("%Y%m%d_%H%M%S_%Z")

                        db = self.login.open_session_db()
                        #query = 'select datname from pg_database'
                        query = "select tablename from pg_tables where tablename not like 'pg_%' and tablename not like 'sql_%' and tablename not like '%_utc';"
                        our_tables = db.queryAndFetchAll(query)
                        mod = ""
                        for table in our_tables:
                                table = table[0]
                                ”’
                                This table grows forever. Never recover it
                                from snapshot.
                                ”’
                                if table in (’user_change’):
                                        continue

                                mod += "create table %s_%s as (select * from %s);\n" % (table,snaptime_str,table)
                        print "Taking snapshot: %s" % mod
                        db.executeMod(mod)

                except:
                        raise
                        db.close()
                        ft=open(TemplatePath + ‘/login’)
                        showPage=ft.read()
                        ft.close()
                        showPage = "<P> Invalid User Name or Password.</P>" + showPage
                        return showPage

                db.close()
                return snaptime_str

        take_snapshot.exposed = True

Small snippets on the key concepts of flushing/retrieving snapshots from/to disk:

         def archive_to_disk(self,snaptime_str):
                        [...]
                        for archive_table in archive_tables:
                                archive_table = archive_table[0]

                                shmod += "mkdir -p %s; pg_dump -U tp -t %s -d -f %s/%s %s\n" % (archive_dir,archive_table,archive_dir,archive_table,dbname)
                                mod += "drop table %s;\n" % archive_table

                        print "Archiving old snapshot tables to disk: %s" % shmod
                        os.popen(shmod)
                        [...]

        def revert_to_snapshot(self,snaptime_str):
                        [...]
                        for revert_table in revert_tables:
                                ”’
                                Move current tables to current snapshots.
                                Copy older snapshot tables to current tables.
                                ”’
                                revert_table = revert_table[0]
                                table = re.sub(’_%s’ % snaptime_str,”,revert_table)
                                mod += "alter table %s rename to %s_%s; create table %s as (select * from %s);\n" % (table,table,curr_snaptime_str,table,revert_table)

                        print "Taking snapshot, and reverting to old snapshot: %s" % mod
                        db.executeMod(mod)
                        [...]

Why spend so much Web 2.0 article space talking about database concepts and client data handling? Because while designing Web 2.0 applications, all matters involving client data take up somewhere between one-third to 50% of the development time and energy. Client data is the very reason the application exists in the first place. The more time and energy placed into making sure it is kept, stored and manipulated in the best possible way for the application, means less headaches and refactoring as the product matures.

Once the database schema and wrappers were in place (much of those details were mercifully left out of this article), the business logic was next. When designing a RESTful application, a great bit of thought goes into the design of the state transactions between the browser and server. For our needs, passing massive XML structures back and forth to communicate state seemed like overkill. Passing JSON dictionaries with well defined fields back and forth seemed like a better solution, for speed and ease of customization.

Once the ‘how’ part is resolved, the next question is what will be passed back and forth between browser and server. In a REST implementation, the goal is to pass a complete transaction from browser to server, and a complete transaction from server back to browser. The definition of a transaction will vary based on the application. For RSS feeds, a transaction is only one-way, where the server constructs and sends a complete XML document to an RSS feed reader. This is probably the simplest REST implementation in existence right now. I’ll be addressing a more complex transaction model, where client and server pass each other data, and either side can manipulate the data and pass it back to the other side.

To achieve a RESTful state transaction model, in a nutshell, means passing enough data back and forth so that a user can click on any feature or function of the application, in any order, and the server will (1) know what to do every time without fail, and (2) will not ‘remember’ each client’s previous states. This implies that the server is ’stateless’ with respect to specific client data. The server has a state of it’s own, and knows it’s own transaction state, of course. But it is only ‘aware’ of each client’s state when it is contacted by the individual clients, and the client notifies the server of it’s state.

It helps tremendously to work out the ‘chatter’ between the client and the server in human language before designing it. Here is an example of RESTful chatter between a client inventory application, and it’s corresponding server application:

server’s current state: “My current state is fifty boxes of paper clips in inventory.”

server: “Received client one’s request. Hi client one, you are properly authenticated, so I’ll look at your state. You want sixty boxes of paper clips. You will settle for a minimum of forty. My current state is only thirty in inventory. Your order is not fulfilled, and I have thirty left. Bye.”

Compare this to non-RESTful chatter between client and server:

client two:”Hi server.”

The point of the chatter is to ensure that the client is passing a complete transaction with every request, and that the server passes a complete transaction with every response. Similar to the CRC cards for OO design, the Chatter ensures that you have your states, transactions and interfaces well defined.

Note that a crash of the server in a REST implementation means that the client can’t reach the server with it’s complete transaction, so it waits and accumulates transactions, sending bulk transactions when the server is available once again. The crash of a client in a REST implementation means that if the server cannot acknowledge the client’s transaction, it can either drop it, knowing it will arrive again, or process it and ignore the duplicate transaction that may arrive again when the client is back up. There are no “partial states” which need to be resolved, the transaction was either accepted and completed, or rejected. Recovery in a REST implementation becomes trivial because of this simplicity, and this saves a tremendous amount of development time and energy.

Once your chatter seems complete, it can be translated into a header and data components. For this application, this means that the JSON structure coming from the client to the server has fixed fields with expected values, and data arrays/dictionaries with predefined formats.

The server receives the JSON data, decodes it into Python dictionaries and arrays, and first evaluates the ‘header’, as defined by the developers. The header tells the server the client’s current state, as well as the server’s last known state when the client last contacted the server. (For our application, this level of detail was necessary. Not all applications would need to know about the server state, and this needs to be decided while devising the chatter.)

Once the client’s full state is known, the client’s data can be processed against the server’s current state data.
The server constructs a reply header and data, as defined by the developers, comprised of expected fields and proposed values. In the case of inventory, let’s say client two needed to be notified of client one’s transaction, because these are franchise stores, and all inventory data is shared between them. The returned data set from server to client one would be the acknowledgment header (processed_order=False) plus the data received from other clients (other_orders_since_you_last_contacted_me=client two placed this order at that date and time).

CherryPy turned out to be an excellent framework for this task. It’s thin, fast, threaded architecture, and clean session handling made REST implementation quite easy. The same methods are called by every client, to contact the server and pass across it’s data. The same method is called by the server threads to respond to every client. The API between client and server is reduced down to about five methods. Granted, the object which checks the client’s current state against the server’s current state is quite large. But the transaction recovery code is quite small, and the objects are extensible, to handle new transaction types.

This article assumes the model of user authentication separate from transactions. The subject of cookieless authentication won’t be addressed here, but it is an option to consider, based on the application requirements.

This article also only addressed a Python implementation. Rails developers have a bit of an advantage, having a built-in REST methodology. I hope some articles to come will address Rails implementations.

http://www.parallelpython.com/

He installed it on his XP desktop, I installed it on my Fedora Core 6 Desktop, and on our Linux test server with eight CPUs. The installation in an existing Python distribution took five minutes. Reading the daemon instructions and starting Parallel Python daemons on all of these machines took another three minutes.

We modified a test example, a distributed sum of primes, to run for many minutes, giving it a huge upper limit. On the Linux machines, running the top command, then pressing the number 1 reveals the activity on all CPUs. We started the modified test example, closely watching top and the task table on XP. We saw an even distribution of jobs across machines, and a consistently even balance amongst all CPUs. Parallel Python is written so well that it balances jobs according to a machine’s capacity and speed.

I decided to integrate it into my web scraper project. I need to spawn jobs to scrape different pages in parallel. The problem I had was mapping my OO model to the existing procedural examples. I have a scraper class, and I want Parallel Python to run it’s run() method, and see it’s self variables.

Looking for help, I find a help forum on the Parallel Python web site. Not a bad start. I post my question, and get a response within 24 to 48 hours from the most helpful geek I have ever dealt with.

Vitalii Vanovschi, the author of the examples, and probably also the author of most of the Parallel Python code, offers damned near flawless, quick, and ego-free assistance with this software. To answer my question, he rewrote an example ‘on the fly’, to show how classes can be used in Parallel Python. His responses to everyone are consistently respectful, concise, and helpful, with a strong undertone of of the genuine love for the Python language and it’s capabilities. This is the best tech support I have ever gotten for any product, EVER.

To top it off, there is a link on his main page, asking people to talk about what drew people to Python. This adds a ‘Ruby-like’ social element to his web site which is still unfortunately rare in the Python realm. I was thrilled to find some social aspect to this web site and it’s subject matter. It is refreshing to be able to talk about the technical and social aspects of the language and the tools, simultaneously, with the same crowd of people. Rubyists seem to understand this, and Vitalii’s hope to bring this aspect into the Python world does not go unnoticed.

Vitalli has brought so many good things together in one application, on one web site. He has created a simple, easy-to-navigate web interface with related forums, great documentation and examples. The software itself is robust, well-written and so easy to use that one can integrate it into an existing project very quickly.

This project is an excellent example of good software, good presentation, good tech support, and good social support. I think Python programmers can learn a lot from this example. My hopes are that more Python project developers make better attempts at the social aspect of their projects, asking people to share their experiences with the project and the related languages in an ego-free environment. Maybe user communities will form from these environments. And maybe, just maybe, Python programmers can one day experience the sense of community that Ruby developers already enjoy.

Vitalli, kudos, and thank you for such a great, well-rounded project. Thank you for setting the the technical, social and presentation precedents. May all new projects follow this model.

With respect to the web framework itself, certain things swayed my decision. We had an existing database and a whole suite of home-grown Psycopg objects to access this data the way we wanted. I was very reluctant to rewrite or replace all of that, because it contained much of the business logic in a way that the entire group understood and was used to seeing/changing.

I was tempted by Django, but realized that we would not need or use it’s very nice database administrative interface (even though I had no trouble whatsoever retrofitting our existing schema into Django. That was quite impressive). At the time, I wasn’t sure if we would use Django’s template language either. Then I discovered that Django did not have multiple database support. These factors made me shy from Django and back toward a thinner Python web framework. I wanted next to no explicit database support, and the freedom to choose whatever template language I wanted.

My best choices at the time were PythonPaste and CherryPy, which were running neck-and-neck in terms of rolling out new ease-of-use features, and WSGI support. I thought PythonPaste had a slight advantage over CherryPy at the time. I used it in earnest, only to find out that the bugs I posted to their user’s list were ignored. There were also times where the new release egg files were broken, and brought my prototype down to a grinding halt. I also found that the configuration file syntax was a bit obscure, and documentation was sparse. I was foced to ask the one guru, Ian Bicking, for help, and hope that he’d answer my post. Sometimes he did, and sometimes he did not. This was not comforting, to say the least. “Did he respond to your bug post yet?” My boss would ask. “Nope, I have to wait another day and see..” I responded time and again, until I finally dumped PythonPaste and gave up.

CherryPy had it’s own quirks. I experienced some header bugs, some oddities in cookie formatting between it and PHP (only to discover that CherryPy did it according to the official spec, and PHP did not). Its configuration syntax was also a bit obscure. But then CherryPy 3.0 was released, and this changed everything. WSGI support was easier to implement than ever in PythonPaste (although I think PythonPaste put the pressue on the CherryPy group to add this functionality, so PythonPaste did serve a good purpose in that respect). The configuration syntax, the root instantiation syntax, object and URL nesting all became much easier in 3.0. The support list is usually excellent, where several developers post responses in a day or two. There was no doubt that this was the direction to take, and it is still my favorite thin web framework.

CherryPy imposes no requirement to use it’s own template language, called CherryTemplate. This is a great! It gave me the opportunity to evaluate template languages as a separate entity. Here is what I found.

One of the goals of the new web framework was to migrate ourselves away from Zope. Much of our administrative interface was, and still is written in Zope using DTML. As part of the evaluation of web frameworks, I tried learning ZPT, which is a tag-based template language. Its syntax was a bit obscure, and it’s python interface was not as robust as I had hoped.

We were not generating dynamic XML content, so a tag based template laguage was not for us. I could see this as being a handy tool for XML content, but then again, there are other Python XML content generators with less obscure syntax. DTML is similarly obscure (although it has try/except syntax, which comes in handy when testing variable values). DTML is not pretty. It’s SQL interface is handy from within Zope, but not handy outside of that scope. Despite the fact that we used it for most of our web interface within Zope, none of us enjoyed using it. I found the magic DTML document online, and became decent at DTML, but I still did not like its obscure syntax. Also, all of us at various times were left wondering “Where did it get THAT value from?” I blame Zope for the variable ‘magic’ that had to constantly be deciphered when passing data to DTML, but it still made the whole experience one that none of us wanted to repeat.

In the end, it was difficult to justify either DTML or ZPT outside of the Zope scope, so both were out of the question in our new web framework.

I tried Cheetah, which seemed simple at the time, but too procedural for me. I wanted a template language which was more object-oriented in design, and as close to the Python syntax as possible. My logic was “Why should I learn/impose a new language on web developers if they already know and use Python?”, because in our small shop, the developers also handled web content.

Kid was the next template language I played with. It seemed interesting as an XML based template language. It has separate template and template generation components, nice for XML generation of fixed templates. But in our non-XML environment, many of our page templates are dynamic. They change based on whether the user is logged in as an administrative user, their group rights and access, etc. Kid turned out to be overkill, since for our application, it is much easier to dynamically generate both the template and the content based on the user’s login mode and group settings. Kid is nice, but not for this non-XML, dynamic template application.

MochiKit was next. I was thoroughly impressed with it’s ability to make javascript usable by a Python developer. Plus, the signal library sounds nifty. Contact the server when certain events happen, instead of pinging the server every N seconds, and letting the server figure out what event happened? Wow, how neat. Unfortunately this too was overkill for our particular application, and I could not justify it in the scope of the project. “Hey, who wants to learn javascript?” did not fly well in our group, so I skipped it.

Lumped along with CherryTemplate were some others, whose names escape me now, with the same philosophy: Provide Python-like syntax and variable expansion. This doesn’t offer much to native Python developers, where variable expansion is already so easy.

I ended up using no template language. Python seemed to serve our needs very well. I defined our own /Templates subdirectory for each module/URL path in CherryPy. Each template file contained syntax like this:

<table border="0" cellpadding="0" cellspacing="0">
<tr><td>

        <p>Client ID: %(client_id)s</p>
        <p>Account Expiry Information (%(VEDIT_OPTION)s)</p>

        <a href="%(STATIC_URL)s/%(inventory_link)s/ViewInventory?client_id=%(client_id)s">Unfulfilled Inventory Orders</a>
        <a href="%(STATIC_URL)s/%(history_link)s/ShowHistory?client_id=%(client_id)s">View Inventory History</a>

        </td>
        <td>

        etc...

The %(variable)s Python template syntax expands out to the variable name by the following syntax:

class MyCherryPyPageGenerator:

        def index(self):
                STATIC_URL = "https://my_site.whatever"
                TemplatePath=os.path.join(os.path.dirname(__file__)) + '/Templates' # set TemplatePath to current dir/Templates
                client_id=505
                inventory_link= 'manage/inventory'
                history_link = 'manage/history'

                if user.admin_mode:   # Optionally set certain links based on login mode.
                   VEDIT_OPTION = '''
                      a href="%(STATIC_URL)s/admin/EditAccount?client_id=%(client_id)s">Change Your Account Settings
                   ”’ % locals()    # Expand STATIC_URL variable in the VEDIT_OPTION variable now.
                else:
                    VEDIT_OPTION = “Access Denied”

                fd = open(TemplatePath + ‘/template_file’)
                this_page = fd.read() % locals()
                fd.close()
                return this_page

        index.exposed=True

This syntax opens the template file shown above, conditionally sets the required local variables, expands them using the locals() syntax, and returns the CherryPy page to the browser. Ignoring the CherryPy syntax, you’ll see that the template logic involves setting local variables, reading in a template file, and expanding those variables.

This syntax, combined with CSS macros for standard page formatting, proves to be very powerful and simple. What is not showing up in my template, above, are CSS id tags, such as <td id=”leftcolumn”>, which are CSS macros, defined in a separate CSS template file.

For our needs, this seemed perfect. It is entirely in Python, not too bad to read when compared to reading another template language, and easy for everyone to modify and maintain. The advantage to this approach is that the templates themselves can be as dynamic as you wish. Entire section of a page can be displayed or not, based on your explicitly defined variables and logic. The formatting is entirely controlled by CSS, and can be overridden in Python when appropriate.

The downfall to this approach is that complex pages generate very long pieces of code, where the template and the business logic are intertwined. Is this a downfall? I am not sure, because I have yet to find any single template language which can cleanly and easily separate business logic from template code, and leave you with something legible.

My Conclusion

Each template language exists for a reason or a season. I think ZPT and DTML have had their days in the sun, and those days are now over. As AJAX grows in popularity, MochiKit seems like something very worthwhile, but requires an entirely separate development effort. It is also up against a very powerful Javascript library called Dojo, which offers a browser based editor and graphics package in addition to standard Javascript functional libraries. This is tough competition.

Many template languages are simple enough to provide the template abstraction you need, but are not robust enough to justify not using Python directly. So many of them promise to separate business logic from template design. But in reality I have yet to see this happen. Very often, the person making the template change has to know about the method returning it’s data, or even about the db schema itself. With the exception of CSS designers, I have not seen a template designer who did not need some level of programming experience and access to the interface code.

I see many template languages tout their speed and similarity to Python. My immediate question is: “Well, then why should I not simply use Python instead of your template language?” My wish for the future of template languages is that they do/don’t do the following:

1. Either make more use of CSS templates, or use a similar CSS ‘#’ macro syntax which can inherit from a CSS template passed into the template language. That would be so nice. CSS is not going anywhere anytime soon. And let’s face it, it rocks if you know how to use it.

2. Don’t just brag about how similar it is to Python, and how fast it is. Show me how it can help me truly separate business logic from content, without adding a level of obscurity.

3. Make it support DOM2, please! This one is personal for me, since I am doing DOM2 server-side manipulation in my current project. A template language with a good DOM2 structure, possibly one which can be passed to other Javascript libraries such as Dojo, would be so very nice.

Comments are welcome. Examples will be incorporated into this article with author credits, so please post or email some useful examples.

Did you work with a template language not mentioned here? Please let us know about your experience.

In short, lots of time was burned doing repetitive things in supposedly great languages. At least we all got paid to do it.

Then along came Perl. Perl rocked my world for many reasons. It was interpreted, so no long compilation cycles, hooray! It was very much like many of the FSF tools I had been using in UNIX for years. It was like the ultimate super sed/awk/grep tool, introducing new meta characters for things like whitespace. Hooray again! No more hours typing sed commands which check for one or more space, followed by one or more tabs, again followed by possibly more space characters, or some convoluted combination of space/tab/space…ARGH! Perl introduced this: ‘\w’, which caused me to have what seemed like an out of body experience, because I was so damned happy.

Then a person named Lincoln Stein wrote a perl module which made me quite a bit of contract money. It was called cgi.pm, and it was stunning. It allowed me to roll out CGI based web interfaces in week instead of months. I wrote many automation tools for AT&T, Lucent, Motorola, and many other companies, one after another, and loved every bit of it. I personally thanked Lincoln via e-mail, then started to pester him with questions, and did not hear back from him. But at least I was able to express my gratitude.

I was happy for a while. To me, Perl was no more convoluted than any other piece of UNIX scripting tool I have used before, so I adjusted well. But as new Perl modules started to appear, Perl started to be used for more complex tasks, which became a bit messy. Perl was still a procedural scripting language. Granted, any experienced developer can encapsulate with the best of them, and this is how procedural languages were kept under control. But it would be so nice to have a truly OO scripting language, many of us sighed.

Then along came Python. The best of all worlds, a byte-compiled scripting language, very OO, run time type checking, small, fast, robust, full support of OS and system calls, it had it all, in one neat package. It was love at first sight for me. I started using it for simple scripting tasks, such as directory walks, file searching, simple ten line scripts. I then started to learn TkInter, then WxPython, and before I knew it, I wrote a touch screen application in three months, entirely in one language. I no longer needed different tools, and wrappers, and interfaces from C to something else. It was all here, and the syntax seems almost intuitive. I once again started to find the fun in programming. Much like a long term relationship, we all need those reminders every now and then, so we can proclaim to ourselves “Ah, yes, NOW I remember why I fell in love with you in the first place”. Python was that catalyst for me. It really changed everything.

I now only program in Python. I have yet to find the need to use anything else. Even embedded systems support thin Python ports. Life is good, and I genuinely love the language and the tools I work with, warts and all.