One browser tab for each source file

2012-08-11T02:44:00.000-07:00

If people want to show a source file in a new tab, it usually works for text files. But if CPP, H files or like are involved then those are offered as a download! One possible solution to this problem is to convert those source files to HTML using tools like pygments, which allows you a conversion like:

pygmentize -f html -O full -o cpp-source.html source.cpp

This will create an HTML file cpp-source.html from source.cpp, where the latter can then be served with whatever method. code2html is a small web application demonstrating this approach; follow the instructions below the get it running:

$ git clone git://github.com/hsk81/code2html.git code2html.git

Clones the GIT repository code2html from GitHub; you'll only have read access.

$ cd code2html.git

Switches to the repository folder code2html.git, which has been created at the first step.

$ ./setup.sh

Installs the code2html dependencies Flask and Pygments. For this step to work you need a python interpreter with the virtualenvenvironment builder properly set-up.

$ source bin/activate

Activates virtualenv for the project; this step is necessary since the previous one has installed our dependencies into a virtual environment.

$ ./code2html.py

Executes the code2html web application; you should see an output like running on http://0.0.0.0:5001/. Point your browser to the address and play around with the demonstration.

After you're done with the demonstration hit CTRL+C to kill code2html and run deactivate to exit the virtual environment.

Flask Guide for Web Application Development

2012-07-26T00:26:00.002-07:00

A friend of my mine has recently dropped me a "Quickest way for someone who has never built a web app to do so?" email asking for some input. I provided that, and would like to share it here.

The problem with writing web sites is that many newbies think it is easy. Well, my experience tells me that it is not. Especially if people would like to display something that's data driven (using tables or diagrams) since it becomes quickly complex: It is then appropriate to talk about web application development rather than using the misleading notion writing a web site.

Fortunately, many, many, extremely many libraries provide excellent help, in building a nice web application. But unfortunately the plethora of the choices is also a problem for a newbie, since it can be difficult the evaluate the various libraries, frameworks and approaches, if one does not know some basic foundations, on which a modern web application is build upon.

Therefore, before I explain you the guide/tutorial towards the end of this post, I've to dive into those basic foundations.

Foundations

Here you see the architecture of a recent web application of mine; I'm going to discuss the foundations of a data driven web application development based on this sketch:

Web Application Architecture

Everything starts with data! God said in the beginning "Be!", provided the data (using JSON, not SQL - he's a modern God) and we became. Almost every system needs a database (DB) to store stuff. This data ends up being shown inside a reader's browser in various forms like trees, tables, lists, text and diagrams.

So there must be something that takes your data from your database, does some magic with it and gives you your web site. This magician is usually labelled as a framework: It renders the data into a view-able format.

But the magician needs some help from another wizard, since it cannot alone deliver the rendered web site to the reader. That big helper is a webserver, which usually sits on an anonymous remote machine (a server) somewhere out there: Its job it to send the stuff to the reader.

The reader has a client machine (e.g. a laptop or a tab) that runs a browser - on top of an operating system (OS) - to display the web site.

So after establishing these very basic notions, I'll explain which software I'm using for each part to build my web applications:

Database: SQLite

For my database during the development stage I almost always use SQLite. It is a single file based sytem, which means that there is almost no installation and maintenance efforts involved. SQLite (like many other DBs) uses the query language SQL: It's a so called declarative language to tell DBs what kind of data you'd like to have. The DB usually responds with zero or many rows containing the entries you've asked for.

Database ORM: SQLAlchemy

Usually querying a DB directly with SQL is considered low level. Post stone-age people use modern tools which are so called object relational mappers (ORM) like SQLAlchemy that maps the relational model of a common database into an object (or class) model of an object oriented programming language.

A relational model is just a fancy name for various data stored inside tables. There is actually a whole mathematical theory hidden under the carpet, but that shall be of no relevance here. Just remember: A bunch of data that is saved in tables.

Now an object oriented (OO) programming language is a language that allows you to write software using notions like classes and objects: E.g. my cat "Billu" would be an object of the class Cat. Using a modern OO language like Python I've the possibility to write:

class Cat:

def __init__ (self, name):

self.name = name

def __repr__ (self):

return '<Cat %r'> % self.name

Having defined such a class Cat I'm now in the position the create various Cat objects, like:

billu = Cat ("Billu")

billa = Cat ("Billa")

Now, I've two cats living in the memory of my computer system; let's print them:

print billu, billa

>>> <Cat 'Billu'>, <Cat 'Billa'>

So this is the minimal explanation of OO programming I could think of; I hope it's enough to give you at least an idea what we're dealing with here.

Very well, but what happened to our fancy relational database? How can we connect our cats to a DB and fetch their names from without us hard coding their names inside the programming language? To do that, we need to extend the above Cat class with something like this (pseudo-code!):

class Cat (db.Model):

id = db.Column (db.Integer, primary_key=True)

name = db.Column (db.Unicode)

def __init__ (self, name):

self.name = unicode (name)

def __repr__ (self):

return '<Cat %r'> % self.name

Now, provided your database is initialized with our Billu and Billa and correctly connected to, you can execute the following DB query:

print Cat.query.all ();

>>> [<Cat 'Billu'>, <Cat 'Billa'>]

The ORM working behind the scenes translates the Python statement Cat.query.all () to the SQL statement SELECT * FROM "cat" ; the result is a table of rows:

1|Billu

2|Billa

This table is translated back by the ORM into Python objects and then printed using the print statement as [<Cat 'Billu'>, <Cat 'Billa'>]. You can see, this greatly simplifies the interaction with the DB and enables us to write relevant business logic instead of dealing with the profanities of SQL and DBs.

Note: The above example works only in combination with the framework I'm going the explain below (you need the Flask-SQAlchemy ORM extension for the shown queries).

Framework: Flask

We use the magician Flask. This sorcerer describes himself on his own web site like this:

Flask is a micro-framework for Python based on Werkzeug, Jinja 2 and good intentions.

Wow! As we know Flask is in the business of dark magik, that means his ways are not always clear to outsiders, and many need a little help to understand what the hell this is all about ..

Let me translate the tech talk for you: Flask is the name of the framework. It is a so called micro-framework, which means that it does not try to solve everything for you, but provides an extendable core of functionality. It is written in the programming language Python and is based on the libraries Werkzeug and Jinja 2.

We can ignore for sake of clarity Werkzeug since it would lead to too much technical details here. But we've to talk a little bit about Jinja 2: It is a so called template engine, meaning it allows us to embed data into other stuff.

So what does our magician friend Flask exactly do? What is this mentioned extendable core functionality about? Its main job is to take the data from the DB (by querying it with SQL by means of an ORM) and put the acquired information into other formats (by using Jinja 2) that can actually be used by a browser to display your web site.

So another description of a framework could go like this: It takes raw data from a DB and uses templates to convert it into a view-able form. Like a magician that mixes some ingredients (data) according to some recipe (template) and come up with a good drink or a nice looking gem stone (web site). I hope I did not stretch the analogy to much and, but you should be getting my point.

HTML, CSS, JS & JSON

But what are these formats a browser can display? Well these are the infamous HTML, CSS and JS triple. HTML stands for hyper text markup language and it is used the write the structure (and content) of a web site; CSS is an abbreviation for cascading style sheets and describes how a web site should look like; JS stand for JavaScript and it is used to implement the dynamic behavior of web site.

Combining all three elements a browser is able the render an nice looking, interactive web site, web page, web application etc.

Hummh, but what about data that might be asked for (by the client to the server) after the web site has been rendered? How should that piece of information be sent? That is the role of JSON, where this JS object notation is used to send bits and pieces to the browser on the client side. An example:

A table (e.g. showing weather for various cities) is structurally described with HTML, styled with CSS and has a fetch data button implementing well the fetch data functionality with JS. Now, when the user clicks on the button the browser sends a request to the server side asking for the weather information.

On the server side the Python framework Flask takes this request and queries the SQLite database using SQL for the latest weather conditions (let's say temperature and humidity). The framework converts then the query result (which is a list of temperature and humidity numbers for each city in the DB) to JSON and sends that back the the client. In this case there are no Jinja 2 templates involved.

GET & POST Requests

If a client's browser communicates with a server there are basically two kind of requests: GET and POST. GET requests should be used to ask the server for information and POST requests to send new information to the server.

Well, I hope this has clarified somehow what a framework really does; since it's working behind the scenes, it's not very obvious for newbies to see what it is really all about.

Web Server: Cherokee

Between the client's browser and the framework usually there is another entity sitting: the webserver. In the development phase of a web site a web server is usually not required, since most the the frameworks have a built-in web server.

But once a project goes into production, it is highly recommended to use one for performance, security and administrative reasons. I personally prefer to use Cherokee: It is fast, but compared to others (like Lighttpd or Nginx) not as fast, but provides an excellent graphical web based user interface to tweak the various configuration parameters of the web server.

Client: Browsers & OS

Well, use your favorite: Chrome, Firefox, Opera, Safari, Explorer etc. I prefer Chrome and Firefox, and make sure that my web application are running smoothly in those. Usually, I totally ignore the others. My development and test machine is GNU/Arch Linux.

JavaScript Libraries: jQuery & ExtJS

Now, there is one more thing you need to know: Designing an appealing web site with pure HTML and CSS is an art and not very easy. Therefore you've many, many helper libraries which are actually implemented in JavaScript. A lot of those offer also integrated UI widgets which can relieve a little the burden on the HTML author and CSS designer. My choices at the moment are jQuery and ExtJS but with a strong desire to learn more about AngularJS.

jQuery (see also jQuery UI) is a small library which helps to inspect and change the structure of and HTML document and its CSS. Plus it has a small but very useful set of widgets like a button, calendar piker, slider, tabs etc.

ExtJS is a full blown JavaScript library (framework) that allows you to build complex web applications that look similar to desktop applications. It has a very good documentation but also a steep learning curve.

Its strong point is that it shields people with less experience in HTML and especially CSS from UI design decisions, since most of them are given by the library, but that is at the same time also a negative, because many pages or applications utilizing ExtJS look somewhat similar.

AngularJS is something quite interesting, that does not support nice UI widgets, but instead offers you so called bi-directional bindings: An easy way to connect your input fields with your views lists, tables etc.

There are many more JavaScript libraries competing for the attention of a web developer, and one has to do a lengthy (on sometimes frustrating) research which library or a set of libraries one is going to use, before starting a new project. Important is in my opinion the documentation and licencing model (and of course the feature list).

Guide/Tutorial: Hello World with Flask

You need to run a GNU/Linux or Unix compatible operating system, like Ubuntu or MacOS. Plus the python, virtualenv and git packages need to be properly installed and set-up. Then execute:

git clone git://github.com/hsk81/flask-prj.git website.git

This will copy an empty project environment onto your system and name it website. But website's repository will be connected to flask-prj's. You should severe that connection with:

cd website.git && git remote rm origin

Now execute:

./setup.sh "website"

which will setup virtualenv and flask for you properly. virtual-env allows you to setup a small Python based virtual environment on your machine and helps you to keep your system installation clean. Change into your virtual enviroment with:

source bin/activate

Read now exactly the following Flask links in the given order; you've already done most of the steps in point (4) by executing ./setup.sh "website", but don't skip it for the sake of completeness (and for you to understand what actually happened inside ./setup.sh) :

This should give you a first impression on how to develop a web application with Flask; after that you should venture into accessing the database. Read

http://packages.python.org/Flask-SQLAlchemy/

to understand of the framework Flask interacts with the ORM SQAlchemy. It will provide you with a very nice overview how to get data into you application.

Final

From here on, you can apply your knowledge you've gained reading the Flask documentation and decide how you'd like to continue with the client side (front-end) development: Either you stick with pure HTML/CSS or you punch your head through some of the ExtJS, JQuery, AngularJS or other JavaCript libraries.

As long as you're not comfortable with the overall approach, I'd recommend to stay with pure HTML/CSS/JS, since the front-end JS libraries can be a pain sometimes.

If you'd like to see full blown system - a web based text editor - still in development, then you can visit http://github.com/hsk81/webed and checkout the source code the get an idea about ExtJS in combination with Flask.

Enjoy ;P

Web Development