Ventanazul

The Django gunicorn fabfile project

alexis — Wed, 04 Jan 2012 22:21:00 +0000

Since I read about Green Unicorn, better known as gunicorn, a Python WSGI HTTP server that goes along very well with Django, I knew I had to try it, and trying it I did. And I liked it. There were many steps to make a server work but overall it was cleaner than other approaches, including the Django deployment using Nginx, Apache and mod_wsgi that I described long time ago. And many reports indicate that a Django setup with gunicorn performs better than one using Apache.

But when you have to setup many servers with the same configuration, the process gets boring and repetitive, and when you're bored you make mistakes. This is the perfect scenario for automating with Python and a Fabric script.

I wanted the initial version of my fabfile to do the following:

Start with a clean install of Ubuntu 11.10, just with an openssh-server running.
Install all the Ubuntu packages needed for running a basic Django site, these include Nginx, virtualenv, some Python development tools, and PostgreSQL.
Create virtual environments and manage them with virtualenvwrapper.
Install a few Python packages in the virtual environments: Django, gunicorn, iptyhon, psycopg2, to name a few.
Create the configuration files to manage both staging and production sites with upstart.
Grab the code for a Django project from a git repository.
Keep the project's specific settings.py outside of version control.
Make everything work with just one or two commands from the shell.

And that's what the first version of this fabfile is doing. I don't have the time to explain all the details of the setup, Senko does a good job with his Django setup using Nginx and gunicorn and I was inspired by him (thanks dude), but I can give you the code I wrote and commited to GitHub.

I've called this the Django gunicorn fabfile project, so go get it and play with it.

Drupal queries and the IN SQL operator

alexis — Wed, 30 Jun 2010 14:44:06 +0000

One of the most used functions in Drupal's database abstraction layer is db_query, which allows passing an SQL string and corresponding arguments to send a query to the database. I'll give you a quick overview of how db_query works before showing you how to drupalize a query such as:

SELECT field1, field2 FROM table_name WHERE field1 IN (value1,value2,...)

Drupal database abstraction layer and db_query

As many other content management systems and web development frameworks, Drupal implements a database abstraction layer that you, the developer, can use for writing code that works with different database servers. For this to work you need to follow certain rules and this starts with the way you pass queries and arguments to the db_query function.

Arguments in a db_query call need to use sprintf style specifications, placeholders such as %d for integers and %s for strings. This allows Drupal to avoid SQL injection attacks and perform other security checks.

Let's review a simple example:

$sql = "SELECT v.vid FROM {vocabulary} v WHERE v.name = '%s'";
$vid = db_result(db_query($sql, $vocabulary_name));

This code will get a vocabulary's id searching by its name, which is passed as a string. Notice the curly braces around the table's name, they need to be there if you want Drupal to provide table prefixing, this is important so get used to always do it this way.

Many Drupal beginners may opt for what they consider a simpler approach:

$sql = "SELECT v.vid FROM vocabulary v WHERE v.name = '" . $vocabulary_name . "'";
$vid = db_result(db_query($sql));

Even if it may look simpler now you're bypassing Drupal's security checks and your query will start breaking up as a series of concatenated strings which is not good for code readability, and this gets more confusing with more complex queries.

Passing arguments to db_query as an array

Arguments can be passed one by one or contained in an array. Let's slightly modify our example query:

$sql = "SELECT v.vid FROM {vocabulary} v WHERE v.name = '%s' AND v.vid = '%d'";

Now we're being more specific, looking for this vocabulary using v.name and v.vid, pay attention to the placeholders, and we could get our result passing each argument to db_query like this:

$vid = db_result(db_query($sql, $vocabulary_name, $vid));

or we could build an array with both arguments like this:

$args = array($vocabulary_name, $vid);
$vid = db_result(db_query($sql, $args));

I prefer the array approach for queries where I have to pass more than a few arguments and that's often the case when we use the SQL IN operator.

The SQL IN operator and Drupal

Every good Drupal developer has to be highly skilled in writing SQL and that means there will be times when you need the SQL IN operator, which compares a field to a list of values. Let's say you want to get all nodes of types page and blog, you're looking for a query like this:

$sql = "SELECT n.nid, n.title FROM {node} n WHERE n.type IN ('page', 'blog') AND n.status = 1";

If you've tried this before you may have experienced escape quotes hell and opted for the non Drupal way of concatenating arguments in the query, at least I did it until I read about <a href="http://api.drupal.org/api/function/db_placeholders/6db_placeholders.

This is how I build my Drupal queries with the SQL IN operator now:

$types = array('blog', 'embedded_video', 'list', 'node_gallery_gallery');
$args = array();
$args[] = $tid;
$args = array_merge($args, $types);
$args[] = $status;
$args[] = $limit;
$sql = "SELECT n.nid, n.title, n.type, c.comment_count FROM {node} n INNER JOIN {term_node} tn
ON n.nid = tn.nid INNER JOIN {term_data} td ON tn.tid = td.tid LEFT JOIN
{node_comment_statistics} c ON n.nid = c.nid WHERE td.tid = %d AND
n.type IN (" . db_placeholders($types, 'varchar') . ")
AND n.status = %d ORDER BY n.created DESC LIMIT %d";
$result = db_query($sql, $args);

This is a bigger query with more arguments and I'm not only looking for nodes of certain types (listed in the $types array) but also a specific term in the taxonomy ($tid) and a published status ($status). I'm also adding a LIMIT clause at the end ($limit).

db_placeholders takes care of adding the correct number and type of placeholders based on the contents of the array $types. In this case it will add four %s because I passed varchar as the second argument and there are four elements in the array.

The $args array is built based on the order in which the arguments appear in the query, notice how I add $tid first and then use array_merge to add $types, then I add $status and $limit at the end.

So Drupalish, isn't it?

Keep your database in sync with MySQL and PostgreSQL

alexis — Mon, 11 May 2009 03:28:13 +0000

If you write web applications you may already be using a version control system to keep code in sync between your servers, but what about your MySQL or PostgreSQL database?

Sure, there is replication, one master database server with one or more slaves is a possible case, but that may be overkill for most simple projects. You just want to make sure the database is the same in your development, staging and production servers at a certain time, like after some major changes in code or before a new release.

Worry not my friend as you just need a few quick commands to keep your valuable data in sync. Yes, it's command line time again.

The dump, drop and create cycle

As most developers I create my web applications on a development server and enter content and settings to be stored in the database. Then, when I commit my code to a version control system on a staging server, Bazaar is my tool of choice these days, I have three options for moving my data:

Enter all the data again, which is too much work and prone to error.
Duplicate the entire database, the option we'll use, and
Duplicate just the tables you know have changed, a slight variation of the second approach.

I'll focus on duplication of the entire database for now as adapting for specific tables is just as easy.

For this method to work you need to use the same database name and user credentials in all your servers. Let's imagine we want to update the database on the staging server with the latest version from the development server. Once you have all your data ready in the development server these are the steps to follow:

Create a dump file of the database in the development server.
Move your dump file from the development to the staging server, I usually use scp but version control should be fine too.
Drop the database in the staging server, optionally creating a backup dump.
Recreate the database in the staging server and feed it with the dump from the development server.

You can write a script to automate the process if you want. Now, I'll take a look at what we need to run on the command line, let's start with PostgreSQL.

Sync your database with PostgreSQL

The administrative user for PostgreSQL is usually named postgres. If you haven't already done so let's use this user via sudo to create the owner of your database in all your servers, in my case the owner will be alexis:

sudo -u postgres createuser -P alexis

And now create a database owned by that user, I'll call mine cataybea:

sudo -u postgres createdb cataybea -O alexis

Once you have some data in the database of the development server get a dump file by running:

pg_dump -f cataybea.sql cataybea

The cataybea.sql file contains all the SQL needed to recreate your database. Now transfer it to the staging server, backup the database you already have there and drop it:

dropdb cataybea

Now recreate the database with the same name, the permissions are already set, and feed it with the dump you got from the development server:

sudo -u postgres createdb cataybea -O alexis
psql -U alexis -d cataybea -f cataybea.sql

Finally, confirm you've got the data correctly by running a few queries:

psql -d cataybea

Sync your database with MySQL

The administrative user for MySQL is usually named root. If you haven't already done so let's create the database in all your servers, in my case its named cataybea and the owner will be alexis:

mysqladmin -u root -p create cataybea
mysql -u root -p
mysql> GRANT ALL PRIVILEGES ON cataybea.* TO 'alexis'@'%' IDENTIFIED BY 'secret';

Once you have some data in the database of the development server can get a dump file by running:

mysqldump -u alexis -p cataybea > cataybea.sql

The cataybea.sql file contains all the SQL needed to recreate your database. Now transfer it to the staging server, backup the database you already have there and drop it:

mysqladmin -u root -p drop cataybea

Now recreate the database with the same name, the permissions are already set, and feed it with the dump you got the development server:

mysqladmin -u root -p create cataybea
mysql -u alexis -p cataybea < cataybea.sql

Finally, confirm you've got the data correctly by running a few queries:

mysql -u alexis -p cataybea

Help on its way

You can sync all your databases in PostgreSQL and MySQL just by following the steps above and replacing your database name and user credentials; however, if you need some context, you can get all the details for any of the commands mentioned using the --help option, for example:

psql --help
mysql --help
createdb --help
mysqladmin --help

And that's it. Happy database syncing!

Markdown and url filter enabled

alexis — Fri, 08 May 2009 16:57:49 +0000

I had noticed that many of your comments included urls but no link and I understand that not everybody is used to thinking and writing on HTML all the time, that's why I just enabled Markdown and a url filter to the comments textarea.

Enjoy.

Is social media taking the comments away?

alexis — Tue, 05 May 2009 15:48:58 +0000

It seems to me that everyday there are more discussions going on in Twitter and FriendFeed than in regular websites or blogs. I don't have numbers at hand or any other way to support this idea, and maybe I'm wrong, but I'd like to know what you think.

Have you noticed less comments on your sites lately? Are Twitter and FriendFeed taking all the comments away?

Video editing with free software: Kdenlive 0.7.3

alexis — Sun, 03 May 2009 20:17:07 +0000

Long time GNU/Linux users may know there are two types of applications where we are not there yet compared to other operating systems: games and video editing. Well, the video editing part has started to change with the recent release of Kdenlive 0.7.3.

Kdenlive is a free software video editor that works with both GNU/Linux and FreeBSD, running either KDE or Gnome. It can edit footage from DV, HDV and AVCHD sources and uses other projects, like FFmpeg and the MLT video framework, to get its job done.

Kdenlive has a very simple and easy to understand user interface that will take you from capturing footage to editing and rendering in just a few steps. I've used Kdenlive to produce some short and silly clips and a few one hour long episodes of my videocast In Silico.

I know my video editing needs are basic, they couldn't be considered professional at all, but I'm sure there are many of you with similar needs out there. I usually capture some video from my Panasonic PV-GS80 MiniDV camcorder, add some music, a few titles and render to a format that can be uploaded to one of the many video services online.

Let's see how to install Kdenlive.

How to install Kdenlive on Ubuntu

I'm using Ubuntu 9.04 with Kdenlive 0.7.3 and my camcorder connects via Firewire so the first thing to do was to make sure I had the needed modules running. Edit /etc/modules and add the following lines:

raw1394
video1394
ohci1394
dv1394

Then load the modules with:

sudo modprobe dv1394
sudo modprobe video1394
sudo modprobe raw1394
sudo modprobe ohci1394

And allow read and write access to your device:

sudo chmod a+rw /dev/*1394

I have to run this command everytime I need to enable my camcorder, for some reasons my permissions are resetting, but once done it works for the whole session. If anybody knows how to fix this problem let me know.

Now let's install Kdenlive:

sudo apt-get install kdenlive

You will find Kdenlive under the Applications > Sound & Video menu.

The first time you run Kdenlive you will see a configuration wizard and a few codecs will be enabled. If you're running Ubuntu 9.04 you may be lacking a few important ones, such as mpeg-2, mpeg-4 and h264. This problem is related to an outdated version of the MLT video framework (the relevant discussion here) and other libraries.

You have two options to install a newer MLT on Ubuntu 9.04, adding repositories to your apt setup or getting a .deb file. Let's see the first one:

Edit your apt /etc/apt/sources.list file and add these two entries:

# for kdenlive
deb http://ppa.launchpad.net/sunab/ppa/ubuntu jaunty main
deb-src http://ppa.launchpad.net/sunab/ppa/ubuntu jaunty main

Then update and install:

sudo apt-get update
sudo apt-get install libmlt++1 libmlt++-dev

The second option is easier, just grab and install the .deb with the updated MLT.

That's all with MLT, now for the extra codecs install libavcodec-unstripped-52 and its dependencies:

sudo apt-get install libavcodec-unstripped-52

Now start Kdenlive and run the configuration again to enable your new codecs: Settings > Run Config Wizard. That's it! You're ready to start editing and having fun, don't forget to consult the documentation for Kdenlive to know what's possible.

Mplayer: a nice video player

I'll talk about publishing to online video services in a future article, for now I suggest you to get a nice video player to watch your creations. I like Mplayer, to install just use:

sudo apt-get install mplayer

Show us your clips

And there you are. Go get your camera rolling and start playing with Kdenlive. Oh, and don't forget to share the links to the clips you've created with it in the comments.

System76 Starling: a new Ubuntu friendly netbook

alexis — Tue, 28 Apr 2009 03:02:18 +0000

System76, the company that focuses on Ubuntu based computers, has just launched its first netbook model, the Starling.

At first glance this netbook offers the default features:

10.1" widescreen glossy screen with a 1024 x 576 resolution.
Intel Atom N270 CPU.
1 Gb DDR2 RAM.
160 Gb SATA II hard disk.
802.11 b/g WiFi.
VGA and 3 USB 2.0 ports, headphone and microphone jack, SD cards reader.
6 cell lithium ion battery.
Intel audio and 0.3 mp webcam.

Priced at US$ 359 the little gizmo comes with Ubuntu Netbook Remix 9.04 pre-installed and System76 has written drivers to make sure all the hardware works correctly out of the box. I have no doubt the Starling seems like a good deal. I'm the happy owner of one of Systems76's laptop models, the Pangolin, and have had a great experience with the hardware and the company; hence, as soon as I knew about the Starling I wanted to know more and asked Carl Richell, System76's President, a few questions:

Alexis Bellido: How does the Starling compare to other netbooks in the market? Not including Windows is a great point. Many GNU/Linux users, like me, don't want to waste time removing it. What else does the Starling bring to the netbook arena?

Carl Richell: Above all, the Starling features Ubuntu Netbook Remix whereas many netbooks are pre-loaded with the 7 year old Windows XP operating system. Ubuntu Netbook Remix represents the latest technologies, is optimized for Intel's Atom platform, and features a user interface geared toward netbook devices.

System76 also supports its customers through email, forums, and over the phone. Buying an Acer or Asus product usually means the end of your interaction with those companies. Buying a System76 computer is the start. Quality support is fundamental to the successful introduction and the lasting usage of Ubuntu in the broader market.

AB: We, netbook users, are constantly on the move and battery life is a great issue. Asus promises 9 hours of baterry life for its Eee PC 1000HE. What does the Starling offer? Do you sell extra batteries and if so how much do they cost?

CR: The Starling runs about 6 hours on its 6 cell battery. This may be longer. Our 3 cell test models run about 4 hours. We will offer extra batteries once they're in stock. We don't have pricing available yet.

AB: How does you keyboard compare to those of other netbooks in the market? 92% of the standard size seems to be the norm these days.

CR: The keyboard is nearly full-size and very easy to type on. You can easily switch between your laptop and Starling netbook keyboards.

AB: Is the Starling upgrade friendly? How easy is to change the hard disk or upgrade to 2 Gb of RAM? I didn't notice an option for extra RAM in the order page.

CR: Not particularly. The netbook isn't designed to be opened up and memory is integrated onto the motherboard.

Thanks Carl, congratulations for the launch, I'm pretty sure the Starling will find its way to many Ubuntu users.

Why you should follow me on Twitter?

alexis — Mon, 27 Apr 2009 18:47:25 +0000

Your time is valuable and you need valuable content in your Twitter timeline. Here some reasons why you should follow me on Twitter:

I just share knowledge and ideas that I think may be helpful for my followers.
I spend a lot of time teleworking, programming, designing and consulting on Internet projects. Many useful thoughts from all that turn into tweets.
I like to watch good movies, listen to great music and enjoy the best books I can find. Even if your tastes are different than mine you may find something worthy.
I follow a very small group of very interesting people. You may find some very smart guys and gals thru me.
Once in a while I may have a few questions that you could answer. I may retweet your answer and we both can help a lot of people.
If I find your tweets valuable I may follow you. Many of my followers could do so as well.
We can become good friends and possibly collaborate on a project later.
I keep personal messages in private and won't clutter your timeline.
I tweet in both English and Spanish. If you're learning either of those languages this is a good way of doing it.
I try to keep updated on the latest news all over the world. You may hear it from me before you watch it on TV or read it on news sites.
Finally, I have a good sense of humor so you can mock me and I won't send my goons to your house, or nearest ISP.

Wanna try? Follow me on Twitter

Versión en español: ¿Porqué debes seguirme en Twitter?

Best settings to export video from Adobe Premiere to Dailymotion and Vimeo?

alexis — Thu, 23 Apr 2009 07:51:16 +0000

I'm looking for the best settings to export my videos to services like Dailymotion and Vimeo.

I'm using Adobe Premiere CS3 and a basic miniDV Panasonic PV-GS80 camcorder but I think I'm not exporting correctly as my videos don't look as good as others. Compare for example to this one, the SD version of course, not the HD version.

Am I at the limit of SD and should think about buying an HD camcorder? Don't think so as the source video looks pretty good in Premiere and I know I'm losing quality in the encoding process.

I think there are some settings that work best for certain video sites, correct?

I'm not a video expert, not yet, but would like to get the best video I can to continue producing new episodes of In Silico.

Thanks for your help.

Social networking no longer a phenomena

alexis — Tue, 21 Apr 2009 20:11:14 +0000

True, we can no longer consider social networking, or should I say social media?, as something restricted to a group of geeks without a life. Social networking has reached Joe and Jane; mom and pop and your kids. It's already changing lives and helping causes all over the world.

In this first episode of In Silico in English I offer a quick look at the evolution of social networks.