Weekend Codes

Django Admin on Google App Engine

2011-03-14T00:00:00-03:00

Django Admin on GAE

Did you ever wanted to run Django’s Admin interface on Google’s App Engine?
Now you can and we have made a live sample so you can clone, try and use it.

Here is what you want to know

This is running thanks to the projects listed below:
djangoappengine
django-nonrel

The real thing, a live sample

https://github.com/stockrt/djangoadminongae
http://djangoadminongae.appspot.com/admin

Starting your fresh project

Here you can find a djangoappengine bootstrap script to start a fresh
project like this one but with the newest possible versions of all deps. The
same script was used to create this sample project.

Traduzindo com pytranslate diretamente do Google

2010-01-06T00:00:00-02:00

Resolvi traduzir alguns textos em lote, e para isso utilizei esse módulo chamado pytranslate.

Ele é um wrapper para o Google Translate e permite que você traduza textos com facilidade. O retorno dele sempre será a sentença completa (frase) traduzida.

Utilização:

import pytranslate

print pytranslate.translate('hello', sl='english', tl='portuguese')
print pytranslate.translate('hello', sl='auto', tl='portuguese')
print pytranslate.translate('hallo', sl='auto', tl='portuguese')
print pytranslate.translate('hallo', sl='auto', tl='french')
print pytranslate.translate('Bonjour', sl='auto', tl='dutch')

A instalação é simples:

wget --no-check-certificate https://github.com/stockrt/pytranslate/tarball/master -O pytranslate.tar.gz
tar xzvf pytranslate.tar.gz
cd stockrt-pytranslate*
./setup.py install

Você pode obter o código do pytranslate aqui

Navegando com segurança e sem tarifação em cafés e hotéis

2009-09-20T00:00:00-03:00

Quando estiver pensando em se hospedar em um hotel, ou utilizar redes de cafés, pergunte primeiro o quanto custa o minuto ou diária de acesso à internet.

Normalmente o custo é alto. Se este for o caso, opte pela conexão tarifada por minuto, e use o seguinte truque para economizar alguns trocados:

Logue no sistema do hotel normalmente, optanto pela menor tarifa para pouco tempo de utilização;
Execute um dos comandos abaixo, ssh ou putty;
Faça logoff do sistema de tarifação;
Configure seu browser e demais aplicativos para usar proxy SOCKS na porta local escolhida.

Usando linux

Se estiver usando linux no seu notebook, você poderá fazer o seguinte:

ssh -D 9999 user@servidor

Usando windows

Se estiver no windows, utilize o putty:

putty.exe -D 9999 user@servidor

Nota: Você precisa, claro, ter um “servidor” rodando um sshd e uma conta nele :) Talvez o seu linux que fica rodando no desktop, em casa? Ou aquele servidor da faculdade/trabalho?

Outra dica é configurar o putty para manter a sessão sempre aberta, com keepalive (Category / Connection / Seconds between keepalives / 5) ou manter a sessão ativa deixando um comando qualquer rodando no shell, que gere tráfego no terminal:

while true; do date; sleep 5; done

Depois de criado o proxy dinâmico

Agora você pode fazer logoff do sistema de internet tarifado e então apontar o firefox, ou qualquer outra aplicação (uma nova sessão de ssh, por exemplo), para o proxy dinâmico local, que agora ouve na porta local 9999.

Outros posts também descrevem essa técnica.

Com essas medidas você pode economizar no acesso, além de garantir uma navegação muito mais segura na internet, pois tudo será tunelado e codificado pelo ssh.

Enjoy the hacking.

handling html forms with python mechanize and BeautifulSoup

2009-09-14T00:00:00-03:00

In the post about emulating a browser in python with mechanize I have showed you how to make some basic tricks in the web with python, but I have not showed how to login a site and how to handle a session, with html forms, links and cookies.

Here I will show it all for you, let’s see it.

First, you must install some dependecies:

easy_install BeautifulSoup
easy_install html2text

Then, let the code speak:

import mechanize
import cookielib
from BeautifulSoup import BeautifulSoup
import html2text

# Browser
br = mechanize.Browser()

# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

# Follows refresh 0 but not hangs on refresh > 0
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

# User-Agent (this is cheating, ok?)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

# The site we will navigate into, handling it's session
br.open('http://gmail.com')

# Select the first (index zero) form
br.select_form(nr=0)

# User credentials
br.form['Email'] = 'user'
br.form['Passwd'] = 'password'

# Login
br.submit()

# Filter all links to mail messages in the inbox
all_msg_links = [l for l in br.links(url_regex='\?v=c&th=')]
# Select the first 3 messages
for msg_link in all_msg_links[0:3]:
    print msg_link
    # Open each message
    br.follow_link(msg_link)
    html = br.response().read()
    soup = BeautifulSoup(html)
    # Filter html to only show the message content
    msg = str(soup.findAll('div', attrs={'class': 'msg'})[0])
    # Show raw message content
    print msg
    # Convert html to text, easier to read but can fail if you have intl
    # chars
#   print html2text.html2text(msg)
    print
    # Go back to the Inbox
    br.follow_link(text='Inbox')

# Logout
br.follow_link(text='Sign out')

The basic flow is:

Open the site and login;
Session is handled by cookiejar, automatically;
We list the first 3 mail messages;
For each mail message, we open it and read it’s contents;
We go back to the Inbox and to the next mail message;
All done, we can logoff;
The first 3 mail messages will have it’s status changed to “read” if you look at it in the gmail web interface.

You may ask, how do I know the name of the form fields to fill?

We can inspect it, before filling:

# Open the site
br.open('http://gmail.com')

# Forms
for f in br.forms():
    print f

The output contains the fields to fill in the form:

...
<TextControl(Email=)>
<PasswordControl(Passwd=)>
...

And the rest you already know.

Why gmail? It was only an example. I know we have libgmail , but again, it was only an example, with login forms and a session to handle :)

git branch in your shell prompt

2009-08-31T00:00:00-03:00

Every time (every few minutes?) I was looking at my git versioned projects (all my projects?) I encounter myself in doubt:

Am I at the branch I think I am?

And there was I, issuing a “git branch” command to check it…

That is enough! I said.

Looking at the github guides I found this tip very interesting, so I decided to bettered it, and to publish as a tip here.

It will show in your prompt which is your current branch, when your current work directory is a git initialized one.

To use it, just place this line inside your ~/.bashrc or into /etc/profile.d/git-branch.sh or /etc/bashrc or even /etc/profile, the choice is yours:

# git branch
parse_git_branch() {
    git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/(\1) /'
}
PS1="\$(parse_git_branch)$PS1"

This will give you a prompt like this one:

stockrt@jackbauer ~ $ cd Dropbox/stockrt/git/stockrt.github.com
(master) stockrt@jackbauer ~/Dropbox/stockrt/git/stockrt.github.com $

Here you can see a “normal” prompt and then, when I enter one of my git versioned directories, a “git branchned” prompt.

Way cool.

Browser cache invalidation with Javascript and querystring

2009-08-30T00:00:00-03:00

Some time ago I started my blog here at github and noticed that new posts didn’t come live right way I published them.

Quickly I spot the problem: They are sending HTTP Cache Headers for the index.html and all pages served by github, a 24 hour cache.

The problem

$ curl -I http://stockrt.github.com

HTTP/1.1 200 OK
Server: nginx/0.6.31
Date: Sat, 22 Aug 2009 01:36:49 GMT
Content-Type: text/html
Content-Length: 66829
Last-Modified: Sat, 22 Aug 2009 01:12:50 GMT
Connection: keep-alive
Expires: Sun, 23 Aug 2009 01:36:49 GMT
Cache-Control: max-age=86400
Accept-Ranges: bytes

So, to overcome this “problem” I made this tiny trick, and published it to others to take advantage of it, in case your are hosting your pages behind an web server with Expires configured.

The trick

Go and clone cache_invalidation and start using the provided javascripts in your site, this way:

<html>

<head>
 <script src="http://your_site/javascripts/querystring.js" type="text/javascript"></script>
 <script src="http://your_site/javascripts/cache_invalidation.js" type="text/javascript"></script>
</head>

<body>
</body>

</html>

Set the desired TTL inside de cache_invalidation.js file:

// TTL: set your cache threshold here
var ttl = 300;  // seconds

And it is all set.

But, why does it happen, and how it works?

It does happen because their web server (the great nginx) is configured with what we used to call mod_expires in Apache. This module activates the Expires HTTP Cache Header.

If you look at the response headers I got before, you would see:

Date: Sat, 22 Aug 2009 01:36:49 GMT
Expires: Sun, 23 Aug 2009 01:36:49 GMT

and:

Cache-Control: max-age=86400

Notice that:

$ bc <<< 86400/3600
24

They are saying to my browser that it should use the local copy, for the next 24 hours, when accessing this site. More precisely, when accessing index.html of this site.

I think that, for a blog, this is a pretty big time to update the user’s cache. This cache header means that if a reader accessed you site just before your posted something, and returned to your site after you posted, he would not see any difference. He would only notice your new post the next day.

But, you can bypass that, just passing any query string within the site’s address to the navigation bar in your browser.

This tricks the browser to go in the source and to fetch the page, instead of using a local copy. It would only use a local copy if you have no query string or if you have already cached that url with that query string (say, in a second time you visit the same query string).

Just because the browser would cache the same query string in a second access, I made the script to vary it on each access, and also it forces a refresh when accessing a querystring that is TTL seconds older than the current time, even if it is already cached from a previous access, say, when clicking a bookmark.

As a front end engineer I am, I only pray to my web developer colleagues don’t find this post, ever :)

Finding the next prime number from a given number

2009-08-29T00:00:00-03:00

Finding the next prime number online is useful if you do not have time to calculate it but need a good seed for your hash.

export start_number=250000
curl -s "http://www.numberempire.com/primenumbers.php?action=next&number=${start_number}" | sed -n -e 's#.*The smallest prime greater than.*<font color=.*>\(.*\)</font></div></td></tr><tr>.*#\1#p'

This can be good the help you finding which number to use when tuning Varnish for better performance with the classic hash algorithm, avoiding bucket collision when having a big number of objects, making hash lookups faster.

sed quick tips

2009-08-28T00:00:00-03:00

A little collection of sed tips.

The basic one, substitution

You already know this one:

sed -i 's/old text/new text/g' file.txt

Deleting all lines containing an specific text

sed -i -e '/this line will disapear/d' file.txt

Deleting blank lines

sed -i -e '/^$/d' file.txt

Filtering text between delimiters

All matching text into the first defined group “()” will be printed:

curl -s -L http://www.terra.com.br | sed -n -e 's#.*\(http://.*\.\(js\|css\)\).*#\1#p'

Printing only from a given line number to another

Prints from line 20 to line 30:

sed -n 20,30p file.txt

More tips can be found here.

Do you know some trick that is worth sharing? Please post it as a comment!

Sunset in Porto Alegre

2009-08-16T00:00:00-03:00

Beautiful sunset in the capital city of Rio Grande do Sul, Brazil.

My wife and I decided to take some of our spare time to appreciate that.

Do you think it was worth?

Emulating a Browser in Python with mechanize

2009-08-16T00:00:00-03:00

It is always useful to know how to quickly instantiate a browser in the command line or inside your python scripts.

Every time I need to automate any task regarding web systems I do use this recipe to emulate a browser in python:

import mechanize
import cookielib

# Browser
br = mechanize.Browser()

# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

# Follows refresh 0 but not hangs on refresh > 0
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

# Want debugging messages?
#br.set_debug_http(True)
#br.set_debug_redirects(True)
#br.set_debug_responses(True)

# User-Agent (this is cheating, ok?)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

Now you have this br object, this is your browser instance. With this its possible to open a page, to inspect or to interact with:

# Open some site, let's pick a random one, the first that pops in mind:
r = br.open('http://google.com')
html = r.read()

# Show the source
print html
# or
print br.response().read()

# Show the html title
print br.title()

# Show the response headers
print r.info()
# or
print br.response().info()

# Show the available forms
for f in br.forms():
    print f

# Select the first (index zero) form
br.select_form(nr=0)

# Let's search
br.form['q']='weekend codes'
br.submit()
print br.response().read()

# Looking at some results in link format
for l in br.links(url_regex='stockrt'):
    print l

If you are about to access a password protected site (http basic auth):

# If the protected site didn't receive the authentication data you would
# end up with a 410 error in your face
br.add_password('http://safe-site.domain', 'username', 'password')
br.open('http://safe-site.domain')

Thanks to the Cookie Jar we’ve added before, you do not have to bother about session handling for authenticated sites, as in when you are accessing a service that requires a POST (form submit) of user and password. Usually they ask your browser to store a session cookie and expects your browser to contain that same cookie when re-accessing the page. All this, storing and re-sending the session cookies, is done by the Cookie Jar, neat!

You can also manage with browsing history:

# Testing presence of link (if the link is not found you would have to
# handle a LinkNotFoundError exception)
br.find_link(text='Weekend codes')

# Actually clicking the link
req = br.click_link(text='Weekend codes')
br.open(req)
print br.response().read()
print br.geturl()

# Back
br.back()
print br.response().read()
print br.geturl()

Downloading a file:

# Download
f = br.retrieve('http://www.google.com.br/intl/pt-BR_br/images/logo.gif')[0]
print f
fh = open(f)

Setting a proxy for your http navigation:

# Proxy and user/password
br.set_proxies({"http": "joe:password@myproxy.example.com:3128"})

# Proxy
br.set_proxies({"http": "myproxy.example.com:3128"})
# Proxy password
br.add_proxy_password("joe", "password")

But, if you just want to quickly open an webpage, without the fancy features above, just issue that:

# Simple open?
import urllib2
print urllib2.urlopen('http://stockrt.github.com').read()

# With password?
import urllib
opener = urllib.FancyURLopener()
print opener.open('http://user:password@stockrt.github.com').read()

See more in Python mechanize site , mechanize docs and ClientForm docs.

Also, I have made this post to elucidate how to handle html forms and sessions with python mechanize and BeautifulSoup

Listing GitHub projects in your blog

2009-08-15T00:00:00-03:00

I needed to list my GitHub projects in a fancy way inside my blog, in the sidebar, all dynamically done. So, I searched a little around and found how.

There are two major badges to do it, here I will describe them:

Dr Nic’s github-badges

This is my choice. drnic’s github-badges is flexible on configuration and also have some code of myself in it.

Here is how to configure it:

<div id="github-badge"></div>
<script type="text/javascript" charset="utf-8">
  GITHUB_USERNAME="stockrt";
</script>
<script src="http://drnic.github.com/github-badges/dist/github-badge-launcher.js" type="text/javascript"></script>

This is the complete list of options available:

GITHUB_USERNAME = "stockrt";
GITHUB_LIST_LENGTH = 10;
GITHUB_HEAD = "div"; // e.g. change to "h2" for wordpress sidebars
GITHUB_THEME = "white"; // try 'black'
GITHUB_TITLE = "My projects";
GITHUB_SHOW_ALL = "Show all";
GITHUB_SHOW_FORK = 1;

subtleGradient’s mootools-github-badge:

This is also pretty neat, subtleGradient’s mootools-github-badge has a great visual effect when it is building the project’s list, using a delayed composal. See this running example.

Here is how to configure it:

<div id="github-badge"></div>
<script src="http://ajax.googleapis.com/ajax/libs/mootools/1.2.1/mootools.js" type="text/javascript"> </script>
<script src="http://projects.subtlegradient.com/mootools-subtle-templates/Source/Plugins/SubtleTemplate.js" type="text/javascript"> </script>
<script src="http://subtlegradient.github.com/mootools-github-badge/mootools-github-badge.js" type="text/javascript"> </script>
<script type="text/javascript" charset="utf-8">
  new SubtleTemplate.GitHubBadge.Basic({ username:'stockrt' });
</script>

Using jekyll and scribbish from stockrt’s site source

If you are using my template to generated a scribbish styled static site with jekyll (pt-BR), all you need to do to activate the use of drnic’s github-badge is to configure this parameters in config.yml:

github_badge:
    enable: true
    username: stockrt
    length: 8
    head: div
    #head: h2
    theme: white
    #theme: black
    title: My Projects
    showall: Show all
    showfork: 0
    #showfork: 1

All said, I must warn you that these two badges do not really behave well on IE 6, but who still uses it today? Better upgrade to Firefox, folks!

Como converter datas e timestamps no linux usando o comando date

2009-08-02T00:00:00-03:00

Se você chegou até aqui, anote esta dica.

Uma hora ou outra a gente sempre chega neste problema: Converter datas com timestamp, seja de timestamp para string ou o contrário.

Pois ai vai a dica para converter datas no Linux:

String da data atual

$ date
Sun Aug  2 03:13:19 2009

Timestamp (seconds from the Epoch) da data atual

$ date +"%s"
1249193599

Timestamp de uma determinada string de data

$ date -d '2009-07-31 12:00:01' +"%s"
1249052401

String de data a partir de um determinado timestamp

$ python -c "import time; print time.ctime(1249052401)"
Fri Jul 31 12:00:01 2009

Iniciando um blog no GitHub

2009-07-19T00:00:00-03:00

Use jekyll, facilita a sua vida.

Estou usando o GitHub para hospedar meu blog/site, mas como ele não oferece nenhuma engine dinâmica de blog, tipo wordpress ou alguma outra feita em php, python, ruby…, estou gerando o site estaticamente e depois faço o upload para o GitHub. Uso jekyll para gerar o site, criar as co-relações entre posts, fazer a formatação, paginação, lista de tags, links e blogroll.

Jekyll é muito legal, vale a pena dar uma olhada. Com esse modelo de geração estática é possível se fazer posts com apenas alguns passos:

cd yournickname.github.com/site
./new_post.sh "Nome do post"
vi _posts/$(date +"%Y-%m-%d")-nome-do-post.markdown
./build.sh; cd ..
git add .
git commit -a -m 'New blog post'
git push origin master

Eu gosto dessa forma de “bloggar” pois assim evito de entrar em editores HTML ou páginas web lentas. Faço tudo daqui do meu shell mesmo, onde me sinto mais à vontade :)

A formatação dos textos é feita usando markdown, bem mais fácil do que HTML para postar da forma como escolhi, pelo shell.

Recomendo ler um pequeno texto que fiz informando sobre como usar o template para jekyll que eu modifiquei:

http://github.com/stockrt/stockrt.github.com/tree/master

Outra dica: No link Projects ao lado você poderá encontrar algo útil. Se sim, me avisa.