Sofeng's Old Blog

Transition: Bye Blogger, Bo*ing

2008-07-23T21:13:00.001-07:00

I've come to a transition point in my blogging and professional life. As for blogging, I am saying goodbye to Blogger and moving on to my new Django blog located at http://www.saltycrane.com/blog/. If everything goes according to plan, this will be my last post here on Blogger. My new blog is pretty much finished, so I might as well start using it for real. Undoubtedly, I will have to make fixes and tweaks, but that's the best way to learn, I suppose. Hopefully, it will serve me as well as Blogger has. To be honest, I hope it will serve me better than Blogger (that's why I'm moving, right?).

Which brings me to my second transition. I am also transitioning out of embedded software and my almost ten year career in the Aerospace/Defense industry to a small web 2.0 startup company. My current company has been great, but kind of like my new blog, I am hoping for something even better. Instead of using C as my primary language, I will be using Python, Django, Ruby, and Rails. It's true I don't have much Ruby or Rails experience, but I plan to learn fast. I am currently in Chapter 5 of Why's (poignant) (and free) (and very funny) guide to Ruby and am starting on my newly purchased Agile Web Development with Rails, 3rd edition Beta. You can expect I will start posting a lot more on Ruby and Rails on my new blog. Hopefully I can add some insightful impressions as a devoted Pythonista. (Though, I don't too much care for that term.)

Well, I'm off! Please come along!

UPDATE 2008/8/6: It looks like I get to mostly stick with Python and Django instead of Ruby and Rails. Cool.

Django Blog Project #9: Migrating Blogger posts with Beautiful Soup

2008-07-14T22:47:00.000-07:00

Last post, I talked about adding comments to my new sample blog application. This was about the last basic feature I needed to add before I started actually using it for real. Of course there are still a number of features I'd like to add, such as automatic syntax highlighting with Pygments, and incorporating django-tagging and some more intersting views, not to mention comment moderation. But I think those will have to wait-- I want to start using my new blog for real sometime.

So for the past few days, I've been working on my Beautiful Soup screen scraper script to copy all my Blogger posts over to my new Django blog. Initial results came quickly (it's pretty cool to see such a huge data dump after only a few lines of Beautiful Soup'ing) but the details (especially with the comments) kind of slowed me down. I've finally got everything copied over to my satisfaction. Below is the script I used to do it. Note, I realize it's not pretty-- just a one time use hack. But hopefully someone else doing the same thing might find it useful.

#!/usr/bin/env python

import datetime
import os
import re
import urllib2
from BeautifulSoup import BeautifulSoup
from myblogapp.models import Post, LegacyComment
from django.contrib.comments.models import FreeComment

URL = ''.join([
        'http://iwiwdsmi.blogspot.com/search?',
        'updated-min=2006-01-01T00%3A00%3A00-08%3A00&'
        'updated-max=2009-01-01T00%3A00%3A00-08%3A00&',
        'max-results=1000'
        ])
html = urllib2.urlopen(URL).read()
soup = BeautifulSoup(html)

for post in soup.html.body.findAll('div', {'class': 'post'}):
    print
    print '--------------------------------------------------------------'

    # save the post title and permalink
    h3 = post.find('h3', {'class': 'post-title'})
    post_href = h3.find('a')['href']
    post_title = h3.find('a').string
    post_slug = os.path.basename(post_href).rstrip('.html')
    print post_slug
    print post_href
    print post_title

    # save the post body
    div = post.find('div', {'class': 'post-body'})
    [toremove.extract() for toremove in div.findAll('script')]
    [toremove.extract() for toremove in div.findAll('span', {'id': 'showlink'})]
    [toremove.extract() for toremove in div.findAll('div', {'style': 'clear: both;'})]
    [toremove.parent.extract() for toremove in div.findAll(text='#fullpost{display:none;}')]
    post_body = ''.join([str(item)
                         for item in div.contents
                         ]).rstrip()
    post_body = re.sub(r"iwiwdsmi\.blogspot\.com/(\d{4}/\d{2}/[\w\-]+)\.html", 
                       r"www.saltycrane.com/blog/\1/", 
                       post_body)

    # count number of highlighted code sections 
    highlight = div.findAll('div', {'class': 'highlight'})
    if highlight:
        hl_count += len(highlight)
        hl_list.append(post_title)

    # save the timestamp
    a = post.find('a', {'class': 'timestamp-link'})
    try:
        post_timestamp = a.string
    except:
        match = re.search(r"\.com/(\d{4})/(\d{2})/", post_href)
        if match:
            year = match.group(1)
            month = match.group(2)
        post_timestamp = "%s/01/%s 11:11:11 AM" % (month, year)
    print post_timestamp

    # save the tags (this is ugly, i know)
    if 'error' in post_title.lower():
        post_tags = ['error']
    else:
        post_tags = []
    span = post.find('span', {'class': 'post-labels'})
    if span:
        a = span.findAll('a', {'rel': 'tag'})
    else:
        a = post.findAll('a', {'rel': 'tag'})
    post_tags = ' '.join([tag.string for tag in a] + post_tags)
    if not post_tags:
        post_tags = 'untagged'
    print post_tags

    # add Post object to new blog
    if True:
        p = Post()
        p.title = post_title
        p.body = post_body
        p.date_created = datetime.datetime.strptime(post_timestamp, "%m/%d/%Y %I:%M:%S %p")
        p.date_modified = p.date_created
        p.tags = post_tags
        p.slug = post_slug
        p.save()

    # check if there are comments
    a = post.find('a', {'class': 'comment-link'})
    if a:
        comm_string = a.string.strip()
    else:
        comm_string = "0"
    if comm_string[0] != "0":
        print
        print "COMMENTS:"

        # get the page with comments
        html_single = urllib2.urlopen(post_href).read()
        soup_single = BeautifulSoup(html_single)

        # get comments
        comments = soup_single.html.body.find('div', {'class': 'comments'})
        cauth_list = comments.findAll('dt')
        cbody_list = comments.findAll('dd', {'class': 'comment-body'})
        cdate_list = comments.findAll('span', {'class': 'comment-timestamp'})

        if not len(cauth_list)==len(cbody_list)==len(cdate_list):
            raise "didn't get all comment data"

        for auth, body, date in zip(cauth_list, cbody_list, cdate_list):
            
            # create comment in database
            lc = LegacyComment()
            lc.body = str(body.p)

            # find author
            lc.author = "Anonymous"
            auth_a = auth.findAll('a')[-1]
            auth_no_a = auth.contents[2]
            if auth_a.string:
                lc.author = auth_a.string
            elif auth_no_a:
                match = re.search(r"\s*([\w\s]*\w)\s+said", str(auth_no_a))
                if match:
                    lc.author = match.group(1)
            print lc.author

            # find website
            try:
                lc.website = auth_a['href']
            except KeyError:
                lc.website = ''
            print lc.website

            # other info
            lc.date_created = datetime.datetime.strptime(
                date.a.string.strip(), "%m/%d/%Y %I:%M %p")
            print lc.date_created
            lc.date_modified = lc.date_created
            lc.post_id = p.id
            lc.save()

I also made some changes to my Django blog code as I migrated my Blogger posts. The main addition was a LegacyComment model along with the associated views and templates. My Blogger comments consisted of HTML markup, but I didn't want to allow arbitrary HTML in my new comments for fear of cross site scripting. So I separated my legacy Blogger comments from my new Django site comments.

models.py

Here are my model changes. I added a LegacyComment class which contains pertinent comment attributes and a ForeignKey to the post that it belongs to. I also added a lc_count (for legacy comment count) field to the Post class which stores the number of comments for the post. It is updated by the save() method in the LegacyComment class every time a comment is saved. Hmmm, I just realized the count will be wrong if I ever edit these comments. Well, since these are legacy comments, hopefully I won't have to edit them.

~/src/django/myblogsite/myblogapp/models.py:

import re
from django.db import models

class Post(models.Model):
    title = models.CharField(maxlength=200)
    slug = models.SlugField(maxlength=100)
    date_created = models.DateTimeField() #auto_now_add=True)
    date_modified = models.DateTimeField()
    tags = models.CharField(maxlength=200)
    body = models.TextField()
    body_html = models.TextField(editable=False, blank=True)
    lc_count = models.IntegerField(default=0, editable=False)

    def get_tag_list(self):
        return re.split(" ", self.tags)

    def get_absolute_url(self):
        return "/blog/%d/%02d/%s/" % (self.date_created.year,
                                      self.date_created.month,
                                      self.slug)

    def __str__(self):
        return self.title

    class Meta:
        ordering = ["-date_created"]

    class Admin:
        pass

class LegacyComment(models.Model):
    author = models.CharField(maxlength=60)
    website = models.URLField(core=False)
    date_created = models.DateTimeField()
    date_modified = models.DateTimeField()
    body = models.TextField()
    post = models.ForeignKey(Post)

    def save(self):
        p = Post.objects.get(id=self.post.id)
        p.lc_count += 1
        p.save()
        super(LegacyComment, self).save()

    class Meta:
        ordering = ["date_created"]

    class Admin:
        pass

views.py

Here is an excerpt from my views.py file showing the changes:

~/src/django/myblogsite/myblogapp/views.py:

import re
from datetime import datetime
from django.shortcuts import render_to_response
from myblogsite.myblogapp.models import Post, LegacyComment

MONTH_NAMES = ('', 'January', 'Feburary', 'March', 'April', 'May', 'June', 'July',
               'August', 'September', 'October', 'November', 'December')
MAIN_TITLE = "Sofeng's Blog 0.0.7"

def frontpage(request):
    posts, pagedata = init()
    posts = posts[:5]
    pagedata.update({'post_list': posts,
                     'subtitle': '',})
    return render_to_response('listpage.html', pagedata)

def singlepost(request, year, month, slug2):
    posts, pagedata = init()
    post = posts.get(date_created__year=year,
                            date_created__month=int(month),
                            slug=slug2,)
    legacy_comments = LegacyComment.objects.filter(post=post.id)
    pagedata.update({'post': post,
                     'lc_list': legacy_comments,})
    return render_to_response('singlepost.html', pagedata)

Templates

In the list page template I used the truncatewords_html template filter to show a 50 word post summary on the list pages instead of the full post. I also added the legacy comment count with the Django free comment count to display the total number of comments.

Excerpt from ~/src/django/myblogsite/templates/listpage.html:

{% block main %}
  <br>
  {% for post in post_list %}
    <h4><a href="/blog/{{ post.date_created|date:"Y/m" }}/{{ post.slug }}/">
        {{ post.title }}</a>
    </h4>
    {{ post.body|truncatewords_html:"50" }}
    <a href="{{ post.get_absolute_url }}">Read more...</a><br>
    <br>
    <hr>
    <div class="post_footer">
      {% ifnotequal post.date_modified.date post.date_created.date %}
        Last modified: {{ post.date_modified.date }}<br>
      {% endifnotequal %}
      Date created: {{ post.date_created.date }}<br>
      Tags: 
      {% for tag in post.get_tag_list %}
        <a href="/blog/tag/{{ tag }}/">{{ tag }}</a>{% if not forloop.last %}, {% endif %}
      {% endfor %}
      <br>

      {% get_free_comment_count for myblogapp.post post.id as comment_count %}
      <a href="{{ post.get_absolute_url }}#comments">
        {{ comment_count|add:post.lc_count }} 
        Comment{{ comment_count|add:post.lc_count|pluralize}}</a>

    </div>
    <br>
  {% endfor %}
{% endblock %}

In the single post template, I added the display of the Legacy comments in addition to the Django free comments.

Excerpt from ~/src/django/myblogsite/templates/singlepost.html:

  <a name="comments"></a>
  {% if lc_list %}
    <h4>{{ lc_list|length }} Legacy Comment{{lc_list|length|pluralize}}</h4>
  {% endif %}
  {% for legacy_comment in lc_list %}
    <br>
    <a name="lc{{ legacy_comment.id }}" href="#lc{{ legacy_comment.id }}">
      #{{ forloop.counter }}</a>
    {% if legacy_comment.website %}
      <a href="{{ legacy_comment.website }}">
        <b>{{ legacy_comment.author|escape }}</b></a> 
    {% else %}
      <b>{{ legacy_comment.author|escape }}</b>
    {% endif %}
    commented,
    on {{ legacy_comment.date_created|date:"F j, Y" }} 
    at {{ legacy_comment.date_created|date:"P" }}:
    {{ legacy_comment.body }}
  {% endfor %}
  <br>

That's it. Hopefully, I can start using my new blog soon. Please browse around on the new Django site and let me know if you run across any problems. When everything looks to be OK, I'll start posting only on my new Django site.

Here is a snapshot screenshot of version 0.0.8:

The live site can be viewed at: http://saltycrane.com/blog

On the quest for consistent keybindings

2008-07-09T17:53:00.000-07:00

I'm trying to get consistent keybindings in the applications which I use the most starting with Emacs, then bash/screen, conkeror (my web browser), and others like KPDF and OpenOffice. I haven't got complete consistency, but I'm making progress. Here are my notes on setting the keybindings for a few of the applications. Maybe when I find the perfect setup, I will write about my complete configuration. Until then, here are some sparse notes. Note, for some reason, I've mixed in vi-style, h-j-k-l, movement keybindings with the standard emacs keybindings. I use control+h,j,k,l to move around by character or line and meta+h,j,k,l to move by word or page. I've been using this for several months and it works pretty well-- much better than using pgup/pgdown and the arrow keys.

Emacs

Here is part of my ~/.emacs:

;; buffer switching
(iswitchb-mode t)
(global-set-key "\C-b" 'switch-to-buffer)

;; movement key bindings (use h-j-k-l for movement like vi)
(global-set-key "\C-l" 'forward-char)
(global-set-key "\C-h" 'backward-char)
(global-set-key "\C-j" 'next-line)
(global-set-key "\C-k" 'previous-line)
(global-set-key "\M-l" 'forward-word)
(global-set-key "\M-h" 'backward-word)
(global-set-key "\M-j" 'scroll-up)
(global-set-key "\M-k" 'scroll-down)

;; rebind displaced movement key bindings
(global-set-key "\C-p" 'kill-line)
(global-set-key "\M-v" 'downcase-word)
(global-set-key "\M-b" 'recenter)

Readline (bash)

To change the keybindings for readline (bash), I edited my ~/.inputrc. Luckily, readline is very similar to Emacs and there are a lot of commands that I didn't know about. Here are a few examples. For a full list of commands see the readline manpage.

# ~/.inputrc
"\C-h": backward-char
"\C-l": forward-char
"\M-h": backward-word
"\M-l": forward-word

GNU Screen

Here are some keybindings I use in GNU Screen. I use C-b to switch windows like Emacs buffers. I enter copy mode by pressing M-k and exit by pressing C-g. The "-m" means the keybinding is for copy mode only. "stuff" is used to stuff some stuff into the input buffer. "^" is used for Ctrl and "^[" is used for Meta (Alt).

# ~/.screenrc
bindkey "^B" select         # like emacs switch buffers
bindkey "^[k" eval "copy" "stuff ^b"
bindkey "^k" eval "copy" "stuff k"
bindkey -m "^[j" stuff ^f   # move down one page
bindkey -m "^[k" stuff ^b   # move up one page
bindkey -m "^[h" stuff b    # move backward word
bindkey -m "^[l" stuff w    # move forward by word
bindkey -m "^j" stuff j     # move down one line
bindkey -m "^k" stuff k     # move up one line
bindkey -m "^h" stuff h     # move back one character
bindkey -m "^l" stuff l     # move forward one character
bindkey -m "^e" stuff $     # move to end of line
bindkey -m "^a" stuff 0     # move to beginning of line
bindkey -m "^f" stuff ^s    # incremental search forward
bindkey -m "^ " stuff " "   # set mark
bindkey -m "^g" stuff q     # exit copy mode

Conkeror

My ~/.conkerorrc keybindings section is pretty long, so I won't include it. But again, I use C-b to switch buffers, and Ctrl and Meta plus h-j-k-l to navigate.

OpenOffice.org

This is directly from Appendix A Keyboard Shortcuts in the Getting Started Guide. Unfortunately, OpenOffice doesn't allow me to use the Meta (Alt) key for custom keybindings so I can't use my usual Emacs keybindings. Apparently, Peter in this forum thread had the same lament. If anyone knows how to fix this, please let me know.

To adapt shortcut keys to your needs, use the Customize dialog, as described below.
    1) Select Tools > Customize > Keyboard. The Customize dialog (Figure 1) opens.
    2) To have the shortcut key assignment available in all components of OpenOffice.org select the
       OpenOffice.org button.
    3) Next select the required function from the Category and Function lists.
    4) Now select the desired shortcut keys in the Shortcut keys list and click the Modify button at
       the upper right.
    5) Click OK to accept the change. Now the chosen shortcut keys will execute the function
       chosen in step 3 above whenever they are pressed.

KPDF

I like KPDF better than Evince. I haven't tried anything else. To change some keyboard shortcuts in KPDF go to "Settings", "Configure Shortcuts..."

Sofeng's *Old* Blog

Transition: Bye Blogger, Bo*ing

Django Blog Project #9: Migrating Blogger posts with Beautiful Soup

On the quest for consistent keybindings

Sofeng's Old Blog