Светлячный Dev Лог

Periodical Tasks For Things

2012-04-11T15:19:00Z

Few days ago, I read a funny document "GTD for hackers" and found an interesting ideas there. One of them is to create the checklists for routines.

I already use Things by Cultured Code, to track my todos. However there is no such feature as periodical tasks. They should behave like periodical events in the calendar — you add it once, specifying a time interval, and task should appear in the list automatically. Things is lack this feature.

But we have this wonderful Apple Script, and it allow to do almost anything!

I spent about hour to generate a 10 lines script. It adds a task to the Things right from the command line. The last part was pretty easy — to run a crontab and to add a one line for each periodical task I want to appear in my checklist.

Finally, I've got a very flexible system, which solves my problem and does not require any additional software.

Sources are available at the GitHub. Fork and edit.

GitHub in Numbers

2011-10-15T09:20:00Z

Some times ago, I had got an idea — to create a new service around the GitHub. In the first place it will be useful to those users who have many repositories or following/watching many users and repositories.

My basic GitHub usage pattern is: to follow other coders to see in the News Feed which other coders they are following and which projects they are watching. This way I'm able to discover interesting people and projects on the GitHub. But this approach works only to a paritcular moment, while amount of data is not very big. When I had got a lot of items in my News Feed, I stoped to read it.

The idea is to aggregate items from the news feed and pop out interesting projects. For example, people who you are following instest can be used or people from the 2nd circle.

Moreover, this service should be useful for watching on the changes in your repositories forks. This feature will be useful for those guys who have tens public repositories. I already implemented this functionality for myself as a simple script. This script generates a RSS feed with new commits from the forks.

Also, there will be a funny medals and ratings. Push a dozen commits at the midnight and you'll become a "Midnight code warrior" :)

But before actual implementation of the idea, I decided to calculate some stats about GitHub's usage. I have to estimate how many people will be interested in such project, because I hate to create useless things.

Progress

At first, I wrote a GitHub's profiles and reps fetcher. It takes one login and downloads a his profile and add all users who he follows to the queue. Then it repeats the process for every login in the queue.

Because the rate limit in 5000 reqeusts a hour, my script worked about 2 days.

Totally about 57 thousand profiles were downloaded and 500 thousand repositories were fetched. I expected that there are much more users, but probably these numbers are correct. After all, if my script didn't download somebody then nowbody follows hib and his account probably abandoned.

This graph depicts a fetching progress. At this graph is well seen the moment when queue stopped to grow and started to fall down.

Watch & Follow

One of the most impotent stats for my project is portion of users who are watching a large amount of repositories or following many other coders:

Here we see that about 50% of users are watching more than 10 repositories and 7% (~ 4000) — are watching more than 100. Also I'm interested in those 20% (~11000), who are following more than 10 people, definitely they are unable to read through all their news feeds. And certanly those 410 users who follow more then hungred other coders will fall in love with aggregation feature. Myself is in the latter category as I'm following about 331 users.

A number of repositories

Average amount of public repositories is 8, 3 of them are forks of someone's else repository.

This graph shows that 60% of users have less than 10 public repositories and 15% does not have repositories at all. But about a quarter of users have from 10 to 100 public repositories. They will appreciate a fork watching feature of my project.

Also, I calculated a part of active repositories. Only 10% of repositories had been pushed at last month.

And more…

As I said before, totally about 500 thousand repositories were downloaded and 40% of them are forks. It is amazing! I thought this number should be much much bigger.

In addition, I estimated how much a 2nd cirle is. Average GitHub user follows 9 people and watches at 33 repositories. But his 2nd circle contains 230 people and 800 repositories. This is average number, but for geeks like me they are much bigger. I have in 2nd cirle 11548 people and 42882 repositories. It is about 1/5 of all GitHub!

And here is how many organisations relative to users:

And some tops, I know, you like them!

Top 20 Companies

Company	Users
	37965
ThoughtWorks	75
Google	65
Mozilla	61
Red Hat	58
Freelance	56
Twitter	40
Japan	40
Yandex	39
Freelancer	36
Globo.com	35
Yahoo!	33
Intridea	31
Facebook	30
GitHub	29
Student	26
Emergya	26
Pivotal Labs	24
Microsoft	23
Engine Yard	23

Top 20 Cities

City	Users
	23657
San Francisco	1441
London	962
New York	578
Paris	474
Chicago	458
Seattle	457
Tokyo	430
Berlin	423
Germany	417
Portland	346
Toronto	317
Boston	288
Austin	280
Sydney	272
Stockholm	261
Japan	244
Los Angeles	230
Brooklyn	226
Melbourne	221

Top 20 "followers"

Login	Follows
snytkine	3242
mtsoerin	1983
webiest	1903
superfeedr	1710
charlenopires	1236
stonegao	1205
Marak	1068
speedygonzalez	1059
tyru	1022
esneko	867
josegonzalez	640
c9s	556
kanzure	555
take-cheeze	517
elliottcable	495
Sannis	475
mattn	462
j2labs	453
dpree	446
rkh	444

Top 20 "who followed"

Login	Followers
defunkt	4005
torvalds	3803
jeresig	3466
mojombo	3248
ryanb	2737
schacon	2429
paulirish	2316
dhh	2170
wycats	2044
ry	2032
rails	1946
facebook	1802
jquery	1767
technoweenie	1572
pjhyett	1563
visionmedia	1554
cyanogen	1410
douglascrockford	1380
tpope	1369
android	1317

Top 20 repository owners

Login	Repositories
gitpan	21976
vim-scripts	3735
emacsmirror	3101
Epictetus	911
panega	612
jenkinsci	602
dev2dev	504
wave2future	411
CyanogenMod	342
MechanisM	329
rjbs	325
tokuhirom	297
rwldrn	297
aculich	287
rainly	282
albertobraschi	278
idega	272
rafl	266
apache	258
kristianmandrup	244

Top 20 "watchers"

Login	Watches
gitpan	21976
vim-scripts	3736
emacsmirror	3588
stonegao	2789
abecciu	2474
igrigorik	2415
charlenopires	2339
stan	2318
matagus	2160
smtlaissezfaire	1955
rmetzler	1916
shanlalit	1897
willi	1896
Epictetus	1821
filipeamoreira	1812
arden	1783
andrew	1746
methodmissing	1665
rkh	1571
lgs	1511

The end

Certanly, some other interesting metrics could be calculated, using my database. If you have any ideas feel comment this post or send me an email.

P.S. — I think that my project have a chance to take off and will be useful for few thousand of people around the world. One more thing to think about is how to monetize it to pay rent for servers.

Git helper for GitHub Users

2011-04-14T12:26:00Z

Found a helper, to work with GitHub from command line. All your need is Ruby (it is already included into the Mac OS X). This script should be downloaded into the PATH, and aliased as alias git=hub.

To use all features, you also have to add you username and GitHub's API token into the ~/.gitconfig:

[github]
    user = svetlyak40wt
    token = a-secret-token

Afterwards, you'll be able to make a quick fork of any GitHub project, for example, mine ForkFeed is a good candidate as it is useful for GitHub users too:

git clone svetlyak40wt/forkfeed
cd forkfeed
git fork

Have a fun!

Asynchronous testing with Twisted

2011-03-01T10:53:00Z

Some time ago, I tweeted that I like to write unittests for programs, using a Twisted framework. Tweets are short and there are not enough space for more comprehensive description, that is why I decided to write a separate post on this theme.

I am developing a distribute, fault-tolerant lock service. It is asynchronous and uses Twisted. Nodes talk to each other using TCP and custom text protocol, but provide a REST HTTP API for clients. It is very important to be consistent and durable, that is the reason I wrote tests before any actual lock implementation. They are functional, rather than unit-tests.

Well, my tests check an interaction among different nodes. Nodes exchange messages via network. How to run such tests? Should I start few separate processes? And if I should, then how to debug them?

Debugging threaded networking code is a complex task. But (drum roll), not with Twisted!

by dirkjanranzijn

First, twisted allowed to run all five nodes in one process on one reactor, and now I am able to set pdb.set_trace() where I want.

Second, I replaced a networking layer with a mock objects, which deliver messages with some random delay. But I can drive the python's random generator which gives me ability to reproduce stochastic errors and debug them. It works as following:

before the test TestCase class calls random.seed(some_value) and remembers the some_value;
it test raises AssertionError besides a normal error message, TestCase prints RANDOM SEED: some_value;
to run test, with random generator initialized explicitly, it should be decorated:
```
@seed(some_value)
def test_my_broken_code(self):
  ...
```

Here are the class TestCase and the decorator seed:

import random
import time

from twisted.trial import unittest

def seed(value):
    def decorator(func):
        func._random_seed = value
        return func
    return decorator

class TestCase(unittest.TestCase):
    def _run(self, method_name, result):
        method = getattr(self, method_name)
        seed = getattr(method, '_random_seed', int(time.time()))
        random.seed(seed)

        def seed_info_adder(failure):
            failure.value.args = (failure.value.args[0] + ' (random seed: %s)' % seed,) + failure.value.args[1:]
            return failure

        d = super(TestCase, self)._run(method_name, result)
        d.addErrback(seed_info_adder)
        return d

Feel free to write your thoughts about this testing method, or send any suggestions.

Yet Another HTTP Lib

2011-02-16T11:11:00Z

Sometimes I see interesting links in the Twitter. For example, Yury Yurevich retwitted few PyPi updates yesterday. One of them is a new HTTP library. This is just a wrapper around urllib and urllib2, to make the life easier. But this bicycle design could be improved.

For example, to use HTTP basic auth, author suggest to create an Auth object and then use it in every call:

conv_auth = requests.AuthObject('requeststest', 'requeststest')
r = requests.get('https://convore.com/api/account/verify.json', auth=conv_auth)

Why? This object just stores a username and a password, why not to use a tuple instead? If you strive for simplicity, why not to go to the end?

And there are some comments, related to the code. It is incompatible with python2.5 because author forgot about from __future__ import absolute_import and uses new style try except blocks. The main method, sending request, is overcomplicated and contains few huge ifelse blocks. Also, he is trying to do some monkeypatching if there are 'eventlet' or 'gevent' modules. But which side-effects I should be aware of if one of these libraries is installed in the system, but I don't mind to use it?

Zen for Everyone

2011-02-15T10:03:00Z

There is a PEP-0020, recommended for every beginners learning the Python right after they learned a PEP-0008. This is a set of rules to follow programming in Python (and not only in Python). These rules were written by Tim Peters and they are available right in the python interpreter. Just do the import this.

Today I want to give you a link to extension of the PEP-0020, written by Hunter Blanks. It is a concise illustration of every item of "The Zen of Python".

Projects evolution at GitHub

2011-02-09T12:11:00Z

Today I released a small tool, useful for all GitHub users. This project's name is Forkfeed. It's target is to watch how your projects' forks are evolved on GitHub.

by GraemePow@flickr

Sometimes somebody forks your project and begins to improve it. But not many people send pull requests. It is hard to track all new forks. That is because I wrote a script which inspects each of your projects at Github and creates an Atom feed for all new commits. My own feed updated daily at a VPS. Now I will be well-informed about all new forks and all new commits in them.

Cony in the Bottle

2011-02-01T00:35:00Z

Two weeks ago, in the "Command Line for the Internet" post, I wrote about useful smart bookmarks service.

Digging into a code of the project, I figured out that it is too complex for such trivial idea. And of course I've decided to rewrite it :-)

My own fork is called Cony that is a synonym for Bunny.

by animalvegetable@flickr

And if Bunny1 has a dependency from heavy CherryPy, then Cony is much more modest. It uses a micro-framework Bottle. Bottle, is only a python file committed right into the repository.

Extend

Because there are not dependencies, installation is quite simple — just clone a repository and run ./cony.py. Adding your own shortcuts is trivial as well — it is enough to create a local_commands.py file in the current directory. Some examples already in examples directory, ready to import into the local_commands.py file. Just do from examples import * to plug them all or import each function separately.

Clone & Share

If you wrote an interesting shortcut don't be lazy, fork the project on GitHub, put your command into examples and send me a Pull Request.

A little about the Bottle

I liked the Bottle. It is minimalistic. There are url router and templates built-in. Of course you are able to use any other template engine and ORM. For small projects, like Cony it suits very well.

OpenSource Translations

2011-01-27T14:55:00Z

About a week ago, I saw an announce at the django-developers mail list. It was about the Django translation process, which is moved to online translation service Transifex.

Transifex provides an interface to work on translations. Here is the process:

You have to sign in.
Then to add a project.
Upload existing translations. Usually it is a gettext "po" files.

Then you need to gather some translators around the project. They must be organized into the language teams.

When translation is ready, you are able to download the "po" files.

For experiment, I added one of my tiny projects django-faces to the service. If you know language other than Russian or English, try to translate it. There is a little amount of text.

By the way, I found the there are about 600 projects at the Transifex, but only half have some translation resources. It means that these developers start to play with the interface, but something prevents them from finishing the process. And interface is really obscure. For example, I do not understand how to search some project to translate it into Russian.

Command Line for the Internet

2011-01-20T11:11:00Z

Do you use search shortcuts, allowing to enter a query directly into the browser's address bar? If yes, then you will certainly be interested to know that you can not only to connect existing search engines to your browser, but also to write a more sophisticated query processor directly in Python.

Yesterday, I found an interesting project on the GitHub. It was developed in the Facebook. The project's name is Bunny1.

by serenionion@flickr

Bunny1, is a web server, which receives a command and a query, and then either redirects somewhere else, or shows a HTML page with the result. If you'll plug the URL of the server as the default search engine, you'll get the powerful command line directly in the browser. And the most useful here is that:

it will work in all modern browsers;
commands can be added by yourself and written in the Python.

For example, you can write a handler that by the request tw Good morning, Moscow! will tweet from your name and redirect to the tweet's page.

Sake of justice, it should be noted that the idea is not new. There is, a service YubNub, which implements similar functionality and a Firefox plug-in Ubiquity. But the former contains a lot of unnecessary commands and can't be extended, and the latter is suitable only for Firefox users.

Debugger With Blackjack

2011-01-18T14:13:00Z

I decided to adhere to a certain post publication schedule. I'll try to publish something interesting every Tuesday and Thursday.

Today I want to tell you about a pretty useful tool — namely, on the debugger. If you're not a fan of full-fledged IDE, you probably used the standard Python debugger pdb. It covers 90% of the necessary functionality to me, but still somewhat inconvenient. For example, there is no auto-complete and it is annoying to run list command after an every few steps just to see the current line's context.

The project pdbpp was created to solve these, as well as some other issues. Here are a few features, which proved to me the most useful:

a sticky mode — in this mode, the debugger redraws current function after each step of the interpreter;
tag auto-complete — you no longer have to call dir(obj) just to view an object's interface;
syntax highlighting — pdbpp uses Pygments, to make the code even more beautiful than it is (your code is amazing, right?).

In addition, there are such things as calling the editor to correct the file is being traced, an intelligent parsing of the command, the command longlist (ll) to display the current function or method.

Here is a typical pdbpp session:

In order to use this wonderful tool, it is enough to set it into the system by any of the methods available to you, then, as usual, you can use it as usual: import pdb; pdb.set_trace().

Update: recorded a short demo of how it works (comments are in russian).

Social bookmarks at the GitHub

2011-01-13T11:17:00Z

Today I want to tell you about an interesting project from the GitHub. It is developed by Hilary Mason and called gitmarks.

This project allows to save URLs and share them with other GitHub users. For each bookmark, script downloads page's content. Also, you could specify the tags and description of the link. All data are saved to the git repository and committed, and of course you could use git grep to search over all saved bookmarks. URLs could be added from the command line or using a bookmarklet, but in the latter case you have to start a local web server.

Image by Ei! Kumpel.

However, this project has a few shortcomings.

First, the code looks ugly because Hilary ignores naming conventions, accepted by the Python community.

Secondly, existing documentation says nothing where to store bookmarks. Because of that, many people start to fork the repository and to save bookmarks in it. Look at the network graph, it is quite cluttered with bookmark commits and it is very hard to find code changes in this heap.

It's worse to enforce the code and content separation, to make script development easier.

And finally, there is a batch of small things which are inconvenient. For example, people could comment bookmarks of one another, using the GitHub's commit comments. However, the commit page looks very scary because it includes a full HTML downloaded from the bookmarked URL. To avoid this HTML have to be converted into the readable text.

I hope, this project will evolve to the usable distributed bookmark service.

Diff Git Cached in Vim

2011-01-09T14:14:00Z

Surprisingly, there is a useful command in Vim for editing git commit messages. It was shipped with Vim starting from version 7.2. It's name is :DiffGitCahced.

Diff could be showed manually, but also, you could write this line in the config:

autocmd FileType gitcommit DiffGitCached | wincmd p

And then diff will be opened automatically after the git commit.

This tip was found in the Vim Recipes ebook.

Image Shortener Wanted!

2010-12-28T01:05:00Z

I have an idea of a new service. I call it "The Image Shortener".

Sometimes, I found an image on the internet, licensed under the Creative Commons and just insert it into the blog post. But to get nice result, image have to be resized according to a blog column width.

I'm wondering if there is already such service exists which is acts like a resizing proxy server and url shortener simultaneously?

For example, if I found this image http://farm5.static.flickr.com/4005/4203341399_c1c99027e5_o.jpg and want to insert in into my blog post, but it should be resized to have no more 600 pixels width:

I go to the shortener and enter the URL (ideally, I even don't have to enter the image's URL (it is should work with flickr's pages too).
Shortener remember the URL and give me the shortened http://blah.img/3bfg, which, by default, redirects to the original image.
Then I insert this URL into the blog post but supply additional parameter: http://blah.img/3bfg?width=600.
When this image will be requested, shortener will have to fetch the original, resize it, save somewhere in the cloud and redirect to this new image.
And finally, I get the profit from this automatization.

What do you think about this idea? Is it a bicycle and already implemented somewhere?

Powered by Pelican

2010-12-25T23:06:00Z

After a few days of work, I rewrote my blog to static blog generator Pelican. Moreover, I transfered not only one blog but two: my english blog http://aartemenko.com and my Google Buzz where I tried to write about software development in russian.

I decided to drop http://aartemenko.com and move this blog to static to reduce a memory usage on my VPS. Google Buzz turned out useless for short programmer's notes, because it even not support a simple HTML tags nor syntax highlight for code snippets.

As the result, I took a new static blog generator Pelican, forked it on the GitHub and patched. Some bugs were fixed and some functionality was added. For example, now it could handle article translation into different languages and separate Atom feeds are available for all translations. About two days I spent, developing my own scheme for the Pelican.

Now I able to edit blog posts in my favorite editor Vim, grep them and do other things, you want usually do with texts. For now, Pelican supports only Markdown and ReST markups, but that is enough for most cases. The other features which I need to implement, are Vim helper to deploy articles to the server and ping the FeedBurner.

By the way, I published the sources, of this blog at the GitHub as an example of the modified Pelican usage.

Python Logging in Twisted

2010-11-28T03:27:00Z

This is a way to turn on a standart pyton's logging support in the Twisted. But it is broken. Really. For example, this is my default log format: "%(asctime)s %(process)s/%(thread)s %(levelname)s %(name)s %(filename)s:%(lineno)s %(message)s", and this is piece of log which I get using twisted.python.log.PythonLoggingObserver:

2010-11-28 03:15:09,027 7571/-1220024640 INFO twisted log.py:508 Log opened.
2010-11-28 03:15:09,028 7571/-1220024640 ERROR blah server.py:37 blah minor
2010-11-28 03:15:09,031 7571/-1220024640 INFO twisted log.py:508 lock.LockFactory starting on 4001
2010-11-28 03:15:09,032 7571/-1220024640 INFO twisted log.py:508 Starting factory <lock.LockFactory instance at 0x8755d2c>
2010-11-28 03:15:09,033 7571/-1220024640 INFO twisted log.py:508 twisted.web.server.Site starting on 9001
2010-11-28 03:15:09,034 7571/-1220024640 INFO twisted log.py:508 Starting factory <twisted.web.server.Site instance at 0x875a18c>

Did you see these log.py:508 everywhere? This is wrong! Wrong! But there is the simple way to fix it:

import os
import twisted
import logging
_srcfile = twisted.python.log.__file__
if _srcfile[-4:].lower() in ['.pyc', '.pyo']:
    _srcfile = _srcfile[:-4] + '.py'
_srcfile = os.path.normcase(_srcfile)
logging._srcfile = _srcfile

After this fix, the log will have the right file and line numbers in it:

2010-11-28 03:18:07,763 7710/-1220380992 INFO twisted server.py:35 Log opened.
2010-11-28 03:18:07,764 7710/-1220380992 ERROR blah server.py:37 blah minor
2010-11-28 03:18:07,766 7710/-1220380992 INFO twisted tcp.py:861 lock.LockFactory starting on 4001
2010-11-28 03:18:07,767 7710/-1220380992 INFO twisted protocol.py:44 Starting factory <lock.LockFactory instance at 0x8755d2c>
2010-11-28 03:18:07,769 7710/-1220380992 INFO twisted tcp.py:861 twisted.web.server.Site starting on 9001
2010-11-28 03:18:07,770 7710/-1220380992 INFO twisted protocol.py:44 Starting factory <twisted.web.server.Site instance at 0x875a18c>

Have a fun, read sources!

Update: I created a ticket in Twisted's trac.

Helper to use SELECT FOR UPDATE in Django

2010-11-16T13:18:00Z

Just wrote a simple helper to use SELECT FOR UPDATE construction in django:

from django.db import DEFAULT_DB_ALIAS

def select_for_update(queryset):
    """ Returns query, rewrited to use SELECT ... FOR UPDATE.
        Can be used in transaction to get lock on selected rows.
        Database must support this SQL statements.

        Example:
        >>> query = select_for_update(MyModel.objects.filter(blah = 'minor'))
        >>> unicode(query.query)
        "SELECT * FROM myapp_mymodel WHERE blah = 'minor' FOR UPDATE"
    """
    sql, params = queryset.query.get_compiler(DEFAULT_DB_ALIAS).as_sql()
    return queryset.model._default_manager.raw(sql + ' FOR UPDATE', params)

Filtering choices in Django's ModelForm

2010-06-04T13:12:00Z

# ...
# There are models Subscription and MailList defined above.
# MailList has a many to many field 'managers'. This field
# defines which manager is able to edit/add subscriptions to
# a maillist.
# 
# To filter choices in the Django's admin, I use following code:

class SubscriptionAdmin(admin.ModelAdmin):
    list_display = ('id', 'mail_list', 'email', 'name')

    def get_form(self, request, obj=None, **kwargs):
        form_class = super(SubscriptionAdmin, self).get_form(request, obj = obj, **kwargs)

        class ModelFormWithPermissions(form_class):
            def __init__(self, *args, **kwargs):
                super(ModelFormWithPermissions, self).__init__(*args, **kwargs)

                self.fields['mail_list'].queryset = \ 
                    MailList.objects.filter(
                        managers__id = request.user.id
                    )

        return ModelFormWithPermissions

site.register(Subscription, SubscriptionAdmin)

# This code creates a special ModelForm class which know
# about the current user, his 'id' and permissions.

Here is a gist file.

10 Minute Mail refresher

2010-03-23T12:13:00Z

"10 Minute Mail" is a service of temporary emails. This script is used to keep mailbox alive while it is opened in the browser.

How to parse CSS colors?

2009-11-30T21:03:00Z

def parse_color(s):
    """ Parses color string in format #ABC or #AABBCC to RGB tuple. """
    l = len(s)
    assert(l in (4,5,7,9))

    if l in (4,5):
        return tuple(int(ch * 2, 16) for ch in s[1:])
    else:
        return tuple(int(ch1 + ch2, 16) for ch1, ch2 in \
                     zip(
                        (ch1 for ch1 in s[1::2]),
                        (ch2 for ch2 in s[2::2])
                        )
                    )

Here is a gist file.