Brabadu Technical Blog

React.js based plugin for LightTable

2015-01-12T23:33:00.000+02:00

LightTable made quite a big fuzz some time ago. Now original authors are working fulltime on Eve, while LightTable is now more driven by community members. And they're doing some pretty interesting work like switching to Atom Shell instead of node-webkit.

I like the idea that text editor is a browser. It gives you ability to do literally anything you want. So I decided to integrate React.js into LightTable plugins ecosystem. One of the most interesting things in latest release is User plugin. It is a place to configure your editor not just by tweaking settings and changing keybinding. You can add new features right there without need to create separate plugin. It is much like .vimrc or .emacs/init.el files. This turns process of configuring your editor into development task.

First let's add User plugin to workspace. Start LightTable and run command Settings: Add User plugin to workspace, open workspace panel (Workspace: toggle workspace tree command). Now you can see User dir with a some files added already.

Let's open user.behaviour file. It is a place to actually change settings and add or remove hooks on different events LightTable and plugins provide. We need to add custom javascript file with React.js and make sure User plugin is loaded too.

[
  [:app :lt.objs.plugins/load-js "/Users/brabadu/react/react.min.js"]
  [:app :lt.objs.plugins/load-js "user_compiled.js"]
  ...
]

Let's open User/src/lt/plugins/user.cljs. It already has an example that shows how to make objects and open new tabs. We'll tweak this example to use React.js.

First we need to define a namespace and dependencies

(ns lt.plugins.user
  (:require [clojure.string :as string]
            [lt.object :as object]
            [lt.objs.tabs :as tabs]
            [lt.objs.command :as cmd])
  (:require-macros [lt.macros :refer [defui behavior]]))

All LightTable plugins are in lt.plugins.* namespace. As you can see, we'll be working with LightTable object, commands and tabs. Next, we'll define couple of helper functions

(defn format-time [d]
  (first (string/split (.toTimeString d) #" ")))

(def el React/createElement)

format-time is for pretty-printing time from javascript Date object. And el is simply an alias to React.createElement.

(def label
  (React/createClass
     #js {:displayName "TimerLabel"
          :render (fn [] 
             (this-as this
                      (el "h1" nil (str "Timer: "
                                        (-> this .-props .-time)))))}))

Our component, that is going to show whole React.js-LightTable integration. I guess nothing stops us from using om/reagent/quiescent/etc. Add it's dependency in project.clj and you're good to go. Our example is simple enough not to bring anything more than bare React.

Interesting thing is how javascript this works. Calling macro this-as with symbol to which javascript this bounds is idiomatic way to access it. Most of ClojureScript libs are hiding this pattern in their innards, so you'll need it mostly when doing interop with javascript.

(defui react-panel [this]
  [:h1 {:id "app-root"} "React + LightTable!"])

(object/object* ::user.react-timer
                :tags [:user.hello]
                :behaviors [::on-close-timer-destroy]
                :init (fn [this]
                        (react-panel this)))

(def react-timer (object/create ::user.react-timer))

Here we define LightTable (not React.js) ui component react-panel and object prototype ::user.react-timer. LightTable has it's own notion of object, that it uses to implement BOT principle. When the object react-timer is created function that is passed with :init is executed.

(cmd/command {
  :command :user.show-time
  :desc "User: Show time"
  :exec (fn []
    (tabs/add-or-focus! react-timer)
    (let [app-root (.getElementById js/document "app-root")
          label-inst (React/render 
                        (el label #js {:time (format-time (js/Date.))} [])
                        app-root)
          refresh-timer (fn []
                   (prn (format-time (js/Date.)))
                   (.setProps label-inst #js {:time (format-time (js/Date.))}))
          interval (js/window.setInterval refresh-timer 1000)]
       (object/merge! react-timer {:interval interval})))})

This code adds new command to LightTable. It creates new tab, then instantiates React.js component and mounts it on rendered by react-panel node.

After that defines refresh-timer function, that updates props in React component. It is called every second as a callback of js/window.setInterval. We could save timer ID in React component's state, but I decided to put it in object, so our component could be simple and it's only purpose would be view layer, while all the business-logic lies separately.

(behavior ::on-close-timer-destroy
          :triggers #{:close}
          :reaction (fn [this]
                      (js/window.clearTimeout (:interval @this))
                      (object/raise this :destroy)))

Now we're taking care cleaning on our tab close event. This is why I added prn to refresh-timer function - to see if it stopped being called by setInterval.

That's it. Code is a bit messy, but hope everything is clear. All code in one place is here

Another useful Ubuntu PPA list

2015-01-09T12:00:00.000+02:00

It's been a long time since I posted my useful ppa list. Looking at that list and my current PPAs shows that almost everything changed and I could share my new favourites.

ppa:ubuntu-mozilla-daily/ppa Unofficial Firefox nightly builds. This is my main browser for about a year now. Even though sometimes it may be unstable it is great browser. It loosed it's speed in development and innovation for some time, but now it becomes good product again
ppa:conscioususer/polly-daily is for Polly Twitter Client which is the only usable desktop twitter client under linux for me.
ppa:tuxpoldo/btsync BTSync
ppa:webupd8team/sublime-text-3 Latest builds of the best text editor in existence - Sublime Text 3
ppa:ubuntu-desktop/ppa Pre-release builds of Unity and other Ubuntu goodness
ppa:webupd8team/java Java release by webupd8 team. I need it to run PyCharm, my main IDE for now
PyCharm has nonofficial PPA on UbuntuUpdates. But I prefer to go to it's EAP download page to get fresh build.

To install any of those PPAs type in terminal:
sudo apt-add-repository ppa:ubuntu-wine/ppa
sudo apt-get update

After that you can install software from PPA with
sudo apt-get install ...

Those things are PPAs, but you have to either add them manually to /etc/apt/sources.list.d/ or with help of this tutorial

http://dl.google.com/linux/chrome/deb/ you can't run away from Google Chrome if you're doing web development these days. Here I also use untested version. And also it is a browser I use when some builds of Firefox Nightly has critical issues.
https://get.docker.io/ubuntu Docker

Yandex Subbotnik

2013-04-28T14:46:00.005+03:00

Today I've been to the 5th Yandex Saturday event - Subbotnik. This is series of technical miniconferences in different cities. Place where yandexoids show other developers how they work, show toolset and other yandexish things. This was second Subbotnik for me. Last year most of the talks were about BEM (their front-end architecture) and toolset around it. So I've lost what they're talking about right from the second talk that day.

Today's Subbotnik was prepared much more thoughtfully. Yes, BEM was mentioned couple of times during each talk anyway, but all the themes were aimed to more broad audience, that doesn't know all of the ins and outs of bem-tools.

This Subbotnik had 2 tracks. One for Yandex's API users, the other was more for general developers with talks about front-end development and cloud services. I watched the second track, as I don't use Yandex API at all, so have no interest in it.

All the talks were in Russian most of the slides (if not all) too. But anyway I think it worth mentioning them in English speaking internet. You never know what may become useful.

Yandex.Disk. The flight is normal by Vladimir Rusinov.
Short introduction talk about Yandex.Disk from it's product manager. Yandex.Disk, for those who don't know, is online backup, storage and syncing service. Something very close to Dropbox. The product is doing fine, usage is growing, other services actively integrate with it using API. There were some numbers, history and funny picture. As it turned out it was warmup before second talk.

Operation and development of fast-growing clouds by Michail Belov and Oleg Leksunin.
First half of the talk was done by Oleg Leksunin who is system administrator of Yandex.Disk. He talked about requirements that were crucial for such service. architecture of the service, how they are organized and deployed. Also mentioned some key parts of the system and talked what each of them does.

There's so called MPFS service, that is a virtual file system for users's files. When file is added to YDisk request to MPFS is sent, which synchronously stores file as blobs on two different servers (with internal project Mulca). When it is saved file's metadata are saved to MongoDB.

Protocol for YDisk is extended WebDAV, that was chosen because it implements most of needed features, had existing working tools like wget or curl. And almost any modern OS has builtin WebDAV support. WebDAV server is written in Erlang. It was described in the post on habrahabr (in russian). MPFS is written in Python with Flask, Jinja2 and is served via nginx. Client applications use XMPP for push notifications.

Overall it was great talk with lots of interesting details and insights. Usually there's not so much you can take from talks like this, but it is interesting to know how big companies make big services. As it turns out everything may be quite simple.

Cryochamber for statics by Alexey Androsov I think was the best talk of the day. Alexey works as a fronted engineer at Yandex.Mail team. He started discussing how usually static files are served. Common case is to set cache period for static files for a long period, e.g. a year or a month, and make url to static resource uniqe by adding version of the app or revision (/static/1.1/app.js or /static/app.js?rev=a7cb9). Problem appears when there's only small change in the fronend app, then version or revision changes and all the static files have to be downloaded again. For big frontend apps this may become big problem, Y.Mail's static resource is more than 1Mb so with often changes to code there's a lot of things to download every time.

Yandex.Mail team took different approach. Now they put all the files for all of the versions into the same folder, but name of the resource is generated by file's hash. And postprocessor substitutes all links to the file in content that is served to user to hash-based new filenames.

So the CSS-code:

.class {
   background: url("1.png");
}

becomes:


.class {
    background: url("freeze/f80abidmv6dlfo9.png");
}

There's some tricks for making all of this possible in JavaScript. That is being done with borschick, the app that does all the magick - renaming, processing links etc. I'll definately take a look at this project it looks very interesting.

Michail Korepanov with a talk on Incremental updates on the client continued discussion on how to make updates of front-apps less problematic. They decided to make browser download not the whole file with new version of app fiiles, but implement diff update.

So now the app doesn't load css and main javascript app as links, but they are downloaded via some javascript bootstrapper and saved to localstorage. After saving resources CSS is inserted inline to page and Javascript is evaluated with "new Function()".

Next time you open Y.Mail bootstrapper sends request with app version saved in localstorage, server responds with diff, bootstrapper patches code in localstorage and injects into page.

Diffs are prepared beforehand for 3 previous versions, when new version is deployed to production. If version stored in localstorage is too old, then server responds with whole latest version of the app. As I said, Yandex.Mail deploy new frontend app version twice a week, so such method saves a lot of bandwith and makes loading new version of the app much faster.

Also Michail mentioned RFC 3229 Delta encoding in HTTP, now finding diff and patching must be done by hand, with JavaScript, but this RFC implementetion could automate most this work. Unfortunately it is not implemented in any browser yet. There's a Chromium-based Yandex.Browser, so maybe this would be a place where this RFC would be first implemented ;)

Now I'm not sure how this is correlated to the previous talk by Alexey, since they solve the same problem from different angles. This talk looked a bit more theoretical, though it is used in production, as I understood.

Next talk after a short break - single-page application using node.js and BEM started by Eugene Filatov.

The main point of the talks was to tell about how yandex makes web apps that may be rendered both by backend and in the browser. Server rendering is needed for search engines to get your content properly.

BEM methodology operates with reusable blocks: news block, search bar block, email list block etc. Every block contains all in one place: html, css and javascript for it to work properly. But since block may be rendered both on server and on client there's some things that may need different handling. In the talk example of setting a cookie was mentioned. On server we have to add Set-Cookie request header, in browser we may write to document.cookies in the simplest case. But there's also a lot of shared code, so they prepare 3 files:

cookie.js - code specific to the browser
cookie.priv.js - code specific to node.js ("priv" is for "private")
cookie.common.js - shared code for both versions

bem-tools that postprocess page would gather bundles for node and browser environments.

Eugene shared a piechart that shows that common code usually takes up to 54% of block code, so it definately worth having similar code organization if you want to do both server and browser rendering.

Next talk was Interface development in distributed teams by Sergey Puzankov.

That was just an overview how the whole process is organized, how communication is made and all the things. Not so much to tell about, everything is the same as others do. Well, there's two things that everybody should, but mostly don't have: regular inner conferences for sharing knowledge and up-to-date docs about systems.

Last talk was Code management or Why SCM by Sergey Sergeev.

Sergey is well-known git advocate, so he told about how his team uses git, best practices and anti-patterns. Now I know that Yandex SERP frontend team uses git-flow :) There was nothing really new here too, but it was nice opportunity to share user stories and ask questions for some people.

This was interesting event with great speakers. Despite there were no groundbreaking revalations, it was great opportunity to gather developers and discuss problems that all of us, both small dev teams and large software giants share.

Thanks Yandex Team for interesting event and great speakers!

Sublime Text 2/3 for Django developer

2011-10-03T21:17:00.000+03:00

Many developers know about this battle between emacs and vim. Well, not exactly between editors themselves, but rather between their fanboys. As for me, I started with gedit, default Gnome, therefore Ubuntu, text editor. After adding a couple of plugins, most importantly, dummy code completion it worked well for couple of months.

A bit later I decided gedit isn't cool enough :-) And made a switch to vim. Can't say this was love from first sight, but as longer I used it, more I liked it. I ended up being completely overtaken by command mode. I don't care if I need to type one more key to yield into system clipboard, I don't need it too often. But copying to internal registers, having them as many as I want, searching right under your pinky is awesome. I started using searching dozens of time more often, then I did in other editors and IDEs. Combining this with remembering of your positions (so you can jump to your previous cursor positions), changes, and thousands of other different things. You know, look at any vim cheat sheet, and you'll be scared and maybe interested :-)

Yes, I know, it takes time to learn it, even more time to get used to it, but when it is in your muscle memory it is extremely comfortable. But I still have different issues with it. I almost can't use Rope, code completion is almost as dumb as gedit's, and configuring it is cargo-cult for me :).

I tried emacs, but it uses many many more times control and alt keys. I don't really like this. Now let's move to the subject :-)

Text editor Sublime Text 2 is starting to gain more and more attention. It is multiplatform, quick, nicelooking editor with a couple of very interesting features.

One of the mostly used ST2 features is it's function for opening files from project. You can press Ctrl+P and start typing filepath and it will give a list of matching files. You have to create project first, to use this function. It works great and became very handy way to move between files. Other options is Ctrl+Shift+P - same kind of text input opens and you can start typing a command and it will filter it for you. For example you can type Rope and it will show you all the commands of rope on first place, and then other commands that have letters r,o,p,e in it. It is hard to explain, you should try it for yourself and you'll understand it quickly.

Opening file from project

Moving between methods is not harder - Ctrl+R and you see a list of methods in current file, can filter and quickly jump to their declaration.

Most surprising feature for me is vintage mode. It is a simulation of VIM's most commonly used commands, such as ghjk-navigation, commands for copying, deleting, pasting, inserting etc. But miracle didn't happen, it is quite restricted. Using motion commands isn't as consistent as in vim. ESC doesn't clears command, so if you typed d, you will delete something, no matter how many times you push ESC button. Vintage mode is made completely on keybindings, so maybe it is possible to fix some things, but I think emulate whole vim's behavior would be overkill.

Other great advantage of ST2 is it's configuration file. You can build your own from scratch, referring to default one, which is pretty well documented. Configuration file is json, so you don't have to learn another configuration language. My (tiny) configuration file is on github.

I have installed couple of plugins for django development:

Djaneiro is nice set of autocompletions, colorscheme for django. I printed their README to see all list of available substitutions, e.g. var becomes {{ }} in template, block becomes {% block %} {% endblock %}
SublimeRope is an attempt to bring powerful library Rope into Sublime Text 2. Rope can do a lot of IDEish things with python code like extracting methods, searching for docstrings, going to definitions, renaming etc. At the moment it can do only renaming and moving to definitions but github repo is being updated time to time, so I hope we will see more features in future.
SublimeLint became a big discovery. I knew lint existed, but never actually used it. It can show you different warnings and errors like those big compilers have: variable declared but never used, unused imports, syntax errors and PEP8. This is not the best explanation what Lint is though :) It can make editor a bit slower, but you can turn off background linting and turn on when you need it again.

SublimeLint shows PEP8 warning

Autocompletions in ST2 is what I still don't understand. Quite often I can't predict what suggestions will it give, especially what it will autocomplete with Tab key.

Everybody is talking about minimap. I can't say anything about it. It is interesting way to find your place in file, no more, no less.

As a summary I think sublime can become my default editor. I use it for a week already, and though I miss some vim specific features, I like it very and very much. I think I'll try to do a reversed path. I want to try recreate all things i liked so much from ST2 in vim. Depending on results I'll choose one of them to use day-to-day.

Even if i will switch back to vim, Sublime Text 2 showed me some things in editor, that I liked and that can make my life easier and nicer. Users really don't know what they want, until they get it :)

Easy REST API with django-tastypie

2011-06-23T10:57:00.001+03:00

One day at work I had to implement JSON response on simple GET request. I was just going to code another RESTish bicycle as my project manager told me to use django-tastypie. I'm sure he knows what he is talking about so I opened google to find what tastypie is.

When spec for my task was being written, author didn't think about what tool will be used for implementation. On the other side tastypie authors didn't know what people will do with their lib. So my task was to make this two worlds meet in one place. On github project page there's a simple example with very basic things showed. The real world is usually more complicated. I'll try to show you how I worked with tastypie.

Useful link
A Brief Introduction to REST

Let's take an example. For this post I created tine django project which you can find at github.
We have some working web application and we need to add REST API to it. Our main models are: Cabinet, Folder and File. Cabinets stand in some office. Every Cabinet has Folders inside, which contains Files of different types.

The task is to implement such response for request GET api/cabinet/list

{
   "cabinets":[
      {
         "added":"2011-06-17",
         "color":"black",
         "floor":2,
         "folders":[
            {
               "files":[
                  {
                     "archive_num":"221122",
                     "content":"Lorem ipsum dolor sit amet
consectetur adipisicing elit",
                     "type":"Evidence"
                  }
               ],
               "name":"XFiles",
               "secrecy":10
            },
            {
               "files":[
                  {
                     "archive_num":"1111",
                     "content":"sunt in culpa qui officia deserunt mollit 
anim id est laborum",
                     "type":"Document"
                  },
                  {
                     "archive_num":"333",
                     "content":"$1000",
                     "type":"Bill"
                  }
               ],
               "name":"Foldy folder",
               "secrecy":6
            }
         ],
         "height":2,
         "width":10
      },
      {
         "added":"2011-06-17",
         "color":"white",
         "floor":10,
         "folders":[
            {
               "files":[
                  {
                     "archive_num":"87878",
                     "content":"Funny picture",
                     "type":"Picture"
                  }
               ],
               "name":"Fun pics",
               "secrecy":0
            },
            {
               "files":[
                  {
                     "archive_num":"23423424",
                     "content":"Photo of some big room",
                     "type":"Photo"
                  }
               ],
               "name":"Old folder",
               "secrecy":3
            }
         ],
         "height":1,
         "width":4
      }
   ]
}

This means that we need whole tree of entries, gathered in one root called 'Cabinets'.

Installing is easy and usual - pip install django-tastypie.

After reading docs we create first resource that looks like:

from tastypie.resources import ModelResource

from app.models import Cabinet


class CabinetResource(ModelResource):
    class Meta:
        queryset = Cabinet.objects.all()

Docs say that we should put this in api.py file in app folder.

Then we should create url handler:

from app.api import CabinetResource

cabinet_resource = CabinetResource()

urlpatterns = patterns('',
    ...
    url(r'^api/', include(cabinet_resource.urls))
    ...
)

This creates a resource at /api/cabinet which returns:

{
   "meta":{
      "limit":20,
      "next":null,
      "offset":0,
      "previous":null,
      "total_count":2
   },
   "objects":[
      {
         "added":"2011-06-17",
         "color":"black",
         "floor":2,
         "height":2,
         "id":"1",
         "resource_uri":"/api/cabinet/1/",
         "width":10
      },
      {
         "added":"2011-06-17",
         "color":"white",
         "floor":10,
         "height":1,
         "id":"2",
         "resource_uri":"/api/cabinet/2/",
         "width":4
      }
   ]
}

Good start but we need more!

We got Meta object, that we don't want to get :), we have resource_uri and id, which we didn't ask either, and most important - we want elements, connected to our Cabinets.

But first let's deal with URI of our resource. Our task is clear - URI must be api/cabinet/list. Resource Meta class has field resource_name, which is clearly described in docs. By default resource name is generated from resource class name by removing word Resource and lowercasing it. But you can specify your own resource name and nothing keeps you from making it as custom as you want, for example - list/cabinet (api part is defined at urls.py), right what we need!

While adding subelements to response I tried some weird hacks. But it turned out to be quite simple. tastypie doesn't make connections automatically. Authors follow the principle "Explicit is better then implicit". So every connection of your resource must be done manually.
Connections are established between resources, not models. So here's a resource for Cabinet's child class - Folder:

class FolderResource(ModelResource):
    class Meta:
        queryset = Folder.objects.all()

Now we need to add field to CabinetResource, what result in something like:

class CabinetResource(ModelResource):
    folders = fields.ToManyField('app.api.FolderResource',
                                 'folders', full=True)
    class Meta:
        .....

Optional parameter full is False by default. In this state folders subelement will only contain links to it's respective Folder resources. To work this way you need to extend your urlhandlers with FolderResource urls. This is described quite well in official docs. But in our case we need to show not just a link, but whole object's content. To achieve it we set full=True. Here's little trick I've noticed while was writing this post. If you don't specify urlhandlers for subelements (Folder in our case) resource_uri parameter for subelements will be empty. So nobody would have a way to get your objects (via this API, I mean)

Having always empty parameter isn't cool. So I wanted to turn them off. First I tried hackish way, deleting this parameter manually on response generation step. But then I noticed a parameter include_resource_uri which prevents adding resource_uri parameter to object.

And at last we would like to remove field id. Showing to the world inner data like ID of object in database is potentially dangerous. This could be done with fields or excludes parameters, that might be familiar to you from Django Forms, that use the same principle. To hide id field we add excludes = ['id'] to our Meta classes.

So now our api.py file is:

from tastypie.resources import ModelResource
from tastypie import fields

from app.models import Cabinet, Folder


class FolderResource(ModelResource):
    class Meta:
        queryset = Folder.objects.all()
        excludes = ['id']
        include_resource_uri = False


class CabinetResource(ModelResource):
    folders = fields.ToManyField('app.api.FolderResource',
                                 'folders', full=True)

    class Meta:
        queryset = Cabinet.objects.all()
        resource_name = 'cabinet/list'
        excludes = ['id']
        include_resource_uri = False

And response at /api/cabinet/list is

{
   "meta":{
      "limit":20,
      "next":null,
      "offset":0,
      "previous":null,
      "total_count":2
   },
   "objects":[
      {
         "added":"2011-06-17",
         "color":"black",
         "floor":2,
         "folders":[
            {
               "name":"XFiles",
               "secrecy":10
            },
            {
               "name":"Foldy folder",
               "secrecy":6
            }
         ],
         "height":2,
         "width":10
      },
      {
         "added":"2011-06-17",
         "color":"white",
         "floor":10,
         "folders":[
            {
               "name":"Fun pics",
               "secrecy":0
            },
            {
               "name":"Old folder",
               "secrecy":3
            }
         ],
         "height":1,
         "width":4
      }
   ]
}

Next step is to add subelements of Folders - Files. We already know how to do it, so let's make it quick.

Adding a resource:

class FileResource(ModelResource):
    class Meta:
        queryset = File.objects.all()
        excludes = ['id']
        include_resource_uri = False

And a field to Folder resource:

class FolderResource(ModelResource):
    files = fields.ToManyField('app.api.FileResource',
                               'files', full=True)

    class Meta:
       ...

And now we have a response like this (excerpt):

....
"objects":[
      {
         "added":"2011-06-17",
         "color":"black",
         "floor":2,
         "folder":[
            {
               "file":[
                  {
                     "archive_num":"221122",
                     "content":"Lorem ipsum dolor sit amet, consectetur"
                  }
               ],
               "name":"XFiles",
               "secrecy":10
            },
            {
               "file":[
                  {
                     "archive_num":"1111",
                     "content":"Duis aute irure dolor in reprehenderit
in voluptate velit esse"
                  },
                  {
                     "archive_num":"333",
                     "content":"$1000"
                  }
               ],
               "name":"Foldy folder",
               "secrecy":6
            }
         ],
         "height":2,
         "width":10
      },
....

Almost there. One issue left about Files model. It has field FileType that tells us quite useful information. It is implemented as ForeighKey, but our specification tells us to integrate it flat, without creating another sublevel. What we can do with it? Let's first create regular subresource.

class FileTypeResource(ModelResource):
    class Meta:
        queryset = FileType.objects.all()


class FileResource(ModelResource):
    type = fields.ToOneField('app.api.FileTypeResource',
                             'type', full=True)

    class Meta:
        ...

That responses with

    ...
"files":[
    {
        "archive_num":"221122",
        "content":"Lorem ipsum dolor sit amet, consectetur adipisicing elit",
        "type":{
            "id":"4",
            "resource_uri":"",
            "type":"Evidence"
        }
    }
 ],
...

We have

"type":{
    "id":"4",
    "resource_uri":"",
    "type":"Evidence"
 }

but we want only "type":"Evidence".

Hopefully tastypie has quite a lot of handy hooks that you can use to modify default behaviour. What we need now is dehydrate method. It is called every time when object is prepared for serialization. dehydrate receives two parameters:

self - obviously, current resource object link
bundle - object of Bundle class. It has two main fields:
- obj - a link to object that is being serialized
- data - dictionary with content of obj prepared for serialization

By default dehydrate simply returns bundle, which is afterwards serialized into JSON dictionary (or other structure with subelements if response is not JSON). So, basically, everything we need is dehydrate to return one bit of data - FileType.type

class FileTypeResource(ModelResource):
    class Meta:
        queryset = FileType.objects.all()

    def dehydrate(self, bundle):
        return bundle.data['type']

dehydrate is the right place to manipulate with data you're going to return to your client. Let's deal with what is left! Default response consist object meta. It is generally useful to have some meta data attached to response. Without it things can become confusing when using offsets or other request modifiers. Nonetheless, sometimes we just have to get rid of it. There's a method for that! Method is called alter_list_data_to_serialize and it starts after all dehydrations, right before serialization. We have to add it to our root CabinetResource. :

    def alter_list_data_to_serialize(self, request, data_dict):
        if isinstance(data_dict, dict):
            if 'meta' in data_dict:
                # Get rid of the "meta".
                del(data_dict['meta'])

        return data_dict

To be clear, dehydrate works on per-resource level, when alter_list_data_to_serialize works with results of dehydrate — generated python lists and dicts. And the last thing. Now root element is called 'objects' and I don't like it. To rename we add following to alter_list_data_to_serialize

    ...
    def alter_list_data_to_serialize(self, request, data_dict):
        if isinstance(data_dict, dict):
            if 'meta' in data_dict:
                # Get rid of the "meta".
                del(data_dict['meta'])
                # Rename the objects.
                data_dict['cabinets'] = copy.copy(data_dict['objects'])
                del(data_dict['objects'])
        return data_dict

Useful link
Mobile API Design - Thinking Beyond REST
not just for mobile, but for overall good REST API design

Renaming is simple, though I still think there must be some option to set it without doing not-so-good things like this. That's it. There's a ton of features I didn't cover. This post is already much larger then I expected. I have to stop here or I will never publish it :)

Enhancing git commit message template

2011-04-23T01:18:00.012+03:00

My faveourite DVCS git has a method to generate (or at least prepopulate) your commit message. I'm going to tell you how it works.

We're using a principle of feature branches at work. Every ticket, that is created on bugtracker receives it's own git branch. Sometimes you have to see all changes related to one ticket. But branches would be merged to master one day sooner or later so commits from different branches would be mixed. To solve this we have a convention, that every commit message has to start with 'ticket:###'.

It is quite easy rule, but sometimes you forget to follow it (you're in a rush, or too sleepy etc). There's a simple answer for such things - automatization :) You can hook many different actions that happen in git. One of them is preparing commit message. So go to your project's dir. Then

cd .git/hooks/

create file prepare-commit-message
put the following in it:

#!/bin/sh
ORIG_MSG_FILE="$1"
TEMP=`mktemp /tmp/git-XXXXX`

TICKETNO=`git branch | grep '^\*' | sed 's/[tT_]/ /g' | cut -f 3 -d ' '`

(echo "ticket:$TICKETNO "; cat "$ORIG_MSG_FILE") > "$TEMP"
cat "$TEMP" > "$ORIG_MSG_FILE"

Code was taken and tweaked a bit from StackOverflow question

This will work if a branch is named so it matches regexp: t[0-9]+_.* (e.g. t777_implement_feature)

Everything is simple:

1. Creating a temporary file

2. Getting a ticket's number from branch

3. Writing "ticket:" and ticket number to temporary file AND content of message template file (it maybe there with this script together)

4. Making temporary file a file, that will be taken as a message template

And don't forget to make it runnable with:

chmod +x prepare-commit_message

There's a lot more interesting things, that you can do with git hooks. Just imagine: start tests on your testserver, automatic deployment, send status emails to your boss ;). Isn't it great?

Android App Inventor

2011-03-29T12:13:00.002+03:00

I had an idea to develop for Android since bought this shiny, speedy, cooly Acer Liquid E. Anyone would like to! It's so smart, powerful and always stays with you literary everywhere! It is a great temptation not try to handle this power yourself.

I installed all requirements for development, but since then didn't have much urge to try do something with it because Android SDK would take time to get into. I am more focuse at Python and Django stuff at the moment. Meanwhile, I remember there's a thing called Android App Inventor - a tool for creating android apps visually, without actual coding. That's an easy way to try something on Android.

App Inventor consists of two parts: interface builder and block editor, that is used for adding logic to elements you've added with interface builder.

There's nothing special about Interface builder, actually. You have a pallete with standard visual elements like buttons, textfields, lists etc and some of non-visual components, like camera, image picker and, mind blowing component as for me - speach recognition. Just imagine, you have an instrument to turn words you speak into text without writing single line of code!!! So I thought that I have to do something with it, no matter will I use it or not. (Last time, when I saw another cool API - face recognition, I made Spycify :) )

I decided to make some kind of a dictaphone app with speech recognition. It has to:

add recognized text to existing text in textfield (standard voice recognition removes previous content from text field, and fills with last recognized)
send whole text to Evernote

Interface should be simple: text field and 3 buttons (start recognizing, send to evernote, clear). Elements positioning options are quite poor, so I couldn't place buttons exactly as I wanted. Text field didn't want to take whole space above buttons, so I hardcoded it's size. Now it looks OK in portrait orientation, but quite crappy in landscape :( Minus to App Inventor.

Let's move to making the real magic happen. Here's the "source code" of my app. Everything is quite simple. These blocks are quite similar to puzzles. You have event's (e.g. button click), you can place actions inside them: set a variable, call something, run activity etc.

As you can see, all you have to do to get your greatest speech recognised - call SpeechRecognition.GetText somewhere and place SpeechRecognition.AfterGettingText handler (square 1). That's it! You have a text that was recognized by those powerfool Google servers in a variable. You can assign it to a textfield, parse it, look for command words, everything you can imagine!

In our case we're checking if textfield (called Note) is empty then just add recognized text. If there's already something in it, add a newline before adding recognized text.

Another question is how to send your text to Evernote. There's one thing in Android called Activity. All the apps are built of them, and you can send special request (called Intent) to any app's Activity from your app, passing some variables to them. Evernote published parameters, that can be passed to their app's (very kind move, thanks Evernote!).

To start Activity (and send your Intent) you have special component ActivityStarter (who could have guessed...). Parameters to Evernote are passed in a special way with dictionary - you have to specify ExtraKey, and it's ExtraValue. For example, to preset a title of a new note, you should set:

ExtraKey=EXTRA_TITLE;
ExtraValue='title you want'

Body of a note is set with ExtraKey=EXTRA_TEXT. If you're makeing standard app with Android SDK you can set any number of those parameters. But ActivityStarter can send only one pair of these ExtraThings. If I'm wrong, please-please, correct me in comments! So my app send only note's body, and it's name I type manually.

That's it. The last handler (square 3) is clearing text field. As simple as it could be.

You can try your app in emulator, or right on your device. Everything you need to know about installing, using and deploying can be found in docs.

Yes, sure, you can't do Angry Birds with App Inventor. Nobody would try to, I think :) Moreover, App Inventor apps can't be published at Android Market (Google says they will allow it some day). There are geeks, who are building a Twitter app with it and even that's kind of crazy, I'd say. But if you are creative enough you can do something like marriage proposal app for the love of your life ;)

Simple automation of virtualenv creation

2011-03-10T22:53:00.001+02:00

Working with python/django related things is a great fun if we're talking about actual development. But there's a couple of things around you have to keep in mind and manage manually. One of the most annoying thing you have to do is activating different virtual environments.

I have a few projects on my local machine, most of them have separate virtualenvs, so when you have to move from one project to another you have to deactivate one, activate other and only then start working. When I was starting developing I thought it's not a big deal - only two commands actually. But after a tens of switches you're starting to get angry. Solution for that was found on Max Klymyshyn blog. There's only one thing to mention - on Ubuntu I placed Max's code at $HOME/.bashrc, because Ubuntu doesn't respect .bash_profile.

But during last weekend I decided to try a couple of projects from github. And for everyone of them I had to create separate virtual environment (and as I'm so lazy I made it automatically started). While creating a new virtualenv, I thought that this whole procedure deserves a bit of automatization too. So I made a little bash script, that creates new clean virtual environment in a new directory. Here's a script:

#!/bin/bash
if [ $1 ]
then
     echo 'Making new dir' $1
    mkdir $1
    cd $1

    echo 'Creating new dev environment'
    virtualenv .env --no-site-packages
    echo VIRTUALENV_PATH=`pwd`/.env > .venv
else
    echo 'Parameter is required for dir name'
fi

It only checks that you've specified a name for a new directory, where virtualenv is going to be created. I put it to $HOME/bin/newenv file, and chmod +x newenv to make it runnable. So now every time I want to try something new from github I have to type only one command, that creates a great place for experiments, which virtualenv is activated right when I cd into this directory.

At last, I tweaked (in russian) my bash prompt. Now it shows if current dir is svn/hg/git repo, and if so, show the name of current branch and if it has not committed changes. It looks geeky as hell, but not so informative and usefull as I expected :) It slows down using of terminal (cause it runs all svn, hg and git twice after every command on terminal: first to get a branch name, and second - for status). And it takes a half of command line space. So, there's more cons then pros, but I'll see, maybe I'll get used to it so wouldn't be able to remove it.

Testing Google App Engine application

2011-02-27T22:18:00.010+02:00

There was a time when I did no testing at all. (As The Kooks sing "You're so naive"). And Spycify was started and achieved it's first releases with about 0 tests. Those first versions were working, but had a whole bunch of annoying bugs. The only way I had is to do massive refactoring of whole app.

Previos versions were checked manually. I started a devserver, clicked through changed parts to see does it works. Sometimes this would not help and I had to upload app to production to test it there, manually, of course. Massive refactoring was a nice chance to start using automate testing. No more manual clicking and repeating myself.

At 42cc we practise TDD. I can't say I'm a big fan of it, though it has it's benefits. I decided that I would not write test for every single bit of code. Unittests for main parts of model and functional test or two for main workflow.

Installing

Testing of GAE apps needs a bit of tools. First we need nice way to run our tests. Plugin nose-gae for nosetest works great. To install it run in console:

pip install NoseGAE

To run tests:

nosetests -v --with-gae

Unittesting

It is done with plain unittest and mock modules. As I was not going to test a connection to face.com, which I am using for actual detecting faces, I saved json output for one of photos. In test I put this json into a model and run a method, that parses json, rejects recognitions with low level of convinience (if face.com is only 50% sure it is face, it is better to leave it alone without putting anything on it. Such results often get shirts, other wierd things) finds coordinates for glasses and moustaches. No actual working with datastore involved.

So, basically, my test only loads input and tests everything is present, checked and understood right in a parsed json.

This tests were not a big deal, actually. But they helped me to find lots of syntax errors, wrong imports and other bugs. As I was working on Spycify on weekends and never started a dev_server with it for about a month. So unittests helped me to find out that my model is generally speaking is working.

Functional tests

First functional test was meant to test that photo can be uploaded, then it processed and made right output. As I'm not going to analyze resulting image - find that we don't end up on one of error pages. This took a little bit more to establish. GAE SDK doesn't have testing infrastructure, as a django. So you have to create your own WSGI application and operate with it. I googled to see how people are doing it.
This is a part of my test_http.py, that checks workflow of Spycify:

import os
import re

from webtest import TestApp
from spycify.main import application

from wsgiproxy.exactproxy import proxy_exact_request

SERVER_NAME = os.environ['SERVER_NAME']
SERVER_PORT = os.environ['SERVER_PORT']

app = TestApp(proxy_exact_request, extra_environ={
                                    'wsgi.scheme': 'http',
                                    'SERVER_NAME': SERVER_NAME,
                                    'SERVER_PORT': SERVER_PORT,
                                    'HTTP_HOST': '%s:%s' % (SERVER_NAME,SERVER_PORT)
                                  })

def test_index():
    response = app.get('/')
    response.mustcontain('Spycify',
                         '<form',
                        )

At first I was trying to run separate testing server on seperate port. But it I couldn't make it run on SERVER_NAME and SERVER_PORT I wanted. Maybe this is not the best decision, but I decided to run a real dev server, and proxy all requests to dev_server.

Testing file upload
I thought I will get in trouble with testing file uploading. But that works the best way I could expect. Webtest, that is used for functional testing, doesn't run javascript, so in test we have to implement all AJAX-requests manually. For example, as I told in previous post, you have to get new upload url for every file upload. Now it is implemented by making separate request, that returns an url for upload. That script inserts url into upload form's action method.

response = app.get('/upload_url')
url_pattern = re.compile(r'http\:\/\/\w+\:\d+(\/.*)')
upload_url = url_pattern.match(response.body).groups()[0]

filename = 'tests/ivan.jpg'
response = app.post(upload_url,
        {'img_type': 'photo'},
         upload_files=(('file', filename),
))

Here we at first making a request to get an url. Then we take last part of url (without server name and port). And then app.post, passing a filename will upload file to app. Easy as pie!

Next page needs some care too. To fit the rule "Show something to user as soon as possible" I'm doing all the picture stuff in background. When user hits "Upload" his photo uploads to the server where it gets a number. Right after user is redirected a page with that number where he sees 'Loading...' message. On this page request is made to a server, that tells him to start photo processing. When it is done a response from server contains image link and the rest of the page.

So for all of this to work we have to strip photo id to make all request, that javascript does, manually. And when all of them are done and we reach last response from server the only thing is left - to check response consist img tag in it.

That's basically all the testing I have at the moment. Next test will be added when new bugs will be found or new features implemented.

There's one problem in such setup. There's no database clearing after each test. My app is not really about data, so I can manage without such feature. But on other apps I'd like to find out the way to do it. There must be something already done for it.

And for the ending a video about testing techniques at Google App Engine. I haven't seen it yet, but going to :)

Spycify them all!

2011-02-17T17:01:00.011+02:00

Recently I've launched my first site, that was revealed to some friends whose critisism I'm not afraid of :). It is called Spycify, and was made just for fun using Python, Google App Engine and some outside APIs.

Actually I saw facedetection API on Programmable Web, and decided that I should do something with it. There was no doubt on selecting programming language. It is Python for sure. As a hosting/framework I've chosen Google App Engine. I see it as a free place for little experiments. I know my service wouldn't need any highscalability, performancability or other cool-abilities. This is just a good place to host a little project, fast to code, fast to deploy. I know that if I place my service on GAE it will be there ~~forever~~ some forseeable future, so I can come back to it, if I want to and show it. There's no need to think about hosting, domain name or something like this - it "just works".

Of course using GAE puts some restrictions and makes you do something in quite wierd way. I'll try to tell about places that made me scratch my nape to think why did Google did it and how to deal with it.

File upload handler. That was one of my first surprises - you need to get special url for POST request from GAE to upload file (see blobstore.create_upload_url()). I have no experience anywhere else, but I can't understand why do I need to do it. At first I placed this call to my view function, that responses with a photo upload page. But soon I realized this is a bad place for it if I want my users to upload more then one photo. The fastest and most obvious way to return on previous page is Back browser button. But because cached page was returned with old, "used" upload url, which you can't use (server raises exception). So I need to update it every time user visits a page. I ended up creating special handler, that only does return of new upload url and requested it in jQuery's ready(). Something like:

$(document).ready(function(){ 
   $.get('/upload_url', function(data) {
      $('div.upload-form > form')[0].action = data });
});

See topic on this issue at tipfy (GAE framework) mailing list for more.

GAE image and blob handling inconsistancy. There are two kinds of model fields for blob-values. One of them is BlobInfo, which is given to you when you process uploaded files. More for that, it is a datastore object itself, so you can get it from the store by key and (most interesting for me) use Picasa infrastructure to serve such images to user. BUT you can't create it manually - it is made only by GAE with uploaded files.

Other field is plain BlobProperty, which only holds a byte-string of something. The main problem is that if you mant somehow modify content of uploaded file and then save it, you can do it only in a plain BlobProperty, no BlobInfo object, no cool, fast, efficient Picasa resources to show your pics. Only reading them from datastore and serving them with an ugly view.

That's all, basically. I'm using web.py + django templates, a which are in a box with GAE. I didn't try to use django ports to GAE, because I don't need them for such small project.

I still have many things to polish and to clean. I'd like to add sharing not only to vkontakte, but to facebook and buzz. But it is already usable. So please, go try it and please tell me what you think about it!

Useful PPA list

2011-01-23T00:12:00.005+02:00

I'm using Ubuntu as my primary OS. It serves me well and I like it. I know some of it's bugs and problems and nice ways to avoid them to live in peace and harmony.

After removing previous version (because it was death slow after my horrible experiments) and installing Ubuntu 10.10 Maverick Meerkat I realised that some of my favourite software is missing. That was because PPA (Personal Package Archive) died together with previous install of Ubuntu. So I had to recollect a list of my daily used software and search for a correct PPA.

So to prevent such failures in future I decided to make a list of useful PPAs:

ppa:docky-core/ppa - is a development PPA for a great nice looking dock, written with Mono, if it means something to you
ppa:ubuntu-on-rails/ppa - a set of plugins, themes and code highlighting schemes for gEdit for RoR development, like TextMate. In fact it suits for Django'ers too, and I was using it for quite a long time. But now a Gemini plugin is broken for me (issue on github) so I'm trying to switch to vim. Process goes slowly and painfully and when it will come to end I'll report it here. This PPA gives you good editor that is both powerful and fast (not to mention monsters like PyDev and PyCharm) to do almost everything you need. With vim I just want some more power and opportunities to shoot myself in a leg :)
ppa:am-monkeyd/nautilus-elementary-ppa patches your Nautilus and makes it sleek and sexy. Many unnecessary elements of it removed, it looks cleaner and more usable. It also contains Gloobus Preview, wich is another nice "MacOS X" kind of app.
ppa:team-xbmc/unstable - fresh version of XBMC - big multimedia combine. (Until now PPA on this position called ppa:team-xbmc/ppa. But they didn't make any updates since Maverick, so I decided to move to unstable XBMC release)
ppa:chromium-daily/ppa - this is what I can't live without. Daily builds of opensource version of Google Chrome - Chromium. It's great to see improvements here and there of the software that is updated every day. The downside - sometimes, when they do really great changes, you can suffer much. It's bleeding edge for sure :)
ppa:ubuntu-wine/ppa - newer version of Wine, that works faster and better.
ppa:ubuntu-desktop/ppa - fresher version of Unity, Ubuntu's default desktop environment. New features and fixes right from developers!
ppa:webupd8team/sublime-text-2 - text editor that I use daily
ppa:mitya57/ppa - PPA which has nice Markdown editor Retext with preview and tabs

There's a few more of PPA in my OS, but I'm not sure what they are doing and do I actually need them :) I guess they were part of my experiments.

To install any of those PPAs type in terminal:
sudo apt-add-repository ppa:ubuntu-wine/ppa
sudo apt-get update

After that you can install software from PPA with
sudo apt-get install ...

But you have to know exact package name of the app in PPA, which is not the name of PPA itself. Chromium package in chromium-daily PPA is chromium-browser. It is much easier to go to Software Center, where all of your PPA are listed and see what you actually need to install.

In the end I want to say that PPA is a handy way to keep your software up to date, see all the new shiny features right away. But quite a big problem with PPA is that they do not support dependencies more then in main Ubuntu repositories. It means it can break your system if you install something from PPA that was made without needed care and love. So watch out!

Tell me other interesting PPAs if you know any!

P.S. As Blogger Stat tells me, some people googles here in searching Opera PPA. I wouldn't comment my attitude to Opera, but I'll try to ease your pain and tell you the answer (taken from UbuntuUpdates) in 3 steps:

1. wget -O - http://deb.opera.com/archive.key | sudo apt-key add -
2. sudo sh -c 'echo "deb http://deb.opera.com/opera/ stable non-free" &gt;&gt; 
/etc/apt/sources.list.d/opera.list'
3. sudo aptitude update &amp;&amp; sudo aptitude install opera

Starting up!

2011-01-22T21:09:00.008+02:00

I was going to create some kind of technical blog for quite a long time. I have livejournal, but that is definitely not the place to write anything IT-specific. But plans are still only plans until you sit and do what you've planned, so now I'm here.

I want to blog in English, though my native languages are Russian and Ukrainian. It will be a challenge for my, but I hope it wouldn't be a problem. Don't be shy to leave comments in any language, I know how to use Google Translate ;)