Professor Ng rules. Concretely.

Permalink | Leave a comment  »

Just have got my Statement of Accomplishment - 95.6%. Better than I expected, worse than I could.

Anyways, at least it was fun. Many thanks to professors Thrun and Norvig and also to all those people who made it possible.

Permalink | Leave a comment  »

Today is my first official work day at BuzzData—a startup company devoted to humanization of data. That means, we facilitate the process of transforming raw data ore into all-powerful fuel of knowledge. We are the context. Although there are newspapers, independent journalists, government agencies, data miners among our clients, BuzzData is perfect for everyone with good will and an interesting dataset. Come and try, it's free.

There was a lot of factors contributing to my decision to join BuzzData, but here is the most important one: an opportunity to join a team of highly skilled professionals and just amazingly friendly people. Also, this is not coincidental that I revived my interest in statistics, optimization, machine learning and related things not so long ago.

The first day was spent on figuring out an algorithm for computing distances in a multidimensional non-isomorphic space. Abstract algebra and all that stuff... Looks like there is a lot of fun there.

I also want to thank Shane Caraveo and David Eaves for heading me up. It wouldn't happen without their support.

Permalink | Leave a comment  »

Today I had an awesome day at the first OpenDataBC hackathon which took place at Mozilla Labs Vancouver.

Tara Gibbs pitched this wonderful idea of consolidating shelter availability data and displaying it on a few window displays, so the homeless people living DTES would not waste their time going from one shelter to another just to find a free spot.

This doesn't solve all the problems of course, but it does solve a little yet very annoying one.

So... At 11:30 we had nothing but an idea. We discussed possible approaches for a while, then came David Eaves and suggested using Twitter as a message queue service.

At approximately 12:00 we still had nothing but a piece of paper covered with boxes and arrows, then we started coding. Tara did the frontend, I was busy hacking the backend and the Twitter stuff.

Four hours later we had a fully functional, production ready system - https://github.com/mikeivanov/vanshelter

How it is supposed to work:

1. Shelters tweet their availability data (they all have internet access)
2. VanShelter monitors -- each of them independently -- receive Twitter updates and
3. Refresh their displays when something changes.

For displays we can use cheap LCD monitors, probably even donated. The software will run on those amazing Raspberry thingies - http://www.raspberrypi.org/, $25 each. This brings the full cost of installing 10 displays down to $250+.

Thank you Tara and David. Also, thank you Jeff and all the people who made this hackathon possible.

Permalink | Leave a comment  »

The point of this rant is to annoy people share my experiences with Ruby.

As strange as it seems, I develop deep emotional relationships with programming languages. I love Python. I truly do. I totally irrationally hate Java (though I'm ok with JVM). 

Now, I like Ruby. 

Well, it has some little warts, but generally it is a very enjoyable language.

I didn't "get" Ruby for quite a long time because I tried to wrap my head around it from the wrong end. Ruby didn't like me, I didn't like Ruby -- it lasted until I realized a very simple thing: Ruby is not "like Perl".

The same happened many years ago with JavaScript. I disliked it so much so couldn't make myself write code in it. I hated it until it occurred to me that JavaScript is not "like Java". JavaScript is a Lisp in disguise. Once I realized that, I stopped worrying and quickly found myself in an intimate, romantic relationship with JavaScript. It is still one of my favourite languages.

Then came Ruby. I had to use it because it was a part of my job. At first Ruby felt like a Perl with broken legs. I tried to make it run and it crawled. I missed one important detail: it didn't have legs at all -- it got wings. Ruby flies.

Here comes my little revelation: Ruby is a Smalltalk. Ruby has much more in common with Smalltalk than with anything else. 

Actually it's a better Smalltalk. Since I realized that, I started to enjoy this beautiful language. Hey, peace and happiness -- welcome back.

Conclusion: it is often our own distorted perception that makes good things look ugly.

Rant mode off.

Permalink | Leave a comment  »

It was a wonderful week. I didn't realize how much I needed to step away from the daily routine. This week was completely spent on (get jealous!):

• sleep
• beer
• football (Lions vs Winnipeg; it was an exciting game, yet a little bit disappointing -- we lost 17:30)
• slacking on the beach
• some hacking (just a little bit)
• meeting some new people (really nice ones)

The stuff I definitely didn't miss:

• nosy office
• commuting to work (2 hours every day)
• stress
• coffee

I'm happy now, at last.

Permalink | Leave a comment  »

This doesn't make any sense to me:

$ irb
ruby-1.9.2-p180 :001 > def nofun(a=nil, b=nil)
ruby-1.9.2-p180 :002?>   puts "a=#{a}, b=#{b}"
ruby-1.9.2-p180 :003?>   end
 => nil 
ruby-1.9.2-p180 :004 > nofun(b="b", a="a")
a=b, b=a
 => nil

Why?..

Permalink | Leave a comment  »

What

This is a very basic Paillier Homomorphic Cryptosystem implemented in pure Python.

The idea is, in short, to encrypt two numbers, perform an "add" operation on cyphertexts, decrypt the result and find it to be the sum of the original plaintext numbers.

How

The code is loosely based on the thep project and a few ActiveState recipes. The code is pure Python and all objects are serializable.

Where

Here: https://github.com/mikeivanov/paillier

Why

I was bored.

Permalink | Leave a comment  »

On my laptop I have two directories:

• ~/activestate, where my work projects reside, and
• ~/me -- for my personal stuff.

The problem is, when I commit changes I want them to be properly attributed. More specifically:

1. everything that belongs to ActiveState should be checked in using my work email
2. all other stuff checks in with my private address
3. I don't want to `git config user.email <...>` each time I clone a new repository ('cause I do it a lot).

Here's what I've done:

• in my home directory I created a file called ~/.gitemail containing just my private email. This address is going to be the default
• to the ~/activestate directory I added another .gitemail with my work address.
• finally, I added this snippet to ~/.bashrc:
  

  alias git='GIT_AUTHOR_EMAIL=$(
        p=$(pwd)
        while [[ $p != "$HOME" ]]; do
          [ -e $p/.gitemail ] && cat $p/.gitemail && break
          p=$(dirname $p)
        done) /usr/bin/git'

The alias scans all the directories up to the home dir looking for a file called .gitemail. When found, it sets the GIT_AUTHOR_EMAIL variable to the file's content. This effectively makes the actual git command use the subtree-specific email. Now, when I'm working e.g. in ~/activestate/stackato, it will automatically pick up my work email from ~/activestate/.gitemail.

No extra efforts, less things to remember, and no history rewriting anymore.

Permalink | Leave a comment  »

Actually, it's very easy. No additional software is required. Just seven easy steps:

1. Attach your USB drive
2. Open the Terminal app (Command-Space, then type "Terminal", hit Enter)
3. Type or copy/paste this command:

   sudo sh -c "mkdir -p /mnt $(mount | grep ntfs | head -n 1 \
      | awk '{ print "&& umount " $3 " && mount_ntfs -o nosuid,rw " $1 " /mnt" }')"

4. Locate your drive in Finder
5. Drag/drop files there
6. Unmount the drive as usual
7. DONE!

The command breakdown, if you're interested:

1. mkdir -p /mnt creates a mount point -- a place in the file system where we will attach the drive
2. the mount command without parameters gives you a list of the currently attached drives
3. grep ntfs filters non-ntfs drives out the list
4. head -n 1 grabs the first line (we're assuming only one ntfs drive can be attached at a time)
5. the awk part produces two commands:
   • umount /Volumes/<name> -- unmounts the drive from its original place
   • mount_ntfs -o nosuid,rw /dev/<device> /mnt -- mounts the drive again, but this time in the read-write mode
6. now, the sudo sh -c "..." thing allows code execution with superuser privileges.

That's it.

Permalink | Leave a comment  »

Recently I started looking for an opportunity to refresh my math/stats skills. I din't do this stuff for a long time, probably more than a decade; so I wanted something really basic for a refresh. Then I discovered this Think Stats book, which I'm going through right now. I find the book useful and interesting. 

To add more fun, I decided to do all the exercises from the book in Clojure instead of Python. This indeed has turned out to be a good adventure. 

My worksheets (which is a work in progress) are on Github just in case it could be it useful for someone. The project is here - https://github.com/mikeivanov/thinkstats.

Permalink | Leave a comment  »

Just have finished transferring my stuff to Posterous.

I like it a lot. The UI is much simpler and cleaner, everything I need is at hand. The TOS is much more straightforward. I like it understands Markdown. It just feels right.

Permalink | Leave a comment  »

Context:

a freshly installed Emacs on Mac OS X.

Itch:

the PATH is the default system path, not that one you get in your terminal:

~ $ echo $PATH
/usr/bin:/bin:/usr/sbin:/sbin
~ $ bash -l -c "echo \$PATH"
/Users/mike/.rvm/gems/ruby-1.9.2-p180/bin:
/Users/mike/.rvm/gems/ruby-1.9.2-p180@global/bin:
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/bin:/Users/mike/.rvm/bin:
/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:
/usr/X11/bin:/opt/local/bin:/usr/local/cuda/bin:/Users/mike/bin
~ $ rvm
rvm: command not found

Not good.

Scratch:

Now here's the choice: either you fiddle with .MacOSX/environment.plist or use a quick and dirty sub-shell hack. I chose the latter for the sake of not touching XML. The idea is to call bash directly from Emacs and ask it for the right PATH:

(setenv "PATH" (shell-command-to-string "/bin/bash -l -c 'echo -n $PATH'"))

Add the line above to your ~/.emacs and the next time you start eshell you will see:

~ $ echo $PATH
/Users/mike/.rvm/gems/ruby-1.9.2-p180/bin:
/Users/mike/.rvm/gems/ruby-1.9.2-p180@global/bin:
/Users/mike/.rvm/rubies/ruby-1.9.2-p180/bin:/Users/mike/.rvm/bin:
/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:
/usr/X11/bin:/opt/local/bin:/usr/local/cuda/bin:/Users/mike/bin
~ $ rvm
= rvm
* http://github.com/wayneeseguin/rvm
...........
blah blah blah...
............
limitations under the License.

Schnell und lustig!

Permalink | Leave a comment  »

A quote that very closely reflects my own feelings: "[It] feels like a minor miracle. It’s an astoundingly high-quality language, sure—in fact, I’m beginning to think it’s the best I’ve ever seen—yet somehow it has still managed to be fashionable. That’s quite a trick. It gives me renewed hope for the overall future of productivity in our industry." -- Steve Yegge in his foreword to "Joy of Clojure".

Permalink | Leave a comment  »

Now it is official.

Permalink | Leave a comment  »

Emacs lisp has no Tail Call Optimization (TCO), neither do many other lisp dialects. The lack of TCO is not a big deal--it's always possible to transform a tail recursive algorithm into a cycle. However, it makes procedures uglier. Here is a very simple method of enabling Clojure-style tail call recursion in Emacs lisp:

;; A very simple linearized Y combinator.
;; All the state management stuff is incapsulated here.
;; Don't call it directly.
(defun rloop- (body &rest args)
  (let ((res nil))
    (while (progn
             ;; here's the idea: we keep calling body 
             ;; while it returns the recursion marker
             (setq res (apply body args))
             (when (and (consp res)
                        (eq :loop-recur-marker (car res)))
               (progn (setq args (cdr res))
                      t))))
    res))

;; Recursion marker factory
(defun recur (&rest args)
  ;; instead of a real recursive call,
  ;; just signal an intention to make one
  (cons :loop-recur-marker args))

;; The form macro
(defmacro rloop (init body)
  (let ((args (mapcar 'car init)))
    ;; a little courtesy to the macro users
    `(let* ,init
       ;; make a lambda from the body and pass it 
       ;; to the combinator function
       (rloop- (function (lambda (,@args) ,body))
               ,@args))))

Here's how to use it:

(defun factorial (x)
  ;; this is the recursion entry point
  (rloop ((x   x) 
          (acc 1))
         (if (< x 1)
             acc ;; done, just return the result
           ;; not done, start the whole rloop block again
           (recur (1- x) 
                  (* x acc)))))

ELISP> (factorial 10)
3628800

The funny part is defun is not necessary. You can have as many sequential inlined rloops as you want. I like this approach a lot: all the state management stuff is off the sight. The procedure is almost identical to the underlying algorithm. Another classic example:

(defun fibo (x)
  (rloop ((x    x)
          (curr 0)
          (next 1))
         (if (= x 0)
             curr
           (recur (1- x) 
                   next 
                  (+ curr next)))))

ELISP> (fibo 10)
55

Nice, eh? Of course, this kind of beauty comes with a price. Here is how the rloop macro expands:

ELISP> (macroexpand '(rloop ((n 0)) (if (> n 5) n (recur (1+ n)))))

(let*
    ((n 0))
  (rloop-
   #'(lambda
       (n)
       (if
           (> n 5)
           n
         (recur
          (1+ n))))
   n))

...which means two extra function calls on each iteration. But realistically, it's not such a big deal. Clarity of the code is way more important.

Permalink | Leave a comment  »

Binary PostgreSQL database modules (psycopg2) are available for MacOS X, Windows and Linux platforms in both 32 and 64 bit versions. The modules are available through PyPM package manager for ActivePython users.

We compiled all the modules, so you don't have to. We statically linked all the dependency libraries (where it was appropriate), so you don't have to build those either. SSL is fully supported on all platforms.

By the way, it's not just about PostgreSQL -- MySQL, Oracle and MS SQL Server modules are also there.

We have even got ODBC drivers for those who lost their hope.

Enjoy.

Permalink | Leave a comment  »

The third part of my Python crypto saga is on the ActiveState blog.

Cryptlib: Peter Gutmann’s cryptlib library -- a powerful security toolkit that allows even inexperienced crypto programmers to easily add encryption and authentication services to their software. The high-level interface provides anyone with the ability to add strong security capabilities to an application in as little as half an hour, without needing to know any of the low-level details that make the encryption or authentication work...

Read the whole article here: http://blogs.activestate.com/2010/05/python-crypto-state-of-the-art-part-3/

Permalink | Leave a comment  »

Cryptlib is a totally fascinating cross-platform crypto library with Python bindings. Unfortunately, the current version (3.3.3) is unusable on 64-bit platforms. The good news though it is easily fixable. Let's look what's going on there:

$ python
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> from cryptlib_py import *
>>> cryptInit()
>>> sess = cryptCreateSession(CRYPT_UNUSED, CRYPT_SESSION_SSH)
>>> sess.CRYPT_SESSINFO_SERVER_NAME = "myserver.com"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 32, in __setattr__
cryptlib_py.CryptException: (-2, 'Bad argument, parameter 2')
>>>

No good. The problem is the way PyObject_AsWriteBuffer (and PyObject_AsCharBuffer) is called. The last parameter is declared as a pointer to Py_ssize_t type, which can be either 32 or 64 bit depending on the platform. Here is the actual declaration in the abstract.h file:

PyAPI_FUNC(int) PyObject_AsWriteBuffer(PyObject *obj,
                      void **buffer,
                      Py_ssize_t *buffer_len);

Now, here is how this function is used in cryptlib_py (python.c, line 21):

if (PyObject_AsWriteBuffer(objPtr, bytesPtrPtr, lengthPtr) == -1)

Where lengthPtr is declared as "int* lengthPtr". See the problem? This is what happens: the function expects a pointer to a 8-byte blob but instead it is provided with a pointer to a 4-byte int. Being unaware of that, the function smashes the variable next to the one pointed by lengthPtr. The solution is to patch cryptlibConverter.py, the script generating bindings/python.c. Although this problem is very likely to be fixed in the next cryptlib release, here is what you do if you can't wait:

mkdir cryptlib; cd cryptlib
curl -O ftp://ftp.franken.de/pub/crypt/cryptlib/cl333.zip
unzip cl333.zip
curl -O http://mikeivanov.com/pc/cryptlibConverter.py.patch
patch -p0 < cryptlibConverter.py.patch
python tools/cryptlibConverter.py cryptlib.h bindings python
make
cd bindings
python setup.py build
sudo python setup.py install

Fixed! The patched version is working just fine:

$ python
Python 2.6.1 (r261:67515, Feb 11 2010, 00:51:29) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> from cryptlib_py import *
>>> cryptInit()
>>> sess = cryptCreateSession(CRYPT_UNUSED, CRYPT_SESSION_SSH)
>>> sess.CRYPT_SESSINFO_SERVER_NAME = "myserver.com"
>>> sess.CRYPT_SESSINFO_USERNAME = "mike"
>>> sess.CRYPT_SESSINFO_PASSWORD = raw_input("pwd=")
pwd=mypassword
>>> sess.CRYPT_SESSINFO_ACTIVE = 1
>>> data = array('c', '\0' * 1024)
>>> recv = cryptPopData(sess, data, 1024)
>>> print data.tostring()[:recv]
Linux myserver.com 2.6 XXXXXXXXX x86_64 GNU/Linux
Ubuntu 10.04 LTS
.......
>>> cryptPushData(sess, "uptime\n")
7
>>> cryptFlushData(sess)
>>> recv = cryptPopData(sess, data, 1024)
>>> print data.tostring()[:recv]
uptime
 18:17:15 up 6 days,  1:54, 13 users,  load average: 0.30, 0.27, 0.26
mike@myserver:~$ 
>>>

Permalink | Leave a comment  »

In a post titled "A Trusted Cloud Entropy Authority" Reuven Cohen writes:

"...maybe there an opportunity to create a trusted cloud authority to provide signed verified and certified entropy. Think of it like a certificate authority (CA) but for chaos. Actually, Amazon Web Service itself could act as this entropy authority via a simple encrypted web service call. I even have a name for it, Simple Entropy Service (SES)."

This is really a good idea. Amazon should have provided such a service long time ago.

When an SSL connection is being established, a browser and a server perform the Handshake protocol. This protocol involves exchanging random bits between the parties. The important thing is that security depends on how random those bits are. If they are not, the connection is effectively insecure.

In the case of AWS, there is no source of true randomness, therefore SSL on AWS is inherently insecure. Moreover, instances running on the same physical machine can affect each other's security by draining the shared random pool in the host system.

Further he writes:

"a website called http://random.org [is] a true random number service that generates randomness via atmospheric noise. Looks cool, maybe this may help solve the problem."

I think that random.org is not a good choice for several reasons.

One problem is a connection to such a service. It should be as secure as the most secure secret handled on your system. If the random bit connection is encrypted with 256 bit AES (and it actually is), this is the highest level of security your system can provide. Plus, there should be guarantee that no unencryped random bits are stored anywhere. The same is true for the proposed SAS service, too.

Another problem with random.org is... well, randomness is perceptive. What you see as "random" can be quite deterministic to the people who run the random.org service. Even though they might not store anything, their present is your future--just think about relativistic effects. A temptation to tamper with someone's future can be, you know, very strong.

The overall quality of the service is not known. There is no guarantee it is random at all. A quote from their FAQ: "Q1.2: Is the source code for the generator available? -- Not currently, no. Maybe I'll make it available as open source some day."

Even though the Whois database indicates the domain name's registrant is located in France, the SSL certificate owner is not specified. I have no reasons for not believing the guy running the service, but I would not entrust my customers' data into a total stranger's hands, even though he or she seems to be a nice person.

So the conclusion is: while there is no trusted entropy generator on the AWS side, we, the AWS customers, are on our own.

Here is a hint: entropy seeds can be generated in-house and smuggled into instances over a secure channel. Then those seeds could be fed to a cryptographically secure RNG like Isaac to produce actual "random" bits. I think there should be a way of injecting those into the instance's random pool.

Permalink | Leave a comment  »