Straw lab

lab portrait, spring 2012

2012-03-12T00:00:00+01:00

Most of the lab at the Vienna City Hall after the awards ceremony for the WWTF Cognitive Sciences grant.

Back row: Andreas Pöhlmann, Lisa Fenk, Karin Panser. Front row: Andrew Straw (with post-doc in training), John Stowers, Alexander Prochaska, Mareike Forthmann. (Unfortunately Lucile was out of town.)

welcome to the class of 2012

2012-02-19T00:00:00+01:00

At the first of this month, the lab grew substantially with four new members. Lucile Belliveau is doing a project as part of her Master’s degree at the École Normale Supérieure, Paris. Andreas Pöhlmann, recently graduated with an undergraduate degree in Physics from Bayreuth, Germnay, is working as a technician. Dr. Lisa Fenk comes to us to do a postdoc after completing a Ph.D. on spider vision with Prof. Axel Schmid at the University of Vienna. John Stowers is expecting a PhD in electrical engineering and control systems from the University of Canterbury shortly and is working as our resident hardware and software engineer.

getting from the airport to the IMP

2012-01-28T00:00:00+01:00

Vienna's otherwise good public transportation system is largely an insider secret, with little in the way of maps and information en route. Here's what you need to get from the airport to the IMP.

Quick summary: take the S7, get off at the Vienna Bio Center St. Marx stop. But read this for important tips.

From the Vienna International Airport (VIE), there are two trains that you can take into Vienna. The most heavily advertised, more expensive, and nicer is the City Airport Train (CAT). The other is the S7. They both share the same track and route, but have different platforms. To get to the IMP, take the S7, as the CAT bypasses the correct stop.

Unfortunately, it's difficult to find the S7 and figure out what ticket to buy at the airport. To find the S7, you can follow the signs most of the way to the CAT, but rather than turning into the CAT departure area, keep walking and you should soon come to the S7 platform.

Buy tickets either before descending the stairs to the platform or from the ticket machines on the platform. Tickets are somewhat complex to purchase, but purchase a ticket for the destination "Wien" or "Wien Mitte". (The same ticket will work on the U-Bahn.) This should cost 3.60. The S7 departs from the airport at 18 and 48 minutes past the hour, and it takes 19 minutes to come to the St. Marx stop.

The IMP is at Dr.-Bohr-Gasse 7, near the end of Dr.-Bohr-Gasse where is approaches Rennweg. The St. Marx stop is a 200 meter walk away across Rennweg.

View Larger Map

lab portrait, autumn 2011

2011-11-29T00:00:00+01:00

From left to right: Andrew Straw, Mareike Forthmann and Karin Panser.

high-level visual behavior grant funded by the WWTF

2011-11-10T00:00:00+01:00

We just received good news from the Wiener Wissenschafts-, Forschungs- und Technologiefonds (WWTF, in English this translates to the Viennese Science, Research, and Technology Fund). Our grant propsal “Algorithms, neural circuitry, and genetics of high-level visual behavior in the fly” was selected for funding.

Together with the ERC Starting Grant FlyVisualCircuits we received a few months ago, the lab is off to a great start.

augmented reality - computing the OpenGL projection matrix from intrinsic camera parameters

2011-11-05T00:00:00+01:00

Summary

Here I describe how the Hartley-Zisserman (HZ) pinhole camera model differs from the OpenGL display pipeline and how to build an OpenGL projection matrix directly from the intrinsic camera parameter matrix of HZ. What is particular about this exposition is that I do this by calculating the matrix elements directly from the camera parameters rather than calling the glProjection() function. This allows for a more general camera model including pixels with skew. I also include the algebraic derivation (as a sympy script) so you can follow my logic exactly.

Update (30 Jan 2013): The various implementations of the coordinate transform pipeline used in this post are available here.

Summary
Note about image coordinates
The OpenGL projection matrix from HZ intrinsic parameters
Comparison with the OpenCV camera calibration
Experimental verification
- Output of CPU implementations
- Output of OpenGL implementation
Download these files
Remaining work: compensating for camera distortion not described by the pinhole model
References

Note about image coordinates

In both OpenGL window coordinates and HZ image coordinate systems, (0,0) is the lower left corner with X and Y increasing right and up, respectively. In a normal image file, the (0,0) pixel is in the upper left corner. We have code paths to deal with this in one of two ways: first, we can draw our images upside down, so that all the pixel-based coordinate systems are the same. This is the code path used when “window_coords=’y up’“. Second, we can keep the images right side up and modify the projection matrix so that OpenGL will generate window coordinates that compensate for the flipped image coordinates. In this “window_coords=’y down’” path, the generated OpenGL Y window coordinates are (height-y).

The OpenGL projection matrix from HZ intrinsic parameters

Enough of the preliminaries. We calculate the OpenGL Projection matrix when window_coords==’y up’ to be:

[2*K00/width,  -2*K01/width,   (width - 2*K02 + 2*x0)/width,                            0]
[          0, -2*K11/height, (height - 2*K12 + 2*y0)/height,                            0]
[          0,             0, (-zfar - znear)/(zfar - znear), -2*zfar*znear/(zfar - znear)]
[          0,             0,                             -1,                            0]

With window_coords==’y down’, we have:

[2*K00/width, -2*K01/width,    (width - 2*K02 + 2*x0)/width,                            0]
[          0, 2*K11/height, (-height + 2*K12 + 2*y0)/height,                            0]
[          0,            0,  (-zfar - znear)/(zfar - znear), -2*zfar*znear/(zfar - znear)]
[          0,            0,                              -1,                            0]

Where Knm is the (n,m) entry of the 3x3 HZ instrinsic camera calibration matrix K. (K is upper triangular and scaled such that the lower-right entry is one.) Width and height are the size of the camera image, in pixels, and x0 and y0 are the camera image origin and are normally zero. Znear and zfar are the standard OpenGL near and far clipping planes, respectively.

This is the cut-and-past output of our sympy script projection_math.py. The approach is that we enter the operations of OpenGL vertex transformation pipeline and the HZ projection model into sympy, a computer algebra system (CAS). We have sympy solve for the OpenGL projection matrix so that the resulting pixel coordinate is the same for both the HZ camera model and the OpenGL pipeline. See the script for the implementation details. Of course this could be done by hand but is tedious and prone to mistakes.

Comparison with the OpenCV camera calibration

Although I have not directly used OpenCV for camera calibration, their parameterization of the pinhole camera is a subset of the full HZ model. In particular, their matrix A corresponds exactly to the HZ matrix K with pixel skew fixed at zero. Consequently, this page can be directly used with OpenCV camera calibrations by setting K01 to zero.

Experimental verification

To verify that this computation of the OpenGL projection matrix accurately captures the HZ camera model, we have calculated the projection of vertices into image coordinates three ways:

A CPU-based implementation of the HZ camera model. This performs matrix multiplication of the eye coordinates by the intrinsic parameter matrix K.
A CPU-based emulation of the OpenGL pipeline. This performs matrix multiplication of the eye coordinates by the OpenGL projection matrix to produce clip coordinates, transforming those to normalized device coordinates, and then finally using the glViewport parameters to establish the window coordinates.
Direct calls to the OpenGL pipeline, presumably running on your GPU. In this case, we directly load our OpenGL projection matrix by calling glLoadMatrixf() and use OpenGL to perform all vertex transformation.

The first two examples are in the calib_test_numpy.py example and their outputs are overlaid. The third (OpenGL) example is in calib_test_pyglet.py. Each of these three examples gives the same results, suggesting that our formulation is correct. All calculations were done with Python scripts, but the concepts and results should be easily adaptable to any language.

Here is a brief conceptual walkthrough of these programs. Each program starts by loading an image acquired by a (crudely) calibrated camera. This image (luminance.png, included in the zip download below) is of a roughly cylindrical object 1 meter in diameter and 1 meter high. Part of the cylinder is occluded and only the front surface is illuminated. The camera calibration is contained within the file cameramatrix.txt. This calibration is decomposed into the intrinsic camera parameters and the rotation matrix, and the camera translation vector. A simple mathematical model of the cylinder is used to generate world coordinates of many vertices. Each of these world coordinates is transformed to camera coordinates – also called eye coordinates in OpenGL. This is done using the extrinsic camera parameters specifying the camera’s pose and is done either as a matrix multiplication with the extrinsic parameter matrix or by loading them into the OpenGL modelview matrix. From there, each of these vertices in eye coordinate is transformed to window coordinates using the methods described above.

Output of CPU implementations

The blue crosses are the vertices after the HZ transformation. The red points are the vertices after the simulated OpenGL pipeline. Calculations done with numpy and scipy, and plotting done with matplotlib.

Output of OpenGL implementation

The green points are the certices after the OpenGL pipeline. OpenGL called through pyglet.

Download these files

In addition to the individual scripts linked above, all the files are included in this zip file.

Remaining work: compensating for camera distortion not described by the pinhole model

You may want to implement the non-linear warping distortions (radial distortion, tangential distortion) that are often used to extend the pinhole model to be more realistic. A GLSL-based shader implementation of warping and de-warping may be the subject of a future post, but is not described here.

References

A nice page on camera models and OpenGL by Wonwoo Lee
A nice page on the OpenGL projection matrix by Song Ho Ahn
A good description of the pinhole camera model by Yannick Morvan
Another description of the pinhole camera model mostly compatible with the terminology here at Wikipedia.
The OpenGL specifications

welcome to PhD student Mareike Forthmann

2011-10-31T00:00:00+01:00

Mareike Forthmann joined the lab this month as a PhD student. She has a masters degree in biology (a Diplom-Biologe, actually) from LMU Munich. Welcome, Mareike!

lab portrait, summer 2011

2011-08-19T00:00:00+02:00

From left to right: Andrew Straw, Natalia Balcu, Karin Panser and Linda Ma.

upcoming conferences

2011-08-14T00:00:00+02:00

The end of August and September are going to be busy for me, with plans to attend five conferences. I look forward to catching up with colleagues I’ve known a while and meeting new ones. Contact me if you want to meet up.

EuroScipy2011 25-28 August, Paris
JEDI conference 2-4 September, Leysin, Switzerland
EMBO conference 10-13 September, Vienna
Champalimaud Neuroscience Symposium 18-21 September, Lisbon
European Drosophila Research Conference 21-24 September, Lisbon

welcome to summer student Linda Ma

2011-06-24T00:00:00+02:00

Linda Ma is studying at Columbia University and will be spending her summer studying fly neurobiology as part of the VBC Summer School. Welcome!

the Grand Unified Fly (GUF) model

2011-03-23T00:00:00+01:00

Note: This page contains the most important information from the original location at the Dickinson Lab Caltech website (http://dickinson.caltech.edu/Research/Grand_Unified_Fly), which is now offline.

Block diagram of “Grand Unified Fly” and a model fly situated in a tunnel environment under closed-loop visual control.

Overview

We are developing a computer model of Drosophila to investigate the aerodynamics, control, neural processing, and sensory inputs of fly flight. The components of the model are individually useful as research tools but also perform as parts in an integrated whole. For more information, please read the papers above.

Our code consists of the following elements:

Drosophila eye map » Viewing directions (eye map) of each Drosophila ommatidium in the compound eye
fsee » Fly-eye view and neural processing simulation code
fmech » Insect flight mechanics simulator.

References

William B. Dickson, Andrew D. Straw, Michael H. Dickinson (2008) Integrative Model of Drosophila Flight. AIAA Journal 46 (9). doi:10.2514/1.29862 PDF

Epstein M., Waydo, S., Fuller, S.B., Dickson, W., Straw, A., Dickinson, M.H. & Murray, R.M. (2007) Bioligically Inspired Feedback Design for Drosophila Flight. Proceedings of the 2007 American Controls Conference. PDF

Dickson, W.B., Straw, A.D., Poelma, C., & Dickinson, M.H. (2006) An Integrative Model of Insect Flight Control. Proceedings of the 44th AIAA Aerospace Sciences Meeting and Exhibit AIAA-2006-0034 PDF