Alex Ott's blog

2012th in overview...

2012-12-31T15:57:00.001+01:00

It was relatively quiet year. It started with 2 weeks vacation in Dominican Republic - it was interesting travel, I and my wife enjoyed it a lot, and want to make it again, may be next year.
Not so much happened on work, most of tasks were not so challenging as in 2011th, although I tried to do some experimenting staff, trying to apply machine learning & related knowledge to my area of expertise. But no of these experiments went to production (I hope, that they will, but who knows when this will happen). I also participated in Microsoft's plugfest - conference about file format & protocols - it was very useful for me, and as usual, most of useful information was obtained during conversations with developers.
I continued to self-study - read a lot of books on different topics: machine learning, natural language processing, big data, software development practices, etc. (you can follow me on Goodreads if you're interested). But pile of books with label 'to-read' is still too big.
I also took several courses from Coursera: Natural Language Processing was very interesting, and right now, I'm doing Heterogenous Parallel Programming - to refresh GPGPU programming skills.
This year I did 2 presentations about Clojure - one for functional programming user group in Bielefeld, and second was done as part of small Russian-speaking conference ITSea-2012, that was combined with vacation in Montenegro.
Tried to participate in different open source projects, mostly in CEDET & Incanter, and some other. And I hope, that I'll have enough time to continue this work, especially for CEDET.
Cycling is still my favorite sport, although I started cycling season relatively late this year, due operation on my right hand, so I managed to ride only 2400 km (last trip was 2 days ago :-)

I want to wish Happy New Year to all my readers, and see you next year!

New version of article about Emacs/CEDET

2012-10-30T19:19:00.003+01:00

I just uploaded to site the new version of my article A Gentle introduction to CEDET. I also left old version of this article, but as separate page.
New version describes CEDET with new activation scheme, so it's now applicable to versions from bzr, and to versions bundled with GNU Emacs (after they will release next GNU Emacs version, where CEDET was updated).
Besides this, I added small description of how to customize CEDET to work with Java, and small section about setting name completion through the auto-complete package.

Instead of my config file, that contains too much not necessary stuff, now it's better to use separate config.

P.S. btw, fresh snapshots of CEDET can automatically detect Maven projects, and get classpath information directly from them. So now, names completion works also for 3rd party libraries. For example, this is name completion during work with source code from Apache Tika:

Getting started with examples from "Mahout in Action"

2012-07-24T10:59:00.000+01:00

I decided to write this post because I saw several similar questions on how to start to work with examples from "Mahout in Action" book (I was technical proofreader for it, and familiar with examples ;-).

Preparations

Complete source code of examples from book is available at separate repository at Github, together with short instruction on how to use them. Please, note that book was written & tested for Mahout 0.5 - stable release, that existed at time of publishing, and master branch in repository contains code for this version. There are also separate branches for code that was modified to work with Mahout versions 0.6 and 0.7 - they are named accordingly.To obtain code, you can use either Git, or use Github's "download source" functionality. Here are links for all existing versions: 0.5 (master), 0.6, 0.7 - download and unpack archives to some location.
To work with examples, you need to have Apache Maven installed (it's better to install it from repository on Mac OS X or Linux systems). Maven is used to compile source code and to create packages. Maven project could be also imported into your favorite Java IDE - Eclipse, Netbeans, or Idea (I will explain how to use Eclipse, but for other IDEs the process is similar). To use Maven with Eclipse, you need to have m2eclipse plugin installed - it will provide import and build functionality.
To run examples from chapter 16, you'll also need to have Apache Zookeeper installed - see instructions in README file in repository - they're pretty detailed.
You also need to download Mahout distribution to run some examples (usually they involve execution of mahout script). Download file mahout-distribution-<version>.tar.gz and unpack it. You can also download file mahout-distribution-<version>-src.tar.gz, although this isn't necessary (it contains Mahout's source code).
I just want to mention, that Mahout is works best on Unix-based systems -- all examples were tested on Mac OS X & Linux. This also applied to Hadoop, so if you're using Windows, it could be better to install Linux in virtual machine and use it for all work.

Build examples

To be able to run examples, you need to build packages (jar files). From directory where source code for examples is located (you should have file pom.xml in this directory) execute following command:
mvn package
It will compile source code and create packages. Compiled packages are stored in the target directory. There are several files created:

mia-<version>.jar contains only examples, to run them you need to specify all dependencies;
mia-<version>-jar-with-dependencies.jar contains examples plus all dependencies - this jar could be run without specifying additional classpath elements;
mia-<version>-job.jar contains examples plus all dependencies, excluding Hadoop -- it should be used for Hadoop jobs.

Use corresponding packages when book refers to them.

Import of example's source code into Eclipse

Import of code into Eclipse is very easy - go to menu File, select Import... item, and then unfold Maven and select Existing Maven Projects from list and press Next. Eclipse will ask you where source code is located - point to directory where you unpacked examples - Eclipse will analyze pom.xml and will display string like: /pom.xml com.manning:mia:0.5:jar, you can press Finish after that.
After import, project will be opened in Eclipse, and you can look into source code, modify examples if you need, and execute them (see below).
If you need, you can also import source code of Mahout itself into Eclipse, the procedure is similar, but this may work for all releases - in some cases, it will give you error that some plugins aren't covered by m2eclipse - you can select Ignore item in Quick fix menu (when you click right mouse button).

How to run examples

You can run examples either from command line, or directly from Eclipse.

Run from Eclipse

To run example from Eclipse, select needed class from browser on left, click right mouse button on it, select Run as..., and from sub-menu, select Java Application.
Take into account, that some classes need to have additional parameters specified - you can customize this by selecting Run configurations item from Run as... sub-menu.
For example, code from chapter 2, expects that file intro.csv is located in current directory (top of the project), while it's located together with source code, so execution without explicit configuration will lead to error. To fix this problem you need to specify that working directory for these examples is in non-default place - go to Run configurations, and select Arguments tab in dialog window. Then change Working directory parameter from Default to Other, press Workspace... button, and select src/main/java/mia/recommender/ch02 directory from tree view. After that you can press Run button, and your example will be executed without error.

Run from command-line

You can run examples from command line either by using java directly, or by using Maven's exec plugin.
To run examples with java, you need to specify package with all dependencies in classpath, and specify class name to execute, like this:
   java -cp target/mia-0.5-jar-with-dependencies.jar mia.recommender.ch02.IREvaluatorIntro
But to run like this, you need to have package recompiled if you did some changes. From this perspective, Maven's exec plugin is more handy - it automatically recompile changed code, and executes it without packaging everything once again. To execute you class with need to issue following command (for this example, you need to copy intro.csv file to top-level directory, or it will fail):
   mvn exec:java -Dexec.mainClass="mia.recommender.ch02.IREvaluatorIntro"
If your class accepts command-line parameters, then you can specify them using exec.args parameter of plugin:
   mvn exec:java -Dexec.mainClass="mia.recommender.ch02.IREvaluatorIntro" -Dexec.args="src"

Conclusion

So, I hope, that this article helped you to start with Mahout in Action examples. Most of examples should work as described here, but some requires more work, but you can find instructions for them in the README file in source code repository.
If you're still having questions, then I try to answer them ;-)

Experience with new mouse

2012-06-18T12:33:00.000+01:00

I had some problems with my right hand at the start of this year, so I decided to remove a part of load from it. I started to use left-handed mouse and used it for several months (it's not always possible to work completely without mouse, especially in Windows).
Week ago I got a new mouse as present - Evoluent VerticalMouse 4 (for left hand), and used it for a week at home, while used old mouse at work. And I can say, that new mouse is much more comfortable for me - hand is comfortably placed on it, there is separate middle button (instead of less comfortable wheel click, although it also works). There are also additional buttons, that could be programmed to do something useful. Mouse works fine on all tested OSes (Mac, Linux).
It was so comfortable, so I ordered the same mouse for work and already replaced my old mouse with it.

ECB & fresh Emacs/CEDET...

2012-06-17T14:18:00.000+01:00

I already twitted about this, and also wrote to ECB & CEDET mailing lists, but I also want to reach Planet Emacs readers :-)
I made small changes in the ECB code that allow to use it together with fresh Emacs & CEDET versions. Modified code is available in my github. I tried this version together with CEDET from trunk, and also with CEDET from Emacs 24.1, and it worked for me.
If you're using ECB, please try this modified version, and leave feedback (either here, or by sending e-mail to me or to ECB mailing list). If you'll find bugs, feel free to file a bug using github's issue tracker.

Jenkins + CMake/CTest

2012-03-23T12:42:00.002+01:00

Some time ago I setup Jenkins CI to compile & test our code. Installation itself was straightforward, matched to documentation on official site (besides official documentation, there is pretty good book "Jenkins: The Definitive Guide"). The only problem was that it didn't work with OpenJDK, so I was need to install Sun JDK.
For C++ components we're using CMake. Out of box, Jenkins has no support for CMake, but there is plugin in repository, that you can install separately. This plugin allows to specify different options for CMake, and will perform compilation and installation of code. (Although, at the end, I setup configuration that explicitly run CMake with needed options, and perform compilation via GNU Make with some additional options). The main problem with CMake (CTest really) was that tests results are in format, that isn't acceptable by default test analyzer that is designed to work with JUnit logs. Plugin repository has xUnit plugin, that should work with other test frameworks, but it also doesn't support CTest log format. Solution was found on StackOverflow - we need to write custom XSL that will convert CTest results into JUnit format. I found stylesheet at StackOverflow and slightly modified it. So, build step for testing now looks following way:

cd Debug

TRES=0

ctest -T test --no-compress-output || true

if [ -f Testing/TAG ] ; then

xsltproc /var/lib/jenkins/userContent/xunit/CMake/2.8/ctest2junix.xsl Testing/`head -n 1 < Testing/TAG`/Test.xml > CTestResults.xml

I was need to rewrite 3rd line this way, because Jenkins aborted job if ctest returned non-0 result if one of tests failed, so other commands weren't executed, and we had no testing results in dashboard.
So now, all our components are compiled & tested immediately after check-in, and we're able to catch error early.

Update: last line will cause build be marked as broken if any test is failing. You can remove it, and in this case, Jenkins will mark build as unstable if any test is failing - it will take this information from tests results

Year's results...

2011-12-30T14:35:00.001+01:00

I think, that today is appropriate time to look back onto what happened in 2011th, and to think, what to do in next year. Many things happened in 2011th...

I got promotion to principal engineer, so some new activities were added to my usual duties, I started to work more closely with other teams around the world.
Traveled a bit - was in England on business trip, together with wife were 3rd time on Canary Islands, spent a week traveling in England, Holland, plus did some trips in Germany (Cologne, Rhein, etc.).
Rode 2000km on my bicycle, and want to make more next year...
Participated in release of several books in different roles: as technical proofreader for Mannings "Mahout in Action" & "Tika in Action", and as translator for Russian translation of famous "Types and Programming Languages", plus I helped slightly in translation/proofing of more functional programming-related books that will be released next year (in Russia).
Open-source activity wasn't so great as before, because of lack of free time, usually did small patches for different projects - muse, haskell-mode, mahout, CEDET, incanter, and several more.
Tried to continue to write articles, so at begin of year, I wrote article about TDD & Unit testing in C++, and at the end of year, wrote (together with Dmitry Bushenko) small tutorial (in Russian) on how to extend Emacs for refactoring of code. We plan to extend this article with more examples, including more tools from CEDET/Semantic, and than translate it to English. Also gave a talk about Emacs for Scala.by user group (via Google+ Hangout)...
Read a lot, mostly technical books...
And one of the most interesting experiences was participation in Stanford's University experiment - online classes on Artificial Intelligence (AI) and Machine Learning (ML). It was very interesting to study there, and they teach me many new interesting things + plus I had a chance to read some books that were bought some time ago, but simply stayed on my bookself, for example, Artificial Intelligence: A Modern Approach. I got only 89% overall score on AI class (I need to check my homeworks more carefully ;-), but anyway it's very interesting experience. ML class was more "easy" for me, but it also was interesting and allowed me to study many new things. I hadn't received my certificate for ML class yet, but I hope that it will be good - I did all homeworks & programming tasks... (If you hand't participated in that courses, I recommend to look onto actual list of cources at ML Class site, maybe you'll find something interesting for you - they will offer much more courses in January-March!).

What's next? I plan to investigate topics that are interesting for me - natural language processing (including study at Stanford's NLP class), machine learning, etc. Plan to participate more actively in open source projects, especially in Clojure-related. Continue to read new books (I already prepared pile of books for reading on vacation). Will try to make my personal record in cycling (I plan to make 3000km next year :-). And do many other things...

And now, I want to wish Happy New Year to everybody!

Planet Clojure's milestone

2011-11-02T21:26:00.000+01:00

Today, Planet Clojure reached another milestone, and now combines feeds from more than 300 blogs (301, precisely).

P.S. I have a question - does anybody interested in Planet Clojure in other languages than English? For example, I maintain Russian Planet Clojure separately, but maybe it will be better to combine all planets in one place, as subdomains of Planet Clojure?

Miscellaneous...

2011-11-01T16:53:00.000+01:00

I hadn't blogged since May, and many things happened since that time.

In June, together with my wife we traveled to England and spent a week, visiting different cities. Although we visited not so much, so we're planning to make another trip to Scotland, and maybe other parts of UK.

Cycling was also big time consumer (this year I made about 2,000km on bicycle), but I like this activity, and planning to upgrade my bicycle next year to something speedy - don't know yet what to select - road bike or time trial bike...

Book-related

As usual, I read a lot, investigating new (for me) technologies & best practices.
Together with Manning's team, I participated (as technical proofreader) in publishing of two books: Mahout in Action, and Tika in Action. First book was very interesting for me, as I planned to study this technology anyway, and work with Manning allowed me to dig more deeply into all examples, code, etc. This book is different from other machine learning books because it shows how to solve ML tasks with Mahout in practice.
Week ago another book-related project also was finished - Russian translation of well-known book on type theory - Types and Programming Languages by Benjamin C. Pierce. This project lasted for 3 years, but at the end we produced high-quality translation of this book (and it's free in ebook form!). I hope that I'll receive paper book shortly.
And more projects are in progress ;-)

Programming & other IT-related projects

I also continued to experiment with different technologies to get new experience (not related to work) - Mahout, Lucene, Hadoop, Erlang, Haskell, and many other.
Some work also was made with Clojure, but not so much, I hope that I'll have more free time at winter.
Also I participate in AI & ML classes from Stanford, and like them a lot, although not so much free time left. First is more hard, as it requires to do more study on your own, seeking for new information, etc. (I forgot many things since I finished university :-) While second is more self-contained - you're usually don't need to search much in external sources to make your homework. Some time was needed to read Octave manual, but this was not so complicated. Although now I'm also studying how to program with R, to help my wife with thesis.

More things happened & happening, than I wrote about, and I'll try to blog more often.

Emacs/Cedet article in Belorussian

2011-07-18T18:03:00.001+01:00

My Emacs/Cedet article now has translation to Belorussian...

Test, please ignore...

2011-05-31T06:07:00.001+01:00

test for clojure planet

Small books review

2011-05-14T18:10:00.000+01:00

I hadn't blogged for a long time, and also wanted to blog about books, read during last 2 months, but currently I have no time - several releases straight at work, plus there are several personal projects, so I decided, that I'll write only about very interesting books. You can find reviews of my read books on Goodreads.

Shortly about read books

During my vacation in March, I finished The Art of the Metabobject Protocol, that was in my reading queue for a long time. This book gives understanding how CLOS was designed, how concrete solutions was selected, etc.
In April I read 2 books from Manning (they asked for review): Tika in Action и ManifoldCF in Action. First wasn't very interesting for me - I'm working in this branch for a long time, plus I already used tika in my projects, but I think, that book will interesting for peoples who will need to extract meta-data & text from files in different formats. And second book was very interesting - I bought it when it was released as MEAP, but I hadn't time to start read it. This book describes ManifoldCF - extensible framework for work with different content repositories, from which you can fetch information for indexing, or similar processing. This framework is very interesting for me, and I hope, that I'll use it in one of my projects.

Advertising in the "Practice of functional programming" journal

2011-04-06T09:27:00.000+01:00

If you want to hire talented software developers in Russia, or other countries of ex-USSR, then you can publish your ad in the "Practice of functional programming" journal. Currently journal has about 10 thousand readers across almost all parts of ex-USSR, and I believe, that they are talented people.
To get more details on pricing, etc. you need to write to following e-mail address.

Unit testing in C++

2011-02-15T10:21:00.000+01:00

I forgot to mention here, that I wrote small article on unit testing in C++ using Boost.Test and Google C++ Mocking Framework. I have plans on extending this article, so comments and suggestions could be very useful

Readings digest. January 2011

2011-02-11T17:03:00.000+01:00

Maybe you noticed that there was no books review for December — I had too much to do, so I hadn't finished reading of any book. But Christmas and New Year brought me more free time, and I finished reading of several books.

DSLs in Action
Camel in Action
Test-driven development: By Example

DSLs in Action

Sometime ago I decided, that I need to read something about domain specific languages (DSL) to find new ideas, compare with my own implementation (I developed several DSLs, that I use in my projects), and maybe improve my solutions. I selected DSLs in Action, written by Debasish Ghosh, who is well-known developer in Scala community, although he is also using other languages, for example, Haskell, Groovy, Clojure. When I was trying to select which book to read, I also thought about Domain-Specific Languages book, written by Martin Fowler, but Manning offered good discount on books, so I bought their book ;-)

This book is written using very simple and clear language, but it provides very interesting information. Author uses in examples different languages — Ruby, Groovy, Scala, Clojure (in this book the only languages that are working on JVM are used, so some things are applicable only to this platform). This allows to demonstrate different approaches to design and development of DSLs (you can also read author's post on why this book is polyglotic).

First part of book is dedicated to DSLs basics — author begins with description what is DSL, what parts it consists from, which types of DSLs are exist, how they are executed, benefits and drawbacks of using DSLs in your projects. Separate section describes how to model specific domains, and how this affect DSLs structure.

Second chapter demonstrates simple DSL, built in Java, and analysis of its drawbacks is performed. In the next section, the similar DSL is created with Groovy, and its benefits are described, comparing with first implementation. In the rest of this chapter the DSLs implementation patterns are reviewed.

The third chapter is dedicated to description of how to integrated DSLs into your project. It describes java scripting engine, Spring-based integration, and some other. For each of approaches, benefits and drawbacks are analyzed, including how it could affect DSL's implementation, and related questions.

The second part (chapters 4-8) concentrates on DSLs implementation using different programming languages. 4th chapter describes common patterns in implementation of internal DSLs — meta-programming, typed abstractions, run-time code generation, etc. And next 2 chapters (5 and 6) demonstrate this with DSLs implementation in Ruby, Groovy, Clojure, and Scala.

Seventh chapter continues DSLs implementation theme with description of how to implement external DSLs. The different types of parsers of reviewed, that could be used to parse DSL's code, and small example of ANTLR's usage is provided. There is also small section on Xtext — Eclipse's framework, that simplifies development of external DSLs.

In 8th chapter, the use of parsers combinators is demonstrated with application to development of external DSLs. Chapter begins with small introduction to parsers combinators, and continues with demonstration of how Scala's parsers library could be used to create external DSL.

The last part of the book provides some thoughts on modern tendencies in DSLs development, support tools, etc.

Besides main material, the book has several appendixes that contain short descriptions of programming languages, that were used in book, so you can get familiar with them, if you hadn't used them yet. Separately, I want to mention "philosophical" appendix A, that dedicated to discussion of role of abstractions in domain modeling, how purity and lack of side effects affect development, and some related problems. Besides this, you can also look on appendix B, that discusses role of meta-programming in DSL's development.

Conclusion: if you're using and/or developing DSLs, or you're interested in this topic, then you need to read this book — it contains a lot of practical information, that could be useful in DSLs development.

Camel in Action

For one of my "pet" projects I was need to create data processing system that will get and process data from different sources, so, after some googling, I found Apache Camel, that was looked pretty interesting and mature project, so I bought at Manning the Camel in Action books, written by Claus Ibsen and Jonathan Anstey.

The book begins (chapter 1) with introduction into Apache Camel — which tasks are solved with it, which components it consists from, etc. The second chapter provides more detailed description of routes, and how you can create them using code in Java or Spring.

The second part of book describes basic Camel's techniques — how to convert data, handle errors, test code, use additional components (such as JMS, databases, etc.).

And third parts provides description of how to work with transactions inside Camel's workflow, how to process data concurrently. There are also chapters on deployment of projects, that are using Camel, and how to monitor Camel in production environment.

Separate chapter (11th) describes development process for projects that are using Camel, including development of new adapters and components. Besides this, there is description of DSL, written in Scala (although, from my point of view, the scalaz-camel project looks more interesting), and also shown how to use Camel with other languages.

Conclusion: if you interested in Apache Camel's usage, then you can acquire this book — it contain pretty good introduction to this system, and could be used as reference to some of the base components.

P.S. The new project was also created to integrate Clojure with Apache Camel. Project is called Hackamore, and if you interested in it, then you can join to discussions in specially created mailing list.

Test-driven development: By Example

Several days ago I made a presentation (for our internal developers seminar) on test driven development (TDD) and unit testing, and during preparation, I decided to read something from peoples, who were founders of this movement. The Extreme Programming series I read many years ago, right after release of corresponding books, so I decided to read the Test Driven Development: By Example book by Kent Beck.

Book isn't big (about 200 pages), but I think, that it still too big for this topic — author took small problem (exchange conversion), and describes test-driven implementation of solution in great details. This example is used in the first part of the book to demonstrate basics of TDD and unit testing.

In the second part, TDD is illustrated by implementing of xUnit-like testing framework for Python language, with detailed description of all necessary steps.

The third part of the book is dedicated to description of base approaches and patterns in test-driven development, how to design code for testability, how to design tests, etc.

Conclusion: this is pretty good introduction to TDD, but I could recommend it only for people who never used TDD in their work — it's too elementary from my point of view. Although, there are some good advices on code organization, workflow, etc. But if you know something about unit testing, then you can find other sources of information.

Additions to haskell-mode

2011-02-01T21:39:00.000+01:00

Between Christmas and New Year I had some time to hack haskell-mode (this was planned for a long time) - I added support for hlint (the hs-lint command) and haskell style scanner (the hs-scan command). As before, hs-lint supports replacement of code with suggestions.
Patches were sent to maintainer, but they not committed into main repository yet. But in the meantime you can use my fork. Please, send me comments and suggestions on these changes...

The 2010th summary...

2010-12-31T12:58:00.000+01:00

This year was very interesting - were many interesting projects, both on work, and personal.
I continued to "play" with Clojure, that lead to writing of big article about it (in Russian) for "Practice of functional programming" journal. I also gave several talks about Clojure at MarginCon 2010, at meeting of Munster Java User Group, and at golodnyj's podcast (1, 2). And I also participated in development of several projects (Incanter, leiningen, swank-clojure, clojure-hadoop, and pair of my own). Although I had no time for Common Lisp & Scheme, but I hope, that I will work with them during reading of "Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp" that I bought week ago.
I also read a lot (corresponding shelf on shelfari, but reading queue is longer).
I visited USA for a first time in my life (business trip), and I want to visit it again, to meet with my friends on west and east coasts.
I have a lot of plans for next year:

continue to work with Clojure with more involvement into different projects;
I want to increase share of Haskell and Erlang in my projects, and maybe look at Scala, if I'll have a time;
continue to experiment with machine learning, and besides this, I have some ideas for natural language processing based projects;
continue to work with hadoop, and other big-data-related things;
update existing articles about programming and Emacs, and maybe write something new;
continue to hack Emacs, and I especially interested in providing support for functional languages in CEDET.

And at the end, I want to wish everybody all the best in the New Year!

Readings digest. November 2010

2010-12-08T15:10:00.001+01:00

It was not so much of free time in November, so I read only one computer-relater book...

Programming Massively Parallel Processors: A Hands-on Approach

To check some of my hypothesis, I decided to play with general-purpose programming for graphics processing units (GPU). I already knew about them, but wanted to make my knowledge deeper, so I obtained the Programming Massively Parallel Processors: A Hands-on Approach book, that should be good introduction into GPU's programming, according to reviews.

The first two chapters describe history of general-purpose programing on GPU, its architecture, and also describe different approaches to parallel programming.

Description of CUDA begins from third chapter. First, structure of CUDA programs is described, and than simple example is shown — multiplication of two matrices. This example will be used in all chapters and will improved as we'll switch to more complex topics. And the third chapter is finished with description of CUDA memory model and CUDA threads together with kernel functions.

Fourth chapter is completely dedicated to CUDA threads — how they are selected for scheduling, how they are organized into thread blocks, how to organize synchronization between threads. And fifth chapter describes all types of memory available for developer, and how these types could be used, taking into account different things, such as performance, etc.

Sixth chapter describes how programs could be optimized — how to achieve full load of processor and memory bus, how to organize data pre-fetching, etc. And seventh chapter is dedicated to problems, related to use of floating-point numbers.

The 8th and 9th chapters are used to show how to use CUDA for real projects. For each of projects, architecture of program is shown, together with description how data are organized, how kernel function is implemented, etc.

The 10th chapter is more philosophical — authors are trying to describe common patterns and approaches to problem solving and algorithms selection for parallel programs.

The eleventh chapter contains short description of OpenCL language with list of differences between it and CUDA. But description is very small, so it's better to look into official documentation of OpenCL, although the book provides very good base for future study.

Conclusion: If you want to become initial knowledge about general-purpose GPU programming, then you can take this book. It contains basic information about GPU's architecture, programming with CUDA, and also many advices on how to develop effective programs for GPU. Besides this, each chapter contains references section that you can use to find additional sources of information. If you want to get more deep understanding of CUDA programming, then you can take the CUDA by Example: An Introduction to General-Purpose GPU Programming book and/or official CUDA documentation.

Future work

I personally more interested in OpenCL, as it could be used from other languages, not only from C. For example, there is the Calx projects, that allows to run OpenCL code from programs written in Clojure. And OpenCL is also working not only on NVidia's hardware. I'm waiting for release of following books in next year: OpenCL in Action (already available as MEAP) and OpenCL Programming Guide, that should contain very useful information.

Additional resources

The very useful resource is the GPGPU site, where GPU-related articles and announcements are regularly published. For example, following articles was very useful for me: GrAVity: A Massively Parallel Antivirus Engine и Regular Expression Matching on Graphics Hardware for Intrusion Detection, because they are related to some of my investigations.

There are also at least two GPU-related courses on iTunesU. And materials of following course are also available.

And there are a lot of documentation and tutorials you can find on NVidia's site — both for CUDA, and OpenCL.

Work/file-formats related, 2

2010-11-12T11:03:00.000+01:00

Microsoft had published videos from lectures on MS Office file formats, that was made during binary file formats plugfest, that I described in previous post

Readings digest. October 2010

2010-11-05T13:42:00.001+01:00

Pro Git

I'm using Git for a long time, when there were no books on it. But not so much time ago I found following book and decided to read to find interesting themes, that may be not familiar with. This book has separate site with HTML version available to read (there is also epub-version of the book), together with blog with some interesting notes. A source code for this book, together with examples is available at github.

This book begins with small review of Git, why it was created, its history, etc. There is also small section on installation and initial setup of the Git.

Second chapter starts with simple Git usage patterns, such as add and commit changes, viewing of history of changes, undoing of changes, etc. There is also necessary information on remote repositories. Third chapter is completely dedicated to branching and related questions, starting with creation of branches, switching between them, and finishing with pretty good description of rebase command. The separate section provides description on using branches to implement different development workflows.

4th chapter is dedicated to creation of remote Git repository, reviewing existing access protocols, providing public access to repository, etc. There is also sections on tools that could be used to make life easier, such as Gitosis, Gitolite, etc. And 5th chapter provides information on how to use Git for collective work, starting with reviewing of different approaches to collective work's organization (shared repository, add separate integrator's role, etc.), and continue with description how project's maintainer and contributor can use Git to perform their work.

6th chapters provide description of some "non-standard" Git's usage patterns — use of external modules (submodules), merging with sub-trees, rewriting history of changes, etc. 7th chapter provides very good description of Git's customization, including creation of hooks. And 8th chapter is dedicated to questions of using Git with SVN, and migration to Git from other version control systems.

And last chapter in the book provides information about Git's internals — how objects and other information are stored, how this information is transferred using different protocols, etc.

Conclusion: this is very good introduction into Git, I can recommend it for all who wants to start work with Git. Developers, who are already using Git also can find some useful information, for example, description of different external tools, etc.

Data-Intensive Text Processing with MapReduce

In my free time (and sometimes at work) I'm solving some tasks, that require lot of data to be processed to be solved. As platform the Hadoop was selected, and usually programs are written with clojure-hadoop, and some other tools.

I already wrote about the "Hadoop: The Definitive Guide" book (btw, the second edition was released not so much time ago, that was updated with description of fresh Hadoop's versions), but in contrast to this book, the Data-Intensive Text Processing with MapReduce book doesn't describe how to program with Hadoop or some other Map/Reduce system, but describes how to design algorithms for Map/Reduce. Book also describes how to implement some well-known algorithms, such as, how to create inverted text index using parallel programming model. This book has separate site, where you can find additional information, and download beta-version of the book. Besides this, to experiment with algorithms, the Cloud9 library was created, and you can use it for your work.

Book begins (chapter 1) with small description of base concepts of Map/Reduce, how tasks are executed in typical framework, and how tasks are separated between mappers & reducers. Besides this, there is small description of main Hadoop's subsystems, so reader will understand how tasks are deployed, how data are stored, etc.

Second chapter reviews different approaches to design of algorithms for Map/Reduce, starting with naive implementations, and after switching to review of optimizations (for example, use of additional combiners, or use of in-mapper caching), sorting, joins, etc.

Chapter 3 explains how you can implement generation of inverted indexing (that is used in full-text search) using Map/Reduce. First, naive implementation is used, and later, discussion continues with optimizations, for example, how to compress index, etc.

4th chapter describes how graph algorithms could be implemented using Map/Reduce frameworks. As first example, calculation of shortest path is described. Authors provide description of parallel breadth-first search, and discuss differences from classic Dijkstra's algorithm, and other issues. In second example implementation of PageRank is discussed. And in last section list of existing issues with parallel algorithms on graphs is discussed, and links to additional literature are provided.

5th chapter is dedicated to discussion of usage of Map/Reduce for machine learning tasks. First section describes expectation maximization algotithms, and second describes hidden markov models as class of tasks, to which expectation maximization algorithms are applicable. And than implementation of expectation maximization algorithms using Map/Reduce is shown.

In last chapter authors discuss different issues with development on base of Map/Reduce, together with alternative computing paradigms.

Conclusion: if you program for Hadoop or other Map/Reduce system, then you must read this book — it provides enough information on proper organization of data processing. Besides this, book has pretty big bibliography, that could be used as source of new information on related topics.

Work/file-formats related...

2010-11-04T14:57:00.000+01:00

Two weeks ago I attended meeting at Microsoft (in Redmond, WA) that was dedicated to Microsoft Office's binary file formats. As I'm working for a long time with corresponding Microsoft's departments, it was very interesting to meet all peoples who are involved in preparation of documentation on protocol and file formats, and also with developers of Microsoft Office.
Meeting was 2 days long, and MS Office's developers gave talks about internal details of MS Office, and how they are related to data that are stored in files. Besides this, it was a good chance to ask some questions directly.
Because this meeting was held for a first time, were not so much external developers (about 15 peoples from different companies), but were many peoples from Microsoft, and many questions were answered almost immediately, and we all had a chance to show broken/problematic files, show snippets of our code, etc.
Channel9 in near future should publish video taken from presentations, and I'll write about this additionally. Besides this, the separate blog was created, where notifications about future events will published. And I hope, that in future, such events will held in Europe...

Readings digest. September 2010

2010-10-21T00:58:00.003+01:00

Some time ago I got an idea, that it could be useful to write reviews on books, that I read. Although I'm using GoodReads and Shelfari, but reviews on these sites are pretty short. I selected month as time frame — this allows me to read 2-4 books, that are worth to mention in reviews :-)

The Joy of Clojure: Thinking the Clojure Way

I bought The Joy of Clojure: Thinking the Clojure Way (by Michael Fogus and Chris Houser) right after as it was announced by Manning, and got updates to it from time to time. But at start of September I got full version of the book (but in draft quality), and decided to read it from cover to cover. This book has separate site where you can find additional information on book, including info on existing discounts on preliminary version (MEAP).

I need to say, that this isn't introduction into Clojure language, and, although this book includes separate chapter with short description of the language, but it's better to read one of existing Clojure books first: Practical Clojure, or Programming Clojure, or not released yet, but available through MEAP, Clojure in Action.
This book tries to explain how to program in Clojure "in right way" — from functional programming and Clojure language perspectives. This includes use of "lazy" data structures, when you can and should use macros, how to organize recursive execution of the code, how different data structures are implemented, when each of them should be used, etc.
There is also pretty detailed description of performance-related questions — which data structures is better to use in concrete situations, how you can improve performance with type hinting, how to use transient data structures, bulk sequences, and memoization.
Book also includes description of concurrent-related topics together with discussion of usage of mutable data in your programs. These topics are discussed in 11th chapter, where you can find description of mutable data structures (refs/atoms/agents), when they could be used, etc. There is also description of parallel programming features in Clojure (futures, promises, locks, etc.).
Besides this, this book describes multimethods, protocols and data types, introduced in Clojure 1.2, together with topics on interoperability with JVM and Java. But these topics aren't described so detailed as other.
Conclusion: this book is recommended for all, who already read something about language, but wants to get more details on some topics, and/or wants to learn to "properly" program in Clojure. I personally found pretty much interesting data in it, and started to use this knowledge in my projects.

Lucene in Action, 2nd edition

I wanted to look onto Lucene for a long time. I wanted to use it for my projects, and knew that there is a book Lucene in Action, but in Internet were many complains, that this book is very old, and doesn't match current version. But second edition of Lucene in Action solves this problem.

This books begins (chapter 1) with short review of Lucene — from which parts it consists, and also describes main parts of search lifecycle — document's collection, text extraction, indexing, and search itself. Besides this, in 1st chapter there are small, but complete examples of how you can index documents and perform search in built index.
The first part of book (chapters 2-6) is dedicated to description of Lucene itself. The Chapter 2 describes indexing — which indexing options are available, their influence on Lucene's work (including performance), how you can index numeric and temporal data, how can you optimize and debug indexing process.
The chapter 3 provides description of base search facilities. This chapter starts with simple application, that is used as a base for all other examples. First, the IndexSearch is described, and in later sections, the other types of queries implemented in Lucene are reviewed, together with QueryParser.
Chapters from 4 to 6 are dedicated to "advanced" topics, and you can skip them on first read, especially if you want to use only standard Lucene's features. In the 4th chapter different text analyzers are described. Besides description of analyzers, implemented in Lucene and Nutch, this chapter provides information on how you can create new analyzers, or extend existing. 5th chapter is dedicated to extended search features — search on several fields and/or indexes, work with term vectors, results sorting, etc. And the 6th chapter is dedicated to description of custom collectors and sorting routines.
Second part of book (chapters 7-11) provides description of the Tika library (chapter 7), which Lucene extensions are exist (chapters 8-9), how you can use Lucene with languages, other than Java (chapter 10), and 11th chapter provides information on Lucene's administration, including performance-related questions.
Third part of the book is dedicated to description of success stories — how Lucene is used in real projects (Kruge, SIREn, LinkedIn). For each of projects, there is description of which Lucene's features are used, how indexing and searching is performed, which extensions were added to perform these tasks.
Conclusion: Big number of examples, and detailed descriptions, are making this book very useful for peoples, who are starting to work with Lucene. And after you'll finish reading, you can still return to it to get answers on questions, not covered in official documentation.

Introduction to Algorithms

I already read this book many years ago, but this was a russian translation of 1st edition. And some time ago I decided to re-read this book to refresh some theoretical topics. I acquired the 2nd edition of this book, but next month after I got it, the 3rd edition of Introduction to Algorithms was releases, that contains some new topics, not covered in 2nd edition.

I can't write too much details about this book, or this review will several hundred pages long ;-) Table of contents can give much better overview of covered topics.
This is classical book on algorithms and analysis of them. Book begins with basic information on algorithm's analysis, and after some examples, started to provide description of different algorithms and data structures (lists, trees, hash maps, heaps, etc.) For each of covered topic, the complexity analysis is provided, the different approaches to implementation are reviewed, etc. Pretty much place is dedicated to algorithms on graphs. Besides of "base" data structures, more "advanced" data structures and algorithms are covered — binominal heaps, b-trees, etc. Almost hundred of pages are dedicated for appendixes, that provide all theoretical information (math, probability theory, set theory, etc), that is needed to read this book.
I need to mention, that there also video-lectures for this course, that was read in MIT (they are also available from iTunesU).
Conclusion: every software developer should read this book at least once — you can use this book as reference, but reading of first chapters is necessary for all.

More work for functional programmers

2010-09-16T12:28:00.000+01:00

I found another vacancy for functional programmers: company looks for programmers in statically-typed functional languages - Haskell/OCaml/F#. And as usual for such languages, this is finance-related works - financial derivatives, stocks, etc. They also have vacancy for junior developers.
The company is located in Tallinn, Estonia. If you interested, write to them directly.

Release of CEDET 1.0!

2010-08-30T08:17:00.000+01:00

After many years of development, numerous 1.0preX versions, version 1.0 of CEDET package was released.
This version differs from previous version - 1.0pre7, and contains many changes in Semantic, EDE, and other subsystems. Full list of changes you can find in official announcement.
Source code you can download from project's page and compile it according to instructions.
And after release 1.0, development team continue to work on further integration of package into GNU Emacs, development of new parsers and other stuff, that will allow to improve work with different languages and build tools.

Russian Clojure Planet

2010-08-11T15:31:00.000+01:00

I created (with Yahoo! Pipes) an aggregator of Clojure-related blogs in Russian language. If you interested, you can subscribe to it here.
If you're writing about Clojure (or about other functional languages) in Russian, then just write to me - I'll add your blog into Clojure planet (or Russian FP Planet)