Alexey Kharlamov

What if coffee houses were like software projects?

2010-11-24T11:47:00.000-08:00

Lately, I've decided to spent sometime on improvement of project management skills. I bought a book of "world level expert" in project management in a hope to find brilliant insights. But I lost my wish to read approximately on 10th page of first chapter.

The author assumed management of software development project is the same as management of a coffee house. Literally. While the book has it value, I still can not funny pictures of coffee houses like software projects:

What if coffee houses were like software projects:
- Each coffee shop employee has to learn her specialty for 14 years in school. After that he has to work at least 5 years before he starts to do it with acceptable quality.
- You have to teach all your new employees to prepare coffee in your way before they start to bring value.
- There shall be thousands of coffee types. And each your employee knows about 10, may be 20.
- Even veterans of coffee making were not able to tell you how long will it take to prepare this type of coffee. They even can not say whether it is possible at all!!!
- 20-30% of time your guys in the kitchen have to spent on learning new ways of making coffee. Otherwise, they can end up on streets without chance to get job.
- Usually, your clients come within groups of several dozens. Each of them has its own opinion about kind of coffee they all want to drink together.
- Clients demand their order have to be done immediately, or in several seconds (see above) from the moment you heard "I want coffee".

Now, do you really think it is completely the same thing?

HighLoad++ 2009: IMDG Presentation

2009-10-24T09:36:00.000-07:00

On 12th of October I gave a presentation dedicated to concept of In-Memory Data Grids on russian conference dedicated to high load systems. My talk laid little aside interests of most participants, but it seemed to induce some interest. I even get an invitation to do the same talk for employees of Mail.ru corporation.

PRESENTATION RUSSIAN ONLY

HighLoad++ 2009 In-Memory Data Grids

View more presentations from aih1013.

Wonderful analogy between programming & sleeping

2009-02-15T18:01:00.000-08:00

Just found a great article about Analogy between sleeping & programming (russian only). When stated, similarity becomes mostly obvious.

Full-text & faceted search over In-Memory-Data-Grid

2009-02-04T21:57:00.000-08:00

Modern in memory data-grid solutions provide different facilities for execution of queries over whole stored data set with different levels of sophistication . Coherence provides Query facilities (one time full scan & continuous quering with Cost-Based-Optimized). Gigaspaces has JDBC Query interface with ability to use hash & B-Tree indexes. The solutions work quite well for big part of problem areas. However for heavy loads & complex multi-createrias queries the facilities may become a bottleneck quickly.

But there is a class of workloads producing high query loads onto IMDGs. Item catalogs (in retailing companies for example) are hit by diverse stream of multi-createria queries.Typically query you see likes :

give me cell phones with MP3 support, Java and in red color.

Fortunately, Compass framework allows to do such queries effectively. It allows to builds inverse indexes with Apache Lucene and store them in a grid. The capability originates in very modular design of Lucene framework. All index IO operations hidden by abstraction of FileDirectory.

For now Compass provides implementations for Coherence, GigaSpaces and Terracota introducing unprecedented ability to build a vertical search solution on top of In-Memory DataGrid.

Moreover, Compass has sophisticated object-to-document mapping system which allows to make stored objects searchable just by addition of Java annotations or XML mapping files. Also mapping can be built in runtime.

However, inspite the great code, Composs has scarce documentation. It may take significant time to dive into the code and docs to get what you want. But results will overcome all your expectations. Search engines on top of data grids easily overcome any old-generation search technologies.

Ship It Review

2009-01-18T06:11:00.000-08:00

Content of the book can be naturally divided into 3 parts:
- Code of Software Developer
- Advices about development tools and considerations
- Trace Bullet Development
I'll review each part independently.

Part dedicated to relations in developers team and management is absolutely brilliant. It gives wonderful inside to team dynamics, main factors influencing changes and intrateam relations. And most importantly it provides solutions for common problems of day-to-day developer’s life. No doubt, the chapter will be interesting for both newbies and seasoned team leaders.

Another part of the book is dedicated to development tools and processes. There is nothing novel here. In our days every developer knows Version Control System should present. And it would be better to have a continuous integration server if your team has more than 1 guy. So the part can be skipped if you have knowledge about modern development processes.

And the last part is dedicated to so-called Trace Bullet Development (TBD) process. While the wording sounds cool, usefulness of the process is rather arguable. Don't get me wrong; described design process (yes, this is not a Development process, this is a Software Design Process) will work. Creation of vertical of functioning system components is a good starting point for any project. But TBD pushes it forward too much. Interfaces created early tend to be error-prone. And after the interfaces were designed, they tend to become unchangeable. So line by line they become monsters which have no connection with the original elegant idea.

Concept of implementation of architecturally important user-stories or use-cases looks much better for me. Needless to say, as in any book there are several out-dated or even harmful advices. For example, "design all interfaces to be remote" . Such methodology used without appropriate care will make system too complex without real need.

Anyway, the book worth reading. This is the best masterpiece I’ve read since "The One Minute Manager".

New Tasks feature in gMail

2008-12-13T06:50:00.000-08:00

The new Tasks feature is gMail is awesome!!! I have waited for it so long. Now I can abandon all substitutes I had tried to use (without any success by the way).

I use gmail as my single access point to work items. Separate tasks management systems was really unnatural for me. And now I can use gmail to do all stuff I need day by day.

It is not very rich from feature set point of view, but usability is great as usual. Hopefully, additional features will appear soon. I'll be happy to have an external access to the task list and tabs for task list switching (google, do you hear me?). :)

No posts for a long time

2008-08-10T03:16:00.001-07:00

Lately, I did not write much. This is due my long summer holidays (I've visited Norway and Altai Mountains btw).

Now I want to inform you about some news. I leave Enterra Inc and Barnaul tomorrow and move to Saratov to work for Grid Dynamics. Hopefully, I'll be able to get a stable internet connection soon. If so, I shall update my blog more frequently.

I plan to share details of my life in Saratov and experience in grid systems building.

GWT: Pros & Cons

2008-05-04T04:58:00.001-07:00

Pros:

Very good perceived UI responsiveness. In a perfect case response times will be even better than on desktop (local Google Gears caching is great).
GWT uses component model for UI, so code reuse is easy. And there are many open-source libraries of GWT components too.
GWT server-side is stateless. So it is highly scalable "by design". In our case we achieved very good performance per single front-end server instance.
You can make your UI similar to desktop GUI with associated sweeties (DND, visual effects & etc)

Cons:

GWT is built around Java -> JavaScript compilation. So if you get a big Java codebase, the result js files will be big. So some old browsers will not be able to load and run them
GWT app loading time may be quite significant for big codebase. Of course, browser caching helps here
GWT renders page content with JavaScript, so indexing of data represented by the app is problematic
While there is browser history control inside the library, there are restrictions introduced by HTTP/HTML stack. In general case, you will not be able to reference pages inside GWT app if you use HTTP redirect (for example during authentication)
HTML Browsers are not too good in visualization of dynamic HTML (IE is the worst, Safari 3 is the best). So you may need some tricks to show big number of items)
GWT components are not easy to debug/implement. They should be cross-browser. And HTML browsers have their own bugs too. So you may need a JS/DHTML guru in your Java team.

SpringSource Application platform

2008-05-03T23:57:00.001-07:00

Lately, several Spring based platforms for application development has arisen. I guess this is a good thing. JEE is a quite rigid and restrictive for modern application development. There are areas were this is good. For example in conservative industries code may live for 10-15 years. So well described standardized platform is a must.

But as I said it has drawback. This is why Spring is the leading framework for application development. It is like LEGO. The framework ties together many different components into a single system and provides unprecedented flexibility in architecture design. In the last two java projects of mine JEE container was just for checkbox. Mostly all functionality is controlled and managed by Spring.

So introduction of the Application Platform by SpringSource is a GREAT EVENT for me! While the platform is on early stages of development yet, it tends to be a great tool!

All components are physically modularized through OSGI. So version control and dependency resolution is inside container already. These things I was doing manually for ages.
Logging is fantastic! The platform separates event streams by originating application and allows to configure details level very precisely. The configuration language is much simpler and comfortable than Log4j XML. Moreover, serious app errors are detected and logged (deadlock & etc).
The platform provides native support for Spring contexts, modules linking through Spring DM. I am happy. :-))))

However, the platform is young still. I did a quick review and got following wishes:

The platform need a bundle distribution infrastructure. I want not put new bundles into all instances manually. Of course, I can automate this. But it would be better to have the feature inside the AP.
I'd like to have Tanuki software wrapper as native runner instead of custom made shell scripts. The wrapper is broadly used by community and it provides very good capabilities for OS integration (services, remote restart, watchdog & etc)

Anyway, this is a big step forward for server application development. You are on good course Spring!

Monitoring that matters

2008-04-21T23:08:00.000-07:00

I am using Hyperic HQ monitoring for more than a year now. And it becomes more and more obvious for me that an important aspect of monitoring is missed by popular solutions. Yes, it is important to know throughput of the system.

But the metric that is really matters is CUSTOMERS EXPERIENCE. And it usually derives for two things:

Service responsiveness
Error ratio

The problem is that these metrics are application level parameter. I mean there is no OS counter that can be easily obtained by software system. And both application and monitoring system developers should pay effort on integration. This is where HQ is good. It is really easy to create JMX bean and an XML plugin to gather application specific metrics.

But there are area for improvement. I'd like to have:

Network level error statistics like number of missed IP packets
Exceptions in the log by type

Usually, errors rate gives more information about system health. But this type of metrics are successfully ignored at the moment.

Gigaspaces monitoring with Hyperic

2008-04-02T03:28:00.000-07:00

A half-hour ago I've released the first public beta of the Gigaspaces monitoring plugin for Hyperic HQ. The plugin provides automatically discovers and monitors of GigaSpaces XAP 6.0 on physical and logical levels. More information from the project homepage.

Farewell, Vista

2008-03-28T09:11:00.000-07:00

Lately, I have purchased a laptop with pre-installed Vista Home Premium. My first impression was quite good. Wow, it is so nice and quick enough.

But problems started to appear soon. I started to get "Not enough memory" errors from Eclipse. But wait, my development environment works fine on my desktop with 1.5GB RAM. Why does it fail on 2Gb with Vista?

The answer is simple. Just after install my Vista allocated 860Mb of physical memory for background processes. So I took my tambourine and file those I had used for Linux a long time ago and fixed it. It had worked for me for sometime.

As I've written already, I am working on a plugin for Hyperic which will allow monitor activity of GigaSpaces clusters. As I was waiting for availability of Ubuntu Hardy beta, I have written the code on my Vista laptop. It was a BIG ERROR.

Hyperic auto-inventory just don't work on Vista. I spent several days of work tring to resolve a error which appears on Vista only. Yes, I am not sure this is Vista's fault. But I have seen several very weird bugs like that. So I will not use it for important things anymore.

Farewell, Vista

GigaSpaces plugin for Hyperic

2008-03-07T04:17:00.000-08:00

I've just started an open-source project aimed to developer Hyperic plugin for GigaSpaces XAP monitoring and management.

See details here.

Ya.ru service validation record!

2008-02-22T00:18:00.000-08:00

Hello!

I have added the blog to ya.ru service. It requires ownership checking. So this post is just for the registration authorization.

22267502.024530d1d30810ca053b429754354a41.1203668219.d3e85a9367d5a2e2633c951543c6ed76

GWT: Improving performance

2008-02-13T00:25:00.001-08:00

Recently, we have published an article about GWT performance problems and ways to resolve them. I think it will be an interested reading as for beginners as for seasoned GWT/AJAX hackers.

Maven: Do you use it? Still?

2008-01-31T04:42:00.000-08:00

Sometime ago Howard Lewis Ship has posted a critique article about Maven builds. I am fully agree with him. I use Maven based build on one of my projects for 1.5 years now. It is a REAL PAIN. A year ago a was so disappointed by it that I had wanted to write a slashing article. But I've abstained. Now I see that my problems are not unique. So I want to add my 2 grams of lead mostly due the comments on the Howard's article.

There are three things those make Maven an bad choice for build tool:

1. Lack of discipline in developers team
Yes, I said it. All other problems are just consequences of this. There is no any release plans, configuration management and communication with customers.

As result, you shall find yorself using SNAPSHOT versions of plugins so explicit version numbers will not work. You need to establish an own controlled repository.

2. Convention over configuration principle
Yes, I am a heretic. I think conventions do not work good enough for large projects. Just because you need account too many of them.

Maven works good for simple projects, where artifact structure is planar. But it become a real mess to manage 3-4 level hierarchy of projects inside one build.

Another bad thing with conventions is easy modification. The Maven dev team can do a small change, and your build will be broken. Actually, the change may even be a bugfix. But you are in trouble anyway. It may pass sometime before one of your team members install Maven with a difference in patch level and the build will stop working for him.

3. Buggy implementation
There is no any software without errors. This is like an axiom in our days. But in the Maven's case bugs are a real problem. Mostly because of lack of appropriate release policy. Some issues stay unresolved for years.

Now, it is unrecommended practice to use Maven for builds inside my company. All new projects use Ivy together with internal Maven repository now. As I can see developers are much happier with it.

Mines in the Field of GWT Development Planning

2007-12-14T01:39:00.000-08:00

Recently, Google Web Toolkit attracted attention of web developers from all over the world. GWT is a great technology for AJAX development actually. It helps to get rid of many head-aches associated with cross-browser development, user interaction and development cycle.

The library provides unprecedented possibilities to build Web 2.0 applications with high levelof interactivity. And there is a trap.

Often, the user interface of web-applications is simpler to build than desktop UIs due to two factors. First, the HTML/CSS ecosystem provides for a great number of tools to easily express designers vision. And there is no need to build/run/debug cycle for the web-pages.

Second, HTML provides for a significantly lower level of interactivity that is encapsulated by browser. So developers don't need to debug each.

Revolutionary AJAX development GWT changes these factors also. GWT apps can be highly interactive and they are component based. So the initial expectation about easy change of web applications’ look'n'feel is not completely true anymore. Now, if you want to change it in the visual design this may require a significant effort. Moreover, as we now use components visual design is limited by available set of controls. A little spicy text edit field invited by a web-designer may create a real mess for the dev team. Isn't it like in the good old times of the desktop application UI?

So GWT development requires very tight cooperation of the design and development teams. Keep these guys in one place, preferably in a room with closed doors and a small window to serve food.

Another feature of a GWT application is interactivity. Yes, this is a great feature and a great head-ache.

As I have said before, web 1.0 applications rarely provide an interactive response on the user's actions. So the first move of web designers is to provide static mockups of the UI. This does not work here. Developers need a specified behavior of the application. And the behavior is a part of the code. That is not an easy thing to change. So you need to have enough reserved resources to debug and tweak the UI.

So let me say this again. Keep your developers and interface designers as close as possible. May be this is why Google apps (gmail, reader and others) so well thought. Usability the team works very closely with developers.

Production system maintenance. Part 2

2007-12-12T01:34:00.000-08:00

In the previous post I've described several organizational moment of the production system update. Now it is time for technical tips and tricks.

Name code equally on all system nodes
Usually, this is a good idea to name all things equally across systems you manage. It can be a real problem to have jboss4, jboss4.0.5 and jboss as names of the same piece of code on you cluster nodes. Use single and simple convention for code naming and placement.

Name resources differently
While single naming schema for code lowers amount of time spent on unproductive things, resource naming is much different. In mature environment code can be easily restored from several places like developer machines, continuous integration, staging servers, source repository & etc. Resources (i.e. data) are different. Information and schema of your database might be unique on some moments. Moreover, the data may represent great value for your company.

So damage to data should be avoided by any means. As the first line of defence name your databases differently depending on type of environment (production, testing, development & etc), contents and schema version. I usually use following db identifiers:

contents-timestamp_of_last_schema_update-database_type

For example: chirp-20071210-prod While looks quite complex, the given notation may protect you from actions done on a wrong data by mistake. Unfortunately, "Oh God, I've dropped wrong database" problem is not so rare I was expected firstly. ;)

Use consistent hostnames
Correctly set hostname is not an absolute requirement for server functioning. But right names may give you some help during exploitation. It is funny to have server called 'snoopy', if you have 3 total. When you have more, their names should be little more transparent and linked with their properties as the hosting data center and ip address.

Also server software need to know name of the server it runs on. That is why hostname and dns name must be the same.

Highlight your current context Most maintenance errors I have seen were "right command with in a wrong context". For example, I have dropped production db while were thinking it was staging environment. (This is why I make backup of all data on the production server before update now).

So put information about your current host, database, directory everywhere.

Put hostname, username and directory must present in the command line prompt.
Put hostname into the xterm title.
Put hostname and database name into mysql client prompt.

And be attentive.
Don't work as root
Actually, this is impossible. Just try to work as superuser as less as possible. root or Administrator can do many dangerous things. You know, one error and your root filesystem is empty. ;-)

Have a remotely controlled power switch
This is actually is not required for VPS. However, this may be essential for
dedicated servers. It is not a so rare task for system administrator to configure network remotely. And this may be very very very inconvenient to have a node with improperly configured network which can be managed by ssh only.

Yes, data center support team may press "Power" for you. But only if you pay for 24x7 support.

Use cluster management software
Not long ago, Debian package of the day published an article about ClusherSSH which I found very useful. There are several similar programs also. While relatively simple, these programs may easy your life and decrease number of errors.

Production system maintenance

2007-12-12T01:05:00.000-08:00

Lately, I've spent around 5 hours recovering production system from failure caused by improper actions of engineering staff during update procedure. I was lucky, so it was not me who has pressed the red button. But after that emergency situation I was trying to analysis causes of the problem and establish a better deployment process for software update. So these are thoughts somebody may found interesting:

Keep people without relevant experience out during important updates
Yes, some are pretty cool in development and they may be very attentive. But production system update require engineer to be a little paranoid. If you have not bad experience in this area, you probably will be unable to predict side-effect and bad-thing-that-might-happen. In ideal case the update must be done by a pair of engineers. Human-factor will be depressed much in that case.

Have a detailed update plan
Well, if the first thing is not the case... Prepare a detailed update plan with all things (shell commands, file & database operation & etc) included. It would be wonderful if you test it in your staging environment first. In ideal case you must have an auto-install package (deb, rpm or what ever).

Have a detailed fallback plan
This is continuation of the previous thought actually. Your update plan should include graceful fall back procedure description. Some errors appear on production system only. Or your development team might miss something, or . So you must have a way to return to previous version. Always.

Have a Plan B
In some cases fall back scenario can not help. For example in case of improper execution of update script. For example, if somebody deleted important files or dropped wrong database. Yes, your update script contains backup step. But what if that step failed silently?
General rule is "wait for trouble, always". So regular backup of production is a must have thing. Some data centers (as ours) even provides daily server backups in default service package.

Are synchronous systems doomed?

2007-11-14T04:38:00.000-08:00

Asynchronism of communication is a basic principle of many distributed systems. Did you ever think why? Why developers do so much staff to handle things asynchronously while sync communications are much simpler?

If you'll see around you shall find many examples of asynchronism. Most organizations exploit this principles, engineers exploit this principle, even nature
has built neural systems asynchronous.

So why? The point is implicit parallelism and reliability introduced by such systems.

Let's imagine a chain of components used by each other and connected by relatively long "links". By "relatively long" I mean information propagation time can be compared with processing time or longer. In synchronous systems the first component in the chain will wait at least while other component shall receive the message. In worst case it will wait for results of the message processing. So transmission delays will sum and increase the first component idle time.

Link failure in that case is a disaster. The only general way to detect it is timeout which increases request handling time (read the fist component idle time). Moreover, there is no way to learn was the request processed or not.

Obviously, synchronous requests are hard to persist. As recovery of distributed state (which in synchronous system is usually represented by execution stack or system components) represents magnificent technical and administrative challenge.

Asynchronous systems are something different. Components work in 'fire-and-forget' mode. So no time (resources) is spent to conserve. Components do not idle and may create additional requests, process data & etc. So it is much more parallel.

Moreover, it is easy to design system in a way when all state is incapsulated inside the messages. Persisting them system may archive significant firmness for link/components failures.

Now take a look onto modern hardware design. In common computer (actually in supercomputers also, but there can be exceptions) CPUs are quicker than communication channels. This has little common with multicomponent architecture. But the trend is not to increase number of CPUs linked with _communication channels_ either on single computer level and on distributed cluster level.

eBay is building their systems in async way. The most part of payment transactions are processed in async way.

Don't you think it is time to learn message passing libraries and designs?

A perfect wiki

2007-10-25T02:06:00.000-07:00

We are using wiki as a knowledge storage on one of the project. While it has many good collaborative features, there are some inconveniences in it.

First of all, it is hard to write rich formatted documents or documents with complex structure. So I my requirements for wiki is all about that.

1. The wiki should provide view representing hierarchy of pages
2. Import/Export to ODF or another word processor format is must have.
3. Easy table editing
4. Wiki must send daily changes summary by email
5. All arbitrary wiki staff like styles and etc.

Things you'd like to avoid during system design

2007-09-07T00:41:00.000-07:00

Lately, I've been working on a geographically distributed clustered web system and got several ideas I would like to share. All of them are written by blood, sweat and pieces of lost dollars.

Don't rely on stability
"Yes, this is obvious" - you may say. But I was knowing this from start the project also. However during deployment and preproduction testing the life has shown many new tricks to me. Building network system all conscious developers will introduce some retries/recovery routines. However, absence of components stability has many implicit consequences.

For example, let's say we have a server on 100Mbit/s channel connection which is enough to serve 50 concurrent jobs. So you are setting up the system to run 50 jobs and go to relax a little. A day after data center company narrows your channel down to 10Mbit/s due the maintenance. Viola! Your servers start dying.

Or another example (this design problem incorporated into network libraries I use):
Let's say you have a network service registry which is used by cluster members to discover addresses of needed services. Everything works well except that all nodes participating in the cluster check connectivity and availability of each other and authorized to delete record about dead servers. But we are in the real world, you know.

Let's say we have 4 servers: A, B, C, D. Now node D tries to ping node A and fails. What will it do? It will kill A's record in the node registry. BUT ping failure does not mean A is dead. It means D can not connect with A. Other nodes (B, C) may work with A without a problem still. Now, when D has deleted A's service record ALL nodes will fail to talk with it! And the only error was to suppose A-D link has the same properties as A-B and A-C links.

So be aware. Don't make any assumptions on stability and properties. It would be better to make the system adaptive.

Don't rely on DNS system
Actually this is a derivative from the previous advice. DNS system was designed a long ago and its problems are well know. In most cases dns servers work over UDP that is not reliable. Even if nobody will DOS attack your dns, it may become source of failures due looses in UDP packets and so on.

But the most important source of risk is distributed nature of DNS. In most cases it does not under your full control. There are TLD server, DataCenter DNS server, servers of domain registrator and so on. When problems occur you may t spent days talking with system administrators and trying to find what is going on.