Collision of Influences

LMN Solutions Office Tour

2010-02-02T19:46:00.003-05:00

LMN Solutions Office Tour

At the LMN Solutions holiday party I was given a Flip UltraHD camera. This is a video of the new LMN Solutions office in the Reston Town Center. I hope you enjoy seeing the place LMN calls home.

Take a look at this blog post for a second opinion.

LMN Solutions - Find out more about us.

LMN Solutions and Social Networking

2010-01-23T19:41:00.003-05:00

LMN and Social Networking

A company is only as good as those working for it. We have a great team here at LMN Solutions and it starts at the top. The owners are L M and N. That is Wajeeha Naeem , Matt McKnight . and David Lee . They really care about doing quality work and finding team members with the same values. They always seem ready to talk and collaborate on new ideas. Each of them work hard to keep us happy. On top of this they keep involved with the real day to day work the company focuses on by designing and coding new software systems.

I have not met an LMN employee I would not want to work with. Each of them is an expert in their field but yet humble enough to learn and expand. We learn from each other and have lots of fun completing our missions.

We have found each other through old fashioned social networking. Most of us know each others work from successful work performed at other places.

We are beginning a new phase in the company where we are increasingly finding each other with internet social networking tools. Tools like LinkedIn have been a great help in my career. I have been able to find great people to work with and for using it.

Come get to know our team on LinkedIn, Facebook, Twitter and LMN Solutions Leadership Team.
And come be part of the team.

LMN Solutions - Find out more about us.

Soft Launches

2009-11-18T17:12:00.001-05:00

A couple of new things coming out soon from the LMN Solutions Crew.

Be on the look out for WhatsFunThisWeek - which is a nice little site we built for a new client to track upcoming local events and keep track of fun lists. It should be launching in December.

Wealso plan on moving a lot of the essay type commentary around particular topics over to our new Agile Blueprints site. Right now, there is nothing there. Soon, it will be a font of knowledge. So, extremely soft launch on that one. By launching something so small, every small improvement we make will seem large.

We're also working hard on a new application in the employment space. This has been around the bend a few times, but we have the right product development team putting it together and will have two interesting products of this effort emerging soon.

In terms of new services, we are making a lot of progress in delivering geographic business analytics as a service. Using ESRI Business Analyst we are delivering geocoding and demographic analysis services. We are using these tools and data to combine what we know about site selection algorithms with commercial problem sets.

Finally, we're developing a high concept product in the text processing and analytics space. We don't have a clear launch for this yet, we're still very early in the customer discovery process trying to figure out who might want it and how they might want it.

In all of these cases, we're trying to lower the barrier between what Merlin calls "blue sky solutioneering" and getting a Minimum Viable Product out the door to test the waters and validate our concept of what the user wants. More to come.

Palantir & Human Effort

2009-09-06T20:17:00.043-04:00

Palantir & Human Effort

Palantir Government is a product we have been working with for some time here at LMN Solutions. Overall it is a great product full of features to aid in investigating events and people. Because of its power and many features it can be hard to learn and applied to situations where it does not fit.

Palantir Government has been used to investigate a cyber attack and many other events. It has an visually stimulating interface. It is centered around three types of objects - documents, entities and events. Each of these are sub divided into other groupings like meetings, bombings, cleric, government officials and so on. Each of these objects can have hundreds of properties such as name, address, longitude, latitude, date, birth date and any other type of data you can think of. It is easy to get lost in the details.

To make it possible there is a good amount of human effort involved to really make Palantir shine. The people using it and the IT folks supporting and loading it with data need to focus on learning how to make it work best for their particular group. Once you get the right data into it and then learn how to filter and search through data the linkages it can build are amazing. It is a "connect the dots" technology.

But it requires patience and time to iterate through the learning process to get the data into the right form and loaded into the system. It takes an experienced staff to really think deep about the ontology needed and how the goal of your group fits into it. It takes an experienced administrators to manage the Oracle database and the other Palantir servers. Experienced personnel to convert data from other systems into rich objects that Palantir loves can also aid the process.

Sometimes it is necessary to remove data from the system or change the object structure or ontology to help the end user, the analyst, work faster. Sometimes too much data can muddy the picture and perhaps slow the system. It takes strong communication with the analyst to figure out what needs to be done. We recommend a collaborative effort. Get the IT folks supporting Palantir right down to the analysts desk.

One of the keys to a successful implementation of Palantir Government is to define the goal. You must not just start ingesting the entire contents of all of the data repositories you have access to. It takes a lot of thought and preparation. Then do not give up, keep trying to add focused data sets that will strengthen the links between events, people and documents. Much of it comes down to old fashioned investigation and slowly building the data to link things together. At times it can just be boring data entry and tagging of documents. At least Palantir makes taging of documents easy.

Palantir Government makes finding linkages between people, places, events and documents easier but investigative work is never easy. It is vital to put in the time to learn the tools that will help find what you are looking for. Palantir has a excellent free training game called Operation Tradestop. It really helps you learn how the search tools work on data from actual events and people.

Palantir supports the use of custom helpers written in Java. They can be deployed right into the Palantir desktop. These are mini applications within Palantir that add additional functionality. With the use of helpers the tasks of analysts can be focused right to the tools they need without having to worry about all of the other features Palantir offers.

LMN Solutions has built a series of Palantir Government Helpers that aid in focusing the tasks of data entry and making sure the analyst does not get lost in a sea of properties.

Jabber Helper - real time communication within the Palantir window. Send Palantir objects straight to a colleagues Palantir graph. Collaborate with colleagues that do not have Palantir

Tasking Helper with work flow coordination. Assign Palantir documents to others and manage tasking.

Link Builder Tool - drag and drop interface to quickly build linkages between objects.

Overall Palantir Government is a useful tool, just be ready to put in the effort to narrow the focus of your task and spend the time needed to fully understand and learn the product. Palantir is not a panacea. One cannot just force all of the data and documents into in and expect it to magically determine the linkages between people, events, places and documents. If you can put the necessary time to see it through Palantir will yield results.

Come visit our booth at the Palantir Government Conference GovCon4 October 13, 2009

LMN Solutions - Find out more about us. Need help with Palantir, contact us.

The Value of Estimating (Even With Software Projects)

2009-08-12T09:41:00.000-04:00

There is a trend in software process commentary to claim that plans and estimates aren't very useful, to admit that most of the plans we make are, in effect, guesses. There is some truth to this. In many cases, the scope of the work is unclear, the staff available to complete it is unclear, and the difficulty of creating the application is unclear. On the other side of the fence, there are fixed budgets that have to be requested, there is the PMBOK which encourages you to build a Gantt chart including every task on the project with no uncertainty, and there are a lot of people out there that want to describe the software development process as something that is more like a manufacturing or construction project. In reality, software development projects lack the predictability of those sorts of projects, because they involve invention of something new.

However, it is absurd to say that estimation is impossible. For example, it is fairly easy to give an order of magnitude size estimate of a project based on other similar projects. Even if your own organization is not keeping track of it's historical project performance, industry standard tools such as Construx Estimate can be used to provide some rough guidelines based on their collection of data. These are overall project estimates, which can be superior to the bottom up estimation that is often done in agile projects. The estimation process by which each feature or story is individually estimated can problematically leave out costs and time associated with missed requirements, tasks that are not particular to a feature, and the overall drift of iterative processes by which the initial implementation of a feature is not final, but must be refined over time with a refactoring cost that is often not estimated. Whole project estimation techniques avoid some of these oversights, and can even be tuned to avoid the error in the initial size estimate for the system.

It is also possible to give estimates that demonstrate the uncertainty of the project by including a probability curve or range. The estimate also should not be static, but rather updated as new data comes in. The Cone of Uncertainty is one common way to represent this process:

This particular manifestation of the cone represents a waterfall style process underlying the estimation accuracy, but we can imagine a similar cone being applied to agile methods. When work is done to begin the bottom up estimation process, there is more information available now to map that information to the estimate, and eventually we can switch over to the bottom up estimate when it starts to become more stable than the holistic estimate.

The key thing to remember here is that an estimate is not a commitment. If you commit to a the 50% confidence estimate, there is a 50% chance you will miss the date. If you commit to a 90% estimate, the proposed timeline may be unacceptable to your customer, especially if you are willing to admit that there is some failure rate involved in software projects and there is a non-zero probability of not delivering at all. Making a commitment to a 90% date also introduces the risk of Parkinson's Law coming into play. When someone writes out a Gantt chart that has the 90% confidence date for each individual task, seemingly magically, the work comes in very close to the deadline. Except for those few tasks that don't, the ones that weren't actually started in earnest until the deadline was right upon the developer, but were more complex than expected. If those were on the critical path, all of sudden your whole project just got pushed out past the 90% confidence date, even if everything else is completed by the deadline. This where Critical Chain Project Management comes into play, but we'll save that topic for a future post.

Selenium Remote Control

2009-08-10T07:15:00.004-04:00

Selenium Remote Control - Web Interface Testing

When you watch this you might want to turn on closed captioning as the volume is very low. To see the actual text full screen mode is best.

Test Automation

Selenium Remote Control has been an extremely helpful tool in increasing the velocity of our Agile Scrum team by shortening the time between development and testing.

It is a tool that actually manipulates the browser like a real person would. This allows the javascript as well as the server side code and database to be tested as a whole.

In conjunction with various Agile techniques the process of building automated GUI tests has opened communication between testers, developers, customers and management. Surprisingly it has been useful as a sort of management dashboard to give high level managers the assurance that development and testing is progressing. People seem to enjoy watching the application run automatically and seeing green and red bars pop onto a report in their web browser.

With some integration with other products it seems to wean folks off of those tractability matrix reports also. Using ant, jUnit reports and annotations has allowed a loose mapping of a story to actual test results.

LMN Solutions has experience in integrating tools to enable continuous integration, automated testing, unit testing to support the Agile process. This demo was built using Eclipse, jUnit, Selenium Remote Control and Selenium IDE. The application under test, LMN Jobs and LMN People was built with Ruby on Rails and jQuery.

On other projects we have combined tools such as ant, Cruise Control, jUnit, Fitnesse, watir, Ruby on Rails and Java to enable continuous integration with functional testing and reporting.

Orthorectification

2009-07-20T08:00:00.014-04:00

Orthorectification

Update: Co-authored paper published by Journal of Applied Remote Sensing: Automated imagery orthorectification pilot

Photo left: Raw Film image from USGS archives.

Orthorectification, just a fancy word for processing images taken from the sky to make them geographicaly correct.

If you have used Google Maps or Google Earth you are a consumer of orthorectified images. Or imagery as the industry calls it. If you ever notice an image of the ground that looks sort of distorted, flat or appears to be taken from an angle it could be a image that has not been orthorectified yet.

To simplify it the output of perfect orthorectification should be an image in which you can point to each pixel and determine the true geo coordinate and elevation for that pixel. It tries to generate an image in which the ground of the image appears to be viewed from directly overhead (commonly called nadir). Of course anything sticking up above the ground will typically not appear correctly without a huge amount of additional processing.

The purpose of doing all of the work necessary is to increase accuracy. When all of the imagery is accurate it is possible to then mosaic it all together so there are no seams between images. It is also possible to take imagery taken from different time periods and layer it on top of each other to watch for changes. It enables all sorts of time saving tasks in many industries such as oil, geological work, scientific, electrical transmission, military, civil engineering and more. In the EPA it has been used to discover where pollution is buried by comparing historical images with each other to detect change over time.

Simplified Process Overview

The most accurate form of orthorectification involves sending surveyors in to the field to collect data points called Ground Control Points or GCP. Using GPS units they collect the longitude, latitude and elevation of points on the ground that are visible from overhead images. These are points such as road intersections or visible land marks. The problem is the cost of this option.

Many resort to the next best solution which is to use existing orthorectified images from the past. These are called control imagery.

The next step involves finding the ground control points from the survey or control images and matching them to the same landmarks in the raw images. This used to be a manual process. Someone had to locate these points and then correlate them together. There is now software that can find these points automatically and one can just accept what the software determines or perform a manual review of the points collected.

One must collect 3 GCP per image otherwise the final warping process will produce some strange results. If you ever took trigonometry 3 points are needed to determine the three dimensions of space. With only 2 points one cannot determine the height the photo was taken from and the scale of the photo will be wrong. Most tools require about 8 or so points to be collected and they should be spread throughout the image.

Two tools I have used for automatic GCP collection are PCI Geomatica and Erdas Imagine. I have also come across a module for ossim that does this also but I have not tested it yet. Other tools will perform orthorectification but require you to collect GCP manually.

It appears that PCI Geomatica collects these points by analyzing the high contrast areas where roads cross or the edges of landmarks.
Example of DEM from NASA.

The next part involves a file that contains the elevation model for the area where the image was taken. Often it is a TIFF file that contains pixels of varied shades of grey. Each pixel represents an elevation value by its color and has a geo-coordinate assigned to it. This allows the software to process the image correctly. This is called a Digital Elevation Model or DEM. If the elevation model is not included the process is sometimes called rectification and not orthorectification.

The next stage involves computing a math model called Rational
Polynominal Coefficients (RPCs) using the control image, the digital elevation model, raw image and the ground control points. Many times the camera model is included. This gives the software the distortion values that a camera lens causes.

Now the software takes this math model and starts warping the image. It moves pixels around to make the image as correct as possible. In areas with lots of elevation changes like mountain ranges the finished product will eliminate areas where optical illusions will make a hill look like a depression.

It has been proven that this entire process can be automated but with less accuracy than with an experienced operator. One of the issues is that each sensor or camera source has to be optimized. Each camera or satellite sensor has its own characteristics and the software has to be re-configured to optimize the results. Processing Ikonos images is going to be different from QuickBird. Aerial imagery from aircraft is even more complicated and has less standardization. This is not really a problem if someone has a large amount of imagery from a small handful of sensors.

To see a visual example of orthorectification view this image.

In a future post I can describe the details of the process used on historical aerial imagery.

Resources

http://www.satimagingcorp.com/svc/orthorectification.html
PCI Geomatica
ERDAS
OSSIM - Open Source
Automated imagery orthorectification pilot

LMN Solutions - Find out more about us.

Automated imagery orthorectification pilot

J. Appl. Remote Sens. 3, 33552 (2009)

http://link.aip.org/link/?JARSC4/3/33552/1

The Agile Toolwright

2009-07-13T13:46:00.039-04:00

The Agile Toolwright
As a child I remember going camping in our multi-colored VW pop top van with our extended family. No, my family was not part of the whole 60’s thing. It was the late 70’s anyway. My father repaired VW’s in his spare time to flip and this one had not been painted yet. As you will see from this story technical improvisation runs in the family.

On the day we were leaving the van would not start. My dad diagnosed the problem to be the solenoid. (One of those key parts that makes an engine start.) Fortunately we had my Uncle Dwight with us. A man who collected things, all kinds of things, scrap metal, screws, bolts, thermostats, and various pieces of "junk".

His collection was housed in a large shed with an estimated 25,000 cubic feet of storage. It was very organized and collected with a purpose. One portion of it held all of the tools he used to craft art, tools and fix things.

Now Dwight did not just collect these parts he used them. He took them and combined them in ways never seen before. Most often they were used in an urgent manner to fix some major problem like broken well pumps. Dwight seemed to be everyone’s fix up man. If something was impossible to fix or you just did not have money for new parts he was the person to call. Other times he used them to weld fantastic looking sculptures or even tools to accomplish a task.

I went with him to his Red Ford F100 and he pulled up the back of the seat to reveal a carefully organized box of parts. Everything from springs, cables, rope, pulleys to a solenoid were there.

A person with traits like Dwight is most helpful for a software development team. They become even more valuable on an Agile team. The Agile Toolwright is what I like to call them. A Wright is a maker of things as in shipwright. The Toolwright is sometimes called a jack of all trades or toolsmith. This idea and word is based on the term Toolsmith from James Bach. http://www.satisfice.com/presentations/agileauto.pdf and http://www.satisfice.com/blog/archives/15

The Toolwright is able to aid the rest of the developers, testers and even the user representative on the agile team get things done faster and increase velocity. It is someone who has experience at all layers of the software stack and may have positions in most of the life cycle roles, from DBA, tester, developer, help desk, system administrator to help desk. They can quickly research all of the software tools at hand and combine them in strange ways to get a task done all focused around software quality.

The Toolwright ends up building small tools or even an entire continuous integration / testing framework to help remove roadblocks in the way of the developers and testers. Some of the tools they may end up building or integrating together are continuous integration platforms integrated with unit testing, automated testing, code coverage and other extras. They help guide developers in making the application testable. Most of the time is spent helping testers with small applications to help them test faster. Sometimes they even help the users in building interim software to allow the development team to speed the time needed to build the feature correctly for the long term.

The person in this role may not contribute directly in the production code but they have the ability to find the gaps in the process that can be made more efficient. The whole process of developing in an agile fashion goes much better if you have a dedicated Toolwright. Perhaps you do not have a full blown Toolwright on your team but there is no reason it cannot be a shared role between all team members. Try to develop the skills needed and the attitude to help others on your team get their job done quicker.

Well that solenoid was not a VW solenoid; it was a bit too large. Good thing VW engines are not that picky about organ transplants. A little duct tape and bailing wire solved that problem. Watch out, many Toolwrights are not opposed to duct tape solutions. In any case, the van started and got us home.

Now for Dwight, being a toolwright of sorts was not his livelihood but he ended up as the community maintenance man, the ultimate handyman on an extreme budget. Often without cost helped neighbors repair their homes, appliances and cars.

I only hope I can do the same with software and code instead of scraps.

LMN Solutions - Find out more about us.

The Power of Pair Programming

2009-05-26T19:41:00.000-04:00

I never wanted to sit with another developer and do my job. I know what I am doing, and google can help me figure out anything I get stuck with, right? This is my code, I know how it works, and I don't want other people messing it up.

It wasn't till I started working with a second developer at my desk for a couple hours a day till I realized the true potential and power that pair programming can bring to a team. We would lay out what we wanted to accomplish and it seemed like we always finished, never hitting road blocks or getting stuck debugging for long periods of time. We would get in "the zone" and just crank out functionality. We would knock out in two hours what could have taken me more than a day to do on my own.

There are many ways to pair with another programmer. Just sitting there while they write code and providing input as an observer. Taking turns on the keyboard with one person driving and one observing, and switching periodically. Playing ping pong, where one developer writes a failing unit test while the other follows with an implementation which causes that test to pass. There are even teams which have put together developer setups which include two keyboards and two mice such that either developer can take control at any given time without having to move over.

Pair programming can bring a team together and spread the knowledge. When a senior developer picks up an easy task, it becomes a great opportunity to pair with a new team member and use it as a teaching opportunity. It is recommended to let the junior developer drive. There is no better way to learn than doing, yet having the support of someone in the know right next to you provides a feeling of security that they can't mess it up. When the next story is taken by someone which may not have the skills to solve it, this again becomes a great opportunity for learning, but also involves team communication to find the individual which may know more about the particular component. This can greatly reduce the "truck factor" of the team.

While all the above are true, and can help justify why pair programming should be done, I must say that from my experience the biggest gain from two people working together is the focus and ability that two people have to stay on task. Working with another person typically does not allow me to be distracted. People also know better than to interrupt when people are working together in front of a computer screen. You put those two things together and a pair can get in the zone and do amazing things.

So pull up a chair and spend some time developing with a teammate.

Photo by by ntr23 via flickr.com

How much will it cost?

2008-08-18T09:21:00.001-04:00

The first thing they want is the estimate: How long will it take? How much will it cost? These are not easy questions to answer- especially when no one knows exactly what the customers want to build. How can we give you an estimate when you don't know what you want?

We are on a project where they hired someone to figure it all out first. The analysis team spent a year, with numbers around one hundred people, learning about the customer, researching technologies, and spewing out documents- in the thousands of pages. Then they came up with a schedule for someone else (eventually including us) to start delivering systems incrementally- and called it Agile (with a capital A): fixed-scope, fixed-schedule Scrum.

Unsurprisingly, it didn't work. The analysis team missed half of the requirements for the most important application. Their technology selection process came up with numerous wrong answers. The first few incremental reviews were perceived as having come from a year of work. A battle began between the implementation and analysis contracts. It got ugly.

So they started over from a contracting perspective. Good-bye to fixed scope. Good-bye to fixed schedule. A new dawn for "real Agile"? Except that now the customer wanted a schedule- they wanted a price and a schedule for scope- scope that was not at all agreed upon.

We had a modest proposal- if you want a fixed cost, fixed scope contract, make it short. Make it a month or a quarter. Give the customer the option to cancel at any time. Use rolling wave planning. Of course, the customer budgets work years in advance, and uncommitted funds often disappear. So you have to write a contract that specifies these iterative and incremental development cycles for about the size of the team the customer can support. Use a little bit of data to at least put you in the wide part of the cone of uncertainty. Refine the estimates with each increment based on real evidence.

Stay tuned to see how it goes.

Are you creating value?

2008-08-06T23:30:00.002-04:00

In the Lean view of the world, the theory of business that emanates from the Toyota Production System, the focus is on delivering a continuous flow of value and avoiding waste. Our business is formed around delivering value to customers. When our team arrives on the scene at a project- often a project that is in trouble for one reason or another, the first thing we look for is the customer. In order to provide value to the customer, one must understand what the customer values.

How do we understand what the customer values? The values must be communicated to us- but, still how, and by whom or what? Too often, we show up on a project and there is a huge document that purports to represent the customer's needs. In some cases, we are told to look at the existing systems they have and recreate them, with changes. In other cases, we are told that some other project functionaries will be spending their time visiting with the customer, and the the people designing and building the system will not be dealing with the customer. What does the lean approach tell us to do?

The Toyota Way recommends a simple thing- "Go and see for yourself to thoroughly undertand the situation." Also known as going to gemba (Japanese for "real place") or genchi genbutsu (Japanese for "real thing"), this allows you to make accurate decisions based on real observations. Rather than acting based on reports or hearsay, we should strive to create real human interactions with the people involved in the mission and understand what they are trying to accomplish. This will help put the technology in the right place.

Your job, your primary goal, is to help accomplish the mission of your customers. If you do not deeply understand this mission, you can only act in a shallow fashion. You will only be providing tools, and you will not be aware of whether or not you are providing value.

What is this about?

2008-08-05T14:29:00.002-04:00

We work on a lot of projects. We see a lot of opportunities for improvement (aka problems). We provide a lot of solutions.

There are patterns to the kinds of things that go wrong on these projects. There are root causes that need to be addressed. There are things that we know really work, and we're trying our best to make them happen.

That process would be far simpler if people already understood what we are trying to do. Therefore, we are kicking off this effort to attempt to work out the specifics of the process and communicate it in an effective manner to the necessary individuals such that everyone's chances for success are improved.