Spatial Intelligence

Why Use GeoEye-1

noreply@blogger.com (Gerry James) — Mon, 03 Jan 2011 23:49:00 +0000

GeoEye-1 is one of the newest ultra high resolution optical satellite platforms for earth observation (EO). With an approximate half meter spatial resolution capability, it gives the user an extremely fine view of their area(s) of interest. The sensor has five spectral bands (red, green, blue, infra red and panchromatic) which makes it ideal for a number of uses including environmental monitoring, mineral exploration and urban planning. The resolution of this sensor allows for extraction of very small features (approaching 1/10 hectare) and provides an economical alternative to air photos.

This half meter resolution GeoEye-1 image shows central Port-au-Prince,
Haiti after a 7.0-magnitude earthquake struck the area on Jan. 12, 2010

Another advantage of GeoEye-1 comes from the orbit performance. With an orbit velocity of 7.5 km/sec, orbiting at an altitude of 681 kilometers, it has a repeat frequency of approximately three days. This means it is ideal for many types of change detection analysis. For example, monitoring extreme flood conditions such as those currently being experienced in Queensland, Australia, is just one potential benefit of this satellites revisit frequency. Imagery from this ultra-high resolution satellite can be acquired at regular intervals to help rescuers identify escape routes, assess areas of highest damage and monitor the changing flood conditions. This is exactly what was done during the Haitian earthquake disaster which took place last year. GeoEye-1 imagery was used to assess areas that suffered the most damage in order to more effectively target the rescue efforts. (see image above) .

Other examples of the utility of this satellite platform can be seen on the GeoEye website. Hatfield Consultants is currently using this and other high resolution sensors on a project to assess the landslide potential from localized forest practices. This work utilizes not only the spectral signature information of the imagery but also the stereoscopic capture abilities of this satellite. Using techniques such as DEM extraction and principal components analysis (PCA), experienced Remote Sensing Analysts can determine the effects of activities such as clear cutting and road building from forestry projects. This can be done at a very fine scale making the output data sets, assuming availability of good ground control, very accurate and highly detailed.

There are many other uses for this imagery including vegetation health assessments, crop monitoring and even construction feasibility. For more information contact Hatfield Consultants Partnership.

A Word About Tile Caching in ArcGIS Server 10

noreply@blogger.com (Gerry James) — Thu, 23 Dec 2010 20:36:00 +0000

I have recently been involved in creating a web mapping application that displays environmental monitoring locations over a very large area (approximately 7 million hectares). Of course one of the primary objectives in developing any application is optimization and performance and, with web maps, that usually means tile caching. So the development team came together (two of us in this case) and quickly decided what our zoom scale requirements were. We then tried to implement this which is where it all fell apart.

Because we were using the Bing Map Service as our base layer we were committed to using the zoom scales dictated within that service. Originally we decided that the minimum scale could be set to 1:9,244,648, which is one of the Bing mid-range zoom scales, however, we felt that the maps should be tiled the the largest scale possible; 1:1,128 in this case. Here in lies the problem. If you do a quick calculation to determine how many tiles are created using this range it is approximately 89 million, YES I said 89 million. Of which 98% come from the scales above 1:9,027. When we tried to tile with those scales the process would run for a number of days without finishing.

So how did we overcome this you might ask. By taking a more realistic view of our requirements and removing the unneeded scales from the tile cache. Once we removed the 3 scale values above 1:9,027 the process took less than an hour. We also made sure to have Tile On Demand checked for each service so that the tiles could be created on an as needed basis, prior to running the tile caching.

Lesson Learned: Before creating your Map Service infrastructure develop a tile caching strategy which should identify the largest scale you NEED, the number of tile cache layers and which layers MUST be cached. DO NOT include scales that you don't need and cache only the Map Layers that will be accessed FREQUENTLY.

Model to Script to Tool: the "Holy Trinity"

noreply@blogger.com (Gerry James) — Fri, 03 Sep 2010 19:47:00 +0000

I have been doing some work for a junior mining company recently involving creating Volume/Elevation curves (see my last post). This work is accomplished in ArcGIS using Spatial Analyst and 3D Analyst which works very well but requires many steps. Over the course of this project, however, I have built a number of Python scripts that are now embedded as tools in a new toolbox in ArcMap. So I am going to take this opportunity to briefly discuss how easy it really is to create sophisticated Python geo-processing scripts in Arc and give a little example of how I recommend approaching this task.

First of all I would just like to say that ArcGIS is still the most powerful GIS in my opinion (yes I am still a fan of open source but you can't argue with the benefits of having a large budget to develop good software). I like Arc not so much for it's ability to perform the geo-processing functions but more for the ancillary tools that help you find the appropriate tools and perform these tasks. Arc has built in search capabilities and very good documentation to allow users with a wide variety of skill levels find the appropriate tools. On top of that, there is a very well developed modeling and scripting environment that gives the user the capability to reproduce these tasks in a relatively easy manner.

Ok, now that's been said lets get on with the main purpose of this post, to show how easy it is to create a tool in ArcGIS using the modeling and scripting environments. For this example I thought I'd use the volume elevation curve tool created for the project I mentioned earlier. I will be making reference to a number of steps that may not be clear if you haven't performed this analysis before but you can read my previous post for clarification.

The Model

So over the years I have found that the easiest way to start any Python scripting in Arc is to look at the modeling environment first and see if the task can be done in there. You may ask "why worry about a Python script if you can create a model", but if you need to add some iteration Figure 1: Surface volume model

to the process (e.g. performing the analysis on successive elevations), for example, then you need to do this with a script. So I started by creating the model shown in Figure 1. The key here is to notice that I have created a number of "model parameters" which will show up as argument variables (sys.argv[]) when the model is exported to Python. These arguments will allow you to attach the fields from a custom tool to them giving the user the flexibility to add their own data, assign a range of elevations to the plane height variable and specify the output location for the delimited text file. You will also notice that there are two height variables (Min Plane Height and Max Plane Height) but only one is actually attached to the model. This is because we are planing to have an iterative model that starts at the bottom of the surface (e.g. 0 cu.m volume) and works it's way up to the top. This requires that we allow the user to assign a lower value and an upper value but because this model is only a single instance of the volume command, it can take only one value. Don't worry though, we will add the iterative properties when we edit the script. Once you have the model the way you like you can export it to a Python script (Model -> Export -> To Script -> Python). Before doing this though, I suggest testing the model to ensure it works to avoid spending an unreasonable amount of time on the scripting.

The Script
Now that we have a script you can open it up in IDLE or whichever Python editing environment you like. The script will look something like Figure 2 after it is first exported.

# ---------------------------------------------------------------------------
# sample_model_script.py
# Created on: Thu Sep 02 2010 04:23:36 PM
# (generated by ArcGIS/ModelBuilder)
# Usage: sample_model_script
# ---------------------------------------------------------------------------

# Import system modules
import sys, string, os, arcgisscripting

# Create the Geoprocessor object
gp = arcgisscripting.create()

# Check out any necessary licenses
gp.CheckOutExtension("3D")

# Load required toolboxes...
gp.AddToolbox("..Toolboxes/3D Analyst Tools.tbx")

# Script arguments...
Output_Text_File = sys.argv[1]

Dam_Containment_Tin = sys.argv[2]

if Dam_Containment_Tin == '#':
Dam_Containment_Tin = "defa" # provide a default value if unspecified

Min_Plane = sys.argv[3]

Max_Plane_Height = sys.argv[4]

# Local variables...

# Process: Surface Volume...
gp.SurfaceVolume_3d(Dam_Containment_Tin, Output_Text_File, "BELOW", Min_Height, "1")

Let's run through the components of this script. First of all you have to import the Python libraries necessary to run this script. In this case it is the system (sys), string, operating system (os) and ArcScripting (arcgisscripting) libraries. Right after this you need to create the geoprocessing object that will do the bull work (gp). Next you need to check out the extension(s) required for the type of geoprocessing to be done. In this case we only need 3D Analysist. Then we asign the different arguments to there own reusable variables (e.g. Output_Text_File = sys.argv[1] which assigns the name of the output file, as provided by the user, to it's own variable). In our example we do the same for the input dam containment tin, the lower plane height and the upper plane height. You may find that you need to change the order in which these arguments appear so that the tool variables can be placed in a logical order. So if you want the input TIN to be first followed by the lowest plane height then highest plane height and finally the output text file name then you need to ensure that you set the index number for the sys.argv[] object accordingly. Finally we run the Surface Volume_3D function using the geo-processing object we created earlier.

This script will do most of the work we need, however, we need to create a loop that will iterate through all the elevation changes needed to create a proper VE Curve. In most cases 9 - 12 points are adequate to create a VE Curve in Excel so we will create a loop that creates 9 points.

Height_Interval = (float(Max_Plane) - float(Min_Plane))/9
Height_Plane = float(Min_Plane)
for i in range(1,9):
    gp.SurfaceVolume_3d(Dam_Containment_Tin, Output_Text_File, "BELOW",
       str(Height_Plane), "1")
    Height_Plane = float(Min_Plane) + (float(Height_Interval) * int(i))

You can see here how we have set up a variable for the plane height and calculated it by finding the elevation difference between highest and lowest values and then dividing that by 9 (number of points for the curve). Then we create another variable to represent the current plane height (initially equal to the lowest elevation in the containment TIN). Next we set up a loop that iterates through 9 times for which we calculate the volume below the reference plane and increase it's height each iteration by the Height_Interval. In the end we have a script that, given some user input for each of the variables, will calculate a volume elevation curve and append each value to an output text file.

Creating a Tool

Now that we have the script the way we want it we need to create an ArcToolbox tool. We do this by simply importing it. I recommend adding a new Toolbox first by right clicking on the root of the ArcToolbox window and selecting New Toolbox. You should make sure to give the toolbox an appropriate name so that it is obvious what it contains. Next create a new toolset by right clicking on the new toolbox and selecting New -> Toolset. Finally, add your newly created script by right clicking on the new toolset and selecting Add - > Script, after which you will navigate to the location of your script. Once the script is added you will need to assign parameters to it so that the user can update the variables when the double click on it (see Figure 2). You get to this dialog by right clicking on the script and selecting Properties then click on the Parameters tab. You can add each parameter to the script in the logical order set out in your script.

Figure 2: Add parameters to the script

Remember that the order the parameters show in this dialog is the order they are passed to your code. This means that the first parameter showing here will be accessed in your code through the sys.argv[1] system variable.

You will also need to make sure that you update all the Parameter Properties such as ensuring input and output data sets are assigned accordingly and that Dependencies are also assigned. This last one comes into play in situations such as the requirement to select from a list of fields.

Now when you double click on the script in the toolbox you should see a dialog something like that shown in Figure 3.

Figure 3: Tool dialog for our Script

Now you should have a workable tool that will produce a volume elevation curve from an existing containment TIN. The intent of this post is not to show you how to do VE Curves but to give you a starting point for creating your own custom script based tools in ArcGIS. Hope you find it useful.

ArcGIS and Volume Elevation Curves

noreply@blogger.com (Gerry James) — Tue, 24 Aug 2010 03:16:00 +0000

I have been thinking about doing this blog for some time and have finally found the time to get to it. I have been doing volume/elevation curves for engineers for some time and have performed them in a few different applications but ArcGIS Desktop does one of the best jobs.

The premise of this particular curve scenario is one in which a tailings pond is being considered for a mining operation (any mining operation). The geotechnical engineers or geologists have a calculated number for the metric output for the mine over a 20 year period and need to make sure that a tailings pond will accommodate this number. Here is my take on the steps involved in performing a VE curve analysis using ArcGIS and 3D Analyst. Keep in mind I am not an engineer or a geologist and, even though I have performed this analysis hundreds of times, it was always double checked by one of these professionals.

Step 1: Creating the base surface.
Assuming you have a decent DEM for the area in question you will need to import it into ArcGIS as a TIN surface. This can be done using the one of the conversion utilities in Arc such as Raster to TIN under the 3D Analyst toolbar. Assuming the data is currently in an ASCII xyz file you will need to import it as a raster first then perform the Raster to TIN function. The resultant surface will be used for determining the containment area of the tailings pond and for performing the volume calculations at a series of elevations starting at the floor of the proposed tailings area and going to the top of the containment. This top elevation will be roughly determined by the desired containment volume and the practical height of the dam.

Step 2: Create the dam surface
The dam surface can be created in a number of different ways. For the purposes of this exercise we will create the dam in ArcGIS. To do this we will start with a center-line that is placed across the opening of the tailings area as discussed in Step 1. The line can be digitized into a "clean" shapefile or geodatabase feature set. You must ensure that this line crosses well over the desired elevation contour that represents the top elevation of the tailings containment. This will ensure that the resultant dam footprint overlaps the containment area. The resultant data set will need a field for elevation because the dam will be a 3D model used to finish the containment surface.


Figure 1a: Dam Geometry		Figure 1b: Dam TIN

Once the centerline has been established, an offset will be needed to create a practical crest for the dam. A typical offset might be 5 meters to create a 10 meter crest. Next the front and back slopes of the dam must be created. This can be accomplished by offsetting the crest to both the front and back by a predetermined distance (see Figure 1a). Because the faces of the dam are usually planar surfaces a single offset is usually adequate. The trick is figuring out how far to offset the lines. To do this you can look at the crest height and, using the slope of the dam face, determine how far to offset the line so that the outside limits fall below the lowest elevation in the containment surface. For example, if the dam height is 50 meters and the slope is proposed at 2:1 then an offset of 150 meters will ensure that the outside line is well below the containment surface (75 meters as in Figure 1) . This step will be needed both in front and behind the dam because the dam is used for both the tailings containment surface and to establish the footprint of the dam. Finally, convert these features to a new TIN surface using the 3D Analyst dropdown menu Create Tin from Features... It is a best practice to look at the new dam surface and the underlying elevation surface in ArcScene to ensure that the dam side slopes extend well below the Elevation surface (see Figure 2).

Figure 2: Dam Surface through Elevation Surface

Step 3: Containment Outline

The next step is fairly straight forward. You need to acquire the area outline of the tailings containment for given "ultimate dam" height. This requires that you create a planar surface at this elevation.


Figure 3a: Features for Planar Surface		Figure 3b: Tin Difference Polygons

You can do this by creating a new data set (shapefile or geodatabase feature set) and add it to the your map. Remember, you will need to assign a valid projection to this surface or you won't be able to create the necessary TIN surface. Now do the following:

Add a field to the data set for the elevation.
Establish the maximum extents for the containment by looking at the dam location and any contour or elevation information available
digitize a few perpendicular lines covering the entire area (similar to how you created the dam geometry - see Figure 3)
Assign an elevation to each of the lines. This will be the maximum elevation of the containment area which is usually the height of the dam minus an appropriate "free-board" or "draw-down".
Create a TIN surface from the this new data set.
Run the TIN Difference function from the 3D Analyst toolbox using the planar surface as Input TIN 1 and the elevation surface created in Step 1 as Input TIN 2.

The resultant data set will contain the outline of the tailings containment but will be "open" at where the dam should be (see Figure 3b).

Step 4: Create Dam Footprint
This is s very simple and short step. To create the dam footprint you simply use the TIN Difference tool and assign the dam TIN you created in Step 2 as Input TIN 1 and the underlying "DEM" TIN as Input TIN 2. This will give you a number of polygons showing the difference in volume between the two surfaces. The most important polygon will be the one showing the Greater Than area which will include the portion of the DAM above the underlying elevation surface. You may also need to dissolve some of the areas that show "No Change" values to make the polygon complete. Now simply edit this new "Difference" or "Dissolved Difference" data set and remove the polygons that do not include the dam "Toe".


Figure 4a: Dam/Surface Difference		Figure 4b: Dam Toe

Step 5: Close Containment Polygon
To close the containment polygon from Step 3, you can simply run a "Merge" on the new "Dam Toe" data set and the "Containment Difference" data set. You may need to expand the "Dam Toe" dataset to cross the appropriate containment polygon to ensure that the merge is successful and you have only one polygon for the containme nt in the end. Now edit this new data set and remove all the polygons that are not your tailings containment area.

Note: the step of stretching the dam toe is likely due to the detail and accuracy of the underlying DEM.

Step 6: Create the containment TIN for volume elevation analysis
Next you will need to create a surface that describes the containment area of the underlying elevation surface with the appropriate dam face in place. This can be done as follows:

Using the containment area polygon created in Step 4, create a raster with the Polygon to Raster tool where each cell has a value of 1
Using the Map Algebra or Times tools in Spatial Analyst, multiply this containment raster with the underlying elevation raster to create an elevation raster of only the containment area.
Convert this new elevation raster to points (Raster to Point)
Create the inside face of the dam

Overlay dam outline from Step 4 and the Dam Geometry from Step 2
Offset inside edge of dam crest a number of times so that several end up within the outline of the dam.
Adjust the elevation of each line so that it lies at the appropriate slope within the face of the dam. (e.g. 2:1)
Trim each line so that it does not extend past the outline of the dam
Remove all lines that are not part of the inside face of the dam

Create a new TIN surface that includes the elevation points created here and the new inside dam face.



Figure 5a: Containment Features		Figure 5b: TIN from Features

Step 7: Create Volume Elevation Curve

You can now create a volume elevation curve. The best way to accomplish this is by using the Area and Volume analysis tool under the 3D Analyst tool bar as follows:

Open the Area and Volume tool from the 3D Analyst toolbar
Assign the new containment surface as Input surface
Set the Height of plain to be the same as Z min
Select Calculate statistics below plane
Z factor = 1
Select Save/append statistics to text file and select an appropriate location for the file
Click Calculate statistics
Increase Height of plane by a predetermined amount (e.g. 1o meters). This will likely be determined by the number of points required to create a good curve (typically 8 to 12 points is good)

Figure 6: Area and Volume Statistics Tool

You will now need to open the text file you created and modify the file so that it can be used in excel. The Area and Volume tool writes a number of statistics we don't need like 2D and 3D areas so you will need to remove these. At the same time you will need to create a comma delimited file that can be opened into columns in Excel. In the end you need only the Elevation and Volume items for each iteration in the previous steps. Save the file and open in Excel.

You will now need to add column headers and create the curve. To do this select the Insert -> Chart -> Scatter Chart. When prompted select your data for the chart. You will likely need to switch the columns in the appropriate series so that the volume runs along the X axis and the elevation along the Y. In the end you should have a decent looking VE curve.

Figure 7: Volume Elevation Curve
I hope this was helpful to those that may be new to this process. Please stand by and I will try to add other useful tutorials in the future.

How to create a multi-unit tracking application

noreply@blogger.com (Gerry James) — Tue, 23 Mar 2010 22:50:00 +0000

I am currently working on an application that will allow users to track any number of GPS enabled radios within a specified group. The key to this application is the proprietary GPWGT NMEA string which not only has GPS location info but also an identifier. These strings are typically transmitted from each radio in a group or polled by a central radio in a group. This string has the basic construct of

$GPWGT,<FixStatus>,DD.MMMMM,d,DDD.MMMMM,d,<Altitude>, <COURSES>,<SPEED>,<NameId>,<SCODE>,<GroupId>,<Status>,<ICON>,<Date>,<Time>*CC<CR><LF>

Where *CC is the "check sum" value and

are carriage return and line feed respectively.

As you can see it is really as simple as parsing out the individual values or tokens from the NMEA string. The GPS microphones being used for this project come with USB drivers that create a vertual COMM port so you can use the .NET commport control to receive the strings. It then mearly requires a way to manage the different units. In our case we created a utility to add new units and store all their attributes in a central XML file. Within the application itself we store GPS MIC classes in a collection that can be accessed whenever one of these GPWGT strings is received. This allows us to plot points for these units with unique symbols and colours.

So you can see that programming to use these multi-unit GPS transceivers is not rocket science. It is quite easy to do and a usable application can be created in a fairly short period of time.

Building an Effective open source GIS application - Chapter I

noreply@blogger.com (Gerry James) — Thu, 20 Aug 2009 16:30:00 +0000

Recently I was thinking about what goes in to building of a GOOD GIS application, open source or otherwise. I have been involved in creating a number of applications, both web based and desktop, and think I am fairly adept at planning the development of such things, but what really goes in to making a useful open source GIS application. This has lead me to dedicate the next four blog posts to looking at the art of designing, developing and deploying an effective GIS using open source technologies. The first chapter touches on the pre-planning and information gathering stage, Chapter II will look at the planning stage, the third chapter is dedicated to the development phase (looking at the process and is not technical), and finally the last chapter will look at deployment and the issues surrounding this critical stage.

Chapter I: Know your audience?
There is an old adage in the newspaper industry that goes something like "Always write to the lowest common denominator". What I understand this to mean is, don't use a bunch of technical jargon which will confuse the general public. This concept is equally important when building a GIS application. You need to understand who will be using the application and what it will be used for.

I guess the first question to be asked is what does the thing need to do. This seems like an obvious question but, in my experience, many application developers don't fully understand what their application needs to do, I mean REALLY understand what it needs to do. This is not an exercise of putting a list of features together that will go into the software. That list should spawn out of a true understanding of what the software needs to accomplish and who will use it. The second part is possibly the most important question that MUST be answered prior to building the application. You must understand whether the application will be used by technical practitioners, non-technical management, non-technical other and so on. Once you understand who will use the application then you can better understand what the application must do.

Example:
Your boss, a technical GIS practitioner, comes to you and says you need to build an application that will allow the user to track a set of assets by grabbing coordinates from a GPS and giving the user the ability to update attribute values on the fly, now go build it. Sounds easy, you need to build an application that can connect to a GPS unit, read the NEMA strings (or equivalent), plot on a map, allow the user to update field values on the fly and convert to a format that can be used in other GIS or web mapping applications. Simple right? Just find out what the assets are and go build it. Now wait a minute. What kind of data collection will it be used for? Environmental monitoring? Engineering? Search and rescue? This is extremely important because it may allow the developer to limit any of the presets to values specific to the target industry. Perhaps some predefined values in pull downs to populate field values and that sort of thing. And what is the level of GIS knowledge of the user? If they are experts you may be able to build the application as a plug-in or add on for an existing application such as QGIS. These are just a couple of reasons why it is imperative to understand the end users needs.

The image on the right is a screen capture of the Quantum Navigator plug-in for QGIS. This plug-in is still in development but will be intended to track GPS input in real time and allow the user to update values, translate between points and tracks and output different data types. The application is being written in Python so will integrate with any operating system. The application is intended to give quick route analysis capabilities to the user will allowing them to add their own data and integrate open street map. This application shows great promise but is intended for the user that may be interested in looking for a optimal route from a specified location to another. This application would not provide all of the functionality that an environmental monitoring team might need for example. Now this application is intended for a very broad audience so it probably meets the objectives of its developers but it helps to illustrate the need to intimately understand your end users requirements. If you were wanting this application to be adequate for an environmental consultant, for example, you would need to add the capabilities to update attribute values and even provide a series of preset values for certain fields. Capabilities that can only be added with intimate knowledge of the intended use.

What should I ask?
So after the this seemingly endless diatribe lets try to understand who the end user is and what questions should I ask that user. So who is going to use this application? Sometimes the initiator of the project will know exactly who the end user will be and what they will user the application for. In this case you as a developer can simply ask them, "Who is the end user and what will they be using the application for". However, quite often it is unclear who all of the users will be and, if you don't know who's using it, how can you know what they need. In these cases it requires some detective work and some of the things you should know up front are:

What information are you hoping to gather?
What asset(s) is this information attached to?
Traditionally, who has collected this data?
Which GIS applications are already being used? (you should probably already know this information)
How will the resultant data be stored (e.g. PostgreSQL, Oracle etc.)

These are just a few of the questions that need to be asked but I hope they provide a good start. This chapter has focused on the pre-planning process, the next chapter will focus on the planning process and will hopefully answer questions like; how do I keep up with clients changing needs, who should be involved in the planning process, should I create a Gantt chart? and more.

I hope you found this chapter to be informative and will come back for the next post.

MapWindow Release Candidate 2 Announced

noreply@blogger.com (Gerry James) — Mon, 06 Jul 2009 14:59:00 +0000

The MapWindow Version 4.7 RC2 has now been released. This is a good sign and seems to indicate the time line for the final release (October 2009) will be attainable. I have notice some bugs have been fixed for this RC including cleaner install on Vista machines and some issues around cartographic elements like exporting the scale bar and north arrrow. A couple of bugs that are still to be resolved at this point are previous zoom issues and incorrect scale bar showing for printing and previewing maps.

All in all this release (4.7) promises to be a significant improvement on 4.5. There has also been significant movement on version 6.0 which is being developed in parallel with the current version. Visit http://www.mapwindow.org for more details.

Converting coordinates to a local SRS

noreply@blogger.com (Gerry James) — Thu, 25 Jun 2009 01:09:00 +0000

I am currently doing some work for a small junior mining company which requires that a number of data sets be translated between UTM NAD83 Zone 11 and a local coordinate system. If you have never had any dealings with mining projects you may not know that it is common practice to create a local coordinate system aligned to the estimated ore body (sometimes called a mine grid). In some cases this CS may have been developed many years previous when the mine was originally developed and, often times, there is no adequate documentation to help define the system. If you are lucky you may be able to establish shift, rotation and scale values but this will likely require the assistance of a surveyor.
In this case, however, I was lucky enough to have the translation parameters. As it is very difficult to use traditional methods to convert from a earth centered to non-earth reference system I decided it would be much easier to do the transformation in a more manual fashion. By this I mean an x/y shift and rotation (no scaling required). Now having said that I also realized that this was a task that would be required many more times as the project progressed. For this reason I decided to roll up my sleeves and create a little stand alone utility. This is always a little daunting during an active project but I felt it could be accomplished fairly quickly and for very little additional cost to the client as I was going to utilize the open source ActiveX MapWinGIS and SharpMap dlls to do most of the work for me. After some fairly basic survey geometry calculations I had an algorithm that could be applied to convert all UTM coordinates to the local grid and vise versa.
Now to put it into a utility that would convert the features in a shapefile using this algorithm. As I am very familiar with MapWinGIS and SharpMap, I decided these would be the ideal ActiveX libraries to perform this task. It was fairly simple as the MapWinGIS dll provides read/write access to many of the attributes of features (shapes) such as the x and y coordinates of the individual vertices (points). The conversions can be applied to the vertices very simply such as:
MapWinGIS.Shape s = osf.Shape(shpindex);
string[] xy = toMine(s.Point(k).x, s.Point(k).y);

Where toMine is a Private Function within the VB.NET class that performs the conversion and s is the current feature (shape). The string array xy() will then hold the converted coordinates that can be applied to the shape point as follows:
s.Point(k).x = xy[0]

When all is complete you will have a new data set with all the features "projected" to the local coordinate system. Now the geologists and engineers using sophisticated modeling software (that apparently cannot easily do this kind of conversion) can quickly and easily convert the data for use in resource estimates and mine design applications.
One other issue lay in the fact that many of the data sets might be provided in other data types (i.e. MapInfo TAB files). For this reason I chose to use the open source GIS ActiveX dll SharpMap. This allowed me to convert these files into a common format (ESRI Shapefile).
This example illustrates how easy it is to create a versatile tool box using open source applications and libraries without having to bother with expensive licensing. MapWinGIS has been developed to work seamlessly within the .NET framework so, once you have referenced the dll, you can easily access all of these calls. I suggest downloading all the files from http://www.mapwindow.org or http://www.codeplex.com/SharpMap and try it out. I think you will find they will become an invaluable tool in your GIS toolbox.

Google Sites with Gadgets for Project Management

noreply@blogger.com (Gerry James) — Wed, 03 Jun 2009 19:56:00 +0000

I know this isn't strictly speaking geospatial, however, I am called upon to provide project management services on a fairly regular basis and I believe any GIS manager, consultant or technician will have to do this from time to time. I have recently been re-introduced to Google Sites as a viable tool for this purpose. My most recent discovery was of a Google Gadget by Viewpath simply called Gantt Chart. This cool little gadget lets you create a work schedule that can be used to create a very nice little Gantt chart. (see the YouTube vid at http://www.youtube.com/watch?v=IGwXQryI2-A).

On top of the gadgets there are lots of resources on Google Sites to add lists, documents, presentations, files and much more. I was recently awarded a development project that is estimated to last 2 1/2 to 3 months and I was able to develop a PM site for it in about an hour. Now there is an ongoing conduit for all players to collaborate, communicate and share. And with the sharing capabilities it is easy to keep the project information private.

I suggest anyone who is taking on projects check out the capabilities of Google Sites.

Leverage the power of PostGIS in QGIS with PgQuery

noreply@blogger.com (Gerry James) — Fri, 08 May 2009 15:35:00 +0000

As many of you may already know I have created a Python plugin for QGIS called PgQuery which gives easy access to powerful PostGIS geoqueries within the QGIS environment. Well it's finally here and you can download it by navigating to the downloads section @ http://www.carteryx.com/.

Today I would like to go over how to install and use this tool with the hope that it will become a great help to all those wanting to leverage the power of OSGIS in their organization.

Install
First method: Installing this package really couldn't be any easier. After downloading the archive file unzip it to the user specific Python Plugin directory (C:\Users\
\.qgis\python\plugins). Currently the plug-in doesn't support installation to the main QGIS Python plugins directory.

Second method: You can also install PgQuery by opening QGIS and selecting the Fetch Plugins item under the Plugins menu. In the resulting dialog select the PgQuery for QGIS entry and select the Install/upgrade plugin button.

After doing either one of these methods you can activate the plugin using the QGIS Plugin Manager (PM). It will then show in PM as PgQuery for QGIS (Version 0.1.1. You can just select this plugin and you will see a new entry for PgQuery under plugins as well as an icon on the tool bar.

Running PgQuery
To run the app either select PgQuery for QGIS from the plugins menu or the icon from the tool bar. When the application opens you will see a number of controls on the main form:

An Output Schema pull down to allow you to select the schema of any output tables

An Output Table box which is self explanitory

A Query box which allows you to create queries from scratch, add query components from some of the other controls, or select a query template from the Query Template pull down.

A Query Template pull down as described in 3. When you select a query template it builds the query for you from the pre loaded queries or from a query that you have built and saved to the default location.

A Functions list box which allows you to double click to add functions to the query at your current cursor location.

A Tables tree control that shows a list of all the tables available in the currently loaded PostgreSQL database. If the table is spatial the geometry type will show in the column beside the Table column. You can also double click the tables to add them to the query.

Finaly the Geometry Column pull down that allows you to select the geometry column from any of the currently selected tables.

You will also find a number of menu items at the top.

Under the File menu there are sub menus to allow you to:

Open an existing project file that will bring you back to a previous state of SQL query development
A Save and Save As menus to save the appropriate project files
And, of course, and exit menu to close the application entirely.

You will find a Query menu containing sub menus that give you access to:

The Run menu which runs the current query as shown in the query box on the currently active database
The Save menu to allow the current query to be saved to a Query Template file for later use.
The Clear menu that allows all query statements to be cleared to start over from scratch.
And a Check Box that allows the user to select whether or not they want to add the resulting spatial table to be added as a geometry data set to the QGIS map view.

The Database menu has a listing of all available PostgreSQL database connections in QGIS that the user can select to make current.
Finally the Help menu gives access to:

Basic Help with PgQuery for QGIS
And a basic About PgQuery statement

This application requires that you have PostgreSQL 8.2 or later installed with PostGIS 1.3.2 or later. You can fairly easily build a good application stack using the PostgreSQL Stack Builder that comes with 8.3. This version comes bundled with the most recent release of PostGIS.

You will also need to have PyQt and Python for QGIS installed. All of these can be installed using the online stack builder that comes with the OSGeo4W installation. Make sure to read the INSTALL.txt file that comes bundled with the plug-in.

You will also find a help document in the "docs" folder of the pgquery_plugin directory. this help doc is called help.html which can be opened in any browser and gives you some guidance for using this tool.

Try it out and see how powerful PostGIS can be in your enterprise GIS environment. This enterprise database application has hundreds of spatial functions that allow you to perform myriad geoprocessing and analytical tasks. For more assiatance on PostGIS you can go to http://postgis.refractions.net/ and look at the documentation available there.

I hope you find this tool very useful and that it brings enterprise GIS into the reach of ANY organization regardless of size of budget.

Who needs ArcMap?

noreply@blogger.com (Gerry James) — Thu, 19 Mar 2009 15:01:00 +0000

Over the past month I have been working on a Python plugin for Quantum GIS (QGIS) that allows the user to access the geoprocessing functionality of PostGIS without leaving the QGIS environment. I call it PgQuery for QGIS for anyone that might care. I have tried to make the plugin easy and intuitive to use with some click/double click functionality to insert PostGIS functions and tables into the query statements. That is the end of my sales pitch for this open source application, it is currently being tested by some associates and I will release it to the OS community very soon.

I mention this application because it illustrates how powerful open source GIS applications have become. I thought I would make a brief comparison here to give a sense of how many of the traditionally ArcGIS functions are actually "doable" in QGIS with one of the myriad plugins available such as GRASS or PostGIS.

This table illustrates a few of the geoprocesses that both the ESRI stack and the OS stack handle very well. The only real difference is the cost. The ESRI stack above represents about $25,000 (requires ArcInfo for Linear Referencing and Spatial Analyst for LOS) where as the OS stack is free. I am a firm believer in the right tool for the job and I believe that ESRI still has an advantage in cartographic output but what I think this illustation shows is that you MUST make a proper needs assessment before deciding on a GIS implementation. You may find that your current and future requirments will fit well with an open source deployment that will potentially save you thousands of dollars.

Don't get me wrong, I am not averse to an ESRI or any other commercial vendor deployment. I am only saying that, when looking at developing a GIS, it is worth considering open source as part of that strategy. In fact larger commercial developers such as ESRI and Safe are starting to support open source platforms such as PostgreSQL/PostGIS.

I will be posting a link to the QGIS plugin I mentioned at the begining on this site very soon so check back often. I will also be announcing it's release on Twitter and in a number of LinkedIn formus such as GIS, Geomatics and GIS Professionals so stay tuned.

Open Source GIS Choices

noreply@blogger.com (Gerry James) — Sat, 28 Feb 2009 22:04:00 +0000

In the last week or so I have noticed a number of posts on Twitter, LinkedIn and the like talking about the number of choices available in Open Source GIS. This list is exhaustive and you can check out The GIS Forum for the most exhaustive list available. So this list is huge, so what? Well this means that there are now enough choices that you can develop and maintain an enterprise level GIS without the costly licensing that is associated with proprietary implementations. As an example a comparison of an enterprise GIS deployment might look like this:

Web Mapping
ArcIMS - $18000
UMN Mapserver $0.00

GeoDatabase
ArcSDE - $18000
PostGIS/PostgreSQL - $0.00

Desktop GIS
ArcMap - $2500
Quantum GIS - $0.00

Total Proprietary - $38,500
Open Source - $0.00

Interoperability

Of course this doesn't even factor in the yearly licensing cost of the proprietary software. Sounds like a no brainer...right? Well maybe not. Companies such as ESRI have spent years and millions of dollars to ensure their software is completly compatable accross the board. What I mean by this is, if you deploy a completely ESRI solution it is very likely that each part will interact with each other part very well. Because the open source projects are run, for the most part, fairly independantly this may not be the case for OSGeo. However, more and more OSGeo packages are croping up that put these software together in interoperable bundles. Some examples are the OSGeo4W project which is a grouping of such GIS powerhouses as Quantum GIS, FWTools/OpenEV and uDig. Another example is the GISVM which is a Ubuntu Linux VMWare image that can be run on Windows, Linux or Mac OS using the VMWare viewer and contains a number of GIS applications such as gvSig, Grass GIS and PostGIS.

Flexibility

The power of any GIS lies in it's customizability. For proprietery platforms they have development environments such as VBA and libraries that contain a number of progamable objects such as ArcObjects. Open source is no different. As an example Quantum GIS has a number of bindings for languages such as Python and C++. These bindings allow the developer to create embeded plugins to extend the functionality of the software as well as stand alone applications that can provide a highly focused set of features to the user.

Meaning?

So what does all this mean. Well it is true that there are many open source choices to help build a highly functional and powerful GIS for your organization, it is still imperative that you perform a comprehensive needs analysis. This can be done completely in-house or with the help of a consultant but is essential. This will help you know exactly what the system needs to produce and allow the developers and deployers to pick the right applications to meet your busines objectives.

Still GeoSpatial Opportunities in Lotus Land?

noreply@blogger.com (Gerry James) — Sat, 31 Jan 2009 17:19:00 +0000

I recently got an email from a LinkedIn contact asking me what my thoughts were on the ongoing opportunities in the geo industry in BC. This caused me to pause and reflect. It was just two short months ago that I was informed my services would no longer be required by a couple of junior mining clients. They had hit extremely tough times and were having to cut back substantially. I have to admit this was one of those rare moments in my career when I said "why did I choose this line of work" and especially "why did I start my own consulting firm". And the beginning of January certainly didn't give me a lot of reason to feel more optimistic.

Leap forward to the end of January: I just recently managed to negotiate a new consulting contract with an engineering firm here in Vancouver and will be presenting a business idea to a few interested investors over the coming weeks (already have them lined up). So what does this mean? Well, my take on this is, while there are definitely doors being shut, there are also many doors and windows opening, especially for consultants. Look for the opportunities that this economic "adjustment" are providing. My business specializes in open source GIS and I think, given the propensity to cut costs, this sector of geospatial will see a marked increase in use over the coming months. (READ THIS) I have also looked for other ways to help companies reduce costs and increase efficiencies using GIS which will be of paramount importance for survival.

So what am I trying to say? I think you need to look at the downturn with cautious optimism and, above all, look for opportunities...EVERYWHERE! I am not advocating quitting a good paying job to start a consulting business (GOD forbid), but I think we all have to think like entrepreneurs. There are opportunities to increase your business, or at least maintain it, during these times and there are opportunities for the entrepreneurial minded to help their employer in this same way, making you the hero.

KEY: Think Outside the Box!!!!!

I am interested to hear what others think about this. Drop me an email gerry.james@carteryx.com.

Create a GRASS Mapset in QGIS

noreply@blogger.com (Gerry James) — Tue, 27 Jan 2009 18:24:00 +0000

I have had the question asked a few times lately of how to create a GRASS mapset with a fixed region in Quantum GIS. I would like to take this oportunity to address this here.

For these instructions I am refering to QGIS 1.0 (Kore).

Creating a mapset in QGIS

Open QGIS and navigate to the menu Plugins -> GRASS -> New Mapset (you must attache the GRASS plugin with the Plugin Manager first)
In the resulting dialog, navigate to an existing directory for the GRASS database or name a new one and select Next >
Select an existing location or check "Create new location" and name a new location and select Next >
In the next dialog you should see in the Existing mapsets at least one mapset called PERMANENT (default GRASS mapset). Now you can name a new Mapset in the box provided. When you select NEXT the new mapset will be created.
Select Finish

You will now need to set the region extents which can be done using an existing GRASS vector or raster layer.

Navigate to menu Plugins -> GRASS -> Open Mapset and open the mapset you just created
Now go to menu Plugins -> GRASS -> Open GRASS Tools
Scroll down to the tree item File - Import vector - v.in.ogr (using vector layer import for this example)
Select the OGR file (shapefile, MapInfo etc.), select a name for the output GRASS vector layer and select Run (note: do not select the check box for "Limit import to current region")
Close the import tool and go back to the Modules Tree to select Config - Region - Region Settings - g.region.multiple.vector
Use enter or select the name of the newly importe vector layer and select Run (note: you can select multiple vector layers and the region will be set to the extents of all)
When you open this mapset you should see a red boundary around the region

You should now have a new GRASS vector mapset with the region set to one or more vector layers. You can now begin adding new data, creating maps and analysing the data with the other GRASS and QGIS tools.

Geospatial in 2009

noreply@blogger.com (Gerry James) — Sun, 04 Jan 2009 23:08:00 +0000

Well we're four days into 2009 and I guess it's time to look at what we can expect in the geospatial world. So, on this line, here are a few of the titles and technologies I have recently discovered or re-discovered for the New Year.

Depiction - a compact desktop application that gives the user easy download access to freely available public data and provides some tools for developing scenarios such as emergency response. Currently this application only gives access to

GeoNetwork - this is kind of a re-discovery for me. "GeoNetwork opensource is a standards based, Free and Open Source catalog application to manage spatially referenced resources through the web". I believe this application is crutial when trying to manage data in an enterprise environment. I allows the user to upload, catelog and metatag geospatial data sets. I think it's worth a first look if you haven't already and a second look if you have.

Google Maps Mobile - this little mobile API is a great little bit of code that enables smartphones to access google maps and derive the users current location with or without GPS. I use this little app frequently when I travel.

These are just a few of my favs for 2009. Let me know if you have any that are must see or must have for the New Year.

GIS in Tough Economic Times

noreply@blogger.com (Gerry James) — Tue, 23 Dec 2008 03:32:00 +0000

I have to say that the slow down in the global economy is affecting almost everyone now in some way or another. I am currently in business development mode and I can tell you it is definately slim out there right now. I think almost everyone is more cautious about taking on new projects and perhaps putting some things off until financing becomes more readilly available again. Having said this I think there is even more of a need for some organizations to look at open source, particularly with regard to GIS. It has been proven time and again that open source can save many companies a significant amount of money due to the low/no cost of open source software. I have been involved in implementing both open source and commercial software for clients and I can tell you the cost differential is truely significant. As an example the following shows the difference in software costs between an open source enterprise geospatial database system and a commercial one:

PostGIS/PostgreSQL - $0
ArcSDE - $18,000+

You also have to keep in mind that ArcSDE is only a geospatial layer that must sit on an enterprise database system like SQL Server or Oracle. This means the cost can be significantly more than this.

Now this cost savings can easily get nickle and dimed away if you spend alot of focusing on trying to get the OS GIS software to do things that it won't easily do. The key consideration here is PLANNING. Make sure you budget for enough up-front planning and the design, development and implementation will certainly be much more efficient. There are many techniques used by GIS architects and designers to make sure that the software you choose, either open source or commercial, will do what the project requires. I always recommend getting the advice of a consultant before taking on a project with technologies with which you are unfamiliar.

Anyway, I urge you, if you haven't already, to consider open source in the New Year for projects that may otherwise get put on hold or even canceled.

ArcGIS Server vs. Mapserver/PostGIS

noreply@blogger.com (Gerry James) — Thu, 13 Mar 2008 03:49:00 +0000

I know that ArcGIS server is supposed to be a true example of SaaS and the Mapserver/PostGIS example I have given is likely considerd something else but I would like to share an experience on a project that spans both worlds. About 5 years ago the company that I was working for at the time got a contract to build an online database/spatial data managment system for a Canadian Government Agency. Originally they required a proof of concept so we decided to build this thing using open source technologies. We very quickly found that UMN Mapserver combined with PostGIS/PostgreSQL and a couple of other nearly free software titles would do everything we needed. By the time we were finished we had a highly interactive tool that gave the user access to such things as dynamic cross sections, dynamic drill logs including down hole temperatures, the ability to upload photos and add them as points on the map and a dynamic 3D visualization tool. The grand total for software expenses was under $2000 CDN.

Anyway, to make a long story even longer, about a year and a half ago the client decided that they wanted to host the application themselves. As they are completely an ESRI shop we were contracted again to migrate the application to ArcIMS and ArcGIS Server. This task did not come without several headaches that's for sure but we managed to get much of the application migrated. The one thing that was impossible was the ability to migrate the 3D visualization tool. Apparently ArcGIS Server won't allow you to consume their ArcScene objects without a thick client such as ArcGIS Explorer or full blown ArcScene. The client did not want their clients to download any software for this purpose so we ended up going with the much less versitile solution of creating a number of static views as jpeg files. This also required some additional programming to create a viewer for accessing the images. Needless to say this required an adjustment to the scope of the project and some creative negotiation with our client.

So why am I telling you all this; keeping in mind that the software we were using with the original application cost about $300 to generate the 3D views. Why is it that ESRI couldn't provide this functionality in thier much more expensive software. Please don't get the idea that I am anti ESRI because I think ArcGIS Server is a phenominal product but I don't understand why, with the advent of such technologies as Google Earth and Microsofts Virtual Earth, ESRI didn't throw in the ability to view snapshots of the 3D scenes without a thick client.

As the project manager for this application I made an official improvement request to ESRI but I haven't heard whether this improvement has been made as of yet. I understand that ESRI as the largest supplier of GIS software can't include everything into thier applications but I just thought it was a no brainer given my comments about virtual earth technology and the web.

Please let me know what you think about this especially if you are involved in creating rich web applications for your clients or employers.

-Gerry-