There are already Windows packages available for MapServer ready for deployment. However if you need to make a modification to the source files, apply a patch, or just want to see how MapServer is made then you can build MapServer from its C source files. Thanks to the ready made build kits this process should only take 10-15 minutes from start to finish.

Prerequisites: Microsoft Visual Studio with C++

1. Download the MapServer SDK (Software Development Kit) from http://www.gisinternals.com/sdk/. I’m using Visual Studio 2010, and building on a 32-bit Windows machine (still using XP..), so I’m using release-1600-dev

These build kits, provided by one of MapServer’s developers Tamas Szekeres,  contain everything you need to compile both MapServer and GDAL. Checking out all the files from the MapServer SVN repository is not enough, as this does not contain the GDAL or RegEx dependencies (DLLs that MapServer depends on), or SWIG – a tool used to create MapScript files for each of the different scripting languages available in MapServer such as Python, Ruby, PHP, Perl etc.

2. Unzip the MapServer SDK on your hard-drive. Make a cup of tea as this takes awhile. In this example I’ve extracted the files to D:\MapServer. You should see a folder structure similar to below.

3. Open up a WIndows command prompt. Navigate to the folder you extracted the files. Now run the following command:

“C:\Program Files\Microsoft Visual Studio 10.0\VC\bin\vcvars32.bat”

This is a Windows batch file that sets up environment variables so you can use the MS Visual Studio tools in your current command session.

If this file is not present then make sure you have Microsoft Visual Studio C++ 2010 installed. You can check this in Add/Remove programs, or through the About menu in Visual Studio. It is not installed by default so you may have to go and find your installation disks or ISO.

4. The nmake tool is a Microsoft program that builds projects based on the contents of a Makefile. It’s the command line equivalent of pressing F5 in Visual Studio. You can look at the contents of the Makefile in a text editor – this is where all the hard work is done. Run the following command:

nmake ms

Sit back and watch the compilation! If you want to record what happened to a log file you can use the command:

nmake ms >> my_log.txt

After this build there should be a new folder D:\MapServer\release-1600\ containing all the files you need to deploy MapServer. When unzipped this folder already contained a build, but the timestamps for the DLLS should now be from a few minutes ago.

Rebuilding MapScript

The Makefile D:\MapServer\Makefile contains details on how to build MapScript. The Makefile commands are great to follow if you want to just rebuild Python MapScript rather than the whole of MapServer. Just search for “ms-python” and you’ll see all the DOS commands you need to rebuild Python. You can then run these “by-hand” as shown below.

My original build produced MapScript for Python 2.6, but I also wanted to create a build for 2.7. To create a Python 2.7 compatible MapScript follow these steps:

1. Open the D:\MapServer\mapserver-6-0\mapscript\python\setup.py file in a Python editor such as IDLE. Change the library_dirs to read:

library_dirs = ['../../','../../../release-1600/lib/']

It should be possible to set these paths outside the setup.py on the command line, but I was not able to get this to work. This change adds in the folder containing all the .lib files that were created when we built MapServer. Check these files are present!

2. Navigate to the D:\MapServer\mapserver-6-0\mapscript\python directory.

3. Run the following commands (taken from the Makefile):

SET DISTUTILS_USE_SDK=1
SET MSSdk=1
SET LIB=%LIB%;$D:\MapServer\release-1600\lib

4. Run the following command:

python.exe setup.py build

You should now have two new files – mapscript.py and _mapscript.pyd (a .pyd file is a DLL with a couple of differences).

Here be Errors

I took out all the twists, turns, and errors I encountered along the way to make the process appear simple, and me to appear to be an expert..however I ran into a myriad of problems, the error messages of which I’ve included below, along with how to fix them.

‘nmake’ is not recognized as an internal or external command, operable program or batch file.

- Make sure you have run the “C:\Program Files\Microsoft Visual Studio 10.0\VC\bin\vcvars32.bat” file in your current Windows command session.

LINK : fatal error LNK1181: cannot open input file ‘gd.lib’
error: command ‘link.exe’ failed with exit status 1181

- You need to edit the setup.py file to include the folder with the .lib files – see “Rebuilding MapScript step 2.” Also make sure you have built MapServer and the gd.lib file is in your D:\MapServer\release-1600\lib folder.

building ‘_mapscript’ extension
error: Unable to find vcvarsall.bat

- When building MapScript you need to set the correct environment variables – see “Rebuilding MapScript step 3”

_mapscript.pyd.manifest : general error c1010070: Failed to load and parse the manifest. The system cannot find the file specified. error: command ‘mt.exe’ failed with exit status 31

- Try cleaning the Python MapScript build folders with the following command:

D:\MapServer\mapserver-6-0\mapscript\python>python.exe setup.py clean

A straw poll was taken on the OpenLayers mailing list in April 2010, and OpenLayers v3 development is taking place in git, with the central repository stored on GitHub. This means working with SVN and OpenLayers may become redundant. However as many OSGEO projects are stored in SVN with no current plans to move the same plugin and knowledge of working with SVN will still be useful.

For further details on working with SVN there is a free e-book “Version Control with Subversion” available at http://svnbook.red-bean.com/

Installing the Plugin

1. Open the Plugin Manager, by clicking on the green jigsaw piece in the toolbar below.

2. Right click and select Install in the plugin manager. A list of available plugins appears.

3. As you can see there are a couple of SubVersion related plugins available.  Fortunately StackOverflow had a question titled SVN plugins for Eclipse – Subclipse vs. Subversive, which leant towards using Subclipse over Subversive. Life’s too short to dwell on possible advantages of using one over the other, so Subclipse it was.

4. Next follow a number of screens related to security certificates, and additional components. Other than selecting a couple of “Recommended” options I left the defaults.

5. Once the plugin was installed and Aptana restarted I now had a new project type = “Checkout Projects from SVN.”

6. The location of the OpenLayers trunk (the latest development version) can be found at http://svn.osgeo.org/openlayers/trunk/openlayers

7. Selecting the root will check out OpenLayers, and all the associated tests, build tools, and documentation.

8. After this you can create a new project location, and the plugin fetches all the latest files directly from the OpenLayers SVN server.

Using the Plugin

Now this is set up the fun can start! The reason I wished to go into the changes made to OpenLayers was that one of my unit tests was throwing an error after I have updated my OpenLayers 2.10 to the latest development version in OpenLayers. The error was:

exception: : object: wkt.replace is not a function

in ( [object Array]) at http://localhost/openlayers/lib/OpenLayers/Format/WKT.js line 59

You can see when files were last modified, and by whom, in the File explorer tab. The most powerful feature though is being able to compare the current files with any past version of OpenLayers. This can be done by right-clicking on a file, and selecting the “Compare With” option.

Now to diagnose my error, I looked to see if any changes had been made from the 2.10 version of OpenLayers(which my unit test passed) , with the current development version. Selecting the “Branch/Tag” option allows you to select a version (based on its tag). Version 2.10 is listed after 2.1 due to sorting by characters rather than numbers.

A comparison showed that the file and line had indeed been changed. A replace function had been added to the code to remove whitespace from the WKT string.

Upon further debugging it turned out the problem was my test had been passing an Array rather than a String, and as an Array object does not have a replace function is was correctly throwing an error. This served as a good reminder that unit tests can themselves have bugs..

On the web system I have an OpenLayers map, with an OpenStreetMap background layer. As with nearly all the online tiling services, these are projected in EPSG:900913 (the Web Mercator / Google projection).

I wanted to display a road network on top of these as a WFS (Web Feature Service). The source data is defined in the Irish National Grid projection (EPSG:29902).

At the same time I also wanted people to be able to connect to the WFS via a desktop GIS client using the Irish National Grid projection.

Multiple Spatial Reference Systems (SRSs)

The Rules for Handling SRS in MapServer WFS are not the easiest to work with. The first paragraph of the rules imply that multiple projections are not available for a single WFS layer available through MapServer:

Contrary to WMS, the OGC WFS specification doesn’t allow a layer (feature type) to be advertised in more than one SRS. Also, there is no default SRS that applies to all layers by default in the OGC WFS spec. However, it is possible to have every layer in a WFS server advertised in a different SRS.

However there is now support for multiple SRSs – added in version 6.0 of MapServer as outlined in this ticket. You can check how a WFS layer is “advertised” to a client using a URL similar to the following:

http://localhost/mapserver/mapserv.exe?map=C:/PathToMapFile/MyMap.map&service=WFS&request=GetCapabilities&version=1.1.0

Each of the layers should have some XML which lists the DefaultSRS, and if you have set it up one or more OtherSRS tags:

<FeatureType>
    <Name>Test</Name>
    <Title>Test</Title>
    <DefaultSRS>urn:ogc:def:crs:EPSG::900913</DefaultSRS>
    <OtherSRS>urn:ogc:def:crs:EPSG::29902</OtherSRS>

To get to this point however is not the simplest of journeys..

How to Set Up Multiple SRSs in MapServer

The SRS rules explain how MapServer advertises the available projections for each of your WFS layers:

• If a top-level map SRS is defined* then this SRS is used and applies to all layers (feature types) in this WFS. In this case the SRS of individual layers is simply ignored even if it’s set.
• If there is no top-level map SRS defined* then each layer is advertised in its own SRS in the capabilities.
• By “SRS is defined”, we mean either the presence of a PROJECTION object defined using an EPSG code, or of a “wfs_srs” metadata at this level.

I recommend you read the further explanations in the official documentation, as there are a lot of other rules on how priorities and conflicts between different configuration settings work. From the documentation:

Confusing? As a rule of thumb, simply set the wfs_srs at the map level (in web metadata) and never set the wfs_srs metadata at the layer level and things will work fine for most cases.

However this only applies if you don’t want to use multiple projections. This cut-down example of a map file may help to illustrate what these rules actually look like in code: 

A few additional notes from spending a number of hours struggling with these:

• The DefaultSRS is always the first item in the wfs_srs list. The order these are listed is therefore critical.
  
• If no metadata is available then the SRS is taken from the Map Projection. In this case there are no OtherSRS values.
• I’d recommend not sharing the SRS definitions between WFS and WMS (you can do this with the ows_srs metadata setting) as the order makes no real difference to a WMS service, but is critical for a WFS service.
• You need to set the PROJECTION object for each WFS layer with multiple projections. Even if the layer is in the same projection as the MAP projection. This was probably the hardest issue to track down. If you don’t set it you end up with the nasty situation of the client BBOX in one projection searching for features in the other projection. It would be nice if MapServer just threw an error at this point. Instead you’ll likely get a message such as:

<gml:boundedBy>
<gml:Null>missing</gml:Null>
</gml:boundedBy>

The WFS Client – OpenLayers

Once you’ve got you DefaultSRS and OtherSRSs set up correctly in MapServer and checked them with the GetCapabilities request, you can to try to connect to them with a real WFS client. The MapInfo desktop WFS involved entering the server name, and selecting layers from a list. The OpenLayers connection was more difficult.

To add a WFS to OpenLayers, you create a vector layer with the WFS protocol. Make sure the WFS version is set to 1.1.0. Why? If you use WFS 1.0.0 the srsName is not supported, so you can’t use OpenLayers to decide in which projection you want your features returned.

protocol: new OpenLayers.Protocol.WFS({
              featureType: “Test”,
              featurePrefix: "ms",
              srsName: "EPSG:900913",
              version: "1.1.0",
              url: url
           })

If you set the srsName in the WFS protocol your WFS requests will include now include the srsName (it can also pick this up from the map, but I find it easier to be explicit about projections). If you check your requests in FireBug or Fiddler you can see the XML that is posted to MapServer when you switch on a vector layer:

<wfs:Query typeName="Test" srsName="EPSG:900913">

A full OpenLayers request looks similar to this:

However there is one more piece of the puzzle left – as adding the srsName value to the WFS protocol has very little effect! Features will be correctly selected, but they will be returned in the DefaultSRS, in this example EPSG:29902, so they will never appear in OpenLayers. Whether this is an issue with MapServer or the WFS specification I’m not sure.

What you need to do is to pass the srsName to the URL of your WFS server. E.g.

http://localhost/mapserver/?map=C:/PathToMapFile/mymap.map&SERVICE=WFS&VERSION=1.1.0&srsName=EPSG:900913

The POST request remains the same, but setting the SRS in the URL forces MapServer to reproject the features. The WFS protocol should therefore look something like this:

protocol: new OpenLayers.Protocol.WFS({
              featureType: “Test”,
              featurePrefix: "ms",
              srsName: "EPSG:900913",
              version: "1.1.0",
              url: url + "&srsName=EPSG:900913",
           })

If you enter a SRS which is not listed for your WFS layer you’ll get the following message:

msWFSGetFeature(): WFS server error. Invalid GetFeature Request:Invalid SRS. 
Please check the capabilities and reformulate your request.

If all has gone well you’ll have a MAP file containing WFS layers that can be requested by clients in a variety of projections – very useful when some people are connecting through a web system based on Web Mercator, and others through a desktop using other datasets projected to national grids.

This question was posed on the GIS Stack Exchange, and the replies include a number of interesting approaches. Below is another one.

Why not just use .NET?

I have several years worth of DLLs built using VB.NET and ArcObjects. Recently I’ve been using more and more open-source libraries, the majority of which can be manipulated using Python.

A major advantage of using a interpreted language (rather than one that needs to compile) like Python is that you can easily change paths and variables in a script.

You can deliver a folder of .NET ArcObjects DLLs, a sample Python script and an end user can automate their workflow themselves. This negates the requirement to build a UI on top of your code as a (Python familiar) user can script it themselves.

DLLs still have their place as ArcObjects includes many functions not available in the ESRI Python libraries. It is also easier to write and test ArcObjects code in Visual Studio with all the advantages of error checking and auto-completion.

Options

Python is a programming language, but it can and has been implemented many times using different underlying technologies. The “default” version of Python is implemented in C (and therefore named CPython). Jython is implemented in Java, and IronPython is implemented in C#.

IronPython therefore seems perfect to use if the majority of your code is based around the .NET framework, however as it is separate from standard Python many third-party libraries are incompatible.

This may no longer be true thanks to the CPython extensions to IronPython, which embeds a CPython interpreter into IronPython – it sounds crazy but apparently works.

Embedding CPython into IronPython relies upon a library called Python for .NET, which also allows .NET libraries to be used in standard CPython. The library can be used deparately from IronPython, so the solution I ended up using was CPython and the Python for .NET library.

Installation

1. Go the download page for the Python for .Net project at http://sourceforge.net/projects/pythonnet/files/, and download the appropriate zip file for your existing Python installation (2.5, 2.6 or 2.7).
2. Unzip the file to a folder of your choice. Ideally this should be somewhere on the Python path, but in the sample code below you can add a reference to this folder by changing the path to the pythonnet-2.0-alpha2-clr2.0_131_py27_UCS2 folder.
3. That’s all you need to do!

The sample script below shows how to add references to your own DLLs and to the ESRI ArcObject DLLs, open ArcMap, and create classes.

As this approach is not ArcObjects specific you can use classes in any of your other .NET DLLs, and even classes in .NET .exes.

I can now script all the classes in my .NET DLLs through Python – instantly removing the need for hastily put together user interfaces, and fully automating previously manual tasks. It also allows easy passing of data between the ESRI and open source stacks. I’ve only scripted a few processes so far, but I’ve had no issues yet, and at the moment it feels like one of the rare programming occasions where getting things to work together was actually easier than expected.

One solution is the ExtJS ScriptTagProxy that can be used to retrieve data from an external domain. However for this to work the server must return executable JavaScript code. For example to access an external WMS capabilities file you’d need to set up a special handler on your server to wrap the data in JavaScript before being added to your web page. This pattern is referred to as JSONP (JSON with padding).

YQL

Thanks to this Unwritten Guide to Yahoo Query Language it became apparent you can get Yahoo to automatically do this wrapping for you. Whilst using YQL is still technically a proxy, it’s a proxy you don’t have to worry about maintaining.

I’m not a huge fan of relying on commercial APIs and services, partly as if they are free then they can also change without warning or disappear, and partly as fully understanding the Terms of Service requires both a law and computer science degree.

However configuring your own proxy and wrapping is a large overhead if accessing data from another domain is the only reason you need server-side code.

YQL looks a lot like standard SQL, but can be used to query data from a URL. Go to the YQL Console and try out the following query:

select * from xml where url='http://api.geoext.org/1.0/examples/data/wmscap.xml'

An External WMS Capabilities Store

This method can be used to easily access a WMS Capabilities file on an external server. I’ve put a small demo online showing a working example. The source code can be found on BitBucket.

The example is based on GeoExt’s original WMS Capabilities demo. The code makes use of the ScriptTagProxy, and a slightly modified reader that takes the results part of the JSON (the XML) and passes it on to the standard WMSCapabilitiesReader:

GeoExt.data.YahooWMSCapabilitiesReader = Ext.extend(GeoExt.data.WMSCapabilitiesReader, {
    readRecords: function(data){
        data = data.results.toString();
        return GeoExt.data.YahooWMSCapabilitiesReader.superclass.readRecords.call(this, data);
    }
});

Yahoo have many “clones” of services offered by the other web giants, which whilst well implemented don’t really interest me – when was the last time you looked at Yahoo Maps?.  However along with YQL, Yahoo have a few interesting and unique developer tools such as Yahoo Pipes and YUI (in fact ExtJS branched out of YUI), so its worth keeping an eye on the often forgotten man of the Internet.

Or perhaps that should be rephrased start browser-based raster GIS?

GIS data is split into two base types – vector data – geometric shapes, usually further split into points, lines, and polygons, and raster data – cell-based or “pixelated” data.

Graphics on the web mirror this divide. On the vector side SVG – scalable vector graphics, is used in many browsers to display geometric shapes. On the raster side “dumb” images come in many well known formats such as bitmaps, GIFs, PNGs, and JPEGs.

Vector geometries are easily manipulated after drawing as they have an abstract model to work with (the SVG, or KML document), which the browser can then convert to the DOM. As an example OpenLayers includes two vector renderers – one for SVG (see source code), one for VML (used by the ever-unique IE), and since the start of this year a new canvas renderer.

The canvas renderer is used to draw features to the new canvas element which is part of the HTML5 specification. This allows access to images loaded into the canvas through new programming interfaces such as the Canvas 2D Context API.  It is this part of the HTML5 specification that could change the way we work with raster data on the web.

The power for raster operations will be based on the compositing functions. These correspond with various map algebra functions that have been used by desktop GIS systems for years. To take a few examples the HTML5 transforms include the following:

XOR – Shapes are made transparent where both overlap and drawn normal everywhere else.

LIGHTER – Where both shapes overlap the color is determined by adding color values.

The clip() function can be used to create masking areas, and to speed up the processing.

These three basic functions cover many of the key uses of raster GIS. By adding or subtracting the colour values in a raster cell, you are in effect changing the raster data itself.

For example it would be possible for example to multiply a soil type raster dataset, with a rainfall raster dataset to produce a third raster dataset giving indications of crop suitability. Or as this is in the browser a user could choose their own weightings to instantly multiple different land use classes to producing a new map.

It’s Already Here…

For excellent examples of these types of pixel manipulations go to Beej’s Bit Bucket and check out his blue goat, scratcher demo, and his challenge to create Sobel edge detection using canvas.

Matthew Trost has created a brilliant path finding demo using canvas and JavaScript. Whilst the target audience for is game developers, it inadvertently implements several of the key functions of raster-based GIS.

It will be coming to OpenLayers soon – the evaluation of HTML5 and OpenLayers is presented here. There is a demo showing layers in a canvas element in krisgeus’s sandbox. And CampToCamp have an impressive demo to dynamically display and graph elevation values.

Challenges and Challengers

Currently it is easier to run raster processing on a server, and return the results to a user. With an increase in cloud-based processing responses times will get even shorter, so developers will have to decide which approach will be easier to maintain and implement, whilst meeting users’ expectations and requirements.

In an ideal world, processing could be balanced between the client and server to bring about the fastest results possible, however coding for two different architectures likely rules this out for all but high traffic sites.

As pointed out here “performance can be a challenge since bitmap operations are very processing expensive, especially at high resolutions.” A major challenge of creating web sites is designing for the lowest common denominator – what happens if a user connects to a site using IE6 on a 15 year-old desktop?

Whether or not it will be more practical to do raster analysis in the browser rather than on the server is up for debate. But home computers aren’t going to get slower, and as HTML5 becomes the norm rather than the exception raster GIS on the web becomes a very interesting terrain.

Linear referencing

In terms of storing linear referenced point features, you only need three fields – an ID of the original line feature, an ID of the  feature to reference along the line, and a M value – the measurement along the original feature. Line features require two measurement fields, a starting distance, and an end distance.

Dynamic Segmentation

Linear referenced features can be stored in a standard table structure. What is more complicated however is displaying these features again – a process (termed, at least by ESRI) as dynamic segmentation – the process of dynamically splitting, or segmenting, an existing feature. It’s a step up from a standard table join, but essentially has the same purpose of relating two records in a database.

SQL Spatial Tools is an open source collection of functions to manipulate spatial data within SQL Server 2008. It contains a  LocateAlongGeom function that “takes a geometry linestring and finds the point a given distance along it.” So that takes care of displaying linear referenced points. What it lacks at the moment (and can be found in other spatial databases such as PostGIS and Oracle), is a way to dynamically create line features using a distance from and a distance to. Fortunately the SQL datatypes and existing spatial tool functions make this easy to add in.

The Code

The CreateLinearReferenceFeature code can be seen in the Function.cs file. The full project is available for download and modifications in BitBucket at https://bitbucket.org/geographika/sql-server-spatial-tools. As I’m never sure if CodePlex accepts patches or not, I decided to create my own fork of the code. I have uploaded the key files to CodePlex where they will await evaluation. I’d imagine however that there will be a new release of the tools by Microsoft for the new Denali release of SQL Server 2008, so this funciton may have to be merged into a new release in the future.

The SQL Spatial Tools compile to a DLL, and the assembly needs to be registered with the database along with all the associated functions. I modified the original SQL script to include the new functions.  There is also an  unregister.sql script to remove the assembly and functions. A few caveats on the linear refence function:

• the code has not been tested on the new SQL Server 2008 curve type
• the project is written in C# and compiled with Visual Studio 2010
• if you give the function a MeasureTo value greater than the length of the feature you will receive the following System.ArgumentException: “MeasureTo value must be less than the length of the feature. ”

To test out the functionality you can use the SQL script below. Feedback and comments appreciated!And no need to store hundreds of little line geometries in the same database as your nice clean line networks.

The script below solves this problem by generating a PNG image of all the PDFs in a folder. The image to the left shows the results of the script when run on a series of UN Mission Maps.

This relies on two programs to be installed. ImageMagick – a free image conversion software package, and GhostScript another free program that can be used to access, read, and create PDF files. You may have to reboot your machine after installing these programs for the script to run successfully.

The script uses a convert utility program which comes with ImageMagick.

A common requirement in GIS is to be able to find the number of points in a polygon to answer a question such as “how many towns are in this county.”

With the spatial operations in SQL Server this can be calculated dynamically, however for large spatial datasets it can often take several minutes to run the query. If a user is running the query through a web interface they will either give up, or the connection will time out.

It can be useful to assign all features to a parent polygon in the database so these calculations are almost instant. To do this run the following SQL:

It would be nicer to have this step automated at the database level, by having spatial query results available in a dynamic calculated column. However at present this is not possible. You will get an error such as “Computed column ‘Centroid’ in table ‘MyTable’ cannot be persisted because the column type, ‘geometry’, is a non-byte-ordered CLR type.” An alternative is to create an Indexed View, or to use a trigger to populate this field automatically.

Update - storing geometry in persisted columns will be available in SQL Server 2008 Denali CTP1.

Displaying Points and Polygons

The image of points and polygons for this post was generated with the following SQL. It takes advantage of the fact that any geometry type (points, lines, and polygons) can be stored in a single geometry field, and therefore be displayed at the same time.

This post also allowed me to create my first Gist in GitHub. SQL code was “prettified” using Simple-Talks online SQL Prettifier.

On the OpenLayers Mailing List Diego Guidi pointed to an opensource .NET proxy. A proxy makes a request to a remote URL, reads the response, and then sends it to the client so it appears all data comes from the same server.

The .NET proxy is written by Paul Johnston, and can be found at http://code.google.com/p/iisproxy/. I made a few minor changes as follows:

• increased the byte size for reading responses to resolve this issue
• added support for PUT requests
• temporarily removed GZIP compression due to invalid responses and a possible bug in .NET

The proxy can be used for any requests and is not limited to just OpenLayers. My source files can be found at http://bitbucket.org/geographika/openlayers/src/bfeab6a9971a/iisproxy/

I’d recommend reading the original project’s README file which goes through compilation and installation, but I’ve added my own notes below.

Compiling the Proxy

To compile the proxy download the source, open the Windows command prompt and run the following commands:

• cd C:\PathToIISProxyFolder
• build.cmd

The build.cmd file is hardcoded to compile the code for .NET 2.0 (located at C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\csc), but you should be able to compile it for higher versions of .NET.

Configuring the Proxy

The important thing to remember is that your folder structure on your local server must match the folder structure on your remote server. So if you wish to get requests back from http://mapserver.org/wfsservice/ you need to create a local application at http://localhost/wfsservice.

In XP / IIS 5.1

Windows XP Professional comes with IIS 5.1. This is not installed by default so you may need to go to Control Panel >> Add or Remove Programs >> Add / Remove Windows Components.

Once installed you can create a new folder in C:\Inetpub\wwwroot and name it wfsservice. Copy the IISProxy \bin folder and web.config into this folder.

Next open the IIS Manager (in Control Panel >> Administrative Tools), navigate to the wfsservice folder, right-click, and select properties. You need to click the Create button to turn this into a web application.

Once the application is created, click the Configuration button.  On the Mappings tab select Add. This step ensures all web requests to your application are sent to the proxy DLL. As the DLL is written for ASP.NET it needs to be run by the aspnet_isapi.dll. Different versions of this DLL may be installed on your machine.

I compiled the proxy for .NET 2 and the DLL I selected is located at C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll.

There is a slight bug in the form which means you may have to click in and out of the Executable box before the OK button becomes active. The extension of .* indicates that any web request (.html, .aspx, .json) should all be handled by the proxy DLL.

I deselected the “Check that file exists” box as I want all requests to be handled remotely even if there is no remote file. I also want the proxy to handle all types of HTTP requests (GET, POST, PUT, DELETE) so selected the “All Verbs” option.

In Windows 7 / IIS7

In Windows 7 the installation process should be simpler when configured using the Integrated Pipeline mode. This article details how using IIS7 Integrated Pipeline mode is more efficient than the wildcard method used above.

Create a folder to match the name of the remote folder, and in IIS Manager convert it to an application. Copy the web.config and \bin folder as before. However the web.config is slightly different and should be similar to below (remember only the root name of the remote server should be used with no backslashes).

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appSettings>
        <add key="proxyUrl" value="http://mapserver.org" />
    </appSettings>
   <system.webServer>
        <handlers>
            <add name="ReverseProxy" verb="*" path="*.*" type="ReverseProxy.ReverseProxy, ReverseProxy" />
        </handlers>
    </system.webServer>
</configuration>

There has also been an issue in IIS7 of the proxy returning security errors such as – “Request for the permission of type ‘System.Web.AspNetHostingPermission, System..”

The fix for this is to set LoadProfile to True for the application pool – via Dave Burke.

Debugging

If you have issues with timeouts and hanging responses you can debug the proxy code on your own machine by attaching to the ASP.NET process in Visual Studio – Debug >> Attach to Process. Set a breakpoint in the code and then trigger a request either in your browser or using Fiddler.

I recently asked a question on GIS Stack Exchange on how to create a  buffer around a point that took into account the curvature of the earth. OpenLayers has support for geodesic measurements, but not creating geodesic polygons. Drawing a standard polygon on a Mercator projected map can produce features with very different measurements from their intended “on the ground” equivalents.

Paul Ramsey pointed out that “the scale errors for Mercator are very high indeed (infinite, in fact, at the poles) increasing as you head north/south from the true scale latitude.” In fact drawing a circle (in Ireland – 53 degrees North) with a 10km geometric radius produced a circle with an on the ground radius of 6km.  A huge margin of error over a very short distance (see a previous post on the same subject).

After a useful answer from Dan Shoutis, it appeared most of the work to implement geodesic circles was already available in the OpenLayers API. The OpenLayers’ geodesic functions are based on code adapted from Chris Veness work at http://www.movable-type.co.uk/scripts/latlong-vincenty-direct.html.

The code to create a regular (non-geodesic) polygon can be seen in the OpenLayers source here. There is a function added after the class OpenLayers.Geometry.Polygon.createRegularPolygon that can be used to “create a regular polygon around a radius. Useful for creating circles and the like.”

This function only required a couple of changes – notably using Longitude and Latitude and rather than X and Ys, to produce geodesic polygons. If you are using the Mercator projection then transformations requires proj4js support.

Code

An online demo of the function is available at http://geographika.co.uk/samples/geodesic/. The adapted function can be seen at http://geographika.co.uk/samples/geodesic/GeodesicPolygon.js.

I have uploaded the full source code for the demo to http://bitbucket.org/geographika/openlayers/src.

Just don’t try drawing circles around the map edges..

Even in the current digital era, being able to print maps from geospatial applications is still a very commonly requested feature. Traditionally, there have been two main approaches to the map printing on browser applications: handling it either on the client or on the server. In the first case, a new page is opened with the suitable size for a common paper format (e.g. A4 portrait) and is printed with the browser’s print function. This is probably the easiest way to go if you have a very basic map and you don’t need a complex layout, and big guys like Google Maps do it this way. The second option is based on sending all necessary information to a remote printing service, which will produce an output file (generally a PDF) ready to be printed. This allows for more complex layouts, different page sizes, etc. but with the drawback of being more difficult to build and maintain.

Fortunately though, the MapFish mapping framework includes a powerful printing module that allows you to define complex layouts and provides a protocol that can be integrated seamlessly with GeoExt based applications. The layouts can include legends, attribute tables, external images and custom variables sent from the client, and are configured via files with the YAML format. The MapFish site has a complete reference of the configuration file syntax.

If you want to use the MapFish printing service out of the box, you need to install it in a Java servlet container like Tomcat or use it as part of the complete MapFish Python framework (See the Installation page for full details). But if you have an existing infrastructure, these two options can seem like overkill. In our particular case, the printing service was needed for an existing application served under IIS with PyISAPIe, so we had a look on how to isolate the printing functionality from the rest of the framework.

It turned out to be relatively easy: the actual work of generating the PDF file with the provided spec is done by a JAR file (a standalone Java application packaged into a single zip-like file). Besides, there is a controller which handles the requests, calls the JAR with the appropriate parameters and returns the response. This controller file is the key, and our work basically consisted of adapting the Paster and Pylons specific code to PyISAPIe code. The same philosophy can be adapted to any language or server: you will need a controller that gets the requests, speaks with the JAR file and returns the output.

The code is Open Source and you can have a look at it on the GitHub repository:

https://github.com/amercader/MapFish-Print-IIS

The following are the steps you have to follow to get it up and running. There aren’t any particularly complicated steps, but you have to be sure to not miss anything.

Requirements

• Python 2.5+
• IIS 7.0 (Not tested under IIS 6.0)
• PyISAPIe 1.1.0+
• Java Runtime Environment 1.5+
• MapFish print module JAR file

Installation

1. Your server will need the Java Runtime Environment (which can be a Windows x64 version). You may need to restart the machine to update the environment variables (JAVA_HOME). If you plan to compile the MapFish print module yourself, you will also need JDK 1.5+.
2. You will obviously need the MapFish print module JAR file. Compile it following these instructions:http://mapfish.org/doc/print/installation.html#compilationIf you don’t want or can’t compile it, you can use a pre-compiled one, available here, but note that this may not be the most recent version of the print module. Copy the JAR file in the directory of your choice.
3. Copy a YAML configuration file for the print service in the directory of your choice and edit it as needed. You can find samples in the ‘samples’ directory of the MapFish print module source downloaded in the previous point.
4. Install and configure PyISAPIe. You can follow the instructions described on this post or in the README file included with PyISAPIe . If you want to run a 64 bit version, have a look at this post.
5. Download the latest version of the package. Copy the files contained (Http and WWW modules and the printer.ini file) where the PyISAPIe DLL can find them. The easier choice is in the same folder as the DLL. You may want to move Http and WWW, e.g to site-packages, but the printer.ini file must be located in the DLL directory.
6. Edit printer.ini to define the configuration options:
   • Path to the MapFish print module JAR file, compiled in the first point.
   • Path to the YAML configuration file for the print service.
   • The temporary directory that will use the service to store the generated files. Please note that the user running the application on IIS must have full access to this directory (i.e. write and delete files). If commented out in the printer.ini file, the application will try to use the default OS temporary directory (the one returned by gettempdir())
7. Create a virtual directory in IIS, and add the PyISAPIe DLL as a Wildcard Script Map.
8. Restart ISS and visit the following URL:http://your_server/virtual_dir/info.jsonYou should receive a JSON response with the capabilities of the printing service.

Troubleshooting

• JAR file testing
  To test if that the JAR file works correctly, you can execute the following command from the command line (replace the paths with your own). You can find config.yaml and spec.json in the ‘samples’ folder of the MapFish print module source:

  java -jar C:/Python26/MapFish/print-standalone-1.2-SNAPSHOT.jar org.mapfish.print.ShellMapPrinter --config="C:/Python26/MapFish/config.yaml" --spec="C:/Python26/MapFish/spec.json" --output="C:/Python26/MapFish/tmp/test.pdf"

• The service returns corrupted PDF files when printing the OpenStreetMap layer
  This is caused by a bug in the MapFish source code. The availabe pre-compiled JAR file fixes this problem (Again, note that this may not be the most recent version).
• IIS returns a 404 error when sending a print request
  If you are using the GET method to request the PDF file (i.e. using the print.pdf end point), you may find that the server returns a 404 error if the query string is too long. That means that the query string (the parameters sent after the service URL) is longer than the IIS security limit. You could try to adjust this limit, but better yet, you can simply use POST requests from your client application, which is the recommended setting. This way you won’t have any problems regarding the length of the parameters sent.
• The application returns an exception when including a local image
  If you try to include a local image in your YAML configuration file, (e.g. url: 'file://${configDir}/north.png' or url: '${configDir}/north.png'), the service will return an exception. This is caused by a bug, and until it’s fixed, a workaround is to use a file accessible via Http (url: 'http://.../north.png').

Feedback

I would love to hear about your experience using it, and if you think of an enhancement or find a bug do not hesitate to contact me (amercadero at gmail.com – http://amercader.net) or even better, fork the code on GitHub and contribute your changes.

The next post will be written by Adrià Mercader who has been working on configuring the MapFish print server to be used on IIS. This fits in perfectly with the recent Window / open source hybrid web-mapping solutions I have been working on.

Adrià is also the author of the WMS Inspector a FireFox plug-in that allows you to view and analyse WMS requests for a web-page. This was recently presented at FOSS4G in Barcelona, near Adrià’s hometown. He has recently moved to Newcastle in the UK, clearly for both better weather and football.

Many popular packages in PyPi have custom Windows installers that take care of both Python files and any associated DLLs. For example Shapely and NumPy.

For other packages the easiest way to add them to your Python installation is to use Setup Tools. Download the Windows setup package corresponding to your version of Python from http://pypi.python.org/pypi/setuptools and double click to install.

This adds an easy_install.exe to your C:\Python\Scripts folder that can be used to quickly add packages.  You run easy_install from the Windows command line followed by the name of the package. It searches the online PyPi for the best match for your Python version, and then downloads and installs it.

For example to install MapFish open a command window and run:

C:\Python27\Scripts\easy_install mapfish

Or for GeoJSON:

C:\Python27\Scripts\easy_install geojson

64-Bit Windows

If you install the 32-bit version of Python on Windows 64 bit then you can continue to install the standard 32-bit packages as above. However if you want to take advantage of a 64-bit version of Python then things get trickier.

If you are lucky an official 64-bit Windows installer is already available for download e.g. Shapely, unfortunately this is the exception rather than the rule. There are however often “unofficial” installers available on the web.

If you try to run a 32-bit installer then it will often say it cannot find your Python installation. This is because the 64-bit and 32-bit Python installations write to different parts of the Windows registry. You can try and fool the set up package into installing using the method outlined here, however if the package relies on DLLs then you could end up with problems. If the package is pure Python (no associated DLLs) then I don’t think this should be an issue.

If you receive a “This application has failed to start because MSVCR71.dll was not found” error then the Python package relies on DLLs written in C++. To use these packages you need to install the Microsoft Visual C++ Redistributable Package. This can be downloaded from here. It contains “runtime components of Visual C++ Libraries required to run 64-bit applications developed with Visual C++ SP1 on a computer that does not have Visual C++ 2008 SP1 installed.”

I recently came across a treasure trove for 64-bit Windows and Python maintained by Christoph Gohlke at http://www.lfd.uci.edu/~gohlke/pythonlibs/. This page contains 64-bit Windows installers for many packages including GDAL (there are further details on using 64 bit GDAL and MapScript here), and a compilation of many packages including geopy, simplejson, sphinx, pytools and many others.

The name of the library is a vestige from when OGR used to stand for OpenGIS Simple Features Reference Implementation. However as OGR is not fully compliant with the OpenGIS Simple Feature specification the name was changed to OGR Simple Features Library (from the GDAL FAQ).

The latest additions to the OGR formats are the SQL Server 2008 geometry and geography. This enables spatial data in SQL Server 2008 to be reprojected, and converted to other formats (shapefiles, MapInfo, KML, GML, GeoJSON and any of the many other formats already supported by OGR) using freely available open source tools. Details on the associated OGR tools can be seen here.

The full MSSQLSpatial OGR driver notes and details are available online.

Where can I get the Driver?

The source code for the new MSSQLSpatial driver can be seen here in the online GDAL subversion code repository. As can be seen by the check-in dates much of the work has been completed in the last two weeks.

The two developers who deserve the credit are Tamas Szekeres and Even Rouault, both members of the GDAL Project Management Committee.  Tamas has blogged about the new OGR driver here. Tamas’s work is becoming invaluable to many of my projects – and he was also responsible for the MapServer SQL Server 2008 driver.

The MSSQLSpatial driver is not currently available  in the latest version of FWTools (2.4.7 at the time of writing). FWTools (Frank Warmerdam tools) is an easy to install package containing both GDAL and OGR, but keep an eye on the http://fwtools.maptools.org/ site for new releases.

The driver was not a part of the GDAL 1.7 release, so if you can’t wait until the next binary (precompiled) release, you have to compile it yourself. However Tamas maintains a page which automatically compiles the latest MapServer source code into binaries at http://vbkto.dyndns.org/sdk/. MapServer includes the latest development GDAL and OGR binaries (gdal18dev.dll) which in turn include the new MSSQLSpatial driver. So this is currently the easiest place to get a compiled version for testing with SQL Server.

Quick Set Up of OGR & MSSQLSpatial (on Windows)

Even though at the time of writing the latest development builds for MapServer failed the GDAL/OGR DLLs and .exes seem fine and can be found in the bin/gdal subfolder of the MapServer packages. To quickly get up and running with OGR you can follow these steps:

1. Download a suitable package from http://vbkto.dyndns.org/sdk/. You will need a development release, or a release with GDAL 1.8 or higher. Click on the information links to see which OGR types are supported in a package.

2. Unzip the package to your machine. It is best to put the package in a folder which does not contain spaces.

3. Run the following commands in a Windows Command Line window. I left the package in my D:\My Documents\Downloads\ folder in this example.

D:
cd "D:\My Documents\Downloads\release-1500-gdal-mapserver\"
SDKShell.bat setenv hideoci

The SDKShell.bat file sets all the GDAL environment variables for your command window session. The hideoci parameter is used to move the Oracle driver plugin out of GDAL – otherwise if you try and run commands you get the following error (unless the Oracle client is installed on your machine):

This application has Failed to start because OCI.dll was not Found. Re-installing the application may fix this problem

The hideoci parameter did not work for me due to the spaces in my folder path, so I manually did what the .bat file was meant to do – move the ogr_OCI.dll, and gdal_GEOR.dll files out of bin\gdal\plugins\ and into a new bin\gdal\plugins-optional folder.

Once this was done I was able to run the following command to check the OGR formats:

 ogrinfo.exe –-formats

It gets even better though! Included in the installation are the latest SQL Server 2008 driver, 64 bit GDAL, and MapScript Python bindings. MapScript allows MapServer functions and classes to be scripted in a variety of languages.

As Python is becoming ubiquitous in the GIS world (opensource and proprietary), if you aren’t familiar with PHP then You can download the zip file and extract to a folder on your machine, in the example below I used C:\ms4w64 (old habits die hard!).

Install Steps

1. Add C:\ms4w64\bin to your system PATH environment variable (right-click on My Computer, select Properties, Advanced, Environment Variables, then in System Variables select PATH and edit. Remember to put a semi-colon ; between folders in the path.

Adding this folder to your PATH will allow Windows to find the MapServer and GDAL DLLs used by Mapscript.

2. Now we want to make the Mapscript Python module available in Python. There are a couple of options here. I took the files from C:\ms4w64\bin\ms\python and copied them into C:\Python26\Lib\site-packages – the default search location for third party Python modules. The files to copy across are:

• _mapscript.pyd
• mapscript.py
• A mapscript.pyc (compiled Python) file will be generated on first use

Alternatively you can add the C:\ms4w64\bin\ms\python folder to the PYTHONPATH Window’s environment variable as in step 1.

Important! You will need to log off, or restart your PC in order for the environment variables to be “refreshed” by the operating system.

Either way you should now be able to type “import mapscript” in IDLE without receiving errors.

Errors?

Traceback (most recent call last):
File "<pyshell#0>", line 1, in <module>
import mapscript
ImportError: No module named mapscript

Python can’t find the Mapscript module. Try typing the following commands into Idle to see the folders that Python checks looking for modules:

import sys
print sys.path

Traceback (most recent call last):
File "<pyshell#13>", line 1, in <module>
import mapscript
File "C:\Python25\lib\site-packages\mapscript.py", line 28, in <module>
import _mapscript
ImportError: DLL load failed with error code 193

It is likely you are using a 32 bit version of Python with 64 bit versions of the MapServer DLLs. You may have reinstalled and reconfigured Python so many times that you have actually forgotten what you installed. A useful program in this case can be downloaded from http://www.mitec.cz/exe.html

This tells you the processor type for a .exe file such as python.exe. Check that it is a 64-bit version as shown below.

• you can cache the service (for example with TileCache) to greatly improve performance
• your application need only worry about a single point of connection, using WMS layers aggregated from many different sources
• you can add additional metadata missing from the source
• services can be reprojected (using GDAL)
• you can merge WMS layers together and server them out as a new map layer

There are also some negatives:

• you have additional server configuration to set up and maintain
• if you are not adding or changing the WMS service you should probably connect to the WMS service directly in the client applications, otherwise there will be a performance hit
• caching may contravene the licensing agreement of the original WMS source

The rest of this post details the steps in setting up a cascading WMS using MapServer. You need MapServer 5.4.1 or higher to implement a secure cascaded WMS service. This example uses a 64-bit version of MapServer 5.6 taken from Tamas Szekeres’s autobuilds site installed in a C:\ms4w64 folder. It also assumes MapServer has been configured with SSL as detailed in another post. In this example I use IIS and Windows Server 2003.

MapFile Configuration

Configuring a MapFile for a cascading WMS service is well-documented on the MapServer WMS Client page. An example is shown below. Note that this service is not reprojected, and the username and password can be passed through the metadata parameters.

LAYER
  NAME "MyLayer"
  TYPE RASTER
  STATUS OFF
  CONNECTION “https://webservices…”
  CONNECTIONTYPE WMS
  #DEBUG 5 #to debug the original layer
  METADATA
    "wms_srs"             "EPSG:2157"
    "wms_name"            "0"
    "wms_server_version"  "1.1.1"
    "wms_format"          "image/png"
    "wms_auth_username" "username"
    "wms_auth_password" "password"
    "wms_bgcolor" "0xFFFFFF"
  END
END

It is vital you set up debugging for the MapFile to see why a connection may be failing. You can set this by adding the following lines to the MapFile:

DEBUG 5 #debugging level between 1 and 5, 5 is the most detailed
CONFIG "MS_ERRORFILE" "C:/ms4w64/tmp/ms.log"
#path to log file - ensure IIS can write to this file

This can be removed once everything is working correctly.

Configuring SSL and MapServer (Part 1)

A secure WMS service is one that uses an https:// address, and may require a username and password. Setting up MapServer to connect using SSL is by far the trickiest part of the configuration. First of all I had to compile my own libcurl.dll to include OpenSSL. Otherwise I got the following error in the log file:

msHTTPExecuteRequests(): HTTP request error. HTTP: request failed with curl error code 1 (Protocol https not supported or disabled in libcurl)

Once I had the new libcurl.dll I replaced the existing file in the C:\ms4w64\bin folder (always make a back-up of the original files first!) with my freshly compiled version. I used Process Monitor to  see if I was missing any other DLLs – I was. The libcurl.dll required some additional OpenSSL DLLs to connect to https addresses. These DLLs were:

• libeay32.dll
• ssleay32.dll

These too I copied into my C:\mapserver\bin folder from the openssl-1.0.0a\out32dll folder created when compiling libcurl.dll. Next I got the following error:

HTTP: request failed with curl error code 60 (SSL certificate problem, verify that the CA cert is OK. Details:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed) for https://…

These errors relate to missing certificates. For more details on security certificates see this page.

Getting a Security Certificate

I needed to get a valid certificate from the web service and point MapServer to this file. I was able to get a certificate by typing in the service;s https:// url in FireFox, and then selecting Tools > Page Info > Security > View Certificate > Details > Export. You can also right-click on the padlock icon at the bottom of the page to access the same option.

When exporting I selected the “X 509 Certificate with chain (PEM)” option. I tried an export without the “chain” but this didn’t allow me to connect.

I’d recommend trying to connect to the site independently of MapServer, so any issues can be narrowed to certificate problems rather than the MapServer configuration. I ran these tests on my development machine – a 32bit Windows XP laptop, following these steps:

1. Install a standalone version of cURL from the cURL download page (I used libcurl-7.19.3-win32-ssl-msvc.zip).

2. Save your security certificate with the filename curl-ca-bundle.crt in the same folder as you unzipped the curl.exe file. Placing it in the same folder, with this name, will mean the certificate can be read by cURL.

In a DOS window navigate to the folder with the curl.exe file and run the following command to see if the certificate works with the site. Add in some valid WMS parameters.

curl https://webserveraddress.com/wms?VERSION=1.1.1&REQUEST=...

If you need to pass in a username and password to the service use:

curl https://username:password@webserveraddress.com/wms?VERSION=1.1.1&REQUEST=...

If you get the following error:

(60) SSL certificate problem, verify that the CA cert is OK. Details:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

It means your certificate is invalid. You can try the command with a /k switch to ignore the security and check that there isn’t a connection problem rather than a security problem:

curl https://username:password@webserveraddress.com/wms?VERSION=1.1.1&REQUEST=... /k

If all went well with the certificate you should get a DOS window filled with random characters:

This is good! It is an image file viewed as text and means you have successfully returned an image from the secure WMS server.

Configuring SSL and MapServer (Part 2)

Now all that remains is to set up the certificate on the MapServer. There are some instructions here and more here. I got a little confused thinking I could point to the certificate file using a CONFIG command in the MapFile to set the environment variable. This is not the case.

It must be set  using a Windows environment variable (although you can set it in Apache if you’re not using IIS). It took a while for me to figure this out, but due to the beauty of OpenSource if you get too stuck you can search the sourcecode itself to see how parameters are used. The relevant sourcecode can be found here.

I created a new Windows environment variable CURL_CA_BUNDLE to point to the .crt file as shown below:

And this time in the log – success!

[Wed Jul 07 16:04:55 2010].384000 msDrawMap(): kicking into non-square pixel preserving mode.
[Wed Jul 07 16:04:55 2010].838000 HTTP: Starting to prepare HTTP requests.
[Wed Jul 07 16:04:55 2010].838000 Using CURL_CA_BUNDLE=C:\ms4w64\bin\curl-ca-bundle.crt

Poor Image Quality?

Sometimes the WMS images lack the quality of the originals when run through a cascaded service. This post attributes it to either reprojection or incompatible output formats. In my case it was the latter. As the forum post writer says – if you aren’t reprojecting or adding information then using a proxy rather than a cascading WMS would be far more efficient.

My original MapFile OUTPUTFORMAT had worked perfectly for all my local WMS services, but with the cascaded service the aerial photography was blotchy. OUTPUTFORMAT NAME ‘AGGA’ DRIVER AGG/PNG IMAGEMODE RGBA END	I changed the OUTPUTFORMAT to the following settings and the images were back to good resolution. OUTPUTFORMAT NAME “png” DRIVER “GDAL/PNG” MIMETYPE “image/png” EXTENSION “png” END

MapServer

libcurl.dll is the library version of the command line tool cURL – a tool written in C (hence the name) that:

Tamas Szekeres maintains a site that dynamically builds Mapserver from source on a nightly basis using different versions of MapServer, and GDAL, for both 32 and 64 bit Windows. Unfortunately for me I wanted to connect to a WMS using MapServer that used the HTTPS (Hypertext Transfer Protocol Secure) protocol. When I tried connecting to the WMS service with MapServer the log file reported the following error:

msHTTPExecuteRequests(): HTTP request error. HTTP: request failed with curl error code 1 (Protocol https not supported or disabled in libcurl) for https://webservices…

The libcurl.dll included in the MapServer build doesn’t support SSL (Secure Sockets Layer). That sinking feeling of knowing I was going to have to get my hands dirty and start messing about compiling things I didn’t understand was soon upon me..

There are many different compiled versions of cURL and libcURL you can download from the download page but unfortunately at the time of writing there were no Windows 64bit versions – however it is worth double-checking this. It is also a good idea to check that the MapServer builds haven’t been updated to include SSL before going through the lengthy steps below. There are precompiled versions of the cURL.exe tool out there in the Internet for 64 bit Windows, but MapServer requires the DLL library as shown below:

So currently if you require 64 bit and SSL it looks like you have to compile your own version, hence the following steps.

cURL Compilation Steps – Part One

These steps are based on information gleaned from this cURL mailing list post and this StackOverflow question and answer. The second link is particularly good, but a few extra details can save many hours of time. If you are familiar with compiling C libraries then its probably best to skip to to StackOverflow now. If, like me, you’re not too sharp on C then follow the steps below. There is also the official compilation steps guide which provides more detail.

1. Download the latest source archives from the cURL download page. At the time of writing links to these were at the top of the page, and in this example the file was named curl-7.21.0.zip. Fortunately it seems that the libcurl.dll version number does not need to match the version installed with the MapServer build.

2. Unzip this file somewhere on your machine. Avoid spaces to avoid future problems. In this example I unzipped the files to D:\Projects\cURL\curl-7.21.0

3. In the root of the folder you unzipped should be a vc6curl.dsw file (a Visual C++ workspace file). I’m assuming you have Visual Studio installed – I’m using VS2008. If you double click this file it should open with Visual Studio – allow it to convert to the new project format.

4. Once the project opens go to Build >> Configuration Manager. Set the parameters to DLL Release, x64.

If x64 is not present in the drop-down list you may have to create a new “Solution Platform”, and copy the settings from the Win32 configuration:

5. Now we have to make sure that the DLL is built with SSL support. To do this we add a couple of “Preprocessor” parameters. To do this right-click on the libcurl project in the Solution Explorer window:

Then in Configuration Properties >> C/C++ >> Preprocessor add USE_SSLEAY and USE_OPENSSL as shown below. The NDEBUG parameter indicates “No Debugging.” Note there is also a WIN32 option – I compiled a x64 DLL successfully using both WIN32 and WIN64. I had a brief look at the source, but it is probably best to set to WIN64.

OpenSSL Compilation Steps

6. Now you’re probably thinking that you can now compile. If you are optimistic then you probably will. But unless I’ve completely misunderstood how all this ties together, you are still many steps away from getting that libcurl.dll file..

If you require SSL, you also need to compile the OpenSSL library. This compilation can all be done from the Windows command line. The source code can be downloaded from this link. I used the latest version available – openssl-1.0.0a.tar.gz. Download this and unzip (use 7-Zip to handle those pesky tar-balls), again unzip to a folder without spaces in the path to be on the safe side.

7. Once unzipped there is a INSTALL.W64 file in the root which provides details on compiling OpenSSL for Windows x64. These steps are fairly well-detailed, but a few screenshots don’t hurt. There are a few warnings in this file not to try and download 4GB files, but as it says it is very unlikely these issues would occur in the real world.

The first step to compilation is to install Perl – a language I’ve never used, but it seems the OpenSSL compilation steps have been automated with Perl scripts. The document recommends using ActiveState Perl, but I read that using Strawberry Perl on Windows would be a better choice. After downloading and installing Strawberry Perl, I logged off and on again to make sure the Windows environment variable were updated and continued the compilation.

8. I opened a Windows command window and made the OpenSSL folder the active directory:

cd D:\Projects\cURL\openssl-1.0.0a

9. Next I set up the Windows Visual Studio environment by running the following command. This batch file assumes that you are developing on a 32 bit machine to deploy on a 64 bit machine. There are a number of similar batch files in the parent folder if your set up is different.

"C:\Program Files\Microsoft Visual Studio 9.0\VC\bin\x86_amd64\vcvarsx86_amd64.bat"

10. Next I ran the Perl script from the Windows command prompt using:

perl Configure VC-WIN64A

11. Then a couple more commands:

ms\do_win64a
nmake -f ms\ntdll.mak

12. In the INSTALL.W64 there are final two steps to check everything was built correctly. As I compiled a x64 DLL on a x32 machine I couldn’t run these, but the lack of show-stopping errors filled me with hope..so back to the libcURL compilation..

cURL Compilation Steps – Part Two

13. A rebuild of libcurl.dll at this stage will lead to “fatal error C1083: Cannot open include file: ‘openssl/rsa.h’: No such file or directory”. The Visual Studio project needs to reference the OpenSSL files we just built. First of all add an “Include Directory” to the libcurl project pointing to the openssl\inc32\ folder as shown below. This is where all the .h header files are kept.

14. Trying to recompile now (I’m an eternal optimist) leads to “error LNK2019: unresolved external symbol DES_set_key referenced in function setup_des_key” errors. The project now can’t find symbols stored in the OpenSSL .lib files. I ran a Windows search on the OpenSSL folder to find these .lib files:

Next I found a handy tool that could export this search list to text, so I could copy and paste references. The list of .lib files is below. I added these to the “Additional Dependencies” list in the project’s “Configuration Properties >> Linker >> Input” option.

4758cca.lib
aep.lib
atalla.lib
capi.lib
chil.lib
cswift.lib
gmp.lib
libeay32.lib
nuron.lib
padlock.lib
ssleay32.lib
sureware.lib
ubsec.lib

I also needed to make Visual Studio aware of the location of these files by going to Tools >> Options >> Projects and Solutions >> VC++ Directories, and pointing to openssl-1.0.0a\out32dll, making sure “x64″ and “Library Files” were selected in the drop-down lists as shown below:

15. Finally I could run Build >> Build libcurl from Visual Studio. A brand new libcurl.dll file appeared in the \lib\DLL-Release folder. Done – for now. There are a number of other steps and DLLs to include to get MapServer working, but that is for another post.

Errors?

There are a number of errors I encountered during these steps that may lead you to this page:

• ‘cl’ is not recognized as an internal or external command, operable program or batch file.

Make sure the Visual Studio environment has been set up correctly using “C:\Program Files\Microsoft Visual Studio 9.0\VC\bin\x86_amd64\vcvarsx86_amd64.bat” or similar.

• error A2088:END directive required at end of file NMAKE : fatal error U1077:

I got this error when compiling OpenSSL (when running ml64.exe), as previous failed compilation attempts had left files in the “openssl-1.0.0a\tmp32dll” folder. Deleting the contents of this folder got rid of this error.

Why move to 64-bit?

If a server’s operating system and hardware are 64-bit it seems a waste not to take advantage of them. It may be better to look at the question why move to 64-bit software, as why wouldn’t you move to 64-bit software?

Its nice to be able to claim one-upmanship over commercial GIS products who are only now bringing out 64 bit versions. However having gone through the pain of setting up Open Source GIS applications on Windows I’m not surprised companies took the time to simplify the process (and wait for Microsoft to do some of the legwork) rather than dealing with all the 32/64 bit support issues too early.

Memory & Performance

Performance-wise one of the main benefits of 64-bit hardware is the extra memory you can add to a server. 64 bit Windows 7 can support up to 128 GB of RAM. GIS applications are used to dealing with large spatial datasets – both raster and vector, and will therefore be able to make full use of additional memory. This is not the case for all programs for example Visual Studio which is unlikely to be multiuser or used to handle large datasets.

A 64-bit server allows files larger than 4GB to be loaded directly into memory. This is very advantageous to video editing, bit also to GISs that use aerial imagery which can be served more quickly if loaded into memory rather than loading in sections.

Databases make use of additional memory to cache data for faster responses to user requests, and to handle more concurrent connections – there is a nice summary post on the benefits of 64-bit for SQL Server here. Spatial databases, with their additional larger and complex data structures, are even better placed to take advantage of 64-bit computing.

Future Proofing

Even though it is not the easiest thing to configure a GIS server, nearly all future software will have a 64 bit release, and in a few years 32 bit versions may start to be phased out. Future-proofing a system currently in development may help your GIS to last a few more years in a production environment.

Another benefit is that you can start designing your infrastructure to scale – all on the same physical machine. You can create a virtual web server, a virtual database server, and a virtual map server and divide up large amounts of memory, facilitated by a 64-bit architecture, between them. If the site needs to allow for more users, you can simply take the virtual server and move it to another physical machine – and (in theory) there is very little to configure once the virtual machine has been moved.

Cost

As noted above the reduced number of physical machines should mean lower maintenance costs. In addition if you can get more throughput from your database then this may require fewer licences (such as for SQL Server 2008 which often comes with a per-processor licence). Of course with a 64-bit server you could have up to 64 processors, although this could well melt the server’s case..

What to Update?

I’m currently involved in a couple of projects, one using Windows Server 2008 and another Windows Server 2003. Both are 64-bit operating systems on 64-bit hardware. As 32-bit and 64-bit applications don’t play very well together it seems it will be an “all or nothing” migration, involving the following components:

• MapServer
• GDAL
• Python (for TileCache and Mapscript)
• Shapely

I’ll try and put steps together for each of these for my future sanity, and also in the help that there will be a few more developers straddling the Open Source and Windows divide. It would be nice at the end to post some benchmarks, but running both set ups on the same server could prove problematic traumatic. If anyone has noticed performance increases when using 64-bit GIS applications please leave comments!

TileCache provides a Python-based WMS-C/TMS server – this is allows you create a local disk-based cache of any WMS server (in my case MapServer), and then connect to it through a WMS-C client such as OpenLayers. Using a cache means that rather than have MapServer create an image for every single request for a tile, the first time a tile is requested the image is saved in a folder on the server. On all subsequent requests for the same data TileCache can retrieve this file without needing MapServer. Performance increase is incredible – however watch out for filling up disk space, and remember to empty the cache if you change symbology or update your data.

As mod_python is now officially dead (and has been moved in Apache’s attic), it is yet another reason if you are using Windows servers to run everything through IIS rather than the rather clumsy Apache on Windows. Why pay for Windows, and then install a web server port, thereby losing all the tools, security, and utilities within Windows to manage and administer the web server?

The aim of this post is to describe the set up of TileCache, running through IIS.

Step 1

Install and configure PyISAPIe

Step 2

Place the following script in an application on IIS – http://bitbucket.org/geographika/mapserver-scripts/src/tip/tilecache/tilecache.py. Set this script to be handled by PyISAPIe as described in the link in Step 1.

In the script update the following line to point to your root MapServer WMS URL. MapServer itself can also run well under IIS.

wms_url = 'http://localhost/mapserver/mapserv.exe?'

A few caveats related to this script:

• the resolutions are hard-coded to the Mercator resolutions used by Google Maps, Bing Maps, OpenStreetMap etc.
• the SRS is coded to EPSG:3785 which is the equivalent of EPSG:900913 – the “Google” Mercator projection. I believe the code has changed again, but in any case it can be set by changing the script
• parsing of the query string sent to the server is done using the Python CGI module. There are newer modules to handle query strings, but these may not be compatible with TileCache which is works with Python 2.5  and 2.6 (I’ve not tried with newer versions)

Step 3

Some minor alterations need to be made to the TileCache scripts themselves, otherwise the following errors will occur:

Traceback (most recent call last):
 File "C:\Python25\PyISAPIe\Http\Isapi.py", line 49, in Request
 raise ImportError, "[Loading '%s'] %s" % (Env.SCRIPT_TRANSLATED, str(Val))
ImportError: [Loading '\\?\c:\inetpub\wwwroot\tilecache\tilecache.py'] 'module' object has no attribute 'argv'

This is due to the following lines in the service.py file:

# Windows doesn't always do the 'working directory' check correctly.
if sys.platform == 'win32':
 workingdir = os.path.abspath(os.path.join(os.getcwd(), os.path.dirname(sys.argv[0])))
 cfgfiles = (os.path.join(workingdir, "tilecache.cfg"), os.path.join(workingdir,"..","tilecache.cfg"))
else:
 cfgfiles = ("/etc/tilecache.cfg", os.path.join("..", "tilecache.cfg"), "tilecache.cfg")

It should be commented out, leaving only the last line as follows:

# Windows doesn't always do the 'working directory' check correctly.
#if sys.platform == 'win32':
#    workingdir = os.path.abspath(os.path.join(os.getcwd(), os.path.dirname(sys.argv[0])))
#    cfgfiles = (os.path.join(workingdir, "tilecache.cfg"), os.path.join(workingdir,"..","tilecache.cfg"))
#else:
cfgfiles = ("/etc/tilecache.cfg", os.path.join("..", "tilecache.cfg"), "tilecache.cfg")

The TileCache set up should now be complete. You can point your OpenLayers WMS tiles to the TileCache URL, and parameters will be passed onto TileCache, which in turn will check its cache and return an image, or call MapServer to request that a new image is generated.

Please let me know of any steps missed or errors encountered and I can update the post and script accordingly.

But no! Five seconds later, all was successful, and everything looked as before. There have been numerous bug fixes and improvements, but after a brief look through the list I didn’t see any “killer” new features.

In the world of GIS however there are a number of important new versions coming up this summer.

OpenLayers

OpenLayers 3.0 development has started. Key features and ideology are listed here. There is also a wiki page. There is a move towards separating the core functions from the UI. I get the feeling OpenLayers could be moving from a homage to the Google Maps API , into a full JavaScript / browser geospatial framework.

The core should focus on the following:

• layer management – tile handling & vector rendering
• map & layer related event handling
• server/client communication – generating requests and parsing responses
• geometry digitization & editing

A separate OpenLayers UI lib should focus on the following:

• rendering of buttons and other widgets
• presenting map information that is not rendered as features (layer properties, feature attributes)
• gathering user input for map configuration, queries, etc.

MapServer

On the server side of things, it seems GeoServer has recently had more traction than the older MapServer. However there will soon be (sometime in July if I understand correctly) a version 6.  This has a number of new items I’ve recently required for a new project, and they are now tantalisingly within reach. These include:

• XML mapfile consumption: run-time compilation (XML/XSLT => .map) or embedded XML parser - it will then be possible to easily generate and manipulate MapFiles from a database, or script (I’m sure these already exist, but moving to XML will mean all sorts of powerful libraries in any language can be used to do this)
• Output formats: KML and GeoJSON – excellent news! I’ve tried WFS, and I don’t like it. Its nowhere near as ubiquitous as KML (a now open standard) , or as easy to manipulate as JSON.
• Tile caching support – whilst it is slightly annoying that having got to grips with TileCache and Windows it looks like it will be integrated into MapServer, one less system component to deal with is always a good thing.

1. Create a new account on http://bitbucket.org/. Then create a new repository for your code.

2. Download the TortoiseHG program for Windows from http://tortoisehg.bitbucket.org/. Install and reboot. This will add context commands to Windows Explorer.

3. Right-click on a folder in Windows Explorer that contains the code files or scripts you want to add to your BitBucket account, and select “Create Repository Here” from the context menu.

4. Next right-click in the folder and select “Add Files.” You can now check which files in the folder you wish to add to BitBucket, and click the “Add” button.

5. Right-click again in the folder and select “Hg Commit.” The screen below should appear. You can enter a text description for the update (“add tilecache script” in the image below). Once you’ve done this click the “Commit” button in the toolbar.

6. Right-click again in the folder and select “Hg Repository Explorer.” Add the name of your BitBucket repository to the address bar as shown below. In this case my repository is http://bitbucket.org/geographika/mapserver-scripts – yours should be in a similar format.

7. In the menu select “Synchronize” and then “Push”

8. You should be presented with a confirmation dialog as below. You will be asked for your username and password for BitBucket (step 1). That’s it! You can then check your code online, and learn the rest at your own pace! Let me know if you’ve posted anything good in the comments below!

• When you improve the code others can see the changes and get the benefits
• If someone else improves the code those improvements can be passed back to you via an automated process

BitBucket is a hosting service for the code-source control system Mercurial. Mecurial is part of a new-wave of distributed source control systems that have the following advantages and disadvantages over regular non-distributed source control:

GitHub a similar service but based on the distributed source control system Git developed by Linus Torvald the creator or Linux. Whilst the debate rages on which of Mecurial and Git is better, I chose Mercurial because Git is MacGyver and Mercurial is James Bond. Also Git isn’t the nicest word in this part of the world.

There are already some GIS related projects in BitBucket such as Descartes, some TileCache Patches, and some OpenLayers HTML 5 code that may be of interest. It’s also free if you use public code repositories (you can have 1 private one) and use less than 1GB of space – which is a lot of source code.

Today people spend hours looking at the aerial photography layer on Google Maps wondering why so many people have outdoor swimming pools in the UK..

People also like free data. And companies like data free from the Google Terms of Service even more. The successful OpenStreetMap wiki refers to what seems to be the perfect datasource – the OpenAerialMap (OAM) project, however it starts with the rather ominous opening statement:

It was available at http://openaerialmap.org/ but…

The reasons for its demise are documented by Christopher Schmidt (a major force in the development of OpenLayers and TileCache) in this post to the mailing list. In summary the main reasons for failure were technical – dealing with the huge amounts of raster data and getting disk space.

Cedric Mouillet asked on the mailing list what was the state of the project, to see if it could be used as reference mapping for the OpenAddresses project. No answers were posted.

There is still however life in the movement. A conference call was made at the start of May 2010 to discuss the hardware set-up of the project. Included in the attendees list for the call and on the OAM Steering Committee are members of OSGeo which may prove critical to the future of the project.

Anyway here’s hoping to a successful second version of OAM. In the meantime Google’s aerial photography service (followed by Bing and Yahoo) will remain way out in front terms of data quality, quantity, and ease of access. Just remember it is only “free” in one sense.

If emails arrive that have to be dealt with then add them to tomorrow’s list – that way you have a chance of completing your tasks for a day, and can switch off the computer feeling that you’re finished for the day.

There is also a chapter at the end of the book that says anytime invested in improving systems and processes is time gained. This fits in with the ethos of many programming books – to automate things as much as possible.

As I work on a number of different client projects I use a time tracking database to enter hours worked on a daily basis. It is fairly trivial to get a summary of hours for a month, but what I needed to provide (for invoicing) was a full list of days in a month and hours worked – whether there were values or not.

Solution

To test the solution you can create a temporary table to mock the hours worked for a project. Running the following SQL should produce a table similar to below:

CREATE TABLE #WorkingHours (
[Date] datetime,
[Duration] decimal(18,2),
[ProjectId] int)
GO

INSERT INTO #WorkingHours ([Date],[Duration],[ProjectId])
VALUES(CONVERT(datetime, '20100503'),8.5,18)
GO
INSERT INTO #WorkingHours ([Date],[Duration],[ProjectId])
VALUES(CONVERT(datetime, '20100504'),8,18)
GO
INSERT INTO #WorkingHours ([Date],[Duration],[ProjectId])
VALUES(CONVERT(datetime, '20100507'),7.5,18)
GO
INSERT INTO #WorkingHours ([Date],[Duration],[ProjectId])
VALUES(CONVERT(datetime, '20100508'),7,18)
GO
INSERT INTO #WorkingHours ([Date],[Duration],[ProjectId])
VALUES(CONVERT(datetime, '20100511'),8.5,18)
GO

SELECT * FROM #WorkingHours
GO

Now to create a list of dates for the previous month use the following SQL:

WITH LastMonthsDates
AS
(
  --first day of last month
  SELECT DATEADD(m,-1,DATEADD(dd,-(DAY(GETDATE())-1),GETDATE()))
  AS [calendardate]
  UNION ALL
  SELECT DATEADD(dd, 1, [calendardate])
  FROM LastMonthsDates
  --less than or equal to last day of month
  WHERE DATEADD(dd, 1, [calendardate]) <=
  DATEADD(s,-1,DATEADD(mm, DATEDIFF(m,0,GETDATE()),0))
)

SELECT CONVERT(VARCHAR(10), [calendardate], 103) AS [Day]
,CASE WHEN SUM(WH.Duration) IS NULL THEN 0 ELSE SUM(WH.Duration) END AS Hours
FROM LastMonthsDates
LEFT OUTER JOIN #WorkingHours WH ON
DATEDIFF(d, [calendardate], [Date]) = 0 AND WH.ProjectId = 18
GROUP BY [calendardate]
ORDER BY [calendardate] ASC

This should produce a record for each day similar to the output below (using the UK date format):

This SQL was built up from a variety of source. To get the last day of the previous month I used the code from here. The code for creating a range of dates is based on the ideas in this blog post.

However the use of recursion to create the calendar table is not recommended for large date ranges due the performance issues discussed here. In cases where yearly reports need to be generated it is suggested creating a full table of dates in your database and joining to this rather than creating it dynamically.

Finally it took me a while to realise that to limit records in the WorkingHours table (for example by ProjectId, or a UserId) the WHERE clause should be added to part of the join, otherwise records are excluded. So rather than:

SELECT * FROM LastMonthsDates
LEFT OUTER JOIN #WorkingHours WH ON
DATEDIFF(d, [calendardate], [Date]) = 0
WHERE WH.ProjectId = 18

I needed to use:

SELECT * FROM LastMonthsDates
LEFT OUTER JOIN #WorkingHours WH ON
DATEDIFF(d, [calendardate], [Date]) = 0
AND WH.ProjectId = 18

Thanks to this forum post for pointing out that the WHERE conditions should be moved to the ON clause.

javascript:navigator.geolocation.getCurrentPosition(
function(position){
window.location.href="http://maps.google.com/?ll="+position.coords.latitude+",
"+position.coords.longitude;});

I believe the navigator.geolocation object is also available in Safari, iPhone, and Chrome. The Mozilla documentation provides more details on its use. This StackOverflow question asks which browsers and devices it applies to, and this is set to grow if it becomes part of the browser standards – the W3C has an online draft Geolocation API Specification. They list the following use cases:

• Find points of interest in the user’s area
• Annotating content with location information
• Show the user’s position on a map
• Turn-by-turn route navigation
• Alerts when points of interest are in the user’s vicinity
• Up-to-date local information
• Location-tagged status updates in social networking applications

There is also a framework API aligned to the W3C specification at http://code.google.com/p/geo-location-javascript/

In the week that ArcGIS is released as a “cloud service”, I believe the real power of the Internet will come more from manipulating and combining data services than putting desktop applications in a browser. Especially if these applications can’t easily integrate open web services. When the full range of ArcGIS geoprocessing tools can be fed streams of GeoJSON, combined with user location, then the geocloud will have truely arrived.

The greatest difficulty when trying to do things in Python is getting the right packages to match your Python version, and making sure all the package dependencies are in place.

Set Up Steps

1. Download Numpy from http://sourceforge.net/projects/numpy/files/NumPy/. Numpy is “a package that defines a multi-dimensional array object and associated fast math functions that operate on it.”

Check that you download the version that matches your Python installation.  In my case I needed numpy-1.4.1-win32-superpack-python2.5.exe.

2. Download MatPlotLib from http://sourceforge.net/projects/matplotlib/files/. I downloaded matplotlib-0.99.1.win32-py2.5.exe. Again there are many different versions that can be downloaded – expand the arrows to see the options.

There’s even a 64-bit version for Windows. However there doesn’t appear to be a 64 bit version of Numpy for Windows, and as Numpy is required by MapPlotLib to run I’m not sure how it could ever be of use. This is where the whole Python packages and 3rd party libraries gets exceedingly confusing..

3. Install the Python setuptools from http://pypi.python.org/pypi/setuptools. These tools include easy_install which “lets you automatically download, build, install, and manage Python packages.” Again get the relevant Python package in my case setuptools-0.6c11.win32-py2.5.exe.

If you have only installed Python during your log in session then reboot to refresh your environment variables. This should allow you to use the easy_install program directly from the Windows Command Prompt (by typing easy_install rather than the full path).

4. We can now install descartes using the easy_install program. Descartes is uses “geometric objects as matplotlib paths and patches.” Type the command below (the full path may not be required) into a Windows Command Prompt:

C:\Python25\Scripts\easy_install.exe descartes

This command should return output similar to below:

If you already have IDLE (the Python editor open) then quit it and restart. If not you are likely to get “ImportError: No module named X” errors, even though the modules have been installed.

5. You should now be able to run the Python script below to generate a PNG image file of England.

from matplotlib import pyplot
from descartes import PolygonPatch
import math
import urllib2
import simplejson

def getData():
    #create the cloudmade query url
    root = 'http://geocoding.cloudmade.com'
    apikey = '85453debd0fc4ad18c5855c3d8eef780'
    query = 'england'
    url = '%s/%s/geocoding/v2/find.geojs?query=%s&return_geometry=true' \
          % (root,apikey,query)

    #open the url to get the geojson
    return urllib2.urlopen(url).read()

def configurePlot():
    #set up the mapplotlib
    fig = pyplot.figure(1, figsize=(10, 4), dpi=180)
    ax = fig.add_subplot(121)
    return ax

def setPlotExtent(ax,data):
    #get feature extents (a property of the cloudmade geojson)
    #note this was previously bbox
    minx = data['bounds'][0][0]
    maxx = data['bounds'][1][0]
    miny = data['bounds'][0][1]
    maxy = data['bounds'][1][1]

    #set the graph axes to the feature extents
    ax.set_xlim(minx,maxx)
    ax.set_ylim(miny,maxy)

def plotFeature(coordlist, myplot):
    #create a polygon geojson-like feature
    poly = {"type": "Polygon", "coordinates": coordlist}
    patch = PolygonPatch(poly, fc='#6699cc', ec='#6699cc', alpha=0.5, zorder=2)
    #plot it on the graph
    myplot.add_patch(patch)

#turn the geojson into a python object
pydata = simplejson.loads(getData())
print pydata

myplot = configurePlot()
setPlotExtent(myplot,pydata)

#loop through each polygon in the MULTIPOLYGON collection
for coordlist in pydata['features'][0]['geometry']['coordinates']:
    plotFeature(coordlist, myplot)

#save the plot as an image
pyplot.savefig('myplot.png')

I now have a way of easily visualising GeoJSON I have a way of checking spatial operations on spatial data manipulated directly from the cloud. More to follow.

Eventually I found the CloudMade’s geocoding developer samples. CloudMade is a company that builds tools and services on top of OpenStreetMap. Even though many of the people involved in both CloudMade and OSM are the same, using the two brands interchangeably has not been without controversy.

Once you’ve signed up for a free account, you can then register for an API key that allows you to call their web services. See here for differences between free and paid accounts. The service I was interested in testing was the geocoding service that returns GeoJSON.

Depending on the type of feature you are looking for you can get different geometry types – the standard point, lines, and polygons, all as GeoJSON objects.

The API uses URLs and GET requests to return data. Calls can be made to either a find.html page which returns the GeoJSON object displayed on a map, as shown in the query for “england” on the left. Alternatively the query can be made to find.geojs to return just the GeoJSON itself. There is also a find.js which can be used to return geometry as JSON, although this seems a little confusing – even more so as the coordinate order is reversed from the standard GeoJSON [lon,lat].

The API can be used in several languages (Ruby, Perl, PHP etc.), my current choice is Python, and the CloudMade Python API documentation is available online here. Retreiving GeoJSON however is easy enough even without this API. For example in Python use:

import urllib2

root = 'http://geocoding.cloudmade.com'
apikey = '85453debd0fc4ad18c5855c3d8eef780'
query = 'england'
url = '%s/%s/geocoding/v2/find.geojs?query=%s&return_geometry=true' \
      % (root,apikey,query)

json = urllib2.urlopen(url).read()

print json

Now that I have a source GeoJSON I can start to use some of the more interesting tools to analyse and manipulate these features.

It can also be used as a barometer to see which current open source GIS projects have gained traction in the mainstream programming world. I took the full list of OSGeo projects from http://www.osgeo.org/ and ran queries on the site to see how many questions there were for each project. The results are as follows (as of May 2010):

A couple of disclaimers – I added the MapFish tag, and tagged a few of the GeoTools questions myself.

Its clear that OpenLayers and PostGIS are way out in front in terms of questions asked, and it may not be too much of a leap to suggest these projects are the most widely used GIS projects outside of the GIS world.

Neither of these projects however match the huge numbers of questions about neogeographer’s tools of choice – Google Maps at 1,369 questions. Poor old Bing Maps (perhaps partly due to being renamed from Live Maps) only gets 71 questions.

Stack Overflow makes use of tags and a REST API so the full collection of questions related to OSGeo projects can be seen using the following URL.

While it looks very unlikely that Stack Overflow will ever replace the OSGeo mailing lists as the first choice for debate and support, it would be good if a few of the FAQ got asked on Stack Overflow to help new users of these projects who aren’t aware of the mailing lists, or the Nabble interface to search them. It would also spread knowledge that these projects exist to mainstream developers and could help some projects gain traction.

You can also upvote (or downvote) my answers too, as there are so few GIS developers using the site that no one can assess the whether my answers are valid or nonsense..

http://en.wikipedia.org/wiki/Mitch_Hedberg

The distance as-the-crow flies between Dublin and Cork is 219 kilometres (136.08 miles in old money) wheras my tool was returning 357km. Having never driven this route it didn’t set off any alarm bells. Its also not the sort of function I’d unit test, – one as its in JavaScript, and two as the sample code seemed to work “out-of the box.”

It turns out measurements are planar by default, as briefly referred to in the OpenLayers API docs (although not on the actual measure tools pages..), and there is a geodesic parameter. By default this is set to false, so in a Mercator projected map (so anything using Google, Bing, or OpenStreetMap background mapping) measurement tools will be wrong. Maybe something to check if you’ve created a system – I’ve been 6 months working with OpenLayers and somehow managed to miss this one. Isaac@MSDN describes Geodesic as follows:

“Geodesic is a fancy (and concise) way of saying a locally path-minimizing curve.  This, in turn, is just a fancy (and mathematical) way of saying that a taught string on a surface between two points will fall into such a path.”

In fairness the OpenLayers Measurement demo makes this clear:

Note that the geometries drawn are planar geometries and the metrics returned by the measure control are planar measures by default.

If your map is in a geographic projection or you have the appropriate projection definitions to transform your geometries into geographic coordinates, you can set the “geodesic” property of the control to true to calculate geodesic measures instead of planar measures.

To set the geodesic option use code similar to the example and pass in the geodesic parameter:

measureControls = {line: new OpenLayers.Control.Measure(OpenLayers.Handler.Path,
{persist: true, geodesic: true})

OpenLayers is not alone in this. From the Google Maps API docs:

Polylines represented on the map are straight lines conforming to the current projection. That is, they appear straight on the map, but may in fact not properly account for the curvature of the earth.

If you instead wish to draw a geodesic (a segment of a “great circle” representing the shortest distance between two points on the surface of the earth), you will need to pass geodesic:true in the GPolylineOptions argument of the GPolyline.

Again code can be found in the docs, and a geodesic parameter must be set to true:

var polyOptions = {geodesic:true};
var polyline = new GPolyline([
  new GLatLng(40.65642, -73.7883),
  new GLatLng(61.1699849, -149.944496)
  ], "#ff0000", 10, 1, polyOptions);
  map.addOverlay(polyline);

I’m not sure if Bing Maps has geodesic properties. This post from Beginning Spatial has a solution for drawing geodesic lines in Bing’s forerunner Virtual Earth. There are also discussions on geodesic measurements on this MSDN messageboard.

Finally in OpenLayers, if you are using the ScaleBar and Spherical Mercator maps then check these recent changes from March 2010.

Some background information – I’ve not done any C++ for 10 years (and even then my most advanced program was to finish off a code sample on a hotel register for pirates – of the traditional kind). Also I’ve never used Subversion. Hence there are lots of pictures, and every step is detailed. If there’s anything missing let me know.

I’m not sure why I started doing this – the performance of Python using 32-bit version is fine, and since I started working on this Phillip Stabon, has mentioned he will be making a 64 bit DLL himself. However as there are many different versions of Python around, and different GIS tools rely on different versions it seemed a good idea to document how to do this from scratch.

In this example I’m using Visual Studio 2008, and a number of other tools that are detailed as I go along. I’ve also got the 64 bit compilation tools installed. There are a few notes from Phillip on compiling it for Windows 64-bit in the discussion lists (see quoted text underneath the post). Since this post was first published there is a newer discussion in the forums.

Anyway “How to Compile PyISAPIe for a 64-bit Server in 5 Easy Steps!”…

1. Get 64-bit Python

You’ll need a 64 bit version of Python. All my scripts were running and developed using Python 2.5.4 so I wanted to compile PyISAPIe to work with this.

The Python downloads are here. I initially thought that as my server has an Intel processor I should use the version for Win64-Itanium users. However it appears Itanium is only used on some very high spec servers, and I should actually use the python-2.5.4.amd64.msi file. To quote the Python site:

The binaries for AMD64 will also work on processors that implement the Intel 64 architecture (formerly EM64T), i.e. the architecture that Microsoft calls x64, and AMD called x86-64 before calling it AMD64. They will not work on Intel Itanium Processors (formerly IA-64).

If you are using the Itanium Python version on a x64 machine you’ll likely end up with the following error during compilation:

Error    10    fatal error LNK1112: module machine type ‘IA64′ conflicts with target machine type ‘x64′    python25.lib

The Python files are downloaded in a MSI (Microsoft Installer) file, and as I’m developing on a 32-bit laptop (to install on a 64-bit server later) I couldn’t install Python to access the files.

To workaround this I found a useful program called Les MSIérables. This extracts the files contained in an MSI without needing to install anything. I Had to click a few times around the interface before the extract button worked, but then I had a shiny new copy of the 64-bit Python files ready on my machine.

2. Get the PyISAPIe Source

I originally downloaded a zip of the latest PyISAPIe source code. However this is not the way to do things.. The build has an automated step which checks something in Subversion (or SVN – an online source control system), so the source code you compile needs to be a check-out. If you don’t do this you’ll get errors later on such as:

Error 2 fatal error C1083: Cannot open include file: 'Revision.h': No such file or directory

To connect to SubVersion I’d recommend downloading TortoiseSVN (in fact as one of its .exe tools is used in a pre-build step later on – its a requirement). This adds a context menu to Windows Explorer that allows you to connect to SVN and get the latest code. Set up a new folder in Explorer, then right click and select SVN Checkout.

The URL of the repository for the latest PyISAPIe code is http://pyisapie.svn.sourceforge.net/svnroot/pyisapie/Trunk/. Entering this URL and Clicking OK should start downloading the latest source files via the magic of the Internet.

3. Configure the Visual Studio Project

Once the source code has downloaded you should be able to open up the PyISAPIe.sln file which references the PyISAPIe.vcproj C++ project file. This is set up to compile the 32-bit version by default. You’ll want to change this using Build >> Configuration Manager >> x64. If this is the first time you are compiling a 64-bit DLL you’ll need to create a “New Solution Platform” as shown below:

If you try to compile now you may get the following error:

Error 2 fatal error C1189: #error :  Please define PYI_PY_VERSION at the project level.

We need to tell the compiler the appropriate Python version to use. This can be set by going to Project >> Properties >>  Configuration Properties >> C/C++ >> Preprocessor. If these options seem to be missing then make sure the project itself is selected in the Solution Explorer when clicking on Properties.

I’m using 2.5.4 so I set this value to be PY_25_4 (a list of valid codes can be seen in PyISAPIe.h from line 25 onwards). Also change the WIN32 variable to WIN64.

Another compile, another error (again apparently fatal):

fatal error C1083: Cannot open include file: 'Python25/include/Python.h': No such file or directory

Looking at line 84 in PyISAPIe.h you can see the header file wants to include the Python files:

#define PYI_PY_VERSION_MAJOR PY_25
#include <Python25/include/Python.h>
#include <Python25/include/structmember.h>
#include <Python25/include/frameobject.h>

To allow these files to be correctly referenced I copied the entire Python25 folder (see step 1) into a new Python\x64 folder. The folder structure should now look something like this:

The project is set up to look for the Python files in the Python\x64 folder (for 64 bit release builds) by default. This setting can be seen in the “Additional Include Directories” shown below. The same project can be used with different versions of Python by adding folders such as Python\x64\Python26 and changing the project environment variables.

4. Making the Pre-build Step

In the original project file there is a pre-build step that checks something in SubVersion (its description is “Subversion Revision Scan”). It can be seen at Project >> Properties >> Configuration Properties >> Build Events >> Pre-Build Event:

It runs the following command from the (Windows not Visual Studio) command line:

subwcrev . PyISAPIe\Revision.in PyISAPIe\Revision.h

Fortunately the command line tool SubWCRev is downloaded as part of TortoiseSVN. If you don’t have the prebuild step set up correctly then you may receive the following errors:

error PRJ0019: A tool returned an error code from "Subversion Revision Scan" 

The syntax for this command is:

SubWCRev WorkingCopyPath [SrcVersionFile DstVersionFile] [-nmdfe]

Where WorkingCopyPath is the path to the working copy being checked. If you aren’t using a copy of the sourcecode from SVN then you will receive errors such as “SubWCRev : ‘D:\PyISAPie’ is not a working copy.” If you have only just installed TortoiseSVN then make sure you log off or restart Windows so that your environment variables are updated.

The prebuild step should work “out of the box”, however if you still get errors in Visual Studio then running the following commands (pointing to your project folders) from a Windows Shell may provide more useful debugging information.

cd D:\PyISAPIe\PyISAPIe
subwcrev \ Revision.in Revision.h

All that is actually required by this pre-build step is to generate a Revision.h file of 5 lines..

5. Compile!

Another attempt at compiling and I get:

Error 14 fatal error LNK1104: cannot open file 'python25_d.lib'

From this post I read “python25_d.dll is the debug version. If you want to compile against python25.dll instead, you need to build a Release build of your application, not a Debug build. Or, you can go to python.org and download the debug python25_d.lib and python25_d.dll.”

I changed my build to Release and on the next compile…it worked! And look I have proof..

Aftermath

I then set up the PyISAPIe.DLL on IIS7 (using the same process as I detailed here), but using my newly compiled 64-bit, and using the DefaultAppPool which should run 64 bit applications. I also installed the Python 2.5.4 for 64-bit AMD processors (the same MSI as detailed above).

Unfortunately I currently get HTTP 404.4 errors, with nothing in the logs of use to help diagnose further. I suspect the references to WIN32 may be the issue, but I’ve no idea what to change them to. I’m posting to the PyISAPIe forum with the hope of trying to get this working, and hope that this is not the final revision of this post.

However firstly don’t run the set up program from the DVD or from an ISO on your hard-disk or you’ll end up with the following error:

‘A problem has been encountered while loading the setup components. Canceling setup’

You need to run the Visual Studio set up through Add/Remove Programs. However once I’d selected the components I got the error below – “A selected drive is no longer valid. Please review your installation path settings before continuing with setup.” Of course there is no option to review or even see your drive settings…

Thanks to this thread, the solution was to uninstall Visual Studio SP1 through Add/Remove programs (check the “Show Updates” box to see service packs and hot fixes for various programs).

Finally I got the 64-bit compilers installed, and could begin trying to compile PyISAPIe for a 64-bit processor which is a whole different set of problems.

One feature I recently “discovered” are the logging options. It’s definitely a timesaver over my previous method of putting temporary alert(vals); throughout my code. As well as debugging the logging is very useful for automatically documenting formulae in JavaScript with worked examples that can then be pasted straight into the documentation.

An example of the logging capabilities can be seen on my sample OpenLayers page. If you already have Firebug installed you should be able to see the output similar to that below when the page loads.

Console.log

The full range of options for logging can be seen in the Firebug Console API wiki page. The sample page has a few different logging options briefly explained below.

To list all properties and functions for the layer object simply add:

console.dir(layer);

To list only an objects properties (without all its associated functions), use a construct similar to:

//an alternative to just list properties
var prop;
for (prop in layer) {
    if (typeof layer[prop] !== 'function') {
        console.log('%s : %s', prop, layer[prop]); //log values to Firebug
    }
}

This example below prints details about the map object. Handily Firebug adds a hyperlink in the output to the line where console.info is called. String substitutions can be used (e.g. replace %s with a string, or %i with an integer) for easier formatting.

// write out some info
console.info("The map projection is %s and has %i layers", map.projection, map.layers.length);
console.info('Layer Name: \'%s\'', layer.name);

Assertions can be placed in code as follows:

// make an assertion
console.assert(map.layers.length === 1);

I was taught to use the assert statement as often as possible in code. Its amazing (and disconcerting) how many times “impossible” situations can occur. Whilst assertions are normally removed from production code for desktop applications, a JavaScript error message may prove invaluable when trying to debug a user’s problem remotely. As with everything in programming its a personal choice whether or not to use assertions – some people clearly assertions are evil.

The console API also supports timers, and writing out the call stack at points in your application for when things start getting more complicated.

Miscellaneous

• This function makes sure that no JavaScript errors are thrown for browsers without Firebug installed. It is included in the sample.
• Sometimes you may have to restart your browser, or make sure there is only one instance of Firefox is running for the logging messages to appear.
• Internet Explorer 8 has similar development tools that can be downloaded from the Microsoft site. It appears to support similar logging functions as Firebug, but I’ve not gone into these in any depth.

To increase the chances pretty pictures always help. If they are also useful then the documentation starts to have real value. Up until now I’ve been using Microsoft’s Visio, but recently I tried out a couple of online tools.

Class Diagrams

The MapServer Mapscript documentation currently uses of diagrams made from ASCII text to illustrate relationships between classes. I’m not sure if these were generated automatically, but the images now appear a little dated. They do  however have the advantage over an image of being able to copy and paste class names.

+-------+ 0..*     1 +-------+
| Class | <--------> | Layer |
+-------+            +-------+

I tried to recreate these diagrams using the online service http://yuml.me/. On the site you enter your model description using a fairly basic syntax. From then on the same diagram can be accessed using different URLs, which are used to control display and formatting options.

The diagram above was created using the syntax below. You can play around with this sample by clicking here.

[Hashtable]1..1->1[Class{bg:yellow}],
[Label ]0..1->1[Class],
[Style]0..*->1[Class],
[Class]0..*->1[Layer{bg:green}]

The class diagram on the left shows the properties of the MapServer Class object. Anyone can add /edit/ to the URL path to make a new diagram based on this. A “clean” format of the diagram can be seen here.

http://yuml.me/ has plenty of samples, but I couldn’t find full documentation on all the available syntax. As this is an online service you can never be sure when, or if, they might start charging. They do offer a download including sourcecode ($50 for a single user) to install locally so you can avoid paying a per diagram charge.

In response to the accusations that UML is evil http://yuml.me/ responds:

UML is evil when you use it too much for your given project; it’s easy to burn hours and hours doing UML stuff when you could be writing code or doing something more useful. Some projects use a lot of UML and that really helps them.

Sequence Diagrams

Next I wanted to make a simple to show the interaction between various parts of a MapServer and TileCache system I’ve been working on. For this I needed to create a sequence diagram that “shows how processes operate with one another and in what order.”

Scott Ambler at Agile Modelling thoughts on these diagrams are:

My advice is to only create a sequence diagram when you have complex logic that you want to think through – if the logic is straightforward the sequence diagram won’t add any value, you had might as well go straight to code.”

Whether or not my example fits into that category I’m not sure – this example was principally to test a different service. This time used I used the online service at http://www.websequencediagrams.com. The whole diagram generation process can be automated using Python – the script used to generate the diagram below can be downloaded here. Again by changing parameters in the URL you can change the “scruffy” appearance to a more clean style depending on the diagrams audience for example a more formal version could be included in a bug report, and a more colourful version in a user wiki.

Shapely

Sean Gillies has just finished documenting the Shapely a “Python package for manipulation and analysis of planar geometries.”

The first impression you get is that it’s attractive. The images of geometry (generated by the Python package descartes) catch the eye, and it was these images as much as anything else that led me to trying Shapely in the first place.

The documentation also has plenty of code samples – which as any developer knows is by far the easiest way to start using a new library or framework. If you can combine writing code samples with generating nice diagrams for a user manual then no wonder he is:

Enjoy(ing) writing some documentation. Crazy, I know.

Hit F5 and let the computer do the work.

This is because NT services like IIS inherit their environment from services.exe, which does not get updated when you change system environment until you reboot.

This knowledge may save a couple of hours of frustration if you are running Python scripts on IIS through PyISAPIe.

If you add paths to your PYTHONPATH or PATH system variables to use a Python package (such as MapScript or GDAL), then the same script may run without problems in IDLE, but continue to produce errors when run through the web server.

If you are running a script in IDLE then you need to restart IDLE itself to refresh the environment variables.

Other Options

I’ve not tried this method myself but R. Teachout suggests a registry hack that allows you to override the Windows environment variables in IIS 6.

You can also override the environment variables in the Python scripts themselves. For example to change the GDAL_DRIVER_PATH variable use:

os.environ[“GDAL_DRIVER_PATH”] = “C:\ms4w\gdalplugins”

To print out current environment variables (assuming you are testing in IDLE):

print os.getenv(“GDAL_DRIVER_PATH”)

I remember being fascinated with the pictures and the theories and solutions bandied about by those older and wiser than I. The majority of programmers seem to have a fondness for puzzles and riddles, and the beauty and success of the book was that it involved transposing clues to maps and real-world locations.

Geocaching

Geocaching could be seen as a successor to Williams’s book.  As someone working in technology and geography it should seemingly be an ideal pastime. However the fact geocaching is a trademarked word owned by a company named Groundspeak kills a little of the mystique. Secondly it appears to me to be the equivalent of driving around in the middle of nowhere with a bad in-car GPS trying to find an address that’s missing from the database.

I wouldn’t go as far as police Inspector Tyrus Cameron who mistook a geocache for a bomb and was quoted as saying:

“They can have their fun, but we can’t be responding in this nature, or it’s a huge issue for the citizens of the city.”

But it does seems to be a mechanical process with little puzzle-solving involved. As I’ve never tried it my opinion clearly counts for little, and I’m open to persuasion.

Easter Eggs

In keeping with the theme of buried secrets and Easter, I’m wondering if the programmer tradition of inserting Easter Eggs into software is now extinct. Microsoft have hidden many eggs over the years, but now officially ban the practice to calm corporate security concerns. And where Microsoft lead, desktop GIS will undoubtedly follow.

Previously typing Jerry in ArcMap version 8.x would pop up an image of the man himself. This has been “fixed” from version 9.1 on as it seemingly caused issues for several users while editing.

Google Maps has a few Easter Eggs, and this year the Australian site gave directions and toll information in Aussie dialect for April Fool’s Day – “Ya might have to cough up some cash along here.” Due to the nature of web applications, and Google’s now familiar quirky approach, they aren’t as as much of a thrill to find as a mode staid GIS application. FME have the odd one-liner in their geo-processing output, and hopefully there are still a few treasures to be found, and a few treasures to place.

Riddle Me This

On my trip down memory lane I found a riddle. A copy of Hareraiser (and Spectrum emulator) for the answer.

There came four men from out the wood
Their way was straight and their strength was good
Then over a bridge the four men ran
To come together as one man
There came a horse without a bone
That made the four men whine and moan
Wood and men and bridge are all but one
Put them together and you’ll find my riddle is undone.

He [the shadow chancellor] was unable to cite specific projects, as opposition parties are not allowed to inspect government contracts.

What would be ideal is if IT projects were able to be scrutinised by the opposition parties, and the public. It appears Australia is far more open with their government tenders, and even include the actual costs involved which would be considered as extremely sensitive business data in other countries.

During the good times IT projects are always associated with savings and efficiencies. During the bad times they now seem to be seen as an extravagance, and prone to failure. A good point made in the comments on The Register article was:

Personally, I’d like all large businesses subject to the same auditing and publishing requirements that government IT projects are so we can get some perspective. Starting with the media. I strongly suspect that government projects are no worse than anyone else’s, but they’re the only ones open to serious scrutiny.

Amongst the projects likely to face cuts is the National Programme for IT a National Health Service IT project with  a budget of around £12bn, that was set to be “the world’s biggest civil information technology programme.” Whether or not patients are likely to benefit is very much up for debate. What does seem to be lacking in this plan is the use of GIS and geospatial technologies, apparently due to a lack of expertise. Maybe ESRI should have sent the NHS their brochure.

Send me the Link

Where the government (or their advisors) may previously have FAILed is their desire for centralised databases and systems. This approach may have recently changed to a more decentralised linked data approach.

[Linked data] is about using URIs to identify things, providing information at the end of those URIs that is self-descriptive, and linking those things to other things through typed links.

Sean Gillies seems to think it’s a step in the right direction – which is always a good thing. Have a look at the new site to judge for yourselves. The use of linked data rather than databases in walled gardens, allows for all sorts of applications to be created by both the public and private sector such as this expenditure map (I’ll forgive the use of Flash just this once..). The amount of mash-ups that can be created is now only limited by the imagination. Notice how maps are high on the list of ideas.

A change that has occurred in the UK that could be seen as on a par with Obama’s health reforms is the freeing of data. Recently the Ordnance Survey released free mapping. This seems to have had a knock on effect with the Post Office relaxing copyright on postcodes (the zip code equivalent). Although it may have helped that this has been available for download on The Pirate Bay since last September.  [Clarification - the Ordnance Surey have released the geographic locations of postcodes, but the Post Office have not released the database linking actual postal addresses to postcodes].

So even though the cuts to IT may have an impact if there is a change of government, the opening up of data will create far more opportunities to geospatial developers, who will now be able to build workable systems and prototypes using real data to proactively drive IT projects rather than wait for the high hand of government to possibly offer them some crumbs from the table.

Alan Skorkin has a great post on “you don’t need math skills to be a good developer but you do need them to be a great one.” Geo-technology developers have similar requirements, and the majority of GISs revolve around CRUD operations. However due to the nature of spatial data we occasionally get to meddle in the world’s of projections, geometry, typology, and set theory more than our counterparts in other sectors.

The Ancient Mysteries

Whilst concentrating on algorithms and recursion is a definite benefit to all developers, us spatial developers need a few more specific tricks.  Euler’s 1736 paper on Seven Bridges of Königsberg is one of the first academic writings on modern topology. The titles of these mathematics puzzles create an aura of mysteriousness and arcane knowledge that definitely sweetens the somewhat dour algebra underneath.

The solution (or non-solution) to this problem underpins many of the more advanced techniques used by GIS packages today. The “Travelling Salesman”, whilst conjuring up less interesting images, relies on similar principles of graph theory.

Traditional desktop GIS sales take a slump..

Graph Theory

The excellent ESRI online documentation provides details on all aspects of GIS, including networks and graphs.

What can happen if you don’t understand typology

Graph theory is the foundation for understanding networks and topology. A geographic information system (GIS) modeller should be familiar with the concepts and terminology of graph theory because it helps to classify and model connectivity and adjacency relationships among geographic features.

The underlying graphs of a network allow us to solve problems such as finding a shortest route, or analysing connectivity between points on a river network. It could certainly help a Chinese Postman.

Whilst being able to implement these algorithms from scratch is beyond the call of duty for most developers (except those who like to reinvent the wheel), an understanding of how they work, and being able to recognise the situations where they can be used, can save weeks of development time and lead to cleaner more elegant coding solutions.

Self-Learning

Once you out in the “real-world” it is highly unlikely you are ever going to get taught this maths, and it becomes a little daunting realising you are now responsible for your own education. Even if you’ve already studied maths to a reasonable level, as the adage says – use it or lose it. This is brought home by my worriying belief that my mental arithmetic was superior as a child through constant use and learning tables by rote, than it is now after 10 years of programming.

Every now and again its useful to buy a book that deals with areas related to computer science, rather than a “How to..” book of the latest programming language or framework, to try and attain a broader understanding of the maths behind the APIs.

Graph-ics

To try and illustrate my point that only a basic understanding of a branch of mathematics is required to produce useful, and sometimes spectacular results I attempted to use a specialised open-source tool armed with only a basic glossary of the terms used in graph theory.

Gephi is “an interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs.”

Amongst the sample datasets there is an adjacency network of common adjectives and nouns in the David Copperfield. I was able to import the data into the program and then export the SVG image shown below. Interestingly “happy” is seems to have as many connections to love as money, which is pleasant.

Gephi produces beautiful data and is definitely a project to keep an eye on – the Road map for version 1.0 includes Geographic data (latitude/longitude) which will open up all sorts of possibilities.

Do the Math(s)?

I’d categorise most people coding in the geospatial area (outside of those in academia or writing the internals of a GIS library), as developers, rather than computer scientists or programmers.

(Developers) are consummate generalists without any truly deep specializations. They are expert at finding ways around problems and plugging components together to fulfil a set of requirements.

A basic understanding of each of the branches of mathematics used by GIS is required to know what exactly you need to input, and to understand the output, of algorithms implemented by others. A grasp of terminology certainly helps navigate the user interfaces of specialised tools, or find help through search engines. At the very least it leads to an understanding of why Manifold is named manifold, and can stop or start us going round in circles.

FAQ

How is SQL Azure Database different from working with a hoster using SQL Server?

Developers do not have to install, setup, patch or manage any software. High Availability and fault tolerance is built-in and no physical administration of hardware, storage or servers is required.

If you have a good hosting solution then the same benefits apply (well maybe not the “self-healing” also mentioned). The only real difference I can tell is that you pay a lot more, to be able to scale more easily. Its unlikely your Local Planning Mapping system is suddenly going to need to scale up to meet the expectations of millions of new users, no matter how great the system looks.

Update: Many thanks to David Chou for his clarifications on this – see comments below. In essence a SQL Azure database uses three replicas of your database with a layer on top that makes it appear to the application and user as a single instance. It is this new architecture that allows for scalability, and the self-healing (which still makes me think of Tekken’s Yoshimitsu).

Windows Azure and SQL Azure SLAs are independent of our on-premises Microsoft licensing agreements.

So if you do plan to continue standard “earth” development, you’ll still be paying SQL Server licences. For most government departments and large businesses there will always be some data that needs to be stored locally.

For Developers

Currently you can’t bring your existing on-premises Windows Server, SQL Server to Windows Azure, SQL Azure.

So it doesn’t seem possible to develop database applications locally and then upload all your views, stored procedures, functions and data in one go to Azure. It appears this only applies to the server software (see comments) – I’d imagine Enterprise level funcitonality such as mirroring or replication. The actual tables, stored procedures, and views will be fine to import.

SQL Server Management Studio will be able to access the online databases, but I would be interested to see how quickly a result set from a query would be returned. Would this data be returned locally and slowly, or would it work in a similar way to Remote Desktop by returning images from the server?

Cheque Please

The pricing structure is as follows:

• Web Edition:  Up to 1 GB relational database = $9.99 / month
• Business Edition:  Up to 10 GB relational database = $99.99 / month
• Data transfers = $0.10 in / $0.15 out / GB – ($0.30 in / $0.45 out / GB in Asia)*

I’ve just completed a GIS project using SQL Server 2008, with around 60 different layers. The database is around 15 GB (this stored procedure returns disk space on a table by table basis). Uploading this data would be cheap enough at only $1.50, but would take hours. It was far easier to go round to the hosts with an external hard-drive, than upload from our network.

Where things get scary for a GIS developer is considering the data transfers out of a system as soon as the Microsoft spatial data types get involved.

The amount amount of bytes sent over the network in SQL Server for each query can be viewed using Client Statistics. Selecting 100 records from one of my tables resulted in 14 KB of data transfer.  Selecting 100 records and their associated polygons resulted in over 10 times more traffic at 151 KB (here’s an online calculator for bit/kilobyte conversions).

Whilst I presume it is possible to set up a WMS server such as MapServer in Azure, it is unclear whether the “data out” costs apply to data that leaves the cloud or to any data that leaves the database (thanks to Ed Katibah for more details – charges only apply to data leaving Azure sub-region (Western Europe is a sub-region). In this second scenario it could well be game over. It is easy to imagine spiralling costs if you wish to display a few dynamic vector layers, and the database takes a hit on every user’s zoom or pan.

It’s (not) all about the Data(base)

The target market for SQL Azure in GIS however may not be individual applications, but rather business to business spatial services. Datasets used for geolocation and navigation could be stored in one place with an API that could then be called by any number of web applications. These “clients” could then be charged a premium on top of Azure’s costs for the additional services. Heavy use of results caching would mitigate the price of data transfer.

Unfortunatley for Microsoft it appears once again Google already has a big chunk of this (spatial) market wrapped up. For free, or $10,000 if charging for your service.  Another example is James Fee’s WeoGeo. I presume by James’s quote about getting your ducks in a row his is clearly a business model that could benefit from SQL Azure. Both examples differ from the SQL Azure model of selling database services in that they both provide data as part of a cloud solution.

There are still many unknowns on how this will develop over time, and who will benefit from Azure. Its nice to know the spatial functionality is out there in the cloud, but for the majority of web mapping applications the risk of sky-high costs may keep it there. Case studies and early-adopter’s experiences will hopefully help paint a clearer picture. Microsoft’s main competitor could well be Microsoft. SQL Server 2008 Express is free (however with a 4GB limit), and includes both spatial functionality and Reporting Services. If you need to scale up it seems simpler to change your licence rather than your entire mindset.

In an effort to move away from MapInfo’s .tab files (the shapefile equivalent) I was keen to test out the direct connection to geometry stored in SQL Server 2008. Until version 9.5 data could only be stored and accessed in a relational database using SpatialWare. Again to use ESRI terminology this is the ArcSDE equivalent. Fortunately MapInfo was quick off the mark in removing this barrier.

Access to SQL Server 2008 spatial data is provided by integrating FME functionality directly into the product, and having a “Universal Data” option on the File menu. There is also a tool named Easy Loader that allows loading .tab files into SQL Server as geometry.

Unfortunately on trying to use the Universal Data menu after installing a fresh copy of MapInfo 10 I was met with the error “This application has failed to start because qfmeext.dll was not found. Re-installing the application may fix this problem.”

Hardly a good premier start..Anyway it appears upgrading to the 10.0.1 maintenance release fixes this issue. No mention of the issue itself in the release notes (Update: It is in the release notes, see comments below, and only occurs when ArcGIS is also installed). There were also a couple of additional fixes for SQL Server (fhe following text direct from the maintenance release notes).

Improving Performance when Connecting to a SQL Server Database

To improve performance when accessing large SQL Server tables, add a Multiple Active Result Sets (MARS) connection setting to the connection string in the live access .TAB file. The connection attribute to update is \DATALINK\ConnectionString, for example:

"\DATALINK\ConnectionString" = "DSN=local-native10;UID=username;
Trusted_Connection=Yes;APP=MapInfo Professional®;
WSID=hostname;DATABASE=test;MARS_Connection=Yes;"

You may also add the MARS connection setting to the connection string of MapBasic Server_Connect( ) command. Note that a MARS connection:

• does not work for SpatialWare tables.
• only applies to SQL Native Client or SQL Server Native Client 10. Other connection types so not benefit from this setting.

Also Fixed..

T25293 Opening a SQL Server table containing a column with type NCHAR causes an error message, “Unsupported data type for column…”
Resolution: Issue is resolved.

1. Get to Know your GIS Application

Most desktop development in GIS involves building extensions to an existing application using an existing programming library. In order to avoid reinventing the wheel you need to be able to use your GIS desktop package.

Before developing any automated tasks, manually go through the process to get a feel for any issues that may arise, and learn some of the terminology used by the program for the various functions. If the application has an option for hatched polygon fills, this option is also likely to be buried somewhere in the API. If you can’t find what you are looking for then run what you are trying to accomplish past a GIS user.

2. Learn some Basic Terminology

A good GIS glossary can be found on the ESRI site, and it is also available in a A to Z of GIS book. Of course nothing is simple as different software companies use different terms for the same thing so a Layer in ArcMap, is the equivalent of an Overlay in Google Maps.

If you want to find all the point addresses within 100m of a road you don’t need to spend weeks writing your own algorithms (I’ve seen this happen..), armed with a good GIS vocabulary you will be aware of the term buffering and can find how to do this programmatically in your software. Your new employer will prefer a basic question related to GIS or geography than 2 weeks developing a function that could be achieved with a single line of code.

If developing for the web there are a plethora of technical terms you will have to learn such as WMS, WFS, GeoJSON, GeoXML (most web standards also seem to take a Geo- prefix), KML, tile caches etc.

3. Get to Know your Datasets

Half the battle with GIS development is getting data into a consistent format so you can run analyses on it. Get ready to start dealing with Excel spreadsheets, strange proprietary GIS formats, and file dumps that may or may not resemble databases. Remember that many of these datasets will be coming from GIS technicians rather than database developers. Safe’s FME can convert pretty much anything to anything else, and the translations can be saved and automated saving plenty of time.

Spatial data is not too complicated. Most features can be represented as points, lines, or polygons. Obviously things can get more complicated such as multipart polygons, doughnuts, geometric networks etc. but these features are not overly common. However learn about projections and how to reproject data. If your data is invisible with no errors then projections are often the issue.

There is also a semi-mythical object called metadata. If this doesn’t exist expect to spend plenty of time trying to track down what various fields mean, if look-up tables are available, and what projection the data is in.

4. Learn the differences between Raster and Vector.

If you want to display cells across a whole country then you will need to user raster, vector data will take an age to draw. Don’t make the mistake or drawing lots of nicely coloured geometric squares.

5. Find New Programming Forums

You may need to look further afield than StackOverflow for answers to technical problems (except maybe Google Maps on Android).  Where to look depends on what you work with. Each major GIS vendor’s site is likely to have a forum which serves as the focal point for all developers using their software, for example ESRI has widely used user forums. For open source mailing lists seem to be the support technology of choice – here Nabble is your friend.

6. Bring your experience

Many people working in GIS come from a geography background, and are self-taught programmers. If you are working with a team then try to introduce concepts that are now commonplace in other areas of software development – source control, unit tests, object programming etc.

You may also see GIS users spending hours doing things by hand that can clearly be automated. If you know some Python then you will be able to save hundreds of man hours with geoprocessing scripts for both proprietary and open source applications. For unknown reasons Python has become the scripting language of choice in GIS. Its good for web GIS too.

Along with software development practices any previous experience with databases will be huge plus. If you know what the following are you will soon become a very productive member of staff – primary key, normalisation, relationships may be new concepts to your new organisation. Geodatabases introduce a few new concepts, such as spatial indexes, projections, and geometry types, but if you have a good grounding in standard databases you will pick things up easily.

Feel free to add anything I’ve missed in the comments below, and good luck!

I’ve come to the conclusion that a detailed and long word document, while it may tick the box on the client’s checklist, is an exercise in futility. No-one ever reads it, you’re lucky if you can even find it a year later, and its out of date the moment its written (or  if its a rehash of the technical specification its never actually in-date).

My key requirements for user documentation are:

• make it easily accessible from within and from without the system
• have only one repository for the user documentation
• allow people other than the developer to create and edit the documentation
• auto-generate when and if possible

I initially started creating web pages, even for desktop systems, as these could be accessed by both project managers and users of the system. This met most of my criteria above, but I knew that if anything in the system changed I’d be the one who would have to go in and edit the HTML. Step forward..

Wikis

Despite a name that makes me cringe every time I say it, an online wiki has all the benefits of using web pages for user help systems, but with an additional  number of advantages:

• non-developers can learn how to use wiki markup much more easily than writing HTML pages
• edits are attributed to users
• edits are stamped with a time and date which gives you an overview of how up to date the documentation is (this may not necessarily be a good thing..)
• inbuilt search functionality
• premade themes and templates
• administration tools

ScrewTurn Wiki

I’ve been moving more and more towards open source tools, and after a brief dalliance with a few different wikis I settled on Screwturn Wiki. It is written in .NET which I am very familiar with, and most of my client’s sites are hosted under IIS/Windows. While some clients do use Microsoft’s Sharepoint, and Sharepoint includes a wiki, it seems overly complicated to develop a help system internally and then try and import to the client’s Sharepoint set up. A new ScrewTurn Wiki site can be set up in about 10 minutes, and an existing site can be installed on a client’s server by simply copying over a folder.

Now the fun part..

While the actual writing of user help is tedious, the creation of a help system need not be. And if you are using a wiki you may be able to “delegate” the user help to someone else (depending on the size of your company). Once the wiki is being populated the more you can use this information throughout your system the better.

For a MapFish based online mapping system I am creating, I was able to automate my help system to display the toolbar buttons, along with their associated icons, as part of my help menu structure. If the icons are changed, the button removed or renamed, or new buttons added, the help system will update accordingly. The next step was to load the relevant tool description text from the wiki when the tool was selected in the menu.

To do this I created basic pages in the wiki, and gave each toolbar button a custom helpPage attribute with the name of the wiki page. I then took my first look at the ScrewTurn Wiki (version 2.0) sourcecode to see how to retrieve the relevant HTML.

I created an ASP.NET handler that would retrieve the HTML from a pre-made wiki page, without requiring IFrames. The class can be seen here.

I had to edit the Global.asax file to allow my wikiHandler.ashx web handler to be accessed (by default all .ashx are seen as wiki pages).

if(ext.Equals("ashx")) {
    if(pageName != "wikiHandler"){
      // Demand the request to Default.aspx
      HttpContext.Current.RewritePath("~/Default.aspx?Page=" +
      ScrewTurn.Wiki.Tools.UrlEncode(pageName) + qs);
    }
 }

I could then make requests such as wikiHandler.ashx?Page=ZoomIn to return snippets of HTML that would be loaded dynamically into my help system. Knowing that wiki text may actually be read helps to get that “documentation” task finally checked off and a project signed off.  A client is also happy to know that they can update and add to the help notes without requiring developers.

The installation went well  on the production server which was also a 64 bit machine, but was using Windows Server 2008. The site had an unpublicised release as it is still in beta, and from this was getting about 60 – 70 visitors a day.

I use host-tracker.com – a web site monitoring service which lets me know if a site is down, and I was soon made aware that it was.. There were no responses at all coming from Apache, even though the Apache service was still running. Not even HTML pages were returned. Restarting Apache returned the service to normal, but this was happening every other day.

The Windows Error Log

The Windows Event Log had lots of the following errors:

Faulting application name: mapserv.exe, version: 0.0.0.0, time stamp: 0x4935ad86
Faulting module name: MSVCR71.dll, version: 7.10.3052.4, time stamp: 0x3e561eac
Exception code: 0xc0000005Faulting process id: 0xe08
Faulting application start time: 0x01caa0e28e6dcbb6
Faulting application path: C:\ms4w\Apache\cgi-bin\mapserv.exe
Faulting module path: C:\ms4w\Apache\cgi-bin\MSVCR71.dll
Report Id: 4bc3555e-0cd7-11df-9e56-00155d002601

The MSVCR71 refers to the Microsoft C run-time libraries, so I *think* the errors are likely to be from a program that uses them (ie. MapServer) rather than the DLL itself. Windows 2003 has this DLL in its system32 folder, whereas Windows 2008 does not, but as long as it is in the application directory (cgi-bin in MapServer’s case) then it should not be a problem. In short I don’t believe this DLL was the root of the problem. I tried turning on MapServer debugging, but there were no errors reported from MapServer itself.

The Apache Error Log

On the other hand the Apache log (C:\ms4w\Apache\logs\error.log) had several warnings such as the following:

[Thu Jan 28 15:37:15 2010] [warn] (OS 64)The specified network name is no longer available.  :
winnt_accept: Asynchronous AcceptEx failed.

The Apache documentation suggests that a possible cause of these errors is due to “virus scanning or virtual private network packages, have bugs that interfere with the proper operation of AcceptEx().” While I could rule the first out, the second was a definite possibility as I was required to use Juniper’s NetScreenRemote to connect to the server. The Juniper site now states “No Longer Available. Use Standards based IPSec client.”

I’ve had horrible problems with this software before, including no longer being able to connect to wi-fi networks (repairing IPSEC was the eventually solution), and corrupting a Cisco VPN client. The 0.5 star review on CNET show I am not alone in this opinion.

I tried using the Win32DisableAcceptEx directive to try and stop these errors and hopefully resolve the Apache crashes. Apache runs in 32 bit mode, so hopefully the 64 bit server wouldn’t mind..In C:\ms4w\Apache\conf\httpd.conf  add the following line, and restart the Apache service:

Win32DisableAcceptEx

Final Outcome

Unfortunately although the Apache changes stopped the errors it didn’t stop the crashing. In the end I had to set up MapServer with IIS and the site has been working fine ever since. As a bonus I was also able to use the 64 bit versions of MapServer.

Why Not CGI?

While IIS 7 has Fast-CGI installed (see this IIS forum), even better performance can be achieved using ISAPI. This answer from ServerFault has a good summary on why ISAPI should be preferred over than CGI. Not only  performance  should be considered – maintenance should also be taken into account. If a web master or hosting service  is familiar with IIS then they are also likely to be familiar with ISAPI.

From the PyISAPIe site:

The reason ISAPI applications have the capability of being better than CGI or FastCGI applications is mostly due to its tight integration with the web server environment. Instead of initializing an entire program from scratch (in this case, the Python interpreter) every time a request is made for a page, an ISAPI extension only has to provide a function that is called upon every request. For interpreting Python scripts on a per-request basis, this means that the interpreter can be initialized once and used many times, creating a very noticeable performance gain.

Some Definitions…

WSGI -The Web Server Gateway Interface defines a simple and universal interface between web servers and web applications or frameworks for the Python programming language.

My interpretation – WSGI glues together IIS and Python scripts, or any web server and Python application combination, in a standard way.

ISAPI – The Internet Server Application Programming Interface (ISAPI) is an N-tier API of Internet Information Services (IIS), Microsoft’s collection of Windows-based web server services. The most prominent application of IIS and ISAPI is Microsoft’s web server.

My uninformed interpretation – ISAPI allows DLLs written in a variety of languages to be loaded into IIS to handle web requests. ASP.NET would therefore be an ISAPI extension,  set to handle requests which end in .aspx

The Python ISAPI Extension

I initially tried isapi-wsgi, but I didn’t have much luck setting it up (I gave up fairly quickly though). It also depended on “Mark Hammond’s Python win32 isapi extension” – and I was trying to keep the amount of packages installed and added to a minimum. I therefore chose to use PyISAPIe which has no dependencies, and only requires a DLL and a folder containing a few .py files.

Documentation and installation instructions are sparse, and assume good knowledge of Python, IIS, and web requests in general in which I was lacking. In retrospect they make perfect sense though! The following links seem to be about the sum knowledge of PyISAPIe on the web – the homepage, the install guide, getting started, the news group, and the API. It struck me a few times while trying to set up Python and IIS that the numbers of people that doing this could be incredibly low. The last Microsoft article I found relates to IIS 5..

Installation on IIS 7

1. First make sure IIS is actually installed (Server Manager >> Roles >> Add Web Server IIS). Ensure the ISAPI Extensions and ISAPI filters are both checked.

2. Download the latest PyISAPIe extension. Make sure you get the correct version for your version of Python. In this example I am using Python 2.5.4 and PyISAPIe 1.1.0-rc4-Py2.5

3. Where to put the files took a while to resolve..in the end it becomes apparent they can be placed anywhere! I got confused by my understanding of packages and by thinking I could test the Python scripts outside of a browser / IIS.

In the end I created a new folder in C:\Python25 named PyISAPIe. I then copied into this the pyISAPIe.dll and HTTP folder.

4. Next you’ll need to set up the handler in IIS. I wanted all files that ended with the .py extension to be handled by the PyISAPIe DLL, so I did the following:

• under sites add a new application (I used the name /apps)
• select the site, and then select “Handler Mappings”
• right-click and select “Add Script Map”
• set the “Request path” to *.py
• set the “Executable” to “C:\Python25\PyISAPIe\PyISAPIe.dll
• give the handler a relevant name such as “PyISAPIe”
• in the Request Restrictions section, I set my handler to run only when a request contains a .py file that exists on  the server. I set it handle all verbs (GET and POST), and gave it Script access.
• select the “View Ordered List” in the Actions panel to see in what order handlers will be applied to a request. Make sure your .py handler has a higher priority than the default StaticFile handler, or it won’t get a chance to handle anything. The StaticFile handler may also appear to be handling requests if your custom handler fails as by default PyISAPIe moves to the next handler on an error.

Important! If you are using a64-bit machine, you will need to create a new application pool, and set “Enable 32-bit Applications” to true. This is false by default. I found this blog post fairly early on describing the issue but, as if by magic, the true setting for my application pool had at some point been set back to false, leading to hours of being confronted by the near useless message:

HTTP Error 404.4 - Not Found
The resource you are looking for does not have a handler associated with it.

Testing

The Getting Started Guide should be followed, but I have a few gotchas, learnt the hard way. First save a Python script into your web application folder with a name such as hello.py with the test code from the guide:

from Http import *
def Request():
    Header("Content-type: text/html")
    Write("Hello, World!")

Note this script cannot be run from a Python editor, you will get warnings such as:

ImportError: No module named Http

Even if the HTTP module is in the PYTHONPATH, or site-packages folder you will still receive an error such as that below, as the DLL needs to be loaded through ISAPI.

ImportError: No module named PyISAPIe

Pointing to http://localhost/apps/hello.py should bring up the output of your Python script. Things get more confusing when you want to debug a more complicated script, as it appears nothing you change in your script has any effect. This is because once the script is run it stays loaded in memory, so no alterations are taken into account. To disable this (for testing only – change it back when everything is working correctly) modify the C:\Python25\PyISAPIe\Http\Isapi.py file.

def Request():
 Script = Env.SCRIPT_NAME
 Key = Name = '__'+md5(Script).hexdigest().upper()
 Handler = Handlers.get(Key, None)
 # the following line will ensure the script is reloaded on each request
 Handler = None

 if not Handler:
 try:
 Handlers[Key] = imp.load_source(Key, Env.SCRIPT_TRANSLATED).Request
 except Exception, Val:
 # trigger a passthrough to the next ISAPI handler -
 # ONLY WORKS FOR WILDCARD APPLICATION MAPPINGS
 #return True
 # or just fail, preferable for an application map
 # show errors in the browser
 raise ImportError, "[Loading '%s'] %s" % (Env.SCRIPT_TRANSLATED, str(Val))

 return Handlers[Key]()

Note the changes you make to the Isapi.py file itself requires IIS to be restarted to take effect. The above modifications only cause the scripts in your www/apps directory to be refreshed on each request.

If you always return true on any exception in the Isapi.py file (rather than throwing the error) then you may be met with the following message:

Possible recursion detected!
You probably did a passthrough with PyISAPIe configured as an application map instead of a wildcard map.

I *think* this is because the first handler failed to return a proper response, and so it goes to the next handler for the application. Just failing should provide you with a better error message from Python.

If I’ve missed out any key steps in this summary feel free to add comments below and I’ll integrate them into the post. Last but not least many thanks to the developer of PyISAPIe – Phillip Sitbon.

My YSlow Firefox plug-in was telling me that I had included too may JavaScript files, and that they should be combined into a single-file, and minified. The default setup of Mapfish has a single script include (mfbase\mapfish\MapFish.js), but this script  automatically loads around 40 further scripts.

I tried a couple of unsuccessful attempts at copying and pasting various Mapfish JS files into an online JavaScript compressor. However when referencing the minified file I kept getting “undefined” errors from various Mapfish plugins and OpenLayer objects.

Initially I was looking to download a ready-made “compiled” MapFish file, but was unable to find a version online. I have uploaded a zipped version so if you want the file without going through the steps below you can download a copy. Note that if you have made any modifications to the MapFish files then these will not be available, and there are no guarantees it will work.

There are instructions on creating a single-file Mapfish client build on the Mapfish site. This process uses JSBuild which is part of JSTools – “a collection of python scripts for packaging and efficiently serving javascript.”
Unfortunately these instructions require a Linux operating system, and I’m using Windows. I eventually managed to “translate” these into Windows commands, and have documented the steps below.

Step 1 – Install Python for Windows

First of all if Python is not installed on your system, then get the MSI installer from the Python site. I used version 2.5.4, so I’m not sure if the steps below work with later versions. Once it has been installed all .py extensions should be associated with the python interpreter. To check this you can run the following commands in a Windows command line prompt:

assoc.py
ftype Python.File

Step 2 – Create a Virtual Python Environment

This step creates an isolated Python environment for JSTools.  Using this “virtual” environment avoids using wrong script versions and references. The Mapfish instructions for Linux were as follows:

$ wget 'http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.3.2.tar.gz'
$ tar xvzf virtualenv-1.3.2.tar.gz
$ python virtualenv-1.3.2/virtualenv.py env
$ source env/bin/activate

These rougly translate into Window’s speak as:

• Download the VirtualEnv installer from http://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.3.2.tar.gz (wget is simply a Linux program that handles the downloading of files).
• Unzip the downloaded file (for example using 7-zip for Windows)
• Run the following command in the Windows command prompt. Change the first parameter to point to the location of virtualenv.py you just downloaded, and the second to a folder where you want to create your virtual Python environment.

  D:\Tools\Mapfish\virtualenv-1.3.2\virtualenv.py D:\Tools\Mapfish\VirtualPythonEnv

• Finally run the activate.bat file which should have been created by the above steps in D:\Tools\Mapfish\VirtualPythonEnv\Scripts\activate.bat

Step 3 – Install the JSTools

Again from the Windows command line, change your current directory to the newly created Scripts folder in your virtual environment setup.

cd D:\Tools\Mapfish\VirtualPythonEnv\Scripts\

Next run the following command (again from the Windows command line):

easy_install.exe -i http://dev.camptocamp.com/packages/mapfish/all/index JSTools

Trying to install the latest version of JSTools from http://github.com/whitmo/jstools/tarball/master caused errors (possibly due to Windows default download paths), but the rest of the steps worked fine with the JSTools available from the Mapfish (CamptoCamp) link used above.

Step 4 – Go Forth and Minify

The final step is to create the minified JavaScript itself. The Mapfish client download contains a configuration file to tell JSBuild which files to include in the single file script. It was at this stage I realised it may have been possible to manually combine these files in the correct order and run the online compressor, but now everything else was in place it was as well to use JSBuild.

First of all I modified the first few lines of the mfbase\openlayers\build\full.cfg file to point to the JavaScript files on my system. This may not have been necessary depending on where JSTools was run from, but this way worked!

[MapFish.js]
root =
 D:/Tools/Mapfish/mapfish-client-1.2/mfbase/openlayers/lib
 D:/Tools/Mapfish/mapfish-client-1.2/mfbase/mapfish
 D:/Tools/Mapfish/mapfish-client-1.2/mfbase/geoext/lib
first =
 OpenLayers/SingleFile.js
 OpenLayers.js
 ...rest of the file remains the same

Next I ran the following command from Windows (making sure my current directory contained the jsbuild.exe). The -o parameter is the folder where you wish the single file to be created, and the second parameter points to the configuration file listing the JavaScript files.

jsbuild -o D:\Tools\Mapfish\ D:\Tools\Mapfish\mapfish-client-1.2\build\full.cfg

A new MapFish.js was created!

I was then able to modify the .cfg file further to add in and minify my own application JavaScript files.

Step 5 – Test in Web Application

The previous reference to Mapfish could now be pointed to the newly created MapFish.js file:

<script type="text/javascript" src="js/MapFish.js"></script>

Although the file is now ~1MB it is smaller than the individual files combined, and it requires less server requests which improves page loading times. Most web servers (including IIS and Apache, but not IIS running on Windows XP) allow gzip compression which will reduce this file to around 250KB.

If you encounter any problems or steps I’ve missed then please let me know!

The GetCapabilities Request

Web Mapping Services (WMS) is an open standard that all major GIS vendors implement in their server software (for example see the ESRI documentation). As a result applications built on these services are able to easily switch from one software package to another – allowing separation of code that allows for easier maintenance and reuse.

The GetCapabilities is one of three requests in the WMS specification. It “returns service-level metadata, which is a description of the service’s information content and acceptable request parameters.”

After eventually being able to see the results of a GetCapabilities request my aim was to generate a report listing the available map layers, and to check I had enetered all the required metadata. I had a look on the web to see if there were any stylesheets that would nicely format the data.

The easywms.com site has an example on how to build a WMS client from GetCapabilities using XSL but I was unable to get the sample working with my GetCapabilities output. XSL is a stylesheet language for XML documents, in much the same way that CSS is used to format HTML. I’ve worked with XSL a little before so decided to write some code myself after not being able to find an existing example.

Adding Metadata to the WMS Server

In order for the GetCapabilities request to return data, it has to be entered in the first place. Each WMS server will be configured in different ways. In this example I am using MapServer. The documentation for the WMS Server can be found here.

WMS services are configured using MAP files. Metadata can bet set for both the overall map and for each LAYER object in the map.Further documentation can be found here. An example of layer metadata would look as follows:

LAYER
   NAME 'MyLayerName'
   TYPE LINE
   METADATA
     'wms_title' 'My Layer Title'
   END
   ...

Using the GetCapabilities Request

To test the MAP file is set up correctly as a WMS Service follow the steps here. You can paste a URL such as the following:

http://my.host.com/cgi-bin/mapserv?map=mywms.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetCapabilities

The map=mywms.map parameter is specific to MapServer, but the rest of the parameters are part of the WMS standards. If your browser keeps trying to download the server response, try sending an incorrect WMS version as explained in my previous post. The MapServer site has a live example of a WMS service you can use for comparison with your own.

Formatting the GetCapabilities Response

The server responds to a valid GetCapabilities by sending back XML. This XML can be formatted in the broswer to produce a report using an XSL file. My  sample page consists of the following components:

• GetCapabilities.xsl - an XSL file which transforms a GetCapabilities XML document into HTML
• GetCapabilities.js – a JavaScript file which can be used to apply the XSL to the XML file. The XML can either be a static saved XML file, or a URL that returns XML. For security reasons, modern browsers do not allow access across domains, so this must be stored on the WMS server. The JavasScript was taken from the W3 Schools site.
• GetCapabilities.css – a stylesheet that applies colours and layout to the HTML (with thanks to Verlees’s blog post).
• GetCapabilities.html – an HTML file that loads the JavaScript and displays the report.

The XSL stylesheet can also be applied directly to the XML by including the following lines at the top of the XML file:

<?xml version="1.0" encoding="ISO-8859-1" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="GetCapabilities.xsl"?>

However this is not an option if the XML is returned dynamically (apart from if you start modifying your WMS server code).

Future Improvements

The current XSL stylesheet is fairly basic, but does provide a quick way to list your layers, and to check that all the required WMS metadata items have been filled in. Possible additions to the XSL sheet include:

• highlighting required metdata parameters that are missing
• collapsing legend images, or allow a trigger to load them (for large legends, and to improve loading speeds)
• allowing a link to create a sample tile of the data, or adding this in automatically – it would be useful to check how features are rendered that services are running correctly
• sorting of table columns
• linking to metadata documents

Any thoughts and feedback appreciated.

GIS systems with organisational data are often only accessible (in theory) through an Intranet, so if the network is well protected then so isthe GIS.

There have only been a few mapping systems I’ve worked on where there is a mix of public data and datasets with restrictive access in the same database. The location of rare bird species (people will go to some lengths for a rare egg) and hallucinogenic mushrooms in a Biodiversity GIS being an example.

I’m currenty designing a MapServer system that will serve out WMS layers to the public, and a few which are limited to certain IP address ranges. WMS and security was something I had never previously investigated. The rest of this post outlines the current situation, and an attempt to restrict WMS access on my server.

WMS and the HTTP Protocol

WMS servers interact with their clients via the HTTP protocol, which uses plain text. For example a client (the user’s browser) would send a “GET” request (along with a number of parameters and headers) such as:

 GET mapserv.exe?map=/ms4w/apps/... HTTP/1.1
 Host: www.example.com

The server would then send a number of response headers as plain text and, for a valid WMS request, hopefully an image.

“The rationale behind this is that text protocols allows you to see what’s going on on the network by just dumping everything that goes through. You don’t need a specialized analyzer as you need for TCP/IP. This makes it easier to debug and easier to maintain.” – PolyThinker

Clearly anything sent between two computers in plain text is never going to be very secure. The OGC announced via a press release last August, that they would begin an “Authentication Interoperability Experiment” on the  2nd of October, 2009. This experiment “will test standard ways of transferring authentication information between OGC clients and OGC services using existing mechanisms such as HTTP Authentication, HTTP Cookies, SSL/X509, SAML, Shibboleth, OpenID and WS-Security.”

The experiments participants include the National Geospatial-Intelligence Agency, and the Carbon Project and its observers include ESRI. Their presence hopefully would mean any resulting decision would be implemented in their software, and lead to the widespread acceptance of a standard. The experiment is led by Jeff Harrison of www.cubewerx.com, who’s website states:

“Concerned about standards? So are we. In fact we literally wrote the book on many of them”

There don’t seem to have been any updates since then, but a good place to keep up with any developments in WMS is through the OGC WMS mailing list – a “discussion list for people implementing and deploying instances of the OGC Web Map Server Interface specification“, and the experiments web page.

Securing MapServer in Apache

In the meantime I was still looking for a way to limit access to my WMS services.  I’m using MapServer for Windows (ms4w) which installs MapServer and Apache on a Windows machine. Apache (and IIS) can be used to limit access to certain files based on an IP address which is exactly what I require.

Unfortunately MapServer runs through CGI which means limiting access to certain .MAP files was impossible. MapServer uses the SYSTEM account in Windows which means it has the same access rights no matter what IP address is used to start the process.

There are ways to run CGI processes as different users (suExec seems to be frequently mentioned), but examples on the web all seem to use Linux, and I’d imagine there would need to be a pre-processing script checking IPs to decide which user account should start the MapServer process.

In the end it seemed the simplest option was to have two instance of ms4w on two different servers (the advantages of several virtual servers on one physical server are becoming clearer and clearer). I could then limit access to the CGI program on the server with the more restricted Map files in the following way:

Open C:\ms4w\Apache\conf\httpd.conf and search for the following text:

#
# Configure MS4W locations and directories
#
<Location "/cgi-bin">
 Options None
 Order allow,deny
 Allow from all
</Location>

The edit these settings to limit the access to the CGI folder to IP ranges, addresses, domains etc. For further details see the Apache documentation and this blog post. In the example below only the computer with the IP 192.168.1.10 can access MapServer through CGI:

<Location "/cgi-bin">
 Options None
 Order allow,deny
 Allow from 192.168.1.10
</Location>

After every change you will need to restart the Apache service to activate your changes. The same settings can also be applied to FastCGI. To see which IPs are accessing your MapServer look in C:\ms4w\Apache\logs\access.log. If you have limited access then IPs without permission should receive a HTTP 403 error> These are also recorded in the access.log file.

Ideally I’d like to be able to set access to each MAP file or directory by IP range on the same MapServer, so if anyone has achieved this with ms4w please let me know!

According to the ModPython site – “Python will run many times faster than traditional CGI and will have access to advanced features such as ability to retain database connections.”

Mod_python is an Apache module that embeds the Python interpreter within the server, and can be downloaded from their site. There are versions for both Unix and Windows, which makes it ideal for MapServer for Windows (MS4W). Ironically after saying that using Python for web applications is many times faster than CGI, the ModPython download page uses CGI..

There are a number of Windows versions available, depending on your version of the Apache server. Make sure you get the right one – at the time of writing I am using MS4W v2.3.1, which includes Apache HTTP Server version 2.2.11. Your version of Python also makes a difference – I’m using 2.5.4 so I downloaded mod_python-3.3.1.win32-py2.5-Apache2.2.exe. My reason for not yet having upgraded to Python 2.6 or 3 was because all the TileCache documentation refers to Python 2.5.

The Mod_python installer allows you to choose your Python and Apache directories which in a standard MS4W installation would be C:\Python25 and C:\ms4w\Apache respectively.

For the rest of the configuration steps follow the instructions here. If you are using MapServer for Windows then a brief summary of these steps is as follows:

1. Open your C:\ms4w\Apache\conf\httpd.conf and add the lines below. A C:\ms4w\Apache\modules\mod_python.so file should have been created during the installation of Mod_python.

    LoadModule python_module modules/mod_python.so
    <Directory "C:/ms4w/Apache/htdocs/test">
        AddHandler mod_python .py
        PythonHandler mptest
        PythonDebug On
    </Directory>

2. Add the following code to a mptest.py file in a C:\ms4w\Apache\htdocs\test folder (check the spaces/tabs for the script are correct using the Python editor IDLE):

from mod_python import apache

def handler(req):
 req.content_type = 'text/plain'
 req.write("Hello World!")
 return apache.OK

3. Restart your Apache server service to apply the changes to the configuration. You can do this through the GUI or using:

C:\ms4w\Apache\bin\httpd.exe -k start -n "ApacheMS4WWebServer"

4. Go to http://localhost/test/mptest.py in your browser (assuming a default HTTP port of 80) and you should see the familiar “Hello World” message. If not..see the troubleshooting page. Note that with this configuration all links ending in .py will be handled with by the same mptest.py script. So if you are editing your Python script and wondering why nothing changes (..said from experience), check your httpd.conf.

You should also be able to see a summary of your system at http://localhost/mpinfo. Setting up TileServer is a blog post in itself and I hope to document that soon.

Finally, if in a fit of frustration with non-running Python scripts you uninstall everything mentioning the name Python from your machine, you are likely to find the error message below in your Apache log files:

[Mon Jan 18 12:28:02 2010] [error] make_obcallback: could not import mod_python.apache.\n
ImportError: No module named mod_python.apache
[Mon Jan 18 12:28:02 2010] [error] make_obcallback: Python path being used "['...
[Mon Jan 18 12:28:02 2010] [error] [client 172.30.99.54] python_handler: Can't get/create interpreter.

A reinstall of the Windows Set Up package should fix this.

Messing with Headers

I had the same problem with both Firefox and Internet Explorer 7 each time I issues a request such as:

http://localhost/cgi-bin/mapserv.exe?map=/ms4w/apps/my.map&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetCapabilities

The browser tried to download mapserv.exe. The “exe” did actually contain the GetCapabilities XML, but it is very annoying to have to manually open it in another program each time.

This is a FAQ on the MapServer site which suggested that a program needs to be manually associated with the MIME type returned by the requests – application/vnd.ogc.wms_xml.

In FireFox I went about associating an application with the MIME type application/vnd.ogc.wms_xml, following the steps outlined here. After a restart of FireFox nothing had changed.

I used FireBug to check that the request was returning the correct MIME type. As shown in the image below the server was correctly returning the correct Content-Type.

I then checked the request parameters sent from my browser to the server. These included:

Accept    text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language    en-US,en;q=0.7...
Accept-Encoding    gzip,deflate

I then thought the reason for trying to download the content was due to my browser not “accepting” the application/vnd.ogc.wms_xml type. I used another FireFox Add-In to modify my request headers to accept the content type (and “chunked” encoding just for good measure..).

The headers sent with my request now looked as follows:

Accept    application/vnd.ogc.wms_xml,text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language    en-US,en;q=0.7,chrome://global/locale/intl.properties;q=0.3
Accept-Encoding    chunked,gzip,deflate

A Hackish Solution

All good “fun” but I was still met with the same dialog box to download the content. FireBug still reported the response data as “The resource from this URL is not text.”

This post hints that the content type defined in the WMS specs was a mistake, and that for the new WMS specifications the content type will be set to text/html by default. It also suggested that adding a &EXECPTIONS=text/html parameter to the request string could override the content type, but this (I believe) is only where an exceptions actually occur s on the server.

I stumbled across a solution by accident. While entering a request string I put an incorrect version number in the VERSION parameter:

http://localhost/cgi-bin/mapserv.exe?map=/ms4w/apps/my.map&SERVICE=WMS&VERSION=1.0.3&REQUEST=GetCapabilities

The response came back in an text/xml format that was displayed in the browser! Whilst not very satisfactory, it was nicer than having to manually open up fake .exe files. If anyone can shed any light on why this is happening please let me know.

A couple of examples of the front-end interface can be found at the EU Inspire Geoportal, and the Marbled Murrelet Mashup. The Mapfish client combines three frameworks into one OpenLayers, GeoExt, and ExtJS.

The client JavaScript is available via downloaded from http://www.mapfish.org/downloads/. At the time of writing mapfish-client-1.2.tar.gz was the most recent version. The files can be unzipped using the open-source program 7-zip that handles both gzip and “tar ball” zip files. Once unzipped there are a number of samples that can be run and edited locally (look for the examples/map/index.html page). This alone is a huge step forward in creating a basic online mapping system that rivals proprietary offerings.

Further details on using the toolkit can be found at the Mapfish tutorial and this blog tutorial. I’ll also be writing some more technical posts, specifically around using Mapfish with Windows, as most of the current documentation assumes people are running Linux.

After turning on logging to diagnose an issue with MapServer I found a more worrying error, that was causing the log file to fill up at a rate of about 2MB a minute! I was reusing a legend for five or six datasets, and it was causing the following debug notes:

[Thu Jan 07 15:28:21 2010].629000 msyyparse():
Expression parser error. syntax error

[Thu Jan 07 15:28:21 2010].629000 msEvalExpression:
Expression parser error.
Failed to parse expression:  ( >= 2.251) AND ( <= 2.5)

Map Server Logging

To turn on logging in MapServer enter the following lines near the top of the .MAP file. The DEBUG value ranges from 0 (only actual errors are logged), to 5 which logs everything (diagnostics, errors, and debug notes). The second line requires the path to a logfile (any path and filename can be used).

DEBUG 5
CONFIG "MS_ERRORFILE" "C:/ms4w/apps/myapp/tmp/ms.log"

Problem Solved

I am using the SQL Server plug-in so initially I thought it could be related to this. I also tried changing the field name used in the expressions in case MEAN was a reserved word. Several sites also suggested the field may need to be in UPPERCASE, but I think this relates only to shapefiles.

MapServer was still generating full images, so data was being read and symbolised. I eventually found the following post which pointed to the problem..nulls. I should have spotted this sooner as the debug output was clear in retrospect. The expression parser was replacing my field name with empty values. So the expression EXPRESSION ( ([MEAN] >= 2.251) AND ([MEAN] <= 2.5) ) became ( >= 2.251) AND ( <= 2.5) wherever the [MEAN] field contained a NULL. All these features turned out to be slivers and could be deleted from the database, allowing my debug files to be readable again. If the features were required I would either have had to create a view of the table in SQL Server and point my LAYER to read from this, or replace the NULL values with a 0 if this was appropriate.

The post mentioned above suggests that in the case of shapefiles NULLS are automatically replaced with 0 for numeric fields, and by ” for strings. I think I’d prefer errors than automated assumptions about the data, but maybe a more obvious message in the log files would be useful.

As for the image – apparently the The Greek lowercase omega (ω) character is used to represent Null in database theory. You live and learn..

Update

Rather than delete null features from the database, you can ignore them completely when displaying your layer using the FILTER keyword – “this parameter allows for data specific attribute filtering that is done at the same time spatial filtering is done, but before any CLASS expressions are evaluated.”

For a SQL Server 2008 dataset to exclude nulls use a statement such as:

FILTER “[FieldName] IS NOT NULL”

I wanted to use SQL rather than FME, or a SQL Server wizard, as it would be easier to script the process for the several tables I had to copy.

The first task was to set up linked servers. I used the following SQL (when logged in to my destination server, and using SQL Server Management Studio) to add the source server:

EXEC sp_addLinkedServer 'MY_SOURCE_SERVER_NAME'

I then needed to allow log-in access to the source server. The SQL below assumes there is a sa user with a password of sa on the source server:

sp_addlinkedsrvlogin @rmtsrvname = 'MY_SOURCE_SERVER_NAME'
 ,  @useself =  'FALSE'
 ,  @locallogin =  null
 ,  @rmtuser =  'sa'  
 ,  @rmtpassword =  'sa'

I thought I was then ready to run a “SELECT INTO” query to copy records from the source table into a new table in the destination database. However I was met by the following error:

Msg 7325, Level 16, State 1, Line 1
Objects exposing columns with CLR types are not allowed in distributed queries.
Please use a pass-through query to access remote object '"MY_SOURCE_SERVER_NAME"."schema_name"."TableName"'.

It appeared that the geometry columns containing the SQL Server spatial types were causing the problem.  After reading up on pass-through queries  OPENQUERY seemed to be the solution.  I tried the following SQL:

SELECT * INTO newschema.MyNewTableName
FROM OPENQUERY([MY_SOURCE_SERVER_NAME],
'SELECT * from [source_database_name].[schema_name].[TableName]');

  

With MapServer 5.2.1  (installed using MapServer for Windows – MS4W v2.3.1) no error message was returned, however neither was any data . I initially thought this was due to the following bug in the plugin, which was reported to have been fixed for MapServer 5.4.

So I tried downloading MS4W v3.0 Beta 7, which included MapServer 5.4, and upgraded my installation. The WMS queries still failed but at least this time I got an error message:

msMSSQL2008LayerGetShape(): Query error. Error executing MSSQL2008 SQL statement:
SELECT convert(varchar(20), FID) from SEI_MyTable WHERE
GEOM.STIntersects(Geometry::STGeomFromText('POLYGON((343410.447904052 262585.540545139,
343410.447904052 262585.540545139,343410.447904052 262585.540545139,343410.447904052
262585.540545139,343410.447904052 262585.540545139))',0)) = 1

-[Microsoft][ODBC SQL Server Driver][SQL Server]A .NET Framework error occurred during execution of user-defined routine or aggregate "geometry":
System.FormatException: 24305: The Polygon input is not valid because the ring does not have enough distinct points. Each ring of a polygon must contain at least three distinct points.
System.FormatException:
 at Microsoft.SqlServer.Types.Validator.Execute(Transition transition)
 at Microsoft.SqlServer.Types.ForwardingGeoDataSink.EndFigure()
 at Microsoft.SqlServer.Types.OpenGisWktReader.ParseLineStringText()
 at Microsoft.SqlServer.Types.OpenGisWktReader.ParsePolygonText()
 at Microsoft.SqlServer.Types.OpenGisWktReader.ParseTaggedText(OpenGisType type)
 at Microsoft.SqlServer.Types.OpenGisWktReader.Read(OpenGisType type, Int32 srid)
 at Microsoft.SqlServer.Types.SqlGeometry.GeometryFromText(OpenGisType type, SqlChars text, Int32 srid)

It appears that no matter what values I passed into my bounding box parameter, a spatial query was made using a bounding box / polygon consisting of 4 identical points, hence the error from SQL Server. I tried a bounding box set to the full extent of my data but still no luck – the bbox values made no different at all.

http://localhost/cgi-bin/mapserv.exe?map=/ms4w/apps/sei/test.map&service=WMS&
version=1.1.1&request=GetFeatureInfo&layers=test&query_layers=test&styles=,
&bbox=0,0,500000,500000&srs=EPSG%3A29902&feature_count=10&x=710&y=132&height=253&
width=991&info_format=text%2Fhtml

This useful site automatically compiles the most recent MapServer versions into binaries available or download, however the most recent versions of the plug-in still fail.

I found a post that had a similar issue using CGI scripts rather than WMS calls. A suggestion was to use the TOLERANCE parameter for a layer, to increase the bounding box size, however this makes no difference to a WMS GetFeatureInfo query.

The MapServer 5.4 sourcecode for the plugin can be seen here, however it is in C..and I can’t seem to find where the rectangle object is passed in or created. It appears Tamas Szekeres has taken over updates to this plugin so I’ll try and contact him to make him aware of this issue. I’m not sure how many people actually use SQL Server 2008 and MapServer, but there is at least one other person!

Update (March 2010): It appears this issue has now been fixed for version 5.6 by Tamas Szekeres.

- Changed the query shape to POINT instead of POLYGON when the area of the query shape is zero (causing a failure at the server).

More details on the issue can be found in the MapServer-dev mailing list.

Geometry and Projections

Make valid geometry – this can often fix ‘invisible’ datasets in desktop GIS packages such as CadCorp.

UPDATE MyTable SET GeomFieldName = GeomFieldName.MakeValid()

Set the spatial reference for all features in a table (STSrid), this is the equivalent of using a .prj file with a shapefile. You could set a different reference for individual features, although why you’d want to do that I’ve no idea. Note that this does not reproject your data, but it is used to tell client applications what projection the data is in. In MapServer I found the SRID has to be set correctly for WMS layers to appear in OpenLayers.  Conversely in CadCorp’s GeognoSIS I had to set these values to 0 for data to display correctly. Change the code according to the projection of your data.

UPDATE MyTable SET GeomFieldName.STSrid = 3785 --change this value to your EPSG code

Feature Extents

The next SQL code gets the extents of each feature, and saves them into new fields. These fields are “calculated” so if the features are modified then the extents will be updated, yet they are also saved to disk rather than calculated dynamically for each query so they are as fast to display as other fields.

ALTER TABLE MyTable
ADD MinX AS (CONVERT(int,GeomFieldName.STEnvelope().STPointN((1)).STX,0)) PERSISTED
ALTER TABLE MyTable
ADD MinY AS (CONVERT(int,GeomFieldName.STEnvelope().STPointN((1)).STY,0)) PERSISTED
ALTER TABLE MyTable
ADD MaxX AS (CONVERT(int,GeomFieldName.STEnvelope().STPointN((3)).STX,0)) PERSISTED
ALTER TABLE MyTable
ADD MaxY AS (CONVERT(int,GeomFieldName.STEnvelope().STPointN((3)).STY,0)) PERSISTED

Then to get the full extent of your data you can use (although there are alternatives):

SELECT Min(MinX)AS MinX, Min(MinY) AS MinY,
MAX(MaxX) AS MaxX, MAX(MaxY) AS MaxY
FROM MyTable

Spatial Indexes

As mentioned in a previous post spatial indexes can greatly improve display and query speeds. To create an index in SQL use the following syntax. You should set the bounding box to the extent of your data, which can be calculated using the query above (and included in the SQL script).

CREATE SPATIAL INDEX MyIndexName ON MyTable(GeomFieldName) USING  GEOMETRY_GRID
WITH (
BOUNDING_BOX =(-1493907.5664457313, 6128509.51667404, -578861.3521250226, 7703103.135644257),
GRIDS =(LEVEL_1 = MEDIUM,LEVEL_2 = MEDIUM,LEVEL_3 = MEDIUM,LEVEL_4 = MEDIUM),
CELLS_PER_OBJECT = 16, PAD_INDEX  = OFF, SORT_IN_TEMPDB = OFF,
DROP_EXISTING = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON)

Spatial Queries

Sometimes SQL Server doesn’t seem to use spatial indexes automatically. You can force a spatial query to use an index using the WITH(INDEX(IndexName) statement similar to below which finds all geometry in a table at a point specified by an X and Y (in a Web Mercator projection – EPSG:3785)

SELECT *
FROM MyTable
WITH (INDEX (MyIndexName))
WHERE (geometry::Point(@x,@y,3785).STWithin(MyGeomField) = 1)

I have found that sometimes it is quicker when querying by points not to use a spatial index, so it is worth experimenting.

To find all features within a user defined bounding box you can create a stored procedure similar to the following:

CREATE PROCEDURE [SearchByBounds]
 @minx nvarchar(max),
 @miny nvarchar(max),
 @maxx nvarchar(max),
 @maxy nvarchar(max)
AS
BEGIN
 SET NOCOUNT ON;
 declare @g as geometry;
 declare @wkt as nvarchar(max);
 set @wkt = 'POLYGON ((' + @minx + ' ' + @miny + ',' +
 @minx + ' ' + @maxy + ',' +  @maxx + ' ' + @maxy + ',' + @maxx +
 ' ' + @miny + ',' + @minx + ' ' + @miny +  '))';
 set @g = geometry::STGeomFromText(@wkt, 3785);
 --print @wkt
 SELECT *
 FROM MyTable WITH (INDEX (MySpatialIndex))
 WHERE GeomFieldName.STIntersects(@g) = 1
END

I haven’t found too many tutorials or blogs on the spatial features in SQL Server 2008. The best resources I’ve found are:

• Jason Follas’s Introduction to SQL Server Spatial
• Issac’s posts on Spatial Indexing
• Bob Beauchemin’s SQL Server Spatial Blog
• Alastair Aitchison’s book Beginning Spatial with SQL Server 2008
• MSDN’s SQL Server Spatial Forums

Feel free to post more links below.

I’ve found a number of tweaks that have dramatically improved the WMS creation speed. In these examples I’m using a table in SQL Server 2008 with 50773 features (Irish Townlands if you’re asking).

To test speeds I use the program shp2img program, which can be found in the C:\ms4w\tools\mapserv folder. This is useful for not only checking if there are errors in your MAP file, but also for checking the speeds of individual layers. I use STATUS OFF for all layers, and then pass in the layer name with the following command (in the MS4W-Shell – a DOS window with environment variables set to various MapServer folders.

shp2img -all_debug 5 -m C:\ms4w\apps\sei\test.map -o -l "Townlands" test.png

Reproject Your Data in the Database

A nice feature of MapServer is that it can reproject your data on-the-fly. However this comes at a performance cost. The original data is projected in the Irish National Grid, but the web viewer is based around OpenLayers and background mapping from Google, which is projected in EPSG:3785 (or EPSG:900913 / EPSG:google – I prefer the vendor free projection myself).

The “web Mercator” projection can be added to MapServer by pasting the following lines into the C:\ms4w\proj\nad\epsg file.

## Google Projection
<900913> +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs
## Google Projection - new code
<3785> +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs

FME was then used to reproject the geometry in SQL Server. A nice feature of using the database as opposed to shapefiles is that two or more geometry fields can be stored in the same table, so different projections can be used for different WMS services or applications. In my MAP file my layer definition is now as follows:

 CONNECTIONTYPE PLUGIN
 PLUGIN "C:\ms4w\Apache\specialplugins\msplugin_mssql2008.dll"
 CONNECTION "server=****;uid=****;pwd=****;database=****;Integrated Security=false"
 DATA "GEOM3785 from Townland"

Reprojected time to create layer: 46.06 seconds

Unprojected time to create layer (all features): 13.469 seconds

Use Fast CGI

Fast CGI is a protocol for keeping cgi-bin style web applications running as a daemon to take advantage of preserving memory caches, and amortizing other high start-up costs (like heavy database connections) over many requests.

Fast CGI can be set up by following the instructions in C:\ms4w\README_INSTALL.html#f-fastcgi

I don’t think differences can be measure using shp2img, so I tested using FireBug’s Net Tab. The following line was added in to the LAYER definition in the MAP file:

PROCESSING "CLOSE_CONNECTION=DEFER"

Without FastCGI: 9.23 seconds

With FastCGI: 8.46 seconds

How accurate these results are I’m not really sure. I guess that improvements in speed will become more noticeable when many layers are loaded in that all use the same database connection at the same time. However even on a single layer the FastCGI was almost a second quicker on every test run.

Use Spatial Indexes

Using a spatial index when querying all layers would have no effect, so to see what difference it made on speed I needed to limit the results to a selected area. I decided to select all Townlands which intersected the county of Carlow. This  returns 1708 records. The following command was used to simulate this in shp2img:

shp2img -all_debug 5 -m C:\ms4w\apps\sei\wind.map -o -e -791171 6884314 -724041 6967856 -l "Townlands" test.png

To select data using a spatial index the following line is used in the MAP file:

DATA "GEOM3785 from Townland WITH(INDEX(Townland3785_sidx)) USING UNIQUE FID USING SRID=3785"

I initially believed that spatially indexes had to be specified explicitly in the MAP file, and to test speeds without an index you could just remove the (WITH(INDEX…) statement. However speeds remained very similar. Disabling the spatial index in SQL Server (an option when right clicking on the index) revealed very different speeds. To re-enable the index, simply rebuild it. I’d leave the (WITH(INDEX…) statement in just to make sure it is used.

Not using a spatial index: 3.806 seconds

Using a spatial index: 0.875 seconds

Any feedback, or more tips much appreciated! There is also an official page on optimisation here. Happy MapServing..

Whilst this is ok for smaller projects where I could make these myself, larger projects require non-programmers to gather data, symbolise it, and handle the map layouts and creation.

I’ve just discovered that another of the OSGeo‘s projects, Quantum GIS (often abbreviated to QGIS) has a tool that can take a map project created and symbolised in the desktop application, and export it to a MAPFILE which can be read by MapServer. This seems to be the missing link in a complete Open Source GIS stack.

I had a few problems finding where this tool was installed. By default the Python files should be in the C:\OSGeo4W\apps\qgis\python\plugins\mapserver_export directory, however this directory did not exist after running through a default installation without paying attention.

OSGeo have a custom installer that you can use to install all their software including MapServer, QuantumGIS, and GRASS. To install Quantum GIS follow the notes on this wiki page. Clearly you need to select QGIS, but the MapServer Export tool also requires python-qgis. I think I missed this the first time round, however you can always rerun the installer to add new packages and libraries.

The OSGeo Installer

Once you have the Python for QGIS library installed open QGIS, and go to Plugins >> Fetch Python Plugins. The MapServer Export plug-in should be installed. Now go to Plug-Ins >> Manage Plugins and check the MapServer Export box as below.

You should now see a new MapServer Export menu item on the Plugins menu.

Further Details

There is a good introduction to using QGIS at http://spatialhorizons.com/category/qgis/

You can also install a standalone version of MapServer Export – http://spatialserver.net/qgis2ms/.

The system documentation is normally both detailed and clear, and FME also has very good support and community web sites.  However in this case it took a lot of documentation reading and trial and error to work out how to change the ouput filename of a model. This may be obvious to experienced users of FME, but it had me stumped for a while.

The steps involved are simple in retrospect, and are as follows:

Create your raster model as normal, but add an additional AttributeSetter transformer, as shown in the screenshot above.

Next create a new Published Parameter that will be used to pass in the output raster filename. The source dataset and destination folder can also be set to be published parameters to provide full flexibility.

Next right click on the newly created published parameter and check the AttributeSetter value item. This creates the link between the parameter and the value for the attribute.

If you look at the AttributeSetter’s properties they should now look similar to below.

Finally look at the properties for your output raster dataset. Make sure the “Fanout By Attribute” is checked and the attribute is set to fme_basename. This sets the output filename to be the same as the fme_basename attribute value, which is now set via an input parameter.

The model is now ready to run from the command line or through a batch process, for one or many raster transformations. The output filename parameter does not require a .tif extension.

    fme.exe raster_sample.fmw
          --SourceDataset_ARCVIEWGRID C:\Grids
          --DestDataset_GEOTIFF C:\Rasters
          --OutputFileName MyOutputRaster

GIS consultants and organisations that use GIS have different needs. Consultants tend to use lots of different software, for short amounts of time, whereas organisations make daily use of software from a single vendor. ESRI have followed Microsoft’s approach of a Developers’ Network (MSDN). This allows the whole ESRI stack to be used for “development, testing, and demonstration” at a reduced price. $1500 dollars gets you an ESRI Developer Network subscription, rising to $2000 if you also want the desktop applications (ArcMap, ArcCatalog etc.).

Getting as many GIS consultants trained in developing for ESRI products will result in more ESRI products being used by the consultant’s clients. However as ESRI also provide consultancy services independent consultants are often direct competitors. This can mean getting a EDN license may not be as simple as signing up on the website.

Smaller players in the GIS market may see a consultancy as a chance to sell a few licenses, especially if the consultants have to develop for top-end products such as web servers. While the client often pays for this license during development, once the system is deployed there is no easy way to provide maintenance on it without continuing to pay for a separate license. I see this as very short-sighted in the long term – the barriers to trying out new software for developers should be removed, rather than added.

All this is against a backdrop of web mapping (aka the geoweb) APIs from Google and Microsoft which are provided for free, along with fast, free, background mapping. The introduction of spatial functions into SQL Server 2008 Express has meant that consultants with MSDN development can be carried out at no cost.

They are very useful if a company is about to invest in a GIS and are unsure what software to use. Point their employees to the following links, create a few datasets relevant to their organisation, and let them get a feel for each and find out their preferences. Clearly it will be the management who will make the investment so there will be a trade-off between cost and how much their employees like a particular product.

Of course you could always just recommend whichever one you feel most comfortable developing in..

“GeoMedia Viewer is an easy to use, FREE GIS software application for desktop viewing and distribution of geospatial data”

https://support.intergraph.com/Product/GeoMediaViewer.asp

“ArcExplorer is a freely available lightweight GIS data viewer that lets you perform a variety of basic GIS functions. With ArcExplorer, you can display, query, and retrieve data. It can be used as a stand-alone application with local data sets or as a client for Internet data and map servers.”

http://www.esri.com/software/arcexplorer/

“Map Reader provides users of Cadcorp SIS with a powerful free tool to effortlessly view, query and print map data. The strength of Map Reader lies in its ability to view over 160 formats..” (requires registration)

http://www.cadcorp.com/products_geographical_information_systems/map_reader.htm

1. VBA is great for prototyping. There is no need to open up Visual Studio, create a custom tool, implement all its interfaces and constructors etc. before you can even get round to writing the code for the OnClick event of a button.

2. The code is saved in the MXD. Prototypes can be easily zipped up and sent to a client for testing without the need to run set up packages, use the add/remove programs tool, and change config files. This allows projects to be developed using an agile approach.

3. It is the easiest entry point for GIS professionals into programming. People with no programming background can use an application they are already familiar with, cut and paste a few samples from the help file and they are away.It is also an entry point into the weird, wonderful, and worringly huge world of ArcObjects.

4. It is all well and good to use Python for geoprocessing, but many tasks require ArcMap’s user interface, and the ability to visualise the data – and maybe even interact with it..!

5. There is a wealth of samples, tools, and scripts available on the web that will now be useless, or require conversion. No more copying, pasting, and running to get a task done quickly. VBA for many people is part of the daily use of ArcMap – basic subs can be bashed out in a few minutes to complete a one-off task and then forgotten about.

6. VBA skills can be transferred to other applications such as Excel, Word, and Access. For many GIS users these are the only other programs they use for daily tasks, so being able to automate them, or modify a recorded VBA macro can greatly increase productivity.

7. It is another nail in the coffin of the VB.Net versus C# argument – if no-one has used VBA then there is no reason to recommend coding in VB.Net due to familiar syntax..however that is just a personal view in an ongoing struggle to avoid having to write all future projects in C# to fit with “company standards.”

In my opinion the ideal solution would be to continue to allow code to be written and saved in the MXD, but replace VBA/VB6 with VB.Net. Why not even go further and allow macros to be recorded in ArcMap?

Still at least we’ll all now be safe from those VBA viruses going around in MXD files..

Update

One item I missed in the original post is that VBA is used for field calculations in ArcMap. I’m sure most GIS users have a few complex operations saved somewhere, all of which look to become redundant. It will be interesting to see whether Python will take its place.

In terms of development the main issue is that there appear to be very few people developing tools around CadCorp’s products apart from CadCorp themselves. This means that the only code samples are in the help system installed with the tool, and a PDF file. Straying outside these samples I found involves a lot of trial and error. There is a forum on the CadCorp site – annoyingly registration is required to read the posts, and there are very few programming related posts. The CadCorp staff do however get back to you on any questions you pose.

Visual Studio and .NET languages can be used to create custom DLLs that integrate with the application. Prior to version 7 (the latest CadCorp release) custom code was run in a separate memory process, and Windows messaging used to pass messages between the custom tool and the CadCorp application. Fortunately this has changed so the DLL can run in the same memory space. The code below shows how to add two new menu items using VB.Net.

Public Sub AddMenus()
  Dim mainMenu As New SisMenuItem("Custom Tool")
  Dim menuB As New SisMenuItem("B", New SisClickHandler(AddressOf b))
  mainMenu.MenuItems.Add(menuB)
  Dim menuB As New SisMenuItem("A", New SisClickHandler(AddressOf a))
  mainMenu.MenuItems.Add(menuA)
End Sub

Strangely the two menu items would be listed on a new menu item alphabetically. Apparently this is by design, and hidden ASCII characters can be used to order the menus..

Each custom menu item has an associated sub procedure that is called when clicked. The example above expects two subs a and b. Each would look similar to:

Private Sub a(ByVal sender As Object, ByVal args As SisClickArgs)
  Dim mm As MapManager = args.MapManager
End Sub

The reference to the CadCorp application is passed in as an argument by default. One of my main gripes with development with CadCorp is that the programming model appears to be a based partly on objects, and partly on passing arcane strings into a huge “application” object. As these are strings they are not picked up be intellisense in Visual Studio and have to be found in the manual. I’m not sure why this is the case, as a simple wrapper could be created to avoid this. Maybe it is in development..

To illustrate my point here is a function to loop through all the layers (aka overlays), and find a layer with a particular name:

Public Function FindOverlayByName(ByVal mm As MapManager, ByVal sLayerName As String) As Integer
 Dim lNumOverlays As Integer = mm.GetInt(SIS_OT_WINDOW, 0, "_nOverlay&")
 Dim iOverlayPos As Integer = -1
 'loop through all overlays checking if the name matches
 For lCount = 0 To lNumOverlays - 1
   Dim sName As String = mm.GetStr(SIS_OT_OVERLAY, lCount, "_name$")
   If sName = sLayerName Then
     iOverlayPos = lCount
     Return iOverlayPos
   End If
 Next lCount
 Return iOverlayPos
End Function

The line to notice is mm.GetStr(SIS_OT_OVERLAY, lCount, “_name$”). There are many cases of passing and setting properties using the GetStr / SetStr approach, each one with a different third parameter.

Collections of objects have to be stored in lists, and can only be referred to by a string name. This makes code inflexible and hard to maintain. For example the code below finds the extent of all features in an overlay:

mm.CreateListFromOverlay(iLayerPos, "AllItems")
If mm.GetListSize("AllItems") > 0 Then
 myExtent = mm.GetListExtent("AllItems")
 mm.SplitExtent(x1, y1, z1, x2, y2, z2, myExtent)
End If

The “AllItems” string is used to keep track of all the features in an overlay. Any string can be used (I chose “AllItems” for readability, and this could be saved in a variable, but it still seems starnge to the object-orientated world of .Net.

The tool is now installed and being used by ~15 users, and has a backend Oracle geodatabase, and seems to be going well. Next I’ll be trying out CadCorp’s web server GeognoSIS.

I wanted to generate a nice diagram of a SQL Server database I had created, and automatically create hyperlinks that would pop up an Ajax box with sample records directly from the database. In order to do this each hyperlink had to have a variable based on the table/entity name. However it soon became apparent that there is no easy way to access the entity name in VBA from its associated shape object. The solution to this is the VisioModelingEngine library which provides access to the UML, but does not provide access to the shapes.

There appears to be very little documentation on using the classes in this library. I found a couple of examples at http://www.eggheadcafe.com/forumarchives/visiodeveloperVisualBasica/Jul2005/post23423796.asp and http://www.eggheadcafe.com/software/aspnet/32121825/forward-engineer-visio-er.aspx

My eventual workaround involves temporarily changing the first attribute in an entity to the entity name, searching through the shape’s text for this name, recording the ID and then changing it back. The idea was based on this post which hinted that was what they did, but then frustratingly didn’t post their solution.

Anyway the main use of the code is to associate a Visio entity (IVMEEntity) with its shape.

The full VBA code can be seen by clicking this link.

A good source of VBA scripts for Visio can be found at http://visualsignals.typepad.co.uk/vislog/ this provided me with the code for looping through shapes on a page.

Public Function GetFeatureLayers(ByRef pMap As IMap) As IEnumLayer

Dim pUID As New UIDClass
pUID.Value = "{E156D7E5-22AF-11D3-9F99-00C04F6BC78E}" 'IGeoFeatureLayer GUID

If pMap.LayerCount > 0 Then
GetFeatureLayers = pMap.Layers(pUID, True)
Else
Return Nothing
End If

End Function

Public Function GetFeatureLayerByAliasName(ByRef pMap As IMap, _
ByVal strLayerModelName As String) As IFeatureLayer

Dim pLayerEnum As IEnumLayer
Dim pFLayer As IFeatureLayer

pLayerEnum = GetFeatureLayers(pMap)
If pLayerEnum Is Nothing Then
Return Nothing
End If

pFLayer = DirectCast(pLayerEnum.Next, IFeatureLayer)

Do Until pFLayer Is Nothing
If pFLayer.Valid Then
If strLayerModelName = pFLayer.FeatureClass.AliasName Then
Return pFLayer
End If
End If
pFLayer = DirectCast(pLayerEnum.Next, IFeatureLayer)
Loop

'the layer was not found
Return Nothing

End Function

Rather than using the layer name visible in the table of contents which is easly changed by a user, I use the .AliasName of the feature class. This is automatically set to the name of the layer when added to a map. A user is much less likely to change this. There is however another name that is 100% certain not to change, it does however require an additional step when adding the layer shown below:

Dim pModelInfo As IModelInfo
pModelInfo = CType(pFLayer.FeatureClass, IModelInfo)
pModelInfo.ModelName = “AnyValueYouLike”

The code to find this layer is similar to the FindFeatureLayer, but relies on the IModelInfo interface. As stated in the documentation - “Object classes and fields in a geodatabase can have between one and three names..The third name is the model name which is a tool for developers of custom objects to use to guarantee the names of objects independent of the true name or alias name.”

This property could be used in conjunction with a GUID to ensure you return the layer you are looking for.

Public Function GetFeatureLayerByModelName(ByRef pMap As IMap, _
ByVal strLayerModelName As String) As IFeatureLayer

Dim pLayerEnum As IEnumLayer
Dim pFLayer As IFeatureLayer
Dim pModelInfo As IModelInfo

pLayerEnum = GetFeatureLayers(pMap)
If pLayerEnum Is Nothing Then
Return Nothing
End If

pFLayer = DirectCast(pLayerEnum.Next, IFeatureLayer)

Do Until pFLayer Is Nothing
If pFLayer.Valid Then
pModelInfo = DirectCast(pFLayer.FeatureClass, IModelInfo)
If strLayerModelName = pModelInfo.ModelName Then
Return pFLayer
End If
End If
pFLayer = DirectCast(pLayerEnum.Next, IFeatureLayer)
Loop

'the layer was not found
Return Nothing
End Function

However it seems that line features can only be rendered as solid lines (no dots or dashes)..a very strange oversight, especially when it appears the outlines of polygons can have linear styling!

I’m going to be installing version 6 soon to see if it has been resolved, and see if there are any new features.

To unhide just change the Trues to False in the code snippet below, or add a parameter to the sub.

Public Sub HideGeoDatabaseTables()

Dim strTableName As String For i = 0 To CurrentDb.TableDefs.Count - 1
If CurrentDb.TableDefs(i).Name Like "GDB_*" Then
strTableName = CurrentDb.TableDefs(i).Name
Access.SetHiddenAttribute acTable, strTableName, True
End If
NextAccess.SetHiddenAttribute acTable, "SelectedObjects", True
Access.SetHiddenAttribute acTable, "Selections", True

End Sub

To view hidden objects in Microsoft Access, go to Options and check the “Hidden Objects” box.

Access, lock files, and Visual Source Safe – an accident waiting to happen! I got the above error, after my code had been fine for weeks. Googling turned up permissions on the folder with the Access database, and closing secondary applications. The solution in the end turned out to be that the .ldb file had come under source control. Removing it from this, and deleting the fle resolved the issue.

It does the majority of the conversion well but I ran into a few minor problems where line breaks were not taken into account.

This tool will be even more useful when the ESRI samples, the majority of which are only available in VB6, are updated to .NET…

However far more amusingly is the generator at http://www.malevole.com/mv/misc/text/. There are some real classics in there…

“Mutley, you snickering, floppy eared hound. When courage is needed, you’re never around. Those medals you wear on your moth-eaten chest should be there for bungling at which you are best. So, stop that pigeon, stop that pigeon, stop that pigeon, stop that pigeon, stop that pigeon, stop that pigeon, stop that pigeon. Howwww! Nab him, jab him, tab him, grab him, stop that pigeon now.”

In my case it has always been related to SQL queries and Access databases. Check the following if your SQL command produces this error when executed:

1. The database is not read only – Visual Source Safe can often be a culprit in setting this property..
2. The SQL query may contain fields that are reserve words (e.g. DATE, TOP etc.). These should all be renamed in the database (I normally add an underscore at the end).
3. Go through a long and tedious process of which Windows user is running the SQL (ASPNET, IUSR_COMPUTERNAME etc.) and check thy have write permissions to the folder containing the database, and the database file itself. Remember Access needs to create those pesky .ldb lock files..

A full archive of previous episodes is available for download (for free), and I found an epsiode that gives a good introduction to Python – it can be downloaded here. I’m still not convinced of the use of Python and ArcObjects as explaining how to run scripts with parameters to GIS users, or creating custom toolboxes, takes as much time as creating a nice UI in Visual Studio. In the show one of the presenters asks “but don’t you miss the Visual Studio environment and intellisense?”, which I couldn’t agree more with. However there is also mention of IronPython, which I knew had something to do with .NET, but it can also apparently be used within the Visual Studio environment which could make life a whole lot easier.

I’m off to download IronPython now…

Writing ArcObjects code and using an Access personal geodatabase will result sooner or later in an error along the lines of “Cannot aquire a schema lock because of an existing lock.” Spending hours going through code, making sure objects are disposed of and that all connections closed often makes very little difference. It becomes apparent some of the problems lie deeper than customised code when the same errors occur when using ArcMap or ArcCatalog without any modifications.

I have been using Unlocker, a freeware program, to solve these issues when developing with ArcObjects (or just using ArcMap..). Once installed a simple right click on an ldb or mdb file can remove any file locks, and your code is free to run again. It works with any Windows file or folder so is a useful program to have around even without geodatabases.

More reviews and an alternative download site can be found here.

I know I should sit down and read all the basics of a language first before using it, but where is the fun in that? I had written my main script, which called a couple of functions declared in the same .py file. I kept running into the following error:

“NameError: name ‘getMyFileTimes’ is not defined”

After looking a few times at how to declare functions, I realised eventually that they have to be declared before calling them. E.g.

def hello (what):
     text = “Hello, ” + what + “!”
     print text
 
hello (“World”)

I guess I am still getting used to writing in a scripting language in what amounts to little more than notepad, and it makes sense that if it is not compiled and read line by line then the function would not be “seen.” Still I miss my autosensing, and Visual Studio debugging tools a lot…

Turns out the problem was with one of the field names being “Size”, once renamed everything ran without problems. I guess this could apply to all reserved words.

The code below should copied and pasted into the VBA Editor in ArcMap, and run when the current view is a page layout. The number of decimal places and distances of the text labels from the grid can be changed easily by modifying the relevant variable.

    1 Public Sub AddCornerText()

    2 

    3 Dim pMxDoc As IMxDocument

    4 Dim pActiveView As IActiveView

    5 Dim pLayoutExtent As IEnvelope

    6 Dim strText As String

    7 Dim pMapFrame As IFrameElement

    8 Dim pGraphic As IElement

    9 Dim dblOffset As Double

   10 Dim lngRound As Long

   11 

   12 ‘this is the offset used so the text does not overlap the map

   13 ‘modify according to needs

   14 dblOffset = 0.2

   15 

   16 ‘this is the number of decimal places used for each coordinate value

   17 lngRound = 0

   18 

   19 Set pMxDoc = ThisDocument

   20 Set pActiveView = pMxDoc.FocusMap

   21 

   22 ‘find the map data frame in the layout

   23 

   24 Set pMapFrame = pMxDoc.ActiveView.GraphicsContainer.FindFrame(pMxDoc.ActiveView.FocusMap)

   25 Set pGraphic = pMapFrame

   26 Set pLayoutExtent = pGraphic.Geometry.Envelope

   27 

   28 ‘create the bottom left point and text

   29 strText = Round(pActiveView.Extent.XMin, lngRound) & _

   30 “:” & Round(pActiveView.Extent.YMin, lngRound)

   31 AddPoint pLayoutExtent.XMin – dblOffset, pLayoutExtent.YMin – dblOffset, strText

   32 

   33 ‘create the top left point and text

   34 strText = Round(pActiveView.Extent.XMin, lngRound) & _

   35 “:” & Round(pActiveView.Extent.YMax, lngRound)

   36 AddPoint pLayoutExtent.XMin – dblOffset, pLayoutExtent.YMax + dblOffset, strText

   37 

   38 ‘create the bottom right point and text

   39 strText = Round(pActiveView.Extent.XMax, lngRound) & _

   40 “:” & Round(pActiveView.Extent.YMin, lngRound)

   41 AddPoint pLayoutExtent.XMax + dblOffset, pLayoutExtent.YMin – dblOffset, strText

   42 

   43 ‘create the top right point and text

   44 strText = Round(pActiveView.Extent.XMax, lngRound) & _

   45 “:” & Round(pActiveView.Extent.YMax, lngRound)

   46 AddPoint pLayoutExtent.XMax + dblOffset, pLayoutExtent.YMax + dblOffset, strText

   47 

   48 

   49 pMxDoc.ActiveView.Refresh

   50 

   51 End Sub

   52 

   53 Private Sub AddPoint(dblX As Double, dblY As Double, strText As String)

   54 

   55 Dim pMxDoc As IMxDocument

   56 Dim pPoint As IPoint

   57 Dim pTextElement As ITextElement

   58 Dim pPageLayout As IPageLayout

   59 Dim pElement As IElement

   60 Dim pGContainer As IGraphicsContainer

   61 

   62 Set pMxDoc = ThisDocument

   63 

   64 ‘create the location for the text

   65 

   66 Set pPoint = New Point

   67 pPoint.X = dblX

   68 pPoint.Y = dblY

   69 

   70 ‘create the text element

   71 

   72 Set pTextElement = New TextElement

   73 pTextElement.Text = strText

   74 

   75 ‘add the element to the layout

   76 

   77 Set pPageLayout = pMxDoc.PageLayout

   78 Set pElement = pTextElement

   79 pElement.Geometry = pPoint

   80 Set pGContainer = pPageLayout

   81 pGContainer.AddElement pElement, 0

   82