GIS Obsessed

Be Transformative!

2014-04-29T13:09:00.000-04:00

I can think of few topics that are more uninteresting on the surface than that of the importance of using the proper (or any!) geographic transformation when using GIS data layers that are in various coordinate systems. Regardless of whether this sounds like a snoozer of a topic, it is actually quite important and it has been a hot topic for the past couple days on the Northeast Arc User Group's Listserv. This post is my relatively non-technical attempt to share some insight into just what happens when we co-mingle data from different types of coordinate systems, without using the proper geographic transformation. In a couple words, please don't forget to Be Transformative!

The example below illustrates what can happen when a layer in a Geographic Coordinate System is displayed with a layer in a Projected Coordinate System, without applying a Geographic Transformation:

The difference is subtle & requires a click to view the image at full scale: Property Boundaries on the left side, in which a transformation is not set, are offset by approximately 3 feet. Meanwhile, the boundaries on the right side align perfectly because a geographic transformation is set. In this case, I'm displaying data in a Projected Coordinate System (MA Mainland Stateplane NAD83 in Feet) with data in a Geographic Coordinate System (WGS84, aka "straight lat/long"). The results of this exercise can vary greatly, although the 3 foot offset is relatively typical (perhaps that's why so few people tend to even notice?).

The transformation that is being set (or not set) is part of the "Project On The Fly" settings in ArcGIS for Desktop. I can't even tell you how wonderful it is that Projection on the fly is something that we can do so easily. It has really become a necessity in the age of web mapping. Unfortunately, it is so easy that the important detail of setting a transformation is very often missed. Also, the default setting for projecting on the fly in an ArcGIS for Desktop data frame is to use no transformation at all! Not good.

Fortunately, it is very easy to set a transformation. The image below shows the Transformations dialog box within the Coordinate System settings for a Data Frame. As you can see, the default of is selected. At version 10.2 (and perhaps as far back as 10.0?), the transformation choices are sorted by suitability based upon the map extent, which makes it quite easy. I usually just choose the top choice. Simple, right?

I'll be honest: without going into the detail of this post, complete with visual examples, I usually tell people to simply Project their data, producing a new copy in the proper coordinate system. I do this because, at least in ArcGIS for Desktop, projecting data requires the user to choose a transformation.

Ok, so don't forget to Be Transformative!

My ArcGIS Online Organizational Account Adventure

2013-10-17T10:37:00.000-04:00

Yesterday was not a good day for our ArcGIS Online organizational account. In a sequence that was documented in another GIS Obsessed post, I complained about Esri's glitch & their slow response time in resolving it. As explained to me by our account rep, our organization's annual maintenance contract with Esri was renewed in July, but our ArcGIS Online renewal, which was covered by the maintenance renewal, was omitted. For reasons that have not yet been explained, it took a couple months before the plug was pulled on our ArcGIS Online organizational account. This "outage" came on without any warning, which was quite unsettling since we could have avoided the whole issue in the first place if a warning email was sent (yup, I did check through the spam archives & nothing was there from Esri).

The initial call was placed to Esri at 9:30am & the organizational account was finally restored at 3:30pm, after what was described as a "significant amount of effort" on Esri's part. Fortunately, all settings, users, web maps, feature services, etc., remained intact when service was restored. During the time that the organizational account was not active, organizational users were unable to access web maps, feature services were unavailable and web maps embedded on our website contained nothing except for basemaps. It was eye opening to see which pieces of our enterprise were completely inoperable because of this glitch.

Fortunately, today is another day and we move on. We learned a lot. I hope that Esri did as well.

That's no way to run a cloud service!

2013-10-16T11:13:00.000-04:00

My organization pays boatloads of $$ to Esri and on most days we receive excellent service. Today, because of an apparent glitch, we are not receiving excellent service. For reasons unknown, our ArcGIS Online subscription expired overnight, without any warning whatsoever. An account rep acknowledged that we were current on our maintenance payments and that "it looks like we forgot to renew your ArcGIS Online subscription when we received your maintenance payment." Our subscription should be restored "within a couple hours."

Ok, mistakes do happen. I can accept that. What I cannot accept is that our subscription went away without warning. Is this how Esri is administering all ArcGIS Online accounts? I hope not. In any case, this is no way to run a cloud service that is being marketed as a foundational service for organizations of all sizes.

Provide Better Service with a custom Error 404 Page

2013-02-28T16:25:00.001-05:00

This post isn't specific to GIS or mapping, but I hope that you will find it to be of use. At work, I manage a site the includes thousands of documents and dozens of pages. In a perfect world, the system that I manage would never generate a dead link to any of these pieces of content. The reality is that dead links are bound to happen, whether through my own negligence, loss of data, outdated external links to our site, or any number of ways that people end up with a dead-end "Error 404" page. Why should it be a dead-end? Everyone is better off when a dead-end can be replaced with a useful way to resolve the issue.

Instead of a dead-end 404 page that most people cannot comprehend, I wanted to provide people with the simplest possible mechanism to report any missing content on our site. I was able to accomplish that via a couple snippets of relatively simple Javascript; one snippet takes the errant address and displays it as a text string on the page, while the other captures the errant address and adds it as a text string to the content of a mailto link. My resulting custom error 404 page looks like this:

The "contact us" mailto link generates an email with a pre-populated subject and message body, with the message body containing the all-important url of the missing content.

Please refer to the links above and the source code of our error page referenced above to see how it all works. One important detail: the custom error 404 page on my web server is set up using a responseMode="ExecuteURL" setting, which retains the errant url as the address, while displaying the custom error page.

So far, in just one day, I've already received the first email from someone who hit some missing content on our site. I was able to restore the content and notify the person that thanks to their willingness to report it, the missing content has been restored. That's just better customer service, thanks to just a little bit of Javascript.

Bye-bye, cryptic message...

Something Amiss on Google Maps?

2012-05-08T14:33:00.002-04:00

One of the regional GIS listservs to which I subscribe has gone wild today with posts about some prominent road name labeling errors that are showing up in Google Maps. It looks like somebody at Google flipped a switch on a map cache update before it was ready to go. Here are some examples:

Portsmouth, Maine: apparently, W City Ave doesn't even exist in Portsmouth

Springfield, MA now contains streets in Nebraska & Illinois!

Hartford, CT is now the east coast conduit to California

This is all amusing & in my opinion, is not worth worrying about. Sure, you can report a problem in Google Maps & if you want to take the time to do so, go right ahead. Based upon the apparent extent of this particular issue, I would expect that these odd street name errors will disappear just as quickly & mysteriously as they appeared. Give it a couple days & big G will make it all tidy again.

If you want more control over a free hosted cached map, might I suggest Open Street Map instead?

Selecting Features Using URL Parameters

2012-01-23T11:34:00.002-05:00

Much of what I want to do on the web with maps is simple: pass some url parameters to a web map that can provide mapped results based upon these parameters. Simple. Easy. Right? We see it all the time on Google maps, with the search text, map type, lat and long right there in the url. Those are url paramters that Google Maps can process to give us the results that we need right on the map. I want the same thing from my web GIS resources and I want it via simple Javascript, without the need for any plugins. Simple. Easy. Right?

Esri is doing a nice job of building up a good solid catalog of code samples for their ArcGIS API for Javascript, which is my preferred environment for developing web maps. It is quite amazing what can be done these days with this API, which makes it all the more perplexing that the simple and often requested capability to select features based on url parameters is not among the current set of code samples at version 2.6 of the API.

I don't consider myself to be much more than a code hack when it comes to developing web maps with Javascript, but I decided to develop my own code sample anyhow. I've posted it to ArcGIS.com and I'm definitely interested in hearing any feedback or suggestions. The sample (direct link here) uses Map and Lot parameters to select/zoom to a parcel while showing some basic information about it. Clicking another parcel on the map revises the url parameters via the relatively new HTML5 history.pushState() method.

Some basic modifications to the code can make it work with point or line features. The Map-Lot combination that is queried by the Map and Lot parameters is a text field, but the code can certainly be modified to query one or more numeric fields. I did get stuck initially when I attempted to use a single parameter for Map and Lot, which are in a single field in the parcel layer that contains a hyphen to separate the two elements. I learned more than I wanted to know or understand about how hyphens cause problems when passed around as Javascript variables & eventually decided to simply pass in two separate parameters, which works quite well.

I hope that this sample is helpful.

-Mike

New Release of MassGIS OLIVER

2012-01-10T12:26:00.001-05:00

MassGIS, the excellent state GIS office for the Commonwealth of Massachusetts, really doesn't receive enough recognition for their work and great public service. I thought that it was worth taking a few minutes to recognize their work today, as they have just announced a new release of OLIVER, the MassGIS Online GIS Data Viewer. OLIVER is an open source web app billed modestly as a "data viewer," but it really does a whole lot more, including GIS data extraction by map extent or shape.

Details of the new OLIVER release are provided below by Aleda Freeman, MassGIS Web Mapping Services Manager.

MassGIS is pleased to release the latest version of OLIVER, MORIS (the "ocean" OLIVER) and SPOLIVER (the "state police" OLIVER).

Developed by MassGIS, CZM and SeaPlan (formerly Massachusetts Ocean Partnership), this version of OLIVER includes many enhancements and bug fixes:

More basemaps including Bing, OpenStreetMap, CloudMade and TopOSM
Better printing
Now 3 ways to extract data (by bounding box, by drawn shape or by selected feature shape)
Ability to change the color of points, lines and polygons (within a limited palette)
Permalink URLs (to send or bookmark) that support color changes and opacity - so that the link you send exactly represents your map!
Ability to add selected external (from non-MassGIS servers) WMS layers
Streamlined data extract process
Bigger, more visible buttons and banner
Dropdown to zoom directly to town (OLIVER only)

In addition to the above standard features OLIVER is now more configurable to create custom viewers.

In addition to custom datasets, titles, logos, default layers and geographic extent there is now in addition:

Draw and edit to live database layers (including optional snapping and splitting)
Configurability for many buttons and tools to be hidden if not needed
Dropdown options to zoom to extents; optionally narrow down dropdown choices within geographies

For state agencies MassGIS can design and host custom "OLIVER"s. For information, contact aleda.freeman at state.ma.us

For more information on configuration options see OLIVER developers documentation and configuration examples.

The OLIVER toolkit is an open source project that can be found on Google code as the "morisoliver" project.

Aleda Freeman, GISP

Web Mapping Services Manager

Office of Geographic Information (MassGIS)

Information Technology Division

Commonwealth of Massachusetts

One Ashburton Place, Room 1601

Boston, MA 02108

http://www.mass.gov/mgis

Putting my geodatabase on a diet

2011-10-26T12:36:00.001-04:00

We've been using ArcSDE with MS SQL Server at my workplace as our platform for storing our GIS data in an enterprise geodatabase for the past 6+ years. I was absolutely thrilled when I loaded our imagery into ArcSDE for the first time & saw how well it performed. I saw the performance boost provided by loading 50+ gigs of 3 inch resolution imagery & I was a believer right away in the power of a relational database for serving up massive amounts of imagery. Since then, we've added imagery from various years and sources to this same geodatabase, causing the database to grow to a size of close to 300 gigs! More than 95% of the space used was consumed by the imagery alone, which is for all purposes read-only data. Differential and log file backups of this database are reasonably sized as well as quick to complete, but our weekly full backups of this database became larger than 250 gigs, even though most of that data remained unchanged since initially being loaded. (by the way, we use Ola Hallengren's excellent SQL Server Backup, Integrity Check, Index and Statistics Maintenance scripts to keep our database backed up & tuned. Check them out if you use SQL Server!)

If I could go back in time to our initial ArcSDE geodatabase setup, I would have set up one database for our editable vector data & a separate database for our imagery data. That would allow for a robust backup plan for our vector data and just one large backup of the imagery data to stow away just in case. Unfortunately, I can't go back in time & since then our organization has created thousands of map document files, most of which include at least some of the imagery layers. The prospect of spending a lot of time repairing our map documents to point to a new source or instructing our busy workforce to load new copies of imagery into their maps was not appealing at all.

So, I've decided to move over to the dark side & tinker under the ArcSDE hood by doing some simple SQL Server tricks. The objective: move the imagery into a new database while providing the illusion of it being in the same database in which it has always resided. I decided to test out an approach that consisted of the following:

Creating a new ArcSDE geodatabase for our imagery data on the same server & adding the same users to this database. Set the recovery model for this database in SQL Server to "Simple" in order to prevent a large amount of growth in the log files.
Copying each raster feature class over to this new geodatabase & setting the appropriate permissions (very important to make sure that any users granted select permission to these layers in the original geodatabase are also granted select permission in the new one).
Replacing the Tables for each raster feature class with Views of the same name. The Views point to the corresponding table in the new geodatabase via a very simple "Select * from..." statement.

Warning: (1) don't do this unless you are very very comfortable working with SQL Server and you thoroughly test it out on a test geodatabase. (2) don't expect Esri to provide support for this sort of stuff...they really shouldn't be expected to, should they? (3) think about the implications down the road when you want to upgrade your geodatabase to a newer release...who knows what kind of a challenge this will present (It's a risk that I've taken on) (4) make sure that you have enough disk space to accomodate a temporary increase the size of your data as you copy large datasets between databases.

Here's some background on how to do this:

Any feature class in ArcSDE is composed of a set of tables in the database. The connection that the client application uses to the connect to the database translates these tables into the layer that you see on a map. For raster data, ArcSDE stores the data in tables with the following prefixes, followed by a set of unique IDs (these IDs do vary depending upon the table):

F
S
I
SDE_RAS_
SDE_BND_
SDE_BLK_ (these are the big tables where the actual data resides)
SDE_AUX_
& the Business Table, which has the feature class name that you see in ArcCatalog when connected to a geodatabase

It took a while to figure out this data structure. I'm sure that it is clearly documented somewhere, but I was only able to piece together through fragments of documentation & through examining a geodatabase under the hood.

The following tables document the name of the feature class & provide the unique IDs:

SDE_Layers
SDE_Raster_Columns
SDE_Table_Registry

The unique IDs are as follows:

SDE_Layers.layer_id is for the F & S tables
SDE_Table_Registry.registration_id is for the I table
SDE_Raster_Columns.rastercolumn_id is for the SDE_ tables

Once the corresponding input & output unique IDs are known for each of the Tables, it's a relatively straightforward task to complete. I set up a stored procedure that would take an input of a feature class name, rename each input Table by adding an "Archive_" prefix (just in case), replace each Table with a View of the same name pointing to the corresponding Table in the new geodatabase, set the appropriate permissions for each View and update the configuration keyword listed for the feature class in each of the SDE tables (I had configuration keywords set for our raster data in order to have them stored in their own database filegroups, separate from the vector data -- I didn't want to encounter any future problems with the data being listed under filegroups that no longer exist). The stored procedure would take about a second to run & I found that it worked even with ArcGIS Desktop users and ArcGIS Server Map services connected to the data. The change was seamless & not a single map document needed to be pointed to a new source. Once the new database was backed up, I deleted each of the original tables that had been marked with the "Archive_" prefix. Once the filegroups in which these tables resided were empty, I was able to delete some very large filegroups & my database shrunk considerably as a result.

The initial result? As expected, there was no degradation in performance. Log file backups & differential backups of our original geodatabase have not really changed in size, since none of the data that was removed was being updated before. The big gain was that the size of the full backup of our original geodatabase is now less than 5% of what it used to be. Whereas the it used to take hours to do a full restore of this database, it can now be restored in a matter of minutes. Restoring the new imagery database will still take quite a while, but at least in the event of an emergency we can be back up and running with our most critical data in a much more timely manner.

Final thoughts: If you're setting up a new spatial database, regardless of whether it's an ArcGIS geodatabase, consider storing your large imagery data separate from vector data so that you don't get into the kind of situation that I got into. If you are using an ArcGIS geodatabase at version 10, take a look at Image Mosaics as an option for dynamically storing data and using the geodatabase as more of an intelligent index for imagery data. I'm looking at that moving forward, but unfortunately that didn't provide the solution to this particular issue.

Happy mapping,

-Mike

Tracking Down 2010 Census Data

2011-04-08T10:11:00.001-04:00

The 2010 block level Census data was recently released for my home state of Massachusetts & I've had the chance to dive right into it as a technical advisor for my community's voting re-districting effort. A couple weeks ago, I began the process of finding the census blocks in shapefile format, which was easy, thanks to the Census Bureau's TIGER/Line Shapefile download site, which is relatively easy to navigate. Within minutes I had the census blocks for my county, completing what I thought would be the first half of the data collection effort.

The second half turned out to be much more difficult than anticipated. After flailing around on Census.gov for far too long, I was ready to throw my hands up in the air & give up. Census.gov has a fantastic amount of data, but like many sites, it has grown into a monster over the years as ever more content becomes available. Quite frustrating indeed, so I did what I tend to do more and more frequently when I can't solve a technical problem: I complained on Twitter. My complaint was not ignored and I was fortunate enough to be pointed to a great post by Micah Williamson on his False Northing blog. The keys in Micah's post are the links to the Census Bureau's FTP site and to their Tech Tips page for working with the 2010 Census Summary files. Thank you, Micah!

The Census Bureau posts the block level summary files for each state in an ascii format without field headings (argh!). Fortunately, they were kind enough to provide an MS Access database shell along with detailed instructions on how to load the data into this shell (both available through the Tech Tips page). Converting the Massachusetts 2010 Summary files into MS Access format took me about 20 minutes while following the instructions provided by the Census Bureau. The instructions were very straightforward & showed me that while Census.gov is about as clear as mud to navigate, the Census Bureau still does have some very competent technical people on staff.

In case you're interested in getting the Massachusetts 2010 Summary files already in MS Access format, I've posted them here as a 25mb zip file. Please refer to the data structure documentation or the Tech Tips page if you have any questions about the structure of this file.

Once you have your 2010 Census data, consider making it easier for those in your community to access. Not everyone has MS Access installed, nor does everyone have the patience or skills to churn through a confusing series of data conversion steps. Consider doing what I did for my hometown: post your 2010 Census data for all to see in Google Fusion Tables. What's that, you ask? Take a look at the Amherst, MA 2010 Census Blocks with basic population data.

Enjoy!

-Mike

ArcGIS.com data downloads for those without ArcGIS Desktop

2011-01-28T10:38:00.001-05:00

We've recently started to make GIS data from my workplace, the Town of Amherst much more accessible via download or extraction. ArcGIS.com has come in quite handy for this purpose, providing a nice way to catalog the various download and extraction options that we've made available. We've made good use of Layer Packages (.lpk) as a way to provide various data sets in a pre-symbolized format for ArcGIS Desktop users and ArcGIS.com makes it extremely easy to share these.

Layer Packages are an ArcGIS Desktop file type, so if you have ArcGIS Desktop 9.3.1 or higher, a layer package will open right up into a map...easy as pie! However, if you do not have ArcGIS Desktop installed, downloading a layer package isn't so simple because of the way that ArcGIS.com delivers them. ArcGIS.com presents an intermediary xml file that initiates the Layer Package download only if you have ArcGIS Desktop 9.3.1 or higher installed! If you do not have ArcGIS Desktop installed, you'll get an xml file that has the true hyperlink to the layer package listed within the packagelocation tag; not so intuitive at all.

So, why would anyone who doesn't have ArcGIS Desktop installed even have a need for a layer package? Well, layer packages are basically zip files that contain both the symbology in .lyr format and the data, which can be in either file geodatabase or shapefile format. If a layer package is provided with shapefiles in it, that data is just as useful to those using AutoCAD or Quantum GIS as it is for ArcGIS Desktop users. The majority of the data requests that we receive at my workplace are from users of various CAD programs that can read shapefiles. Being able to catalog and share data in this format via ArcGIS.com for these folks is something that we would like to do, so now we are, with just a little extra documentation:

Here's a link to our shapefile format layer packages, each of which has some extra documentation that goes like this:

This layer package was posted specifically for people who need the data in Shapefile format. If you do not have ArcGIS Desktop, you can get at the shapefiles through the following steps:

Download the layer package from here: (link provided here)
change the .LPK file extension to a .ZIP extension.
Open the resulting zip file. The shapefiles are stored here: \commondata\data0

It's not pretty, but it gets the job done and we're receiving good feedback so far. It's my hope that ESRI will eventually make it easier to *download layer packages from ArcGIS.com onto computers that do not have an ArcGIS Desktop installation.

*It is worth noting that ESRI's free ArcGIS Explore can open layer packages directly from ArcGIS.com, but what I'm more interested in here is exploring ways to use ArcGIS.com a vehicle to provide data for download purposes.

Peace,

-Mike

Dude, Where's My MXT?

2010-10-22T11:20:00.000-04:00

And now for some Friday levity...

While hurrying to produce a map this morning, I learned something about ArcGIS Desktop 10 that's actually old news by now: Map Templates (MXT files) are no longer supported in ArcGIS 10. Well, that was kind of annoying to learn when I needed my map NOW, but I suppose it's progress in some strange way. It always seemed awkward that there was this template file type that could be opened like a map & could be loaded into an existing map, but wasn't actually the same as a map document. It was also frustrating that for 10 years, this template file type could be counted on to completely blow up the formatting of any legend in a map. ESRI never solved that problem and now I guess it doesn't matter. Bye bye, MXT. Time to convert you into your step-sibling, the MXD. You were always lovable in an awkward sort of way, so I'm not sure whether to miss you.

Ok, so we're in this post-MXT world now. I'm coping. I'll play nice and convert my MXT files into MXD files. Of course, doing so isn't as easy as doing a File -> Open because, well, MXT is no longer a supported format. Converting an MXT requires use of the new Catalog Window. What a clever way to promote a new feature: "look at me! look at me! I can open unsupported formats! will you use me now?" Yes, I'll use you now, Catalog Window, at least until I'm done with this MXT thrashing exercise.

Two different versions of ArcGIS Server on one web server? No sweat.

2010-07-16T16:48:00.004-04:00

I'm testing ArcGIS Server 10 at work, while using 9.3.1 in a production environment. For various reasons, it quickly became a need to have both versions exposed to the outside world. The simplest solution would be to have one external web server for each, but rather than bothering our system admins to deploy another external web server, I decided to get adventurous and try setting up both on one external web server through the use of a reverse proxy.

The reverse proxy approach is part of ESRI's recommended approach to securing ArcGIS Server. A reverse proxy allows the essential elements of ArcGIS Server to be exposed on an external web server without ArcGIS Server actually being installed on that web server. This is achieved primarily by using URL rewrite rules that will take external URL requests and route them to the proper internal server, which sends the result to the external server, which sends it to the outside world. The result is that the external web server appears to the outside world to be running ArcGIS Server. Pretty cool stuff.

There are various ways to implement a reverse proxy, depending upon which web server technology is in use. We're using IIS7 on a Windows 2008 R2 server, which makes this type of task easy when consulting ESRI's instructions. I've had our ArcGIS Server 9.3.1 instance set up in this manner for several months without any issue, but after some quick searching, I was unable to confirm whether it was possible to set up a separate instance of a different version of ArcGIS Server on the same external web server. The only clue that I came across was that it is possible to use a non-standard ArcGIS Server instance name with a reverse proxy (the instance name refers to the /ArcGIS/ portion of the web address, as in http://servername/ArcGIS/). The logic that I applied was that if it's possible to use a non-standard instance name with a reverse proxy, then it's possible to set up reverse proxies to two different instance names. So far, it seems to be working out quite well, having set up a reverse proxy to our new ArcGIS Server 10 test instance. Here are some observations on the process:

Obviously, you need to use a different instance name for each instance that you want to set up on your external web server. We went with the default name of "ArcGIS" for our 9.3.1 instance and "AGS10" for our 10 instance.
Following ESRI's instructions for setting up a reverse proxy server system with IIS7 for both instances, just make sure that you set up two different server farms and name them so that you'll know which is which at a glance because the server farm name is used several times during each configuration.
If you're running ArcGIS Server for Microsoft .NET (as we are) and you run any other .NET applications on your external web server, make sure that your URL rewrite rule for aspnet_client is set for "aspnet_client/ESRI" instead of the default, which is simply "aspnet_client" -- I missed this simple detail while setting up our 2nd reverse proxy & it instantly stopped all non-ArcGIS Server .NET apps on that external web server from working! (oops...at least we figured it out & corrected it quickly enough)
When a 2nd server farm is set up in IIS7, Microsoft Application Request Routing (ARR) (one of the enabling technologies at work here), automatically creates a "LoadBalance" URL rewrite rule for each server farm. I found it necessary to disable both of these because these rules by default create a URL rewrite for "*" -- the wildcard for any text string. This means that if you leave these enabled, all directories on the web server that are not participating in the set of URL rewrite rules used by the reverse proxy setup will become unreachable, as if they no longer exist. I haven't had a chance to do much research on the LoadBalance rules, so if disabling them comes back to bite me in some other way, I'll report it here.
As one of the early steps in setting up a reverse proxy for ArcGIS Server, it's necessary to create the following 3 virtual directories on the external server that are used to serve up ArcGIS Server content: ProxyOutput, ProxyCache and ProxyJobs. These virtual directories sit at the root level of the website in IIS7, so they're not necessarily associated with either ArcGIS Server instance that we're running. I thought that this would create a conflict and it may very well without my realizing it. We're not currently running any cached services, so if you're planning to try this, be aware that the caches from both instances may end up being co-mingled within the same cache directory. I do not know whether this would be problematic, so proceed with caution on this front.

I guess that's a good way to end this post...proceed with caution, but do know that it is possible!

ESRI's Make a Map App

2010-07-06T09:07:00.002-04:00

I just tried out ESRI's "Make a Map" app, which makes it easy to make a quick embeddable web map like the one below. Ray Carnes wrote a nice article summarizing how it works in ESRI's latest issue of ArcWatch. Make a Map currently includes several demographic layers that are available as the basemap, with more to come as ESRI builds upon it. The performance is pretty slick, using ESRI's Flex API for ArcGIS Server.

KML to Shapefile Conversion...does it matter anymore?

2010-05-25T15:50:00.001-04:00

A while back I posted about an online kml to shapefile converter that has long since ceased to function properly. Not that I post all that often or that readership is all that high on this blog, but that post still fetches the most comments of anything that I've posted here, so I thought that a brief follow-up was in order. From my perspective, the thirst for KML to shapefile (or any GIS format) conversion was at its peak in 2007 and has dropped off considerably since then. My reason for believing this is simple: how many converters of this sort are out there? Not many. Where's the demand? I'm not seeing it...are you? I'm not sure why...KML seems to be alive and well, but folks in the GIS world don't seem to have much interest in grabbing data from this format. I know that my interest in it has waned, except for the occasional interest in converting a running or hiking route into a shapefile.

Althought I have given up on any effective online KML to Shapefile converter, there are options. For those interested in a simple and free program that does convert quite effectively from KML or GPX to Shapefile format, I would suggest checking out DNR Garmin, which was developed by the Minnesota Department of Natural Resources as a GIS compatible interface for Garmin GPS units.

Spring NEARC is Soon...Submit a Presentation Proposal Today!

2010-03-15T11:32:00.002-04:00

If you're in the Northeast US and you're using ArcGIS or any ESRI technology, the Northeast Arc User Group (NEARC) Conferences are fantastic. It's like having the best of the ESRI User Conference just a couple hours away and the NEARC community is full of knowledgeable people who know how to have fun. So, please join us at Smith College in Northampton, MA on Tuesday May 11th, 2010 for the 1-day annual NEARC Spring Conference!

We think that you may have something to share as well, so please act now and submit a presentation proposal. Presentations will be accepted in several different formats, including one that is new to NEARC: the Lightning Talk, 5 Minutes of GIS Glory! Here's a motivational video to get you excited about giving a lightning talk at Spring NEARC:

COGO Made Easy in ArcGIS Desktop

2010-03-04T12:05:00.002-05:00

Far too many people using deeds or survey plans to revise a cadastral base end up resorting to some form of sketching, tracing, fudging or what have you. That is really unfortunate & unnecessary. My goal with this post is to help ArcGIS Desktop users to unlock the power and simplicity of using Coordinate Geometry (COGO) in ArcMap. Surprisingly few people even realize that ArcMap 9.x natively supports editing using COGO at all license levels, including ArcView. It is true that ArcInfo and ArcEditor include the additional capabilities of a COGO toolbar that helps to save some clicks & automatically capture COGO data at the feature level. It's also true that the Survey Analyst Extension provides some excellent enhancements to workflows that include COGO. However, it is a complete myth that you're out in the cold if you only have a simple ArcView license. The goal of this post is to help you to make good use of COGO with minimal effort with even an ArcView license:

So, I have a survey plan from a few years back that I've scanned and georeferenced for the purposes of illustrating this process. It is not normally necessary to scan and georeference, but it does make things easier when it's possible. I often will scan a plan & do a quick georeferencing without worrying about it being perfect, simply to be able to better visualize it on-screen. In this example, I'll set up this plan to be incorporated into my existing cadastral fabric. Here's what the plan looks like:

One of the first things that I noticed about the plan is that it uses magnetic north, which changes over time. To adjust for the shift in magnetic north, we'll need to make an adjustment to the bearings to match the bearings of our existing property boundaries. For this reason alone, most people don't bother with COGO in ArcMap or in any GIS platform, for that matter. Fortunately, ArcMap allows a Ground to Grid Correction to be set. The only problem is how to figure out what exactly that correction should be & that is why I decided to write this post in the first place. For folks with an ArcEditor or ArcInfo license, there's a handy Ground to Grid Correction button on the COGO toolbar, but for those using an ArcView license, you'll want to follow along with me. Here's my Ground to Grid Correction trick that will work at any ArcGIS Desktop license level:

Begin an edit session in ArcMap, set a property boundary layer as your target layer & set the snapping preferences for that layer to include vertex and edge snapping. Go into the Units tab of the Editing Options dialog (Editor -> Options) & make sure that your Ground to Grid Correction Direction Offset is set to 0. Also, set the "Direction Type" to Quadrant Bearing and the "Direction Units" to Degrees Minutes Seconds. Close the dialog.
Find a boundary on your plan that corresponds to a reliable property boundary in your property boundary layer (what I mean by "reliable" is one with a bearing that you can trust, to which to set your ground to grid correction). In our example here, I'll use a roadway right of way line that I trust in order to set the alignment of my plan.
Use the Sketch Tool to begin a line anywhere near that existing property line, then right-click & choose "Direction/Length," (CTRL + G will also work) which will bring up this dialog box:

Type in the direction that the plan has for the line, formatted in the manner shown above. The distance doesn't really matter too much, as our primary concern here is adjusting the bearing. Hit enter to draw the first line segment, which will normally show up anywhere from 1-20 degrees off from where it should align (if your line segment aligns perfectly with your existing boundary line, then you don't need to set a ground to grid correction & you can skip the rest of these steps!). Here's what mine looked like, which appears to be a few degrees off. I've superimposed the Direction dialog box over it to emphasize that the bearing of the line matches the plan:
Next, right-click on the existing line that you want your plan to align with and choose "Parallel" (CTRL + P will also work) to set your next segment as parallel to the line that you trust.
Then, right-click and choose "Deflection" (CTRL + F will also work). The deflection dialog box that pops up will contain the deflection from the first segment that you drew to the second segment that you drew. This is the ground to grid correction that you'll need! Just copy it & close the dialog box. You can also delete the sketch, since you've taken what you need from it.
Go into the Units tab of the Editing Options dialog (Editor -> Options) & paste the deflection angle that you just copied into the "Direction Offset." Click OK to save the setting and close the dialog. Here's what it looked like for me just before I closed it:
Now, once you choose a reliable starting point, you can draw in the whole plan using just COGO! Just begin a line at a good starting point, right-click and choose "Direction/Length," (CTRL + G will also work).

A couple tips for doing COGO with plans or deeds:

ArcMap will support several different types of units & you can enter unit types other than what you're using in your map by including a unit abbreviation after a distance. I've found some deeds that have distances in Rods, which are even supported in ArcMap (add rd after the distance).
If you're drawing a line in the opposite direction from the way that a plan shows it, just make sure that you flip the directions in your bearing, ie., if the plan shows the bearing as S 86-12-36 E and you are drawing the line the opposite way, just enter it as N 86-12-36 W.
Use "Tangent Curve..." (available at a right-click while in sketch mode) to draw in arcs with COGO. Most plans will show at least the Radius & Arc Length for an arc, which you can plug right in with the Tangent Curve dialog.

I hope that this is helpful. Please feel free to comment or contact me with any questions.

-Mike

ArcGIS Desktop II course March 15-17 @ Westfield State College

2010-01-11T14:28:00.003-05:00

Greetings,

In collaboration with the Westfield State College Community Eduction Office, I am happy to announce that we will offer the ESRI Authorized ArcGIS DESKTOP II: TOOLS AND FUNCTIONALITY class on March 15-17, 2010. This is an excellent 3-day course created by ESRI for ArcGIS 9.3. This course is designed to prepare people to get right to work with ArcGIS Desktop. Be the first in your organization to take this & at by far the best price going!

Please call (413) 572-8034 to register
(ask about the special course rate for WSC students & Staff)

Course No: CMED 0719
Instructor: Michael Olkin
Dates: March 15-17, 2010, Monday-Wednesday, 3 sessions
Time: 9:00 a.m. – 5:00 p.m.
Place: Bates PC Lab, Westfield State College, Westfield, MA
Tuition: $620 + Fee: $15
*Cost of course includes tuition, books, CDs and evaluation copy of ArcView9.3.
All books and class materials and will be distributed on the first day of class.

Course Description:

ArcGIS DESKTOP II: TOOLS AND FUNCTIONALITY

This course is part of the ArcGIS Desktop Foundational Training Curriculum. ArcGIS Desktop software is an integrated system that includes all the tools needed to get the most out of a GIS. This course teaches the range of functionality available in the software and the essential tools for visualizing, creating, managing, and analyzing geographic data. The hands-on course exercises emphasize practice with ArcMap and ArcCatalog (the primary applications included with Arc-GIS Desktop software) to perform common GIS tasks and workflows. The tools for creating and managing geographic data, displaying data on maps in different ways, and combining and analyzing data to discover patterns and relationships are highlighted, and you learn how ArcGIS Desktop provides a complete GIS software solution. By the end of the course, you will be prepared to start working with the software on your own.

This course is designed for those with an education in or workplace experience with GIS but no ArcGIS software experience. Knowledge of basic GIS concepts is assumed. This course teaches the skills and knowledge needed to take other ArcGIS courses, including ArcGIS Desktop III: GIS Workflows and Analysis.

More information & a full list of topics covered are available here: http://tinyurl.com/67m7pr

Thanks,

-Mike

Great GIS Day Video: Tied Together (with GIS)

2009-11-20T08:50:00.004-05:00

The folks at the City of Austin, TX GIS Department have outdone themselves with this year's GIS Day video. Enjoy!

http://www.youtube.com/watch?v=wDVNBDIVUZo

ArcGIS Desktop Courses This Spring at Westfield State College

2009-10-29T13:03:00.000-04:00

Greetings,

In collaboration with the Westfield State College Community Eduction Office, I am happy to announce that we will offer the ESRI Authorized ArcGIS DESKTOP II: TOOLS AND FUNCTIONALITY class on January 11-13 & March 15-17, 2010. This is an excellent 3-day course created by ESRI for ArcGIS 9.3. This course is designed to prepare people to get right to work with ArcGIS Desktop. Be the first in your organization to take this & at the best price going!

Please call (413) 572-8034 to register

Course No: CMED 0719
Instructor: Michael Olkin
Dates: Monday, Tuesday, Wednesday 3 sessions
January 11-13, 2010 or March 15-17, 2010
Time: 9:00 a.m. – 5:00 p.m.
Place: Bates PC Lab, Westfield State College, Westfield, MA
Tuition: $620 + Fee: $15
*Cost of course includes tuition, books, CDs and evaluation copy of ArcView9.3.
All books and class materials and will be distributed on the first day of class.

Course Description:

ArcGIS DESKTOP II: TOOLS AND FUNCTIONALITY

This course is part of the ArcGIS Desktop Foundational Training Curriculum.
ArcGIS Desktop software is an integrated system that includes all the tools needed to get the most out of a GIS. This course teaches the range of functionality available in the software and the essential tools for visualizing, creating, managing, and analyzing geographic data. The hands-on course exercises emphasize practice with ArcMap and ArcCatalog (the primary applications included with Arc-GIS Desktop software) to perform common GIS tasks and workflows. The tools for creating and managing geographic data, displaying data on maps in different ways, and combining and analyzing data to discover patterns and relationships are highlighted, and you learn how ArcGIS Desktop provides a complete GIS software solution. By the end of the course, you will be prepared to start working with the software on your own.

This course is designed for those with an education in or workplace experience with GIS but no ArcGIS software experience. Knowledge of basic GIS concepts is assumed. This course teaches the skills and knowledge needed to take other ArcGIS courses, including ArcGIS Desktop III: GIS Workflows and Analysis.

More information & a full list of topics covered are available here: http://tinyurl.com/67m7pr

Thanks,

-Mike

Comparison of iPhone Running Apps

2009-10-20T13:01:00.008-04:00

I've spent some time the past couple months trying out iPhone 3GS GPS apps for tracking my running activity. In the past, I've manually traced routes using various GIS tools and online tools in order to determine the distance of various routes that I like. Using a GPS is much easier and it has some other benefits that I've come to really appreciate. I thought that it might be nice to write up a brief review here of the apps that I've tried out. The iPhone 3GS does include what looks like a great app from Nike, but I don't have any Nike running shoes, so I did not get to test it (what can I say? I'm just a New Balance kind of guy). I tested these three apps, in this order:

MotionX GPS Sport

iMapMyRun

RunKeeper

Please don't take any of the following to be a formal endorsement. Also, if any of these seem interesting to you, give them a try for yourself; each of these apps have free versions.

MotionX GPS Sport
I started out using the free version of this app to help support some of my work activities. Our local Conservation Department was conducting a bog bridge assessment on some trails in town & I wanted to see how well my new iPhone would compare with the Conservation Department's Garmin GPS unit. I was pleasantly surprised & I quickly forked over a few dollars for the Pro version. MotionX GPS Sport can be used to collect both tracks and waypoints. It has built-in support for the iPhone camera functionality, allowing a photo to be associated with each track or waypoint (although, wouldn't it be nice to be able to associated more than one photo per track?). Sharing capabilities allow tracks or waypoints be shared in GPX & KML formats via e-mail, Twitter or Facebook. The map display is quite nice, providing the user options between the various Google Maps, Bing Maps & a couple MotionX basemaps. MotionX GPS Sport also includes the capability to import GPX or KML features for display on the map, which is great for locating features. In terms of use for running, my favorite feature in MotionX GPS Sport is the embedded iPod support, which makes it easy to control play lists while continuing to keep the GPS data collection going. However, in terms of utility as a running app, I found MotionX GPS Sport to be very basic; it's a great general purpose GPS app, without anything special built in for runners.

iMapMyRun
The 2nd app that I tried was iMapMyRun, which is more specifically built for running. The app is integrated with a web-based profile on mapmyrun.com, to which my running activities were posted. I found the iPhone app to be easy to work with and the result at the end of my run was a nice summary of my running activity, including distance, time, average pace, calories burned, etc. The app allows sharing via Twitter & Facebook, which I tried out with good results. Of course, being into mapping, I was most interested in the map output, which is accessible on mapmyrun.com with a GoogleMaps base. The map has limited interactive capability, although it does include export to KML & GPX, as well as a static elevation/distance graph. This is a good stable app & I would have stayed with it, if not for the final one that I tried...

RunKeeper
I found RunKeeper shortly after trying out iMapMyRun. Both are quite similar, being simple yet effective apps tied to an online profile. I set up my RunKeeper profile and tried out my first run. The results were quite similar to the iMapMyRun results, with a decent summary of my running activity, but what really drew me in were the mapping capabilities. The map interface on the RunKeeper site is simple yet effective, using a Google Maps base with distance milestones & accompanied by an interactive Elevation/Speed graph. Hovering over any spot on the graph highlights that location along the route on the map, which I find to be very helpful in learning more than I ever knew about my running habits (it's interesting to analyze changes in speed along a run; they're usually attributable to changes in terrain or slope, but I slow down a lot more than I ever imagined when I'm not paying attention to my pace). The map provides the ability to export the route to KML or GPX formats and the app can post to Twitter or Facebook. After my first run with RunKeeper, I was hooked. The map and interactive graph are what won me over. I upgraded to RunKeeper Pro, which includes voice prompts at timed intervals. I have mine set to tell me how far I've gone & what my current pace is every 5 minutes. Between that and posting my runs to Facebook (where my friends can tease me for being slow), I have more motivation than ever to run faster!

ESRI Reveals the Final Stage of Some Legacy Products

2009-09-15T10:30:00.003-04:00

As evidenced in their newly posted Depreciation Plans FAQ, ESRI is finally moving on from some products that have been around for more than a decade, but which have been in steeply declining use in the past few years. Of note are the final stages of support for ArcInfo Workstation & ArcIMS. Neither are surprising at this point.

I can recall how it was a decade ago when command-line driven ArcInfo Workstation was king among GIS geeks & I honestly don't miss it at this point. I guess it's just my preference for working visually, but poring over code to determine why the legend on my map isn't coming out correctly isn't so high on my list these days. I also don't miss wrestling with ArcIMS, although there are days when wrestling with ArcGIS Server isn't a whole lot different.

There's a nice line by line commentary on the depreciation plans in James Fee's Blog. As usual, James does a great job of getting right to the point with some humor mixed in, while not getting caught up in the odd feelings of nostalgia that some people have for old software.

ArcGIS Desktop II course August 17-19 @ Westfield State College

2009-07-21T15:11:00.002-04:00

In collaboration with the Westfield State College Community Eduction Office, I am happy to announce that we will offer the ESRI Authorized ArcGIS DESKTOP II: TOOLS AND FUNCTIONALITY class on August 17-19. This is an excellent 3-day course created by ESRI for ArcGIS 9.3. This course is designed to prepare people to get right to work with ArcGIS Desktop. Be the first in your organization to take this & at the best price going!

Please call (413) 572-8034 to register
Course No: CMED 0719
Instructor: Michael Olkin
Dates: Monday, Tuesday, Wednesday 3 sessions
August 17-19, 2009
Time: 9:00 a.m. – 5:00 p.m.
Place: Bates PC Lab, Westfield State College, Westfield, MA
Tuition: $620 + Fee: $15
*Cost of course includes tuition, books, CDs and evaluation copy of ArcView9.3.
All books and class materials and will be distributed on the first day of class.

Course Description:
ArcGIS DESKTOP II: TOOLS AND FUNCTIONALITY
ArcGIS Desktop software is an integrated system that includes all the tools needed to get the most out of a GIS. This course teaches the range of functionality available in the software and the essential tools for visualizing, creating, managing, and analyzing geographic data. The hands-on course exercises emphasize practice with ArcMap and ArcCatalog (the primary applications included with Arc-GIS Desktop software) to perform common GIS tasks and workflows. The tools for creating and managing geographic data, displaying data on maps in different ways, and combining and analyzing data to discover patterns and relationships are highlighted, and you learn how ArcGIS Desktop provides a complete GIS software solution. By the end of the course, you will be prepared to start working with the software on your own. More information & a full list of topics covered are available here: http://tinyurl.com/67m7pr

Online KML to SHP Converter

2009-03-09T11:04:00.003-04:00

I just found Zonum's new online KML to Shapefile converter at http://www.zonums.com/online/kml2shp.php

I had a KML that I wanted to convert for use in ArcGIS & this did the trick!

Reminders that 1991 was a long time ago

2009-02-20T12:16:00.006-05:00

I'm in the midst of shuffling some old files at work & I just came across a box that contained documentation and manuals from the Town of Amherst's original GIS purchases in 1991. The treasures include a full set of PC ArcInfo 3.4 manuals and the specs from the 386 that was purchased to run it. My, how things have changed...

Geospatial Courses this spring at Westfield State College

2009-01-16T14:08:00.001-05:00

In collaboration with the Westfield State College Community Eduction Office, I am happy to announce two geospatial courses at Westfield State College this spring!

ArcGIS DESKTOP II: TOOLS AND FUNCTIONALITY - this is an excellent 3-day course recently created by ESRI for ArcGIS 9.3. This course is designed to prepare people to get right to work with ArcGIS Desktop. Be the first in your organization to take this & at the best price going!

OUR WORLD IN 3D: WORKING WITH GOOGLE EARTH & GOOGLE SKETCHUP - this is an online course that was offered for the first time this past fall.
The course site from the Fall 2008 course is available at http://envcenter.wsc.ma.edu/courses/3DWorld/ to anyone who is interested in seeing what this course is like.

Please call (413) 572-8034 to register for either course. The Westfield State College Community Education Spring 2009 Course Catalog is available at:
http://www.wsc.ma.edu/Academics/Division_of_Graduate_and_Continuing_Educ
ation/Community%20Education/WSC_Spring2009_CommEdCatalog.pdf

Here are the full course descriptions:

ArcGIS DESKTOP II: TOOLS AND FUNCTIONALITY

ArcGIS Desktop software is an integrated system that includes all the tools needed to get the most out of a GIS. This course teaches the range of functionality available in the software and the essential tools for visualizing, creating, managing, and analyzing geographic data. The hands-on course exercises emphasize practice with ArcMap and ArcCatalog (the primary applications included with Arc-GIS Desktop software) to perform common GIS tasks and workflows. The tools for creating and managing geographic data, displaying data on maps in different ways, and combining and analyzing data to discover patterns and relationships are highlighted, and you learn how ArcGIS Desktop provides a complete GIS software solution. By the end of the course, you will be prepared to start working with the software on your own.

More information & a full list of topics covered are available here:
http://tinyurl.com/67m7pr

This course provides professional development opportunities in the areas of Information Technology, Database Management, Geosciences, Environmental Sciences and Criminal Justice & Public Safety. The class incorporates a combination of lectures, hands-on exercises, and discussion. All participants are provided with course exercise & lecture books, a course exercise CD and an evaluation copy of ArcView 9.3.
Students completing this course will receive an ESRI Course Completion Certificate.

Course No: CMED 0719
Instructor: Michael Olkin
Dates: Monday, Tuesday, Wednesday 3 sessions March 16-18, 2009
Time: 9:00 a.m. - 5:00 p.m.
Place: Bates PC Lab, Westfield State College, Westfield, MA
Tuition: $620 Fee: $15 *Cost of course includes tuition, books, CDs and evaluation copy of ArcView9.3.

All books and class materials and will be distributed on the first day of class.

OUR WORLD IN 3D: WORKING WITH GOOGLE EARTH & GOOGLE SKETCHUP http://envcenter.wsc.ma.edu/courses/3DWorld/

Google Earth and Google SketchUp are powerful free tools for producing high quality interactive 3D geospatial content. Google Earth has become the world's most popular software for visualizing, touring and learning about our world in 3D. Google SketchUp provides and exciting way to render realistic buildings, bridges and other geographic 3D features for using Google Earth. This course provides a hands-on introduction to these systems and their supporting web-based resource infrastructure.
Participants will learn to harness and apply these technologies in powerful ways that can be applied across multiple disciplines. Prior experiencewith Google Earth, Google SketchUp, and ESRI's ArcGIS software are helpful but not required.

The intent of this course is to have fun while learning some powerful 3D Geospatial skills that you immediately will be able to apply to your work. We'll spend the first half of the course learning about Keyhole Markup Language (KML), the data format used by Google Earth. During the latter half of the course, we'll dive into Google SketchUp, a relatively simple 3D modeling system. All aspects of this course will use free software or online services that you can run on either a Mac or a PC. A Google Account will be a requirement of this course; participants will be instructed on how to create one during the first week of the course.

Course No: CMED 0720
Instructor: Mike Olkin
Dates: ONLINE, April 20 - June 1, 2009
Tuition: $200 Fee: $15

Thank you!

-Mike