GeoPDF

US Army Geospatial Center is a MAX Finalist

2009-10-02T09:34:00.002-04:00

The US Army Geospatial Center is a 2009 Adobe MAX Finalist in the Public Sector category. From the site:

From the battlefield to the densest urban center, Army warfighters now have instant access to advanced digital topographic and geographic information delivered in PDF. The application was developed using Adobe Acrobat 9 Pro and Acrobat 9 Pro Extended, as well as TerraGo Composer, a plug-in for Acrobat.

Congratulations to Ray Caputo and all of the hard-working folks at the AGC. Please show your support for their efforts by voting at the MAX Finalist site.

Tomas Lopes' Presentation Online

2009-07-27T08:27:00.002-04:00

Tomas Lopes' webinar is archived. Enjoy! More than 300 folks showed up. I hardly had time to take in any of the presentation because I was typing away at answers to the many questions that came flying in.

Ben Franklin Digs GeoPDF

2009-07-21T08:44:00.004-04:00

FYI, Ben Franklin announced new world use of geospatial PDF at James Fee's Spatially Adjusted. Of course, James used "geospatial PDF", and I'm using GeoPDF to stir up the pot... We like to GeoPDF it up around here.

In the post, James kindly provided a pointer to the upcoming webinar wherein Tomas Lopes plays Ben Franklin, discussing how to integrate Acrobat into work-flows with geospatial contexts.

I think that there is something in the previous sentence that often gets lost in the obsession with the arcane details of format and who should do what for how much by when: Adobe Acrobat offers value for people who work with GIS and geospatial work-flows. Although I haven't seen the script, I'm sure Tom is going to go over this in some detail, so I won't try to grind through all of that here.

It seems that folks like to talk about the things things can't do or don't do to satisfaction. Acrobat, PDF, and GeoPDF are not exempt, nor from what I hear is ArcGIS. If there is something that you wish Acrobat would do, you can tell Adobe. If there is something that you wish Reader should just do for free, tell Adobe. If you want more features added to PDF exported from ArcGIS right out of the box, tell ESRI. If you want ArcGIS to interact more seamlessly with PDF files and Acrobat and LiveCycle work-flows right out of the box, tell ESRI. I can't recommend this more strongly. The more robust the platform, the better. Platform is the job of the platform providers.

If you want to get somthing that finds maps and items of plant using easy-to-use geospatial contexts into the hands of your field crews, talk to TerraGo [mailto]. If there is something you'd like to see in platform that you're not getting from the platform providers, say it here and stimulate some discussion. Maybe TerraGo will implement it in between the platform providers' release cycles, maybe not. Likely no one will if no one knows why it might be a good thing to have.

It would be interesting to hear how you think things should be. The more Adobe and ESRI hear how people want to solve problems, the more they will provide tools and interfaces that play together nicely. Sometimes TerraGo can help bridge the gap between what is being done and should be done by the platform providers.

What would be more interesting yet would be to hear your what data distribution and data collection problems and pain-points are, and who suffers. The ArcGIS jockey may feel no pain in a work-flow, but myriad linemen might be suffering at the other end. That's the person in whom I'm keenly interested.

If you don't want to work with TerraGo for whatever reason, you're always free to roll your own tools and solutions or talk with third-parties to help you do so. Like ArcGIS, Acrobat, and Reader, TerraGo's tools are proprietary. The Portable Document Format and the geospatial extensions thereto (GeoPDF!) are not [OGC] [Adobe/ISO]. I'll even try to hook you up as best I can. GeoPDF is an open platform, regardless of what anyone says to the contrary.

Although I strongly encourage use of this forum, if there is anything that you want to discuss, but don't want to do so publicly, email me directly [mailto], and I'd be glad to discuss it with you.

Getting Geospatial With Acrobat

2009-07-17T14:48:00.002-04:00

Tomas Lopes is putting on webinar about using Acrobat in geospatial contexts. From the abstract:

Join Tomas Lopes, a geospatial database specialist at Farallon Geographics, on Wednesday, July 22, as he demonstrates how easy it is for GIS professionals to facilitate data maintenance workflow for individuals in the field by using Acrobat 9 with professional desktop GIS software. Combining traditional GIS software with Acrobat 9 is proving to be a fast and cost-effective way to create content that is easy to use in the field and easy to maintain in the office.

I'm going to be answering questions during Q&A. I actually have no idea about the details of the webinar beyond what's written on announcement on Acrobat User's, so I'm curious to see how it goes.

History of GeoPDF: PDF Map Books, LGIView, and LGIDict

2009-05-05T07:58:00.005-04:00

GeoPDF has a history that goes back to the 1990s. Using PDF files for engineering applications was the brainchild of Phil Lee, one of the founders of TerraGo. In 1999, Phil, then vice president of Layton Graphics, bid on a project to deliver maps and engineering drawings to telecommunications maintenance crews. The maps and drawings came from scanned paper maps, AutoCAD and Microstation design files and backed by different database systems maintained by different engineering groups. Phil was an avid Acrobat user -- fanatic may not be too strong a word. He knew more about getting stuff done with Acrobat than anyone I knew then or since. Taking all of these heterogeneous formats and data to PDF and exploiting the features of PDF and Acrobat was obvious to him. The PDF solution was compelling because it was not tied to any CAD or database system, secure, and easy-to-use. The solution was to render all of the engineering data to PDF, add bookmarks and hyperlinks to ease navigation, and organize the resulting map book with an interactive index map. My nominal role in all of this was for the rendering AutoCAD DWG files to PDF, along with extraction and analysis of the data that they contained for creating links and metadata. Alan Stewart headed up the Microstation side of things. Alan had been writing software to manipulate and render DGN files for quite a while. He arrived at Layton Graphics with Michael Bufkin some years before in an acquisition of Michael's company Cad Share. Michael ran Layton Graphics' engineering group. Michael, Alan, and I are all still working together at TerraGo.

The PDF map book project was a big success for Layton Graphics. We delivered the first map book set in the fall of 2000 and last in 2005. We built a custom system that consumed the raw design files and databases and churned out 2500 map book updates quarterly, burned them to CD and delivered them to the different maintenance groups. These map books were popular with the crews, and soon received requests for new features. These features were usually exposed by creating plugins to Acrobat. This was the responsibility of JB Freels. It was increasingly obvious that a combination of software and carefully prepared data was a compelling application that could be used in numerous contexts. It seemed that everyone we showed our map books to wanted them.

One of the first attempts to productize a collection of PDF files specifically configured for engineering applications along with Acrobat plugins was called LGIView. Adobe profiled LGIView as a success story. Patrick Graves was brought on board in 2002 to serve as chief software architect and run the software development group. One of the LGIView features was a coordinate finder. Engineering design coordinate systems are almost invariably Cartesian, but they are sometimes aligned with some mapping coordinate system like UTM or a State Plane system. Since the working coordinates were Cartesian, it was a simple matter to embed a rotation, scale, and translation that mapped the engineering coordinate system to that of the PDF. This metadata was encoded in the PDF file as a PDF dictionary called LGIDict. My notes indicate that Alan came up with the LGIDict version 1 schema. I Ling Xu initially documented the schema and I gave its name: LGIDict -- LGI Geospatial Information Dictionary. At the time, I didn't know about Acrobat developer's prefixes, and JB didn't bust me on it.

Even as LGIDict version 1 was being drafted in 2002 -- we had deadlines and product to ship! -- the limitations were obvious. In 2003 I started working on a second version with provisions to drive a coordinate transformation engine. The LGIDict version 2 was cut in November of 2003.

OGC Publishes GeoPDF 2.2 as OGC Best Practice

2009-05-01T12:02:00.002-04:00

After much anticipation, GeoPDF 2.2 has been published as an OGC Best Practice. From the page:

The intended audience of this document is a developer of software for creating GeoPDF. It specifies how to create the necessary PDF objects that identify a region of the PDF page as a map and describe the map’s coordinate systems. Map creation and rendering to a PDF page are not addressed. The underlying PDF file format is not addressed. The file format is specified in PDF Reference .
The reader will need knowledge of PDF objects and document structure. An understanding of cartographic projections and datums will also be helpful. Information about these can be found in Map Projections A Working Manual .
Though written with the PDF 1.7 file format in mind, this Best Practice is believed to be valid for all versions of the PDF file specification prior to PDF 1.7.

Things Get Quiet When Adam's Not Around

2009-03-27T14:43:00.003-04:00

TerraGo's amiable and prolific blog poster and all around social media butterfly Adam Estrada has signed on with a three-letter acronym agency with some sort of if-he-told-me-he'd-have-to-kill-me sort of gig. Everyone at TerraGo and certainly I wish Adam all the best in his adventures.

Despite the quiet on the outside, we're full tilt on the inside working on revs of all of our software and I hope to have some news to share soon. If I can shake free of the mountain surrounding me at the office, I hope to get back to the technical series on how geospatial PDF files work and how to make them. If there is anything you'd like to know about this or anything else GeoPDF or at TerraGo, drop me a line.

TerraGo is at ESRI Federal User's Conference

2009-02-17T12:50:00.002-05:00

TerraGo has got booth 123 at the ESRI Federal User's Conference. CEO Rick Cobb will be there, along with most of our sales crew.

TerraGo Composer for (ArcGIS) Server

2009-02-17T12:41:00.003-05:00

TerraGo is announcing a new product today called TerraGo Composer for Server. It works like this: you interactively select an area of interest (AOI). Composer grovels around ArcGIS Server finding files and documents associated with the AOI. You pick the types of information of interest, and Composer creates a GeoPDF index map of the AOI with hyperlinks to the documents you selected.

First ARGON Mapping Satellite Lauched 48 Years Ago Today

2009-02-17T08:42:00.002-05:00

The first ARGON mapping satellite was launched 17 February 1961. The ARGON satellites carried film cameras and ejected film canisters. The imagery was declassified in the Clinton administration. The CORONA, ARGON, and LANYARD satellites exposed more that 2 million feet of film between 1959 and 1972. You can get the imagery and more information from USGS.

GeoPDF and GeoPS with Adobe-Style Payload

2009-02-13T13:05:00.006-05:00

Here are the companion (Geo)PostScript and GeoPDF maps to those generated for the worked example. It took me less than 10 minutes to map the TerraGo sytle to the Adobe. We'll delve into the details of what I did in a subsequent post.

Isn't that PostScript GeoPS?

2009-02-13T07:59:00.002-05:00

The PostScript file from georegistration worked example post contains a full georegisration. If you put some metadata in there to reliably extract that information, well, that PS is GeoPS!

What's up with the parens in the GeoPDF CTM?

2009-02-13T07:14:00.004-05:00

In the georegistration worked example post, we calculated a CTM that takes the page coordinates in points into projected coordinates in meters:


[35.28267 0 0 35.28267 205188.64 3207094.8]

In the PostScript and PDF files, there are CTM entries:


[(35.28267) (0) (0) (35.28267) (205188.64) (3207094.8)]

Same numerical values, just the values in CTMs in the files have parentheses around them. What's up with that?

PostScript and PDF share a similar object system that provides a rich set of types. We'll go into the details of the PDF object system in a later post. However, we'll talk about the types in the two version of the CTM. There are three types present: numbers, strings, and an array. Arrays are delimited with the square backets []. Strings are delimited with parens (). There are actually two types of numbers: integer and real. These types were implemented in earlier versions of Acrobat as 32 bit objects. Real numbers used a fixed point scheme that traded precision for range. That range was too small to hold values often used in geodesy. More recent versions of Acrobat used IEEE 754 single-precision floats, the precision of which is not sufficient store values for geodesic calculations. Rather than try to shoehorn a syntatic extension that used native PDF numbers directly, we stashed the values into strings and extracted the values based on whether a string appeared in a numeric context.

Bonus link: David Goldberg's What Every Computer Scientist Should Know about Floating-Point Arithmetic [PostScript].

GeoPDF Map for Worked Example

2009-02-12T14:20:00.004-05:00

As promised, I created a PostScript file using the information we calculated in the worked example post. I created a GeoPDF file by running the PostScript file through Ghostscript to create a ~~PostScript file~~ PDF file that looks like this:

PostScript hacking is fine companion to cubicle lunching...

HT Klokan!

Georegistration: A Worked Example

2009-02-12T08:09:00.002-05:00

We're going to work through making a quad sheet and georegistering it. The quad is going to be [90W 29N 89W 30N]. If you're wondering why that quad out of all of the possible quads, it's because I know that 90W 30N hits one of my favorite spots: New Orleans. Let's pick a scale for our quad to be 1:100000, and let's give ourselves one inch margins. We're going to use UTM zone 16 WGS 84 as our coordinate reference system, and I'm going to use GEOTRANS to do the calculations. The the table of longitudes (λ), latitudes (ϕ), eastings and northings is:


  λ  ϕ  E(m)    N(m)
-90 29 207729 3211697
-90 30 210590 3322576
-89 30 307085 3320469
-89 29 305179 3209635

The bounding eastings and northings are:


[207729 3209635 307085 3322576]

Taking the difference between the upper right and lower left values and scaling the result leaves us with a rectangle 0.99356 m by 1.12941 m. To put us into PDF world, we're going to convert the meters to points and add 144 points to the result to give us our one inch margins. The resulting media box is (rounded values):


[0 0 2960 3345]

The neatline falls an inch (72 points) inside the page media box at:
[72 72 2888 3273]
What we've done here is aligned the axes of the projected and page coordinate systems. Moreover, we constrain the scales to be constant regardless of direction. This let's us reduce the problem from solving:


|a c H||h|   |x|
|b d V||v| = |y|
|0 0 1||1|   |1|


|a 0 H||h|   |x|
|0 a V||v| = |y|
|0 0 1||1|   |1|

The solution is given by:


a = (x1 - x0)/(h1 - h0)
H = x0 - a h0
V = y0 - a v0

Using the values:


x0 = 207729
y0 = 3209635
x1 = 307085
h0 = v0 = 72
h1 = 2888

yields:


a = 35.28267
H = 205188.64
V = 3207094.8

In PDF notation, the full CTM would be:


[35.28267 0 0 35.28267 205188.64 3207094.8]

This matrix defines the relationship between a point on the page and a point in the projected coordinate system. It's inverse takes the projected coordinate system into the page coordinate system:


[0.028343 0.0 0.0 0.028343 -5815.5645 -90897.17]

Using this, we can discover where the corners of the map neatline will fall:


  λ  ϕ  E(m)    N(m)  H(pt) V(pt)  
-90 29 207729 3211697   72   132
-90 30 210590 3322576  153  3274
-89 30 307085 3320469 2888  3215
-89 29 305179 3209635 2834    74

In a later post, we'll draw this map...

Why Two GeoPDF Specs?

2009-02-11T17:51:00.005-05:00

An anonymous reader asked

I'm not sure that the community needs two specs. What am I missing?

This is a hot question, but I hope I can cool it off a bit. The community doesn't need two georegistration techniques, I suppose, but it does have two, so as software providers we need to deal with it. And that's only for PDF. How many ways can you georeference, say a TIFF file? Well, there's world file, at least three different combos of GeoTIFF tags, different metadata payloads cooked up by GIS vendor X, etc. DGN? DWG? DOC? XPS? FOO?

What TerraGo has done with releasing the georegistration technique it used to create and have had created hundreds of thousands of GeoPDF files is to document what we did. This is why it will be published as a best practice rather than a OGC-endorsed specification. People have invested heavily in GeoPDF map libraries that use that technique and we're going to make sure that investment is protected. Similarly, GeoPDF files can be created using the extensions that Adobe has cooked up. We can deal with that, too. Ultimately, I hope that the people who ultimately use, enjoy, and profit from GeoPDF maps and imagery neither know nor care about the details of the arrangement of the bits and bytes that make the georegistration -- just rest easy in knowing that GeoPDF is built on open standards.

If you really get down into the details of how the two techniques differ, there are some contexts which one might be preferable to the other -- and vice versa. Neither is fully baked, IMO, so there will be extensions and revisions. However, GeoPDF's situation is much more akin to GeoTIFF's than a VHS v. Betamax. It's more a matter of flavor than variety. From my perspective, both flavors are delicious. From a GeoPDF user's perspective, it all should "just work".

I will be delving into the details of both techniques in subsequent posts.

GeoTIFF CTM in ASCII Art Glory

2009-02-10T08:20:00.004-05:00

Did I mention that I love the GeoTIFF spec. Allow me to reproduce for you the ASCII art version of the section on it's CTM provision:


ModelTransformationTag
      Tag  =  34264  (85D8.H) 
      Type =  DOUBLE    
      N    =  16
      Owner: JPL Cartographic Applications Group

This tag may be used to specify the transformation
matrix between the raster space (and its dependent
pixel-value space) and the  (possibly 3D) model space.
If specified, the tag shall have the following
organization:

      ModelTransformationTag = (a,b,c,d,e....m,n,o,p).

where

        model                              image
        coords =          matrix     *     coords

        |-   -|     |-                 -|  |-   -|
        |  X  |     |   a   b   c   d   |  |  I  |
        |     |     |                   |  |     |
        |  Y  |     |   e   f   g   h   |  |  J  |
        |     |  =  |                   |  |     |
        |  Z  |     |   i   j   k   l   |  |  K  |
        |     |     |                   |  |     |
        |  1  |     |   m   n   o   p   |  |  1  |
        |-   -|     |-                 -|  |-   -|

Some of dat!

GeoPDF Map Frames

2009-02-10T08:17:00.001-05:00

We've discussed the concept and mechanisms of georegistration -- the alignment or registration of two Cartesian coordinate systems (one of which is a projected coordinate system (PCS)) via a coordinate transformation matrix. We take this explicit relationship between a PDF page and a specific PCS, give it a name, delineate a region on the page for which the relationship is valid, and name a preferred display coordinate system, and roll it all up into what we call a map frame. I don't now remember from whence the name, but this one turned out OK, IMO.

So here are the map frame pieces:

map frame description or name
map coordinate system
preferred display coordinate system
page area of map (bounded by neatline)

You'll notice that there isn't anything particular to PDF in there. GeoPDF georegistration and map frames, although talked about here largely in PDF jargon, are by design not solely applicable PDF. You could georegister a portion DVI, AI, (E)PS or HTML page among others, or any file that references some Cartesian coordinate system like DGN or DWG. GeoTIFF has a provision for a CTM that maps pixels to a projected coordinate system. Lovely spec, that. I strongly encourage everyone to read it and profit from the work of Niles Ritter, Mike Ruth, and the GeoTIFF crew. I guess what I'm getting at is if there is a map or image in bits and bytes format, it should be georegistered! Perhaps this post should have been titled The Map Frame Manifesto...

A final note about map frames is that they are only implicitly tied to the content in or image on the page (we'll that's not exactly true with a variation of Adobe's georegistration technique, but I'll leave that for another post...). A page can have several map frames and they can "stack". For instance, maps in that road atlas under the Hanna Montana coloring book and crayons in the back of the car has a map of Utah in it, with insets for Salt Lake City and Provo obscuring Colorado and Nevada. In GeoPDF, that map would have three map frames: a base map for Utah and adjacent states with two insets on top for SLC and Provo, respectively. Map frame stacking is an important implementation detail, because it controls how you determine what you're looking at when, say doing roll-over detection for mouse movement. More on this, later.

So, georegistration is what makes GeoPDF GeoPDF?

2009-02-09T19:25:00.002-05:00

Well... maybe yes, probably no. Or vice versa. It's become confusing. When TerraGo products create a GeoPDF file, they do more than just take data, make a picture of it, and stuff a little georegistration payload in the result. For instance, as Adam has pointed out, we take advantage of PDF features to provide a richer, more interactive, PDF file. However, many features often associated with GeoPDF are not unique to GeoPDF. Layers are not unique to GeoPDF. Object data are not unique to GeoPDF. These are "just" features of PDF that anyone with the time and patience can implement. ESRI, Bentley, Safe, and Cadcorp have all done it, although I'm not certain they all embed georegistration payloads.

In my mind, it's not GeoPDF without georegistration, that's for sure! As we work through the history and some of the implementation details in the history of GeoPDF, maybe we can sort this out to everyone's satisfaction.

Just where are Adobe's geospatial extensions to ISO 32000?

2009-02-09T13:44:00.002-05:00

Reader GCP Maps provides a pointer to the where to actually see the Adobe extensions to ISO 32000, which I'll repeat here: Section 8.8.1 Geospatial Features in Adobe Supplement to the ISO 32000 BaseVersion: 1.7 ExtensionLevel: 3 [PDF]. That's page 49 if you want to jump right to it...

The GeoPDF CTM and Units: An Implementation Note

2009-02-09T08:58:00.002-05:00

Like many projects, GeoPDF escaped from the lab before it was fully developed. An implementation detail that may seem bizarre at first glance are the units of the parameters CTM which defines the a GeoPDF georegistration. Recalling the CTM:


[ a b c d H V ]

the parameters a, b, c, and d are in units of meters per point. Parameters H and V are in meters. A point in PDF is defined as 1/72 of an inch. What's up with that?

If you take it all the way back, it goes back to the arcana of printing. We'll take it back as far as PostScript, where a unit in what's called default user space was defined to be a point, and a point defined as exactly 1/72 of an inch. See the Red Book [PDF] for more. The point unit default user space lives on in PDF, and, in GeoPDF.

Many of the interfaces for dealing with PDF pages take and yield values in points. The GEOTRANS projection engine we used at the time works in meters. The points-in-meters out (and vice versa) interface was convenient for bridging the two worlds and helped to make the georegistration concept -- this part of the page is "registered" to this part of the world -- concrete.

There's even a reason it's meters per point rather than the other way around. The eastings and northings associated with projections used for many practical mapping applications tend to be "large", often ranging around the hundreds of thousands or millions of meters. Points tend to range from zero to the low thousands for most practical page sizes. A ratio of points per meter often yielded fractional values that could not be encoded as a in a PDF file as a PDF number without significant loss of precision. Also, the H and V often took on values that could be recognized as being associated with this or that coordinate reference system. We ran into other issues with limitations of PDF numbers, the solution to which we'll leave for another time.

GeoPDF and Georegistration

2009-02-06T13:11:00.002-05:00

One of the key concepts behind GeoPDF is georegistration. We define georegistration as a specific form of georeferencing in which a projected Cartesian coordinate system is registered to a page Cartesian coordinate system using a linear transformation. That is, the relationship between a point in a map to a point in a projected coordinate system is given by:


x = a h + c v + H
y = b h + d v + V

where (x, y) is the projected coordinate, (h, v) is the page coordinate and a, b, c, d, H, and V are the parameters that define the linear relationship. In matrix notation, this can be represented as:


|a c H||h|   |x|
|b d V||v| = |y|
|0 0 1||1|   |1|

We call the matrix a coordinate transformation matrix, or CTM. We encode this in a more compact form in PDF object notation as:


[ a b c d H V ]

where the square brackets delimit an array. More on PDF object notation later, but Adobe's PDF Architect and Senior Principal Scientist Jim King's tutorials give a nice overview.

Who does geospatial PDF?

2009-02-06T07:49:00.003-05:00

I know that Adobe, ESRI, Safe, and Cadcorp all roll their own geospatial PDF files using Adobe's geospatial PDF extensions. Let me know if you or some one you know does as well.

Adobe groks GeoPDF.

Today GeoPDF files are created from software from TerraGo (naturally!), Adobe, ESRI,Cadcorp, ERDAS, Intergraph, and BAE Systems.

GeoPDF files are consumed by software from TerraGo (naturally!), Adobe, and ESRI.

Since we support Adobe's geospatial PDF extensions and are going to use them under the hood moving forward, the distinction between the georegistration techniques from an us-v-them perspective are, more or less, moot.

How do I get my hands on spec?

2009-02-06T07:33:00.003-05:00

If you would like to create a geospatial PDF file, I strongly recommend georegistration using Adobe's extensions to ISO 32000. Apparently, OGC 08-139r1 has not been posted yet to the OGC best practices repository. I'm lobbying to post it on our corporate website... we'll see how that goes. If you want to get your hands on the spec to get cracking making an indexer or something like that, let me know and I'll get you a copy and some implementation notes. If you want to just see how we did it, open the PDF in a text editor and search for LGIDict... I would like to start dissecting what we did and why on this blog in a series of posts.

Closing out Bug 1584 "Make GeoPDF Open Source"

2009-02-05T16:43:00.003-05:00

Customer IANT logged the following enhancement request:

I know it's not going to happen but had a request to make GeoPDF open source and allow how we store all of the georegistration information publicly available.

IANT, it happened. Bug 1584 is closed, resolved "fixed". Cheers!