Get some data and import it

Grab populated places points

wget http://edcftp.cr.usgs.gov/pub/data/nationalatlas/citiesx020.tar.gz

Grab US counties map:

wget http://edcftp.cr.usgs.gov/pub/data/nationalatlas/countyp020.tar.gz

Lots more data available in the National Atlas

Extract shapefiles from the two archives

tar -xzvf citiesx020.tar.gz
tar -xzvf  countyp020.tar.gz

Ogrinfo the files to have a look at them

ogrinfo citiesx020.shp
INFO: Open of `citiesx020.shp'
      using driver `ESRI Shapefile' successful.
1: citiesx020 (Point) 

ogrinfo countyp020.shp
INFO: Open of `countyp020.shp'
      using driver `ESRI Shapefile' successful.
1: countyp020 (Polygon)

Notice that the cities shapefile is a point layer and the county layer is a polygon layer.

Import the two shape files into your PostGIS db

To get started with PostGIS see my first tutorial.

ogr2ogr -f "PostgreSQL" -s_srs EPSG:4269 "PG:dbname=mydbname" countyp020.shp -nln us_counties
ogr2ogr -f "PostgreSQL" -s_srs EPSG:4269 "PG:dbname=mydbname" citiesx020.shp -nln us_cities

Make sure you change mydbname to your database’s name.

Checking the Import

psql mydbname
mydbname# \d us_cities
                                   Table "public.us_cities"
    Column    |       Type       |                          Modifiers
--------------+------------------+-------------------------------------------------------------
 ogc_fid      | integer          | not null default nextval('us_cities_ogc_fid_seq'::regclass)
 wkb_geometry | geometry         |
 citiesx020   | double precision |
 feature      | character(27)    |
 name         | character(48)    |
 pop_range    | character(21)    |
 pop_2000     | numeric(8,0)     |
 fips55       | character(5)     |
 county       | character(55)    |
 fips         | character(5)     |
 state        | character(2)     |
 state_fips   | character(2)     |
 display      | numeric(1,0)     |
Indexes:
    "us_cities_pk" PRIMARY KEY, btree (ogc_fid)
    "us_cities_geom_idx" gist (wkb_geometry)
Check constraints:
    "enforce_dims_wkb_geometry" CHECK (ndims(wkb_geometry) = 2)
    "enforce_geotype_wkb_geometry" CHECK (geometrytype(wkb_geometry) = 'POINT'::text OR wkb_geometry IS NULL)
    "enforce_srid_wkb_geometry" CHECK (srid(wkb_geometry) = -1)

You can see that ogr2ogr has imported the shapefile, created a primary key (ogc_fid) and created a spatial index. What you can’t see is that ogr2ogr was also nice enough to add an entry to the geometry_columns table so that postgis tools know that there is a geometry column (wkb_geometry) in this table.

Looking at the Data

Lets fire off qgis and see what we have

Next time, querying with spatial predicates….I promise it won’t take six months for the next post.

mkdir ${HOME}/macports
cd ${HOME}/macports
wget http://svn.macports.org/repository/macports/downloads/MacPorts-1.6.0/MacPorts-1.6.0.tar.gz
tar -xzvf MacPorts-1.6.0.tar.gz
cd MacPorts-1.6.0
./configure --prefix=${HOME}/macports --with-tclpackage=${HOME}/macports/tcl \
--with-install-user=${USER} --with-install-group=${USER}
make
make install

You will now have a macports directory in your home directory:

$ ls macports
bin     etc     include lib     libexec man     sbin    share   src     tcl     var     www

Add ${HOME}/macports/bin to your path and you’ll be able to run “port install *portname*” ports you install will get installed in your home directory. Handy for any little tools you need to have available that might not be built on the system you’re using.

sudo apt-get install gdal-bin

Once you have GDAL/OGR installed you get a slew of command line utilities, I’ll try to cover some others in later tutorials, but for now we’re interested in ogr2ogr. ogr2ogr converts between the vector formats that OGR understands.

Example using a shapefile at the city of chicago website:
Download the data

wget http://egov.cityofchicago.org/webportal/COCWebPortal/COC_ATTACH/TIFS2008.zip

Unzip it

unzip TIFS2008.zip

Now we have to do the hard part (not really that hard, but important), look for the projection information in the metadata. I looked in the tifs.shp.xml file and found that the projections is:

NAD_1983_StatePlane_Illinois_East_FIPS_1201_Feet

Google uses WGS84 spatial reference system.
Now we have to lookup EPSG codes that OGR understands for these projections. A good spot to do this is spatialreference.org. EPSG codes provide a short form of expressing projection and spatial reference information.
Once we have that all sorted out we’re ready to run ogr2ogr:

ogr2ogr -f "KML" -s_srs "EPSG:102671" -t_srs "EPSG:4326" tifs.kml tifs.shp

The -f “KML” specifies that we want the output in KML. -s_srs is the source (tifs.shp) spatial reference system and -t_srs is the target spatial reference system found at spatialreference.org, then we specify the output file tifs.kml and the input file tifs.shp. That is it!

Lets shrink it down into a kmz (compressed kml) so that it takes up less disk space.

zip tifs.kmz tifs.kml

Hmmm…I wonder what one can do with this? A development indicator, surrogate for population? Very interesting. This anthropogenic interference gets in the way of soil moisture detection over land, but I wonder if it can be used creatively for something else.

First we need to make sure we have the necessary packages installed.

apt-get install tesseract-ocr sane-utils

The tesseract-ocr package gives us a utility called tesseract which takes a TIFF file as input and will output the OCR’d .txt file of the tiff.

tesseract my.tif output

Now we need a command line method to grab the TIFF, sane-utils comes to the rescue. The command “scanimage” from sane will let us do that. It is a great little utility that I recommend reading up on to learn more about its features and options, as they may vary based on the type of scanner you have. My scanner has an Auto Document Feeder (ADF) so be aware that my instructions are specific to an ADF scanner.

This example is for scanning a letter sized piece of paper in batch mode saving output in the format of a TIFF

scanimage -y 279.4 -x 215.9 --batch --format=tiff --mode Lineart --resolution 300 --source ADF

This will output a new TIFF for each page that is scanned.

The below script combines several steps to output a single PDF document and .txt file for a scan job.

outname=$1
startdir=$(pwd)
tmpdir=scan-$RANDOM
 
cd /tmp
mkdir $tmpdir
cd $tmpdir
echo "################## Scanning ###################"
scanimage -y 279.4 -x 215.9 --batch --format=tiff --mode Lineart --resolution 300 --source ADF
echo "################### OCRing ####################"
i=1
for page in $(ls *.tif); do
        echo -n "Page: $i - "
        #run tesseract on each page and combine the outputs in a single file with a .txt extension.
        tesseract $page $page
        echo "---BEGIN PAGE: $i ---" >> $outname.txt
        cat $page.txt >> $outname.txt
        echo "---END PAGE: $1 ---" >> $outname.txt
        i=$(expr $i + 1)
done
mv $outname.txt $startdir
echo "############## Converting to PDF ##############"
#Use tiffcp to combine output tiffs to a single mult-page tiff
tiffcp -c lzw out*.tif output.tif 
#Convert the tiff to PDF
tiff2pdf output.tif > $startdir/$outname
cd ..
echo "################ Cleaning Up ################"
rm -rf $tmpdir
cd $startdir

I name the above script “scandoc” and it can be run by typing “scandoc myoutput.pdf” which will drop a pdf file (called myoutput.pdf) and a .txt (called myoutput.pdf.txt) file in the current directory with all the pages from the ADF. Very handy!

EDIT:
I’ve added Joe’s contributions in the comments to a gist at github.

sudo ntfsundelete -u -d /destination/directory -f -m *.jpg /dev/sda1

Run without the -u option and -d options to show what can be recovered.

On ubuntu/debian to install it do:

apt-get install ntfsprogs

To get started in ubuntu 7.04 (should work in 7.10 and debian too):

$ sudo apt-get install postgis postgresql-8.1-postgis

Next we’ll start setting up the PostGIS environment.

Then we need to set up a new database, I called mine geodb:

$ createdb geodb

We then need to bind it to the PostGIS libs with the following:

$ createlang plpgsql geodb

And finally add the PostGIS Tables:

$ psql -d mytestdb -f /usr/share/postgresql-8.1-postgis/lwpostgis.sql
$ psql -d mytestdb -f /usr/share/postgresql-8.1-postgis/spatial_ref_sys.sql

Phew….that was a bit of work.

Now we’re ready to go. We have a database called geodb and we’ve told PostgreSQL that it is a PostGIS database.

So now we can use the shp2pgsql command to import a shapefile into the db:

$ shp2pgsql -W LATIN1 -D -I myshapefile.shp mytablename | psql geodb

-W is for setting the encoding type, I had problems with many shapefiles as my locale is UTF8 and PostgreSQL seems to default to the locale. For me LATIN1 has been working better.

-D sets the SQL output to be in dump format for insertion into the database.

-I flags it to compute the spatial indexing upon import. For large complex vector features this can take awhile, so use with discretion.

All of this is piped to

This gives us a new table called “tablename”, which we can see by doing the following:

$ psql geodb
Welcome to psql 8.1.8, the PostgreSQL interactive terminal.
 
Type:  \\copyright for distribution terms
       \\h for help with SQL commands
       \\? for help with psql commands
       \\g or terminate with semicolon to execute query
       \\q to quit
 
geodb=#

At the PostgreSQL prompt you can enter various SQL commands including queries.

geodb=# \\d mytablename

shows the columns and data types of the data in the table.

Notice the column called: the_geom

This is where the GIS Objects are stored.

The sample query there would show all the data in the shapefile’s associated .dbf as well as a column called

Next time we’ll launch into some querying fun.

Onward to Part II…