Going for a walk with the dogs

Calling Image Magick from PowerShell

2017-09-22T15:59:00.000-07:00

I am old enough to know what a BAT file is. I used to know MSDOS 3.0 inside and out and created all kinds of tricky BAT scripts for performing lots of useless (and a few useful) tasks. Anyone else that has ever done this knows that you were likely at some point to be tricked by the command processor's funky syntax and semantics. You would get stuck for hours trying to figure out why something that should be obvious was not working as you expected.

Enter Windows NT and CMD. Things were much improved. There was a lot more power and the chance to create even trickier CMD scripts. But, the problem of being out-tricked by the tool was ever present. It seemed I was bound to pull some amount of hair for any non-trivial script I would create (as evidenced by minimal hair remaining on my head).

Fast forward to present day and Microsoft has deprecated all those old scripting technologies in favor of PowerShell. I love it! The days of tricky code are gone. Modern language features, predictable syntax and semantics, powerful tools to perform typical functions, it's all there... until I tried to call an EXE.

The days of me spending hundreds of hours to become an expert in any of this technology are long gone, but it still has its uses and total expertise is not needed (not to mention that Mr. Google is a close friend of mine). I have written scripts to do things I would have never imagined in a BAT or CMD. I can call remote web services with minimal effort. Two lines of PowerShell code can do what would have taken 200 in a BAT. Then I needed to call Image Magick.

Image Magick is a powerful and free tool for processing images. It does hundreds of things I can never imagine needing and a number I can see using all the time. This time I just needed to combine single pages of scanned TIFF files into a multi-page TIFF file. Magick can do this with a trivial command line statement.

magick file1.tif file2.tif filecombined.tif

Actually I needed a few options to really do what I wanted, but still pretty straight forward.

magick -quiet file1.tif file2.tif -compress JPEG filecombined.tif

I thought it would be simple to add to my PowerShell script just like calling a CmdLet. Not so much.

What I really wanted to do in PowerShell was something like the following.

magick -quiet $ListOfFiles -compress JPEG $Destination

Where my list of input files and my destination file were stored in variables. Well, it was obvious early on that Magick was having a hard time to digest the parameters I was passing. A little playing around with quotes and things seemed to get it sorted out, but I never really understood why it worked or why it did not work as the case may be. However, I was ready to put my script to its one-time use.

I rather like PowerShells Start-Transcript CmdLet. It really simplifies logging for one-time scripts like the ones I often need to create. What I did not realize was that I had accidently developed my script on PowerShell v4, which does not support this handy tool. So, I upgraded to v5 only to find that everything else fell apart.

It turns out that not really knowing why my script was calling Magick correctly or incorrectly meant that a subtle change in EXE processing between v4 and v5 PowerShell was enough to break everything. Well, I went back to my brute force method of trying all sorts of variations only to be stymied at every attempt.

Off to ask Mr. Google about my problem. Turns out that calling an EXE in PowerShell is something special and sometimes it works and sometimes, not so much. If fact, there is a long list of methods of making such calls published by Microsoft. TechNet has an article on the complex topic as do many other blogs and StackExchange questions.

PowerShell: Running Executables

Unfortunately, I could not find one article that would directly solve my issue. The Magick method of using the @ sign to get a list of input files from a text file looked promising, but only resulted in really strange Magick errors like "magick.exe: unable to open image '@z:ÿþz'". What?

What finally did bare fruit was using an array to pass the parameters to the executable along with the call (&) operator. Once I got this setup, it worked perfectly (and logically) every time. Now my code looked a little more like the following.

$MagickParameters = @( '-quiet' )

$MagickParameters += 'file1.tif'

$MagickParameters += 'file2.tif'

$MagicParameters += @( 'compress', 'JPEG' )

$MagickParameters += 'filecombined.tif'

&'magick' $MagickParameters

Simple. I realize my method of recreating the array to add new parameters is not super efficient, but performance was not a factor here and a more efficient method would have been a waste for a one-time script. Obviously, there was other code intermixed for actually getting the filenames, but I have saved you from the unnecessary complexity for this example.

Unlike my tricky BAT and CMD scripts from the past, the PowerShell solution is pretty clean once you know the trick.

Accessing the encrypted file path in ImageNow

2015-01-10T00:44:00.000-08:00

Intellectual property protection

Many years ago I took on a project to determine the protocol used to communicate between a Norand hand-held computer and a Norand 4820 printer. Norand called this protocol the Norand printer control protocol (NPCP). This was a turnkey proprietary solution with Norand controlling hardware and software. It was designed for the connection between a portable computer and printer for route accounting and sold primarily to bakery and beverage delivery companies.

As the Norand solution became older and the cost of functionally similar printers decreased well below the Norand product cost a demand opened up for third party printers that could be used in place of the Norand printers. The only problem was that the portable computers would only print using NPCP; a protocol that was unpublished.

At the time the world was starting to see the first free and open source software. Proprietary computing systems were giving way to DOS, the new Windows, Unix, XMODEM, and TCP/IP. The proprietary Norand protocol seemed like a dinosaur. After a bunch of digging through serial traces and obscure NEC uPD7810 microprocessor code I was able to create a Norand printer emulator. What became clear in this process was that Norand designers had intentionally obfuscated the NPCP protocol and the code that implemented it.

In many ways this is no different than HDMI encryption or software anti-piracy mechanisms. The intellectual property owners want to protect their property.

Content as a hostage

Wind forward to today and I have a customer that would like to extract all their documents from a document management system called ImageNow from Perceptive Software. They have added millions of pages to their ImageNow system over the years, but the annual cost for maintaining the software is prohibitively high and much lower cost and equally capable products are now available in the marketplace.

The documents, the “content”, in ImageNow is unquestionably owned by my customer. The software is installed on a Windows server, the data is stored in an SQL Server database, and the pages are stored in files in the Windows file system. A reasonable expectation for them would be to terminate their contract with Perceptive Software and be able to access their content without use of ImageNow or further cost from Perceptive. Unfortunately it seems this is not possible as Perceptive has encrypted one critical piece of information that describes the relationship between the documents stored in ImageNow and the pages stored in the Windows file system: the image file path.

I cannot think of any reason that this encrypted file path adds value for any customer of Perceptive. It appears to be in place to force end users to engage Perceptive for additional software and services to extract their documents. The content that is owned by my customer, not Perceptive, is effectively being held hostage by this encrypted file path.

Perceptive does sell an option called ImageNow Output Agent that can export mass documents and may help for exporting documents, but customers in this situation probably do not own this option.

A content lock is a fairly dangerous situation for users of document management systems. End users are left with a difficult or impossible situation if the manufacturer of the software goes out of business, gets acquired, or otherwise changes their plans to continue to support the software. My customer is in a little better position with Perceptive, but one that will require them to spend new money at a time they are trying to downsize their document management system expenditure.

Unlocking the encrypted file path in ImageWare

The good news for my customer and others in this situation is that there is a way to get their file path without engaging Perceptive or purchasing any other modules of ImageNow. They will, however, need the skills and time of a Javascript programmer.

ImageNow includes a scripting language called iScript. This is basically Javascript with some libraries specific to ImageNow. This scripting environment can access many or all of the objects stored in ImageNow.

A search of the ImageNow 6.x specific object documentation will find that the INLogicalObject provides information about the actual files stored in the file system. However, it does not contain any information about the file path. A little closer inspection under the hood of the object reveals that it does have a file path field and the value is not encrypted. It is a member of INLogicalObject. The following very simple example shows finding a single document and displaying its file type and unencrypted file path on the console.

 // get a single document  
 var results = INDocManager.getDocumentsBySqlQuery( "", 1, var more );  
 if ( results )  
 {  
     var doc = results[0];  
     doc.getInfo();  
     // get a single page for the document  
     var logob = INLogicalObject( doc.id, -1, 1 );  
     logob.retrieveObject();  
     printf( "file type: %s\n", logob.filetype ); // this member is in the documentation  
     printf( "unencrypted file path: %s\n", logob.filepath ); // this member is not in the documentation  
 }

If you would like to tackle iScript, the documentation appears to be available with an ImageNow installation. There is a small amount of iScript information on the web including a nice introduction from Blaine Linehan of Wichita State University. He has a blog on programming with iScript for beginners.

Bose Quiet Comfort 3 are the Greatest Headphones Ever!

2013-02-08T23:48:00.001-08:00

So often you read product reviews from people that just acquired the product. They have only had a limited opportunity to put the product through its paces and are really only able to give a first impression. At best you might read a review after a few months of use.

I have been using the Bose Quiet Comfort 3 headphones since 2008, nearly five years. It all started when, for the first time in my career, I did not have my own office with a door. The workplace had gone open-concept and the associated noise was a new experience. I wanted something to dim the drone and I had heard that noise cancelling headphones might do the trick.

Someone else in the office had received a set of Bose QC3s for a gift. I gave them a try and they were great; then I saw the price. That got me looking around. Surely I could do better. I started researching headphones and found a lot of reported downsides to the Bose units: they were too fragile, they used a proprietary battery, and they were expensive. I did find some Sennheiser headphones that sounded pretty good and they were half the price of the Bose, so I bought them; big mistake. They just did not cut it. They were not very comfortable and they had no-where-near the noise cancellation of the Bose headphones I had, had a chance to try.

I decided to take the $400 (plus tax) plunge when Bose had a $50 accessory deal. You would get your choice of accessory to a value of $50; I went for the spare battery after hearing horror stories of dead batteries on long plane flights.

From day one I was not disappointed. They worked just like the first ones I tried and they were comfortable enough for me to use all day long. I was expecting to be swapping out batteries everyday, but to my surprise the battery lasts forever. I listen to Radio Paradise when I work and I can play that all day long all week long on a single charge. I can make 10 hour each way round trip flights and not worry about running out of power, though I have the spare battery charged and ready to go if I need it. The airplane adapter is great cleaning up the signal on the old school planes that were built before the iPod generation.

My very first flight with the headphones was almost a disaster. They were new to my travel kit and I left them on the plane after an Air Canada flight to San Francisco. I figured I had just thrown $400 down the drain, but I left my name with Air Canada operations and they got them back to me via Montreal a few days after I returned. They come with a nice little case including a spot for a business card, so it is pretty easy to keep all the parts together and identifiable.

After about three years I noticed that the ear pads had started to wear and needed to be replaced. For about $40 including shipping, Bose sent me a new set of pads that were easy to replace for the old ones. After nearly five years one of the two cords that came with the headphones started to separate between the wire and the connector sheath. Unfortunately I could not find a replacement at bose.ca, but I found something compatible on eBay from China. For $7, including shipping, I got a new cable. It is not quite the quality of the original, but it is working just fine.

I loved my Bose QC3s from the first time I tried them and I still love them five years later. They have only needed minor maintenance and a little care. They are always put away in their custom fit case after use. They can still be purchased today. They are not any cheaper today than they were five years ago, but I must say they are worth every penny. I see they now include an iPhone/iPad adapter. My total cost over five years has been just about $500. At 200 work days per year, averaging about 5 hours per day of use, I figure I have paid about 10 cents an hour for a comfortable and quiet working environment.

While I cannot claim to have tested a wide variety of headphones, I sure think my Bose QC3s are the greatest headphones ever!

Breaking the Folder Habit : Document Management Without Shackles

2013-01-24T09:42:00.001-08:00

An Electronic Document Filing Disaster

Did you ever create folders on your computer to help you organize your documents? Did it seem like a good idea at first, but later you realized your folders did not fully categorize your ideas, so you added some more folders inside your existing folders? Did you notice that you repeated this process a few times and after a while you had so many folders that you stopped putting documents in the correct folder when you were busy, because it was too much trouble?

If any of this sounds like you, do not worry; you are not alone. Many individuals and companies have gone through this cycle; often many times. For some, the end is to purchase a document management system. These systems offer the promise of better organization for documents in addition to security, audit controls, and much more.

With a shiny new document management system installed, many library administrators proceed to setup their ideal library structure. They have had the benefit of creating and recreating the structure many times on their old shared folder based systems, so they are experts. Of course, their structure is an elaborate and accurate taxonomy of their document management requirements.

Soon they discover users are misfiling documents or not putting them in the document management system at all. Others are complaining about how long it takes to file a document. Upon investigation it turns out that some users are unsure how the complex taxonomy applies to their document or their document should be in two or more places at once. In any case, it is enough trouble that are more likely to file it on their local drive “temporarily” and get around to putting it in the document management system when they have more time.

In the same environment, when someone wants to find a document they are faced with digging through the structure to get what they want. The only other option they have is to perform an unstructured search and hope the document contents will help to reveal its location. A user wanting to find all timesheets from last week in my example screen shot above would need to drill into each project down to the timesheet folder and then look to the document name or modified date to hopefully find the timesheets with last week’s entries.

Of course, the obvious solution to finding the timesheets is to add a ninth level of hierarchy for each week and ask users to put their timesheet into the correct folder. I think you get the point.

How did we get to think this way? Do not blame yourself. Organizing files into a hierarchy started with Multics and Unix back in the early 1970’s and dramatically broadened with MS-DOS 2.0 in the early 1980’s and nearly every operating system since. We all have many, many, years of conditioning.

Easy Document Filing

You can avoid the look of befuddlement with a small change to your thinking. First, extract the metadata from your folder hierarchy. It is not as tough as it might seem at first and there is a bit of guidance here. You can organize this metadata according the types of documents you are storing; some will need more and some less. You may decide that some metadata should be mandatory and some optional.

Your document management system will almost certainly give you a place to configure this metadata; after all, that is what they are supposed to be good at. Some document management systems will directly use the term metadata. Others will substitute indexes, properties, tags, labels, and possibly others to mean the same thing. The point is that this is information that describes the document without the need to modify the contents of the document.

Folders are very good for setting access control for a variety of documents. Create as many as you need, but no more. You do not want users to need to think too hard about putting documents away. By restricting permissions for users to only areas related to their job, they may only need to look through a very small list of folders; perhaps only one!

With a metadata based approach, when a user stores a document they will be asked to identify the document type. Some document management systems may call this the document schema or document class. They will also be asked to add values for the metadata that has been specified for the document type. Rather than navigating a tree of folders to find the right place to store the document they will see one piece of information at a time and the specific location where the document is stored will be less important. Where a set of metadata values can be predetermined the user may be able to select from a list to help ensure accuracy. Depending on the source of the document and the capability of the document management system it may be possible to automatically extract metadata directly from the document.

I often hear library administrators complain that they cannot rely on users to fill out metadata. Unfortunately these are the very same users can will not search the folder hierarchy to store the documents in the correct place either. The advantage with the metadata approach is the question and answer style of collecting the metadata. In the worst case, if users are in too much of a hurry to enter all the metadata, at least they would be able to easily put the document in the document management system rather than their local drive.

An added benefit of adding metadata to a document is that the information lives with the document regardless of where the document is located. When a folder structure is used to represent meta information about a document the relationship is indirect. Moving a document from one part of the library to another will cause the original meta information to be lost and new meta information to be added regardless of whether or not that is what was intended.

Locating Documents Easily

Library administrators will often cite the ease of locating documents as the reason for creating the deeply structured folder hierarchy. The reality is that is far from the truth. When was the last time you browsed through a tree of options to find something on the web? You almost certainly have a few personal bookmarks, but you probably find most things using a search engine like Google. This sort of searching capability is the hallmark of many document management systems.

Most document management systems have a method for individuals to bookmark documents that are important to them. Other names for a bookmarking like feature include shortcuts, favorites, and virtual folders. In this case it is not up to the library administrator to predetermine what you should care about and how you should find it. The individual user gets a chance to customize their environment according to how they work.

When it comes to searching, I know many of you are thinking that Google does not always find everything you are looking for. You do not want to loose your critical documents when you cannot think of just the right search term. It is true that document management systems can often use the text contents of documents to perform a full text search, but even better results are possible when the search is looking for specific metadata. In fact, you can expect that with metadata searching your chances of finding a document will never be worse than with hierarchical folders and will generally be much better.

I use Gmail extensively and it has a very good search capability, but I can enhance it. Gmail allows me to add one or more labels (metadata) to my email message and it allows me to qualify my search by specifying the labels in addition to my text. This already provides a huge improvement on helping me find my documents and it is an enormous leap over sorting my emails into individual folders in Microsoft Outlook.

Back in the document world, users have typical ways they need to find their documents on a day-to-day basis. These need to fit with the operational processes that they support in their jobs. They tend to remain fairly constant. In my previous example an accounting clerk was looking for last week’s timesheets for all projects. They do this search every Monday to update their project accounting, so they do not want to worry about forming a special search each week. Document management systems provide a solution here as well.

Many different types of software systems containing files have a method of saving a search to use over and over again. It is a way of looking at your documents as though they were in a folder, but the folder is virtual and dynamic rather than hard wired into a hierarchy. One of the first examples of this was the saved search in BeOS. Apple MAC OS X borrowed this as “Smart Folders” and Microsoft Outlook has now has “Search Folders”. Adobe Lightroom has “Smart Collections”, M-Files has “Dynamic Views” and FileHold has “Saved Searches” to name a few.

A common complaint about using metadata instead of a folder hierarchy is that simple metadata fields cannot convey a complete taxonomy that is best reflected in a tree. Document management systems can often solve this problem by simply providing a tree structured metadata list such as the drill down fields in FileHold.

In Conclusion

A document management system will help you store and organize your documents.
Computer users have years and years of hierarchical folder training in their heads that will get in the way of implementing a better approach to storing documents. Help them get over it.
Use folders to control access not to represent meta information.
Find a document management system that gives you the capability to create folders that are virtual and dynamic based on search criteria against metadata.

Exporting Metadata from Canon ImageWare Document Manager

2012-10-19T13:25:00.000-07:00

Note: Canon ImageWare Document Manager (iWDM) 4.1 Workgroup Edition was used for this analysis, but I expect the details are the same or very similar for other editions and minor versions.

[Updated 2015-04-22]

ImageWare is like many document management systems sold by scanner companies. They tend to be focused on finding a place to store the mountains of scans the scanner companies' devices can produce. They can be difficult to use and have a number of limitations compared with general document management solutions. As a result many users of these systems would like to switch to more powerful and comprehensive products like Sharepoint, FileHold, or LaserFiche.

The problem with making the switch is getting the thousands of documents from the old system into the new one. In the case of ImageWare there are some complications. All the documents are stored in a proprietary volume block file format with a dot IMG extension. While Canon does provide a method for exporting these in their original source format it requires a fair bit of manual effort for anything but the most simple repository and it does not export the documents' metadata.

The metadata is often the most important part of these documents. It is typically created using optical character recognition when the document is scanned or it is manually entered when the document is filed. Either way it is a valuable commodity that should not be lost. The good news is that the metadata is stored in a Microsoft SQL Server database. With the right technical skills the metadata can be extracted and prepared to import into a new document management system.

iWDM stores all its files in a cabinet. Each cabinet has one or more folders. The folders can be nested. Underneath the covers the cabinets are stored in the file system the cabinets in a folder called iW DM Cabinet. Each cabinet is stored in a sub-folder named Cabinetx where the x is a number that is incremented by the system. If your document repository is in the default location on your C drive and you have a single cabinet called Accounting you would find it in the following location.

C:\iW DM Cabinet\Cabinet1

You can find this information in the cabinet properties in iWDM user interface as the actual location and names are provided.

The Cabinet1 folder would have one or more sub-folders containing your files in the proprietary IMG format. At the root of the folder you will find the all important Microsoft database files. There should be four files with the extensions MDF or LDF. The MDF files are database files and the LDF files are the data log files. The file named iWDM_Accounting_Data.mdf is your accounting cabinet database. RM_Accounting.mdf is the database used for maintaining the full text search indexes.

After all that preamble we can get to the metadata, which is stored in the cabinet database. A quick view of the iWDM_Accounting_Data.mdf file with a tool like SysTools MDF Viewer will show a number of tables. The key table is Document. There is one row in this table for each document in the system. There are three key fields that will associate the database rows with the files that were exported. The first one is FolderIndex, the second is Name, and the third is Creator. The Creator is effectively the file extension such as .tif or .jpg. However, there is a special internal Canon image format with a creator value of .image. When these files are exported they will be converted into the image format you select.

Documents are stored in iWDM in a hierarchy that looks like a Windows folder structure with the cabinet at the root level. When you export a folder that structure is maintained when the documents are put into the Windows file system. The FolderIndex is the key to finding the folder the document will be in. It is a link into the Folder table. The Folder table includes the name of the folder and the tree structure. Folders have a FolderType column that can contain one of five values.

0 - Cabinet
2 - Main trash folder
4 - Hidden user trash folder
5 - Normal folder
9 - Deleted folder

Note that trash folders are a special case of folders. They are created automatically as needed by the system. When a document is deleted it gets moved to a corresponding trash folder. It still exists in the Documents table, but it is not possible to export it any longer. When the document is deleted from the trash folder it is removed from the Documents table. The trash folder you see in the user interface contains a hidden trash folder for each user. When a document or folder moves to the trash folder the Location column gets changed from 0 to 2.

All that remains to match the document in ImageWare to the file that was exported is the filename. The filename in iWDM is stored in two columns. The base name is in the Name field and the file extension is in the Creator field.

Now that we have the basic relationship between the exported file and the database we can start to find the associated metadata. There are three sources of metadata in iWDM: document properties, system index, and user index. The first two can be found in the Document table. Document properties like author or create date are available as columns of this table. System index refers to three predefined category fields. An index to the category values for each document is stored as Category1, Category2, and Category3 fields with each document. The index provides a reference to the values in the Category table.

The user index is slightly more complicated. The Document table provides no direct link to the user index. This work is done by the DocUserIndex table that provides a multi-way link to the Document table, the UserIndex table, and then to one of several user index data tables. There are eight different types of user indexes and six corresponding value tables.

User Index Data Type	Value Table	User Index Type
fixed string	FixedStringIndexValue	0
fixed maximum string	FixedStringIndexValue	1
variable string	StringIndexValue	2
date	DateIndexValue	3
integer	IntIndexValue	4
unsigned integer	IntIndexValue	5
floating point decimal	FloatIndexValue	6
boolean	BoolIndexValue	7

There is a special case when a user index has been defined as a "selectable list". In this instance the administrator predefines the possible values when the system is setup. The user can only choose from the list when they set the value of the user index. These fields have the UserIndexValueType set to 1; all other types are set to 0. The corresponding value table contains each possible value in the selectable list regardless of whether or not any documents have been assigned the value. Boolean indexes are a special case as they are always a selectable list with the values TRUE and FALSE predefined.

As an example, the following SQL query will return all string user index values for the given document:

SELECT d.Name 'Doc Name', f.Name 'Folder Name',
u.Name 'Index Name', u.UserIndexType 'Type',
fs.Value 'Fstr (0-1)', s.Value 'Str (2)'
FROM DocUserIndex di
LEFT JOIN Document d on di.DocumentIndex = d.DocumentIndex
AND di.FolderIndex = d.FolderIndex
LEFT JOIN Folder f on f.FolderIndex = d.FolderIndex
LEFT JOIN UserIndex u on u.UserIndexId = di.UserIndexId
LEFT JOIN FixedStringIndexValue fs on di.ValueId = fs.ValueId
AND di.UserIndexId = fs.UserIndexId
LEFT JOIN StringIndexValue s on di.ValueId = s.ValueId
AND di.UserIndexId = s.UserIndexId
WHERE d.Name = 'document name' AND f.Name = 'folder name'

That is all that is needed to extract the metadata for each of your documents. Each comprehensive document management system has its own method of importing documents and metadata. For example, in FileHold you would create a document.xml file with the metadata and the document locations at the root document folder and import it using managed imports.

It is unfortunate that the documents are stored in a proprietary format. This makes it difficult to automate the entire process. It appears as if the only change to the file is that iWDM adds 54 bytes to the front of the file. It may just be a simple matter of stripping this data off, but that is an investigation for another day. If you do export the documents using the iWDM export function you will likely need to reformat or compress the output files using a tool like Batch TIFF Resizer or TIFF Junction.

[Update]

I have taken a little closer look at the proprietary IMG files. As I suspected, removing the header (55 characters) of the IMG will reveal the embedded file. At least this is true for one type of IMG file; it turns out there are two: modifiable and non-modifiable. The modifiable version contains a single file and striping the first 55 bytes off will give the original. The non-modifiable format can contain multiple files and the 55 byte rule only works if there is exactly one file in the IMG.

I did dig into the format a little deeper and there are a few interesting titbits.

The first 32 bytes of the file appear to be the volume name; a 31 character ASCII null terminated string. This is followed by 5 bytes whose purpose is not known to me. Next we have what appears to be four 32 bit integers: a pointer to the end of the file, the IMG file number, the length of the first embedded file, and a pointer to the IMG file number (always seems to be 41); little endian format. Finally we have "VU" (hex 5655).

In the case of the modifiable IMG files the image file number is translated to base 36 and used for the file name. For the non-modifiable format the image file number relates to the file number in the IMG file. The file numbers start at 0.

For the non-modifiable format there is a secondary header at the start of the second and subsequent files. It starts with three 32 bit integers. The first one is the IMG file number, the second is the length of the embedded file, and the third is a pointer to the IMG file number. As before, the header ends with "VU". I have not investigated, but I suspect there is a flag in the header somewhere for deleted files.

There is an option to encrypt the IMG file. If encryption is in place, all bets are off for recovering the embedded files.

For the embedded files the format seems unchanged with the exception of images. If the images are converted to the iWDM image format they still seem to be stored their original format. You can check the file signatures at www.filesignatures.net or similar sites to confirm this even though you have lost the original image type in the database. Interestingly binders are stored as self extracting executables. When you run the binder code it extracts a PDF file with each of the images in the binder on each page.

Capturing Metadata from Folder Hierarchies

2012-09-04T17:04:00.000-07:00

Document management solutions have come a long way from a time when dropping critical files into a network folder was state-of-the-art. Common solutions like Sharepoint, FileHold, LaserFiche, and others have been around for many years. Even with the availability of these systems, there are still large numbers of companies that store their documents in a hierarchy of folders.

Users of GUI operating systems like Windows have been trained for a few decades about how to use folders to store information. A big challenge with making the move to a real document management system is to get away from folder mind block. This is the condition where a user wants to have a folder for everything and put everything in its folder.

Using folders to organize documents creates a numbers challenges including the following:

The connection between the folder and document is tenuous. If the document gets moved or copied the information that the folder provided about the document is lost.
The visual hierarchical nature of folders provides an impediment to storing the documents. The user must find the right spot to drop the file. Folders tend to look the same; choosing the wrong one is easy. Or, one slip of the mouse and the document is dropped in the folder next to the intended folder.

Document management systems (DMS) tend to use methods other than folders for storing and retrieving documents. Metadata is the most universal of these. Other names for metadata include tags, labels, and properties.

Metadata is information that describes a document. For instance, a document could be a project plan. If the project plan document had a metadata field describing the document type and the value was project plan, it would be very easy to find this document in a search for project plans. Additional metadata could include fields that describe the project name, client name, or whether or not the document was a final or a draft version.

Some metadata is explicit and some is implicit. Explicit metadata is expressly defined for the document and implicit metadata is derived. The previous metadata fields could all be considered explicit. Implicit metadata could include the file type (Word document, JPEG image, etc.), the number of words in the document, or the last user to modify the document.

When documents are moved from a folder hierarchy to a DMS, metadata can be implied from the structure of folders. It possible to preserve this implied metadata when documents are moved to a DMS. This enables users to find documents using the same information as before while using a more efficient document repository. The following example demonstrates this. The Excel document plan.xlsx in the legacy file share has the following path:

\\fileshare\Projects\Monkey Express\PRJ089\Project Management\Project Plan

From this structure we can imply the following metadata:

Client = Monkey Express
Project Code = PRJ089
Document Type = Project Plan

A DMS has methods to import this metadata when the document is moved to the new repository. Once imported, a user can easily find all the project plans in the repository, only the projects plans for Monkey Express, or the specific project plan for project PRJ089 using the DMS search capabilities regardless of where the plan is stored in the DMS repository.

Shoot Me Now!

2012-07-01T18:02:00.000-07:00

It has been a bad week for things breaking. I found out my Nikon 70-200AFS f/2.8 lens, which stopped focusing, was not going to be repaired under warranty ($600 plus tax); the air conditioner stopped working a week after getting repaired; something got stuck in the central vacuum pipes in the wall and it lost suction; and, my photo edit suite workstation died.

Despite feeling the weight of an old Vista install, the edit suite had been running fairly well after I solved an intermittent reboot problem about a year ago. Then my wife reported a reboot while she was working the other day. I did not think too much of it until I saw it stuck on the RAID controller BIOS page the following day; one of my striped arrays had a drive with the word "ERROR" next to it. As it happened it was the OS drive, so I was completely dead.

After playing with the computer for a while it seemed clear the error was not a one off situation, so I went looking for the Western Digital drive diagnostics (for my Raptor drives) to be sure what was working and what was not. I needed a DOS bootable USB stick to run the thing, which turned out to be an annoying problem to create from my other Vista computer. I eventually got it sorted out and determined that one drive was dead (time to see how the 5 year WD warranty works) and the others were fine.

I decided a quick fix would be to simply loose the broken RAID array and go with a mono drive for the time being. I went to grab my Vista install disk only to be reminded that I had originally purchased a 32 bit OEM disk and downloaded the 64 bit version, which I then installed. Not only could I not find the backup DVD I created for the downloaded version, but I could not find anywhere at Microsoft where I could download (no MSDN subscription) a new copy. Since I was having to go through the pain of a clean install anyway, I thought I would say goodbye to Vista and hello to Windows 7. Off to buy a new license.

Up to this point, my elapsed time on this whole project till now is about 3 days. Mostly because I have had other things to do, but there has also been a fair bit of time sitting in front of a screen waiting and or scratching my head. I was pretty happy when I could finally say I had all my ducks in a row and could begin my clean Windows 7 install. After a fairly quick book from the install DVD, I was surprised to find that it seemed to know all about my Intel(R) Matrix Storage Manager RAID controller as all my drives appeared on the install menu. I saved finding and loading the drivers. My pleased look soon wiped away as a new problem seemed to be forming.

Two of the four partitions displayed had a note that said they were not compatible with being a Windows system drive. This was okay as this note was not present on the one I really wanted to use. However, when I tried to go to the next install step I got an error: Setup was unable to create a new system partition or locate an existing system partition. It seemed simple enough at first until I realized that no deleting, formatting, or creating new partition from the menu would help me.

A quick search on Google demonstrated that I was not alone, but nothing suggested seemed to help. I thought there may have been an issue left over from the RAID controller. I used the WD diags to clear the drive. SHIFT-F10 brought me to a command line where I could use DISKPART to try and clean things up, but no help there either. There were recommendations to disconnect all other drives, but that did not seem to help. I installed a drive I had not been using thinking there may still be some residual RAID "effect", but no.

Many hours and many more reboots had passed. I was just about to dump the SATA ports on my ASUS motherboard and find an old IDE drive to try when I thought I would give Google one more shot. I dug a little deeper through the search this time and found more of the same stuff, but one very short post solved my problem. It repeated an old refrain, "unplug all your other drives", with one minor difference. In parenthesis it said "external" and "flash". I looked down and saw that my USB stick was still installed from days before when I was still testing the failed drive. I pulled it out, clicked NEXT and nearly fell off my chair when the Windows 7 install continued! I guess a drive is a drive is a drive. (I did not need to unplug the DVD drive with the install DVD in it though, so maybe all drives are not truly equal). Doh!

As a side note, I found a great piece of software during this ordeal. TestDisk is used to repair problem drive partitions among other things. I had to break a RAID 0 array in the course of things and TestDisk repaired it perfectly. I would tell you about how I solved the making-a-bootable-DOS-USB-stick-under-Vista problem, but it was more fluke than process and I had no energy to go back and recreate it, so I will be labeling and keeping my DOS stick intact.

The Shaw Gateway Six Months On

2012-04-12T23:31:00.001-07:00

We have had the gateway and three portals for about six months now. My first impressions of it’s-just-okay have not changed for the most part, but there are a few things I love and a similar number I hate.

Things I Love

Let’s start off with the good.

The multi-room PVR is huge. No more contention to watch recorded shows. Starting a show in the theatre then taking it into bed is great.
There is only been one time where six tuners was not enough. It was a rare occurrence where there were several shows with weird start and stop times and the software was too stupid to make some common sense decisions about dealing with the conflicts in the guide.
Speedy (relatively) video on demand (VOD) interface. We watch way more VOD now (mostly no extra cost) as it is no longer a painful experience.

Things I Hate

It would be great to have a product that just impresses you; the Shaw Gateway system is not quite there yet.

Network instability; at least that is what I assume what it is. Far too often the picture freezes. It is most common when you are changing something like switching from a recording to live TV or skipping ahead on a show. It just gets stuck. There is still the occasional “Cannot find the gateway” message, but they have become less frequent.
The user interface was designed by a team of (poorly) trained monkeys attempting to copy someone else’s UI. The time-delay-action drives everyone nuts. Unclear and inconsistent menus are a constant source of error and frustration. Who’s brilliant idea was a remote with no EXIT button? There are so many issues, I will list them below.
In the summer, Shaw was talking about feature updates in fall 2011; so far, nothing. There have been two software upgrades, but they have only fixed bugs or corrected minor feature issues.

Why The User Interface Is So Bad

There are plenty of examples of good and bad media device interfaces out there. You would think Shaw/Arris could come out of the gate with something much better than what we have today.

You must wait for things to happen. The horizontal scrolling menu waits for you to land on an option for a short period of time before you are able to see the options. For example, if I want to see the channel list I need to land on the channel list option and wait. After what seems like an eternity (from a UI perspective) you see the channel list. This waiting theme is used all over the UI. It is really confusing when you are learning how to use the device and really annoying once you know.
Inconsistent menu operation. When you get into a submenu you are treated to a confusing array of options to select from. Some of the options are actions (they do things) and some are parameters (they change the behaviour of actions). Some have obvious side effects and others, not-so-much. I have included some examples below.
Round lists. I am a big fan of the round list. Who wants to get to the end of a long channel list only to find they must return from where they came. The problem with these round lists is that I can never seem to tell where I am or where am I going. Some start in the middle and some at the top of the screen. Sometimes a thin yellow line tells me where the top and bottom meet. Sometimes there is a menu block with more options breaking up the top and bottom. Which way is up and which way is down; I can never seem to tell without looking at individual list items. What order (if any) are the items in and why can I not change the order? Why does Blackhawks at Canucks come after Luck; everything else seems to be alphabetical.
It looks like a child scribbled it. I mean; really? See the Sony PS3 if you are interested in what elegant looks like.

[Update 2012-04-26]
I just discovered another annoying UI glitch. I assume it is a bug; if not it is at least an inconsistency between the portal and the gateway. There is a menu option to delete all episodes from the same program at one time, but it only works if there are 35 episodes or less. If you have 36 or more episodes and select delete all you will get a server error 3308. The only work around I could find was to delete enough episodes individually to get down to 35 or less before selecting delete all.

Menus That Make Life Difficult

A big part of the UI problems can be found in how the menus are designed. I use the term “designed” loosely as in many cases it is unclear that any thought was given to the consequences of the menu operation.

You must wait for things to happen. Okay, I already mentioned this, but it is much worse than it seems. If you try and stop a recording or delete a program the UI appears to accept your request without a problem, then after a few moments you get the are-you-sure menu. What is worse is that when you confirm the operation it appears to accept your selection, but nothing has really happened. The UI returns to normal operation, but the action did not complete. As a result you can select options in the menu that no longer make sense.

I often record a hockey game to skip through the commercials. I cannot be sure when it will end, so I add extra time to the recording settings. If the game ends on time, when I finish watching the recording is still going on, so I choose the option to stop recording and confirm it. Then I return to the menu for the program in order to delete it, but I still have the stop recording option showing. I know I stopped the recording, but the gateway seems to be working on things in the background and the menu is not updated.

Delete has a similar problem. The worst issue as a result of this is when you want to delete more than one program at a time. We may have saved three episodes of Breaking Bad until everyone has watched them. When the watching is done I want to delete the old programs. However, if I select one of the episodes to delete and confirm it, nothing happens right away. I return to the menu immediately, but if I try to delete a different episode I will eventually get the message episode does not exist. After the background stuff finishes everything works fine, but you must sit there and wait for the delay to pass.
What does close mean? Most of the meaty activity in the UI happens in the little vertical menus that pop up for many things. If you press info on a program you will get a little menu that gives you some options for what you can do to or with the program. Unfortunately the behaviour on these menus is very inconsistent.

On the info menu for a program, there is typically an option to record once or record series. Each of these options behave differently if you press the right arrow button or the okay button. Press okay and it immediately sets a recording with the default options. Press right arrow and you get to see the list of options and have a chance to change them first. That seems to make sense, however, as soon as you get used to the right arrow digging you deeper into the menu you discover that most of the time you press it you will exit from the menu completely. If you are in the middle of changing the parameters for a recording, your changes will be lost without warning.

There are other annoying things about these little menus. In some cases, items on these menus are actions that cause things to happen and others are parameters that you can change. Unfortunately there is no way to tell the difference unless you have already trained yourself in advance. It took me far to long to discover that when you drill into the record once menu in order to change the parameters, you must select record once again in order to actually set the recording. Most of the items on this menu are parameters, but there is another action item: close. As you might guess, close will cause exit out of whatever little menus you have drilled into. The only side effect will be that you loose everything you had changed up to that point. If, on the other hand, you go to a scheduled recording to make a change and close will not cancel your changes, it will confirm them. In fact, any key I use to get me out of that menu will confirm my changes.

If you bring up the info on the current live program you will get an option to close and watch (close and resume on a recorded program). I have no idea what the difference is. Both seem to do exactly the same thing. What a waste! A redundant menu option to bypass with the up/down key.
Strange things happen. I understand that the MoCA network is running on potentially shaky cables and this may have some impacts on playback performance, but does it really need to affect the function of the UI? Very small amounts of simple data should be all that is necessary to communicate commands between the portal and the gateway. In a world where I can send commands between computers in all corners of the world it seems odd that I cannot get them over 20 metres of cable in my house. The picture below is what I saw when I tried to set a recording. It seemed to display scheduling forever, then it stopped.

After a moment (the famous delay) I saw the following screen.

There is no reason that the UI cannot be quickly updated according to the functions that have been executed on the gateway. These problems are so common across the system it seems as though the software architecture has been designed in many layers with only the upper layers having responsibility for providing a good user experience.

Things the Cat Loves

The cat’s opinion does not factor into too many of my purchase decisions, but if he had a vote he would say the portal is a comfortable place to sit even if it is not as hot as the old DCT.

Photographing the Night Sky

2012-02-05T23:03:00.001-08:00

Something appeals to me about photographing the night sky. I am not exactly sure what it is, but it probably has something to do with being able to create something the casual viewer will never see. While I love pictures taken with telescopes of far off nebulas and galaxies, I do not have the equipment or late night patience to make that happen.

Copyright © 2008 Russ Beinder. All rights reserved.
Nikon D700 30s, f/2.8, ISO3200, 50mm. My camera was rested on the ground, focusing was manual (several attempts), and the shutter was triggered by directly pressing the shutter release.

Equipment

Wide field photography can be done with fairly basic equipment; you will need a camera with a time exposure feature and a method to keep the camera steady at a minimum. As the name implies, wide field images show off a large section of sky at one time. This gives many compositional possibilities for terrestrial additions to your image.

Many digital point-and-shoot cameras can open the shutter for up to 15 seconds. With the lens zoomed out as far as possible, many will have an aperture of f/3.5 or bigger (smaller number). This is sufficient to make a photograph of the night sky. You will probably need to set the ISO to 3200, but trial and error is your friend here. You can take a fair number of 15 second shots to find what works best. Noise will certainly be a problem with less expensive cameras, but can be hard to see the difference between noise and star, so no one may notice.

A digital SLR camera will give you more flexibility. They will usually have a BULB option for extended exposures and the possibility of adding a remote shutter release. Some will also have an interval timer. If you do not have a remote shutter release you can use the self timer to trigger the shutter without touching the camera.

A camera with a BULB setting will be a must if you are shooting with film as will a cable release with a lock as you will not likely have ISO 3200 film (nor do you want it). If you have an ISO 100 film you will need more that 15 seconds. Since you will record about 1/6 of light as the digital ISO 3200 you will need about 480 seconds of exposure to capture the same amount of light. Check your film specifications for reciprocity failure. That will tell you what will happen to the colour balance and exposure time. You may need to add more exposure to compensate.

Everything is Spinning

It may not be obvious everyday, but the earth is turning on its axis. We are in constant motion relative to the stars. This means one of two things to the night photographer: the stars will seem to speed across the sky or you will need a tracking mount so your camera keeps up with the stars. I do not have the latter, but there is a side effect to this approach. If your camera is spinning with the stars you have no chance to include an interesting foreground.

There is one really important fact to know about the spinning; there is a point in the sky that appears to be the center of the circle: Polaris for the northern hemisphere and Sigma Octantis in the south. It is important because if you know where the center is you can compose your shot according to the hidden arc of the stars.

Combination of 204 discrete exposures. Nikon D700 30s, f/2.8, ISO1000, 14-24mm.

There is Too Much Light

Whether it is light pollution, the moon, or the dawn, too much light is not your friend. Your nice black sky will be completely washed out by light sources other than the stars. You do need enough light to expose any foreground you many have, but anything beyond that is too much. Digital cameras offer a solution that film never could. Many digital images of stars can be combined into a single image using a tool such as Photoshop or Startrails. The trick is to find an exposure that will keep the background dark with some brightness in the stars. You can add a separate exposure for the foreground.

I walked in a large circle with a small flashlight. The light is broad as I moved away from the camera and narrow as I walked towards it. I waved the flashlight up and down some of the cacti to fully illuminate them. Nikon D700 30s, f/2.8, ISO1000, 14-24mm.

If you control it, extra light can play a part in the terrestrial portion of your image. A flash, a flash light, or just about any light source can play a part. You can light up a very large area with a single light source as you can move the light while your shutter is open.

Copyright © 2008 Russ Beinder. All rights reserved.
Passing car headlights provided the lighting for this old dead tree.

Final Notes

Stars are not the only thing that lights the night sky. If you live in an area with northern lights they can make a spectacular subject. Remember that they are constantly moving; you will need to use a fairly fast shutter speed to unless you want a smoother look.

The moon can also be a good subject. Like the stars it is also on the move. The good news is that it is very bright. An exposure of 1/250 of a second, f/5.6, and ISO 200 is all you need.

If you want to have a clear image of the moon in a shot with other night elements you will likely need to combine multiple exposures as the brightness is so different.

This image is a composite. The moon is exactly where it appeared as the sun went down, but it was exposed for a much shorter time than the rest of the image to ensure it had definition.

One warning about digital cameras over film is that the image sensor will get increasingly hot as it is energized. This means that the longer the exposure you make, the hotter it will get. The hotter it gets, the more noise that gets introduced.

Cheese Please!

2011-11-10T00:17:00.001-08:00

Everyone here loves bocconcini. We eat it on paninis, with tomatoes and balsamic vinegar, or straight out of the brine. The problem is that you pay anywhere from about $5 – 7 for about 180 grams of the stuff and often it is just okay quality. I have seen cheese made on TV plenty of times, so I thought I would give it a go myself.

I did a bit of research and found lots of recipes and lots of complexity. The biggest problem has been finding ingredients or substitutes locally (Metro Vancouver); I do not keep goats in my backyard.

I did bump into David B. Fankhauser, Ph.D on the web and he seems to be the authority on making cheese at home. Some of his recipes are included with one of the key ingredients, Rennet, from Redco Foods, Inc. under the brand name Junket. His material is pretty thorough, but it leaves a bunch of questions for you to solve yourself. I did find a “foolproof” recipe here, but it calls for using citric acid to make your cheese. I want bocconcini, not a pizza blanket, so I moved on.

As I said, no goats or other milkable animals at my house or in my neighbourhood. That means a trip to the local supermarket for the basic ingredients. The main item I was struggling to find was Rennet. There are places on the web to get it like Danlac in Airdrie, Alberta or Glengarry Cheesemaking in Lancaster, Ontario. Shipping is huge at these places relative to a tiny order for Rennet, but I found that Bosa Foods sells the tablets ($2.99 for a box of 8). This was good as Dr. Fankhauser recommends these. I am sure there must be others places around town. I will have to keep my eyes open for something closer.

The next item was plain old milk. Pasteurized is probably best for a fresh mozzarella, but most of the milk I can easily and inexpensively find is homogenized also. I guess technically I need water buffalo milk to do this properly, but I have no clue where I will find that and I did want to keep this cheap.

Most recipes are from the USA and call for whole milk. As close as I can tell this is the same as homo (3.5% butter fat) here in BC. The nearest milk was Island Farms Homo, so I grabbed a 4 litre jug ($4.99 at Langley Farmers Market). Most of the recipes call for quantities in US gallons, which is just about 4 litres.

I thought I was ready to go, but as I read a little deeper into Dr. F’s material, I realized I needed a starter. Cultured buttermilk will do as a starter, so I picked up small container of Lucerne 1.5% buttermilk ($1.29 at Safeway). The name did not say “cultured”, but a quick look at the ingredients said it contained bacterial culture. Personally I have never seen the stuff that comes directly from making butter, so perhaps there is little chance of confusion here. Dr. F also said that you may need to use calcium chloride to enrich the store bought milk, but I thought I would take a chance without this.

I will not get into all the gory step-by-step details here as you can read them at Dr. F’s site, but I thought I would let you know what did or did not work. The beginning is pretty straight forward; warm the milk, add the starter, and wait. You are not supposed to scald the milk and you definitely need to keep the burner lower than you might expect. A little patience is a good thing here. I used a digital wireless thermometer I bought for the barbeque. It worked great for this.

I waited two hours, which was the minimum time before I checked for a “clean break”. Something had definitely happened with the milk, but there was no clean break, so I waited another hour. By the three hour mark it looked as though there might be a clean break, but I could not be sure, so I waited another hour. After four hours it looked like the conditions were right.

One thing I noticed right away was that my curd did not seem as firm as Dr. F’s curd. I thought maybe the calcium chloride might have made a difference, but I also started to think I may have been keeping the milk too warm. The instructions say to keep it warm, but provide no specific temperature. There is also information that says to keep it a room temperature, but I could not say I find room temperature warm. I decided to us my oven. I warmed it to 100F then turned it off and put the mixture in. The temperature after four hours was still at 89F. Seems warm, but also certainly hotter than room temperature.

I should mention that I am fully metric where ever practical. Dr. F’s instructions are in mostly in imperial and metric, but I keep my thermometer on imperial as those are the numbers I remember when I am cooking meat.

Next I was suppose to mix it and let it sit a bit, so the whey and the curds could separate. At first it did not seem to be happening, but eventually the curds fell to the bottom and the whey sat on the top making it easy to pour off and save for making ricotta.

I was not super excited about my curds at this point. They seemed pretty weak, but I left the pot overnight for “acidification”. About 12 hours later I took a look at the batch and nothing had happened to make the curds any better than they were before. In fact, they seemed worse, almost ricotta texture. I followed some of the next recipe steps, but it did not take me long to give up and just drain it; I was left with about 400g of ricotta like stuff. I think we will have lasagne next week.

I am guessing that my fundamental problem was that I was not getting good curds in the first place. The milk was relatively fresh; at least right from the store. The same was true for the buttermilk, which I am assuming was good enough as a starter. More rennet might have helped, but I did a little research (There is another good website with cheese making info called cheeseforum.org) and found that chlorine will inhibit the enzyme from doing its work. I used tap water, so no shortage of chlorine in there as we are on Metro Vancouver water.

There is also the calcium problem I mentioned earlier. Store bought milk is pasteurized and that processes causes a loss of calcium. You can restore the calcium with calcium chloride, but this is another strange ingredient to source. You should use food grade calcium chloride. It is used to keep pickles firm among other things, but you are not likely to find it in a regular grocery store. Apparently it is used when making beer, so a brewer’s supply house might have it. Non-food grade calcium chloride is used for lots of other things, but it is probably not best to use it to make cheese.

I managed to find the right stuff at D.C. Duby where you can order it online ($6.25 + shipping), but I also found it at Famous Foods on Kingsway in Vancouver ($1.29 + gas). I want to give the cheese making thing another shot tomorrow, so I made a quick trip to Famous Foods; open till 9pm. As it turns out, it is a very cool place. Many hard to find ingredients. They even have the rennet. Nothing like one stop shopping.

So, fingers crossed. No chlorine and a bit of calcium. Let’s hope that does the trick.

My New Shaw Gateway

2011-10-26T13:55:00.000-07:00

It seemed like Shaw had been in the entertainment dark ages forever. We went digital as soon as the service was available many years ago. We waited a little while after PVRs hit the market as the price was very high and I thought I would wait until I had a high definition television and could better justify the new equipment. We got our HD TV and PVR in 2008. It did not take too long to notice the biggest flaw with the system.

In the old days we had two or three VCRs. There was nearly one on every TV. We could record a program at one location then stick the tape in any machine to watch. If we started watching on one TV we could finish watching on another just by moving the tape. You could not do that with the PVR! In the PVR world we have the ability to record many more shows much more easily, but we can only watch them in the same place we recorded them.

The Tipping Point

Telus Optik TV has the ability to record in one room and watch in another, but it has very limited capacity. You can only record two HD shows at one time, which was no better than my current PVR. Lately two shows has not been enough as more people in the house are pushing the record button.

I had been watching the development of MoCA (Multimedia over Coax Alliance) for some time as it offered the possibility for multi-room with a traditional cable system. Shaw has been talking about it for a while, but just introduced it for the first time in May 2011. It came to Vancouver and my house in July 2011. Basically it turns your point-to-point TV cable wiring into a high speed network.

Unfortunately the equipment cost was initially a little high: $598 for the gateway and a single portal and $178 for an additional portal. The other day I noticed that Shaw had an ad in the Vancouver Sun. It was advertising a Gateway with two portals for $98.90 per month.

I gave Shaw a call to find out what was included in the price, but the agent did not know anything about the offer I saw in the paper. Eventually she determined that they were advertising the regular price and they had made a misprint. It should have been $99.90.

I have seen the same ad at least twice since then and the price is actually correct, but a little tricky to confirm. If you go to their web site and price the basic HD TV package with Broadband 50 internet, remove the home phone, and add the gateway and two portals on a monthly payment plan, you will get $98.90.
One good thing I learned was that the gateway and portal prices had dropped to a more reasonable level. Now, the gateway and first portal was only $348 and a second portal $148. A quick calculation led me to determine that if I switched off my Telus account and used Shaw phone (hardwired is old school I know, but we do not have great cellular coverage in our house) we could save $15 per month including the new equipment costs. That was enough to convince me to place the order for the gateway.

Is There Such a Thing as an Easy Install?

I put the order in and prepared for the week long wait for the installer. He was intended to arrive between 0800 and 1200 and stay for about 4 hours. Fortunately he was at the house by 0830 and ready to start.

The first order of business was to setup the telephone. The Shaw phone works by adding a Telephony Modem TM602G from Arris in the garage near the Telus box which is mounted on the outside of the house. A coax cable goes into the box and a phone cable comes out and gets attached to the Telus block. The phones in the house work as normal.

It took a while, but soon enough the phones were up and running. At this point I realized I should give the installer a diagram of where all my jacks were and what I want connected in each spot to simplify the conversation we were having. He started testing the various lines (there are some dead ones in the outside cable box) and proceeded to begin the Arris MG5225G gateway install.

But wait! I forgot to put the cable for the FM feed to my receiver on the diagram. The installer told me that Shaw does not support FM, but he would see what he could do.

Things really slowed down at this point. The hold up seemed to be a result of the installer waiting on hold to the Shaw office to get things enabled. You would think he would have an app for that on his iPhone, but no such luck.

With the gateway installed he went to plug in our first portal at the bedroom TV. We had a digital terminal there before, but it was a standard definition unit. Unfortunately the TV is a bit old and only supports component video and the portals only ship with an HDMI cable. The good news was that I had some spare component cables and the physical install was completed.

Next we moved to the family room. We are planning to keep our digital terminal there for now and install a portal in the near future. Due to this he added a filter of some kind to the coax outlet near the TV. Apparently the gateway generates a signal that will mess up everything else, though I do not fully understand what it is doing as the portal in the bedroom did not get a filter and the digital terminal in the spare bedroom did not get one either, so I am at a loss to know what it is for. The same filter is on the coax going to the cable modem.

Finally off to the theatre in the basement where the second portal was installed. It went in easily enough, but he asked me if the portal was connecting directly to the TV or not. When I said it went through my Denon AVR-3808CI receiver he told me I should expect the screen to go black from time to time as there is so much compressed data going of the HDMI link that the receiver would not be able to keep up. That was not a good thing to learn, but so far there have been no such blackouts, so I am starting to wonder if he really knew what he was talking about there.

With this last portal attached, the installer proceeded to do a series of configuration steps. The TV image jittered a fair bit while this was going on, but he explained this was normal when a lot of data was being sent by the gateway and would not happen for normal viewing. Sure enough, the configuration ended and so did the jitter.

Finally we tested out the internet. As an added bonus with the phone and gateway we are also now getting Shaw’s Broadband 50 service with about 50 Mbps down and 3 Mbps up. It worked and he was done.
I had forgotten all about the new DOCSYS 3.0 modem. He installed it when he installed the gateway. Both devices are hidden away in the same cabinet. The new modem is an SMC D3GN router. I did not want a new wireless router as my D-Link DIR-655 is configured and working well. Shaw told me they could configure their router as a plain modem, so I was not too worried. However, the installer told me how to connect to the Wi-Fi and how to log into the router administration GUI. This had me a bit worried, but the internet was working, so how bad could it be.

In the end, the whole install took just about the 4 hours that were promised. The installer was telling me that technicians are complaining about some of the future changes to this configuration as eventually the gateway will do everything: phone, internet, etc. They get paid per piece they install, so this is not good for them. I cannot imagine a customer complaining about less equipment and less install time though.

Now We Can Find Out What is Not Working

Finally on my own I could finish the system setup. The first piece was the old PVR. There are a large number of shows on the hard drive and the extender and there does not seem to be any way to get them onto the gateway. As a result I was going to hang onto the old Motorola 6416 until we had watched the shows.

Shaw will give upgrades from PVRs to portals; basically one-for-one. Seems like a fair deal to swap the PVR I paid $606 for, for a $148 portal. There is a catch, you only have 30 days following a gateway install to take advantage of the upgrade, so I wanted to get this box setup quickly.

I got a bit of a surprise when I turned it on. The power had been disconnected and there was an on screen message telling me I need to hook up the cable in order to use the device. I did not need a cable as I was only planning to watch the shows already recorded, so this was also not on the cable diagram I had given to the installer. He knew what I was planning to do with the PVR, but it obviously did not ring any bells with him either.

I gave Shaw a call to find out how to override this message, but it turns out there is nothing you can do except attach a cable. Fortunately I had a long piece of coax sitting around, so now there is an ugly cable running across my floor; at least it is temporary. It turns out that after the PVR downloaded everything after power was restored I could disconnect the cable and still access the recorded shows. Maybe the ugly cable can go away sooner than 30 days.

While I had the tech on the phone I also asked about the internet performance. By this time I had, had a chance to run a speed test and I had discovered my new Broadband 50 was no faster than my old speed. It was a simple fix. When he confirmed everything was setup correctly on their side I just had to reboot my D-Link router. The speed went up to 46 Mbps down and 2.3 Mbps up. Not bad, but not as fast as when I had a Motorola DOCSYS 3.0 modem temporarily back in the spring. A bit disappointing, but not the end of the world.
Now it was time to finish up the master bedroom. When the installer had hooked it up before there had been some strange behaviour from the portal and the TV when he was trying to set the HD mode to 720p. In fairness to Shaw, I know the TV up there is a bit weird. It is a Philips 23PF8946A that we got a good deal on several years back.

I dragged out the manual for the TV and found that there were two sets of component video jacks; one for standard definition and one for HD. Things were much better when I made the switch; however, there was a problem. For some reason, every time the TV is turned on, the user must press the HD button on the Philips remote to get a picture.

The remotes that Shaw ships with the portals look pretty fancy, so I figured there may be a way to setup a macro to turn the TV on properly. The remote would not turn the TV on at all. Volume worked, it turned the TV off, just not on. I had a similar problem with the remote for the old digital terminal, but I could always set the old remote to TV mode and then power on. There is no concept of TV or any other control mode on this new remote, so it would seem I am stuck with two remotes or I need to buy a smart remote to do them both.

The Internet Is Down

Nothing worse than a call from home to tell you the computers do not work. Well that is what I got on day two. I had been having problems with the internet all day. My D-Link router would just seem to disconnect from the net randomly. Sometimes I could not even access the administration GUI on the D-Link router. I rebooted various things a number of times, but the problem remained.

By that evening the internet was completely down. I remembered the installer had been explaining how to get access to the administration GUI for the SMC router. I connected a computer to have a look and I quickly figured out that the SMC router was acting more like a bridge as the IP addresses I was getting were from the Shaw network, not the SMC router. There was no chance to get to the SMC GUI. I started pinging the Shaw network gateway and it ran for some time with no problems. So, Windows Vista computer worked; D-Link router did not. Another call and another long wait on hold to Shaw was in my future.

I am expecting the support guys at Shaw will assume it is me or my equipment that is failing, but I know that the only thing that changed in my home network in the last couple of days was all Shaw. Before I called I tried to reconfigure the SMC to give me some control, but no luck. Every time I set it to factory defaults to get assess to the GUI, the new configuration would only last for a minute or so. After that the device would reboot and be back to the old configuration.

I got the Shaw tech on the line and spent about 30 minutes to decide it was probably the SMC equipment and he would need to send a tech. When I told him the Motorola DOCSYS 3.0 modem I had in the spring worked great he said he would try to get one; either way it was going to take another week of waiting. I did not want the internet to be down for a week, so I asked if he could reconfigure the SMC to act like a router and I would switch my network temporarily.

After the switch I was able to get into the SMC GUI and nose around. Like many of these sorts of routers it has a DMZ capability, so I thought I would try and use that to connect to my D-Link router and save me messing with the network. It was a fairly simple matter. I changed the SMC network address to 192.1681.0 as it was the same address as on my D-Link. I gave my D-Link a static address on its WAN port of 192.168.1.2 and everything came up just fine (though still slow). No other changes were necessary other than to disable the SMC Wi-Fi, which is poor and I do not need it.

So How Does The New System Perform

I guess the whole point of the gateway is entertainment; so, how does it perform. In a word: okay. Sorry, not amazing or even great, but okay. It does take care of my original problem very well. It is totally transparent moving from room to room to watch recorded programs. I tried recording six programs at one time; this also worked very well. All-in-all, a huge improvement in utility, but not much else.

The picture quality is about the same. At least that is what I can see without any test equipment. The image still shows compression artifacts as it did before. The video stops occasionally as it did on the old box. The user interface is different, but it would be hard to say it is much better.

There is a guide as there was on the old UI, but it appears to be there for familiarity reasons. It is more-or-less the same as the old one except you can no longer page through the channels. You can scroll fairly fast, so I guess they figure paging is not needed.

The main interface for selecting programs is much different. You are given a variety of categories of entertainment to select from. Things like HD, music, movies, video on demand, PVR all channels, etc. are presented in a horizontal display.

One of the biggest issues I have with the interface is the time sensitive option activation. When you move to a menu option nothing special happens unless you wait there for a second or so. Most of the time the consequences of this are not too bad, but there are a few situations that are annoying. You can press a key and what you expect to happen is not what actually happens because the menu timed out and did something for you automatically.

One area of the user interface that is a huge improvement is the Video on Demand. Previously this was painfully slow, now it is about the same speed as the rest of the GUI. If they can find a way to drop the exorbitant price of a HD movie down to the Apple TV price as a maximum we might start using it.

Despite the new speedy menus, VoD still suffers from issues we saw on the old PVR such as “We apologize for the inconvenience, please try again. If the problem continues call customer service [and wait for hours]”. I added that last bit, but you know they wanted to put it on screen.

It took a little while for me to figure out that there is no longer an exit button on the remote and no obvious way to get out of the menu. It turns out that exit has been replace by zoom. If you are in a menu and press zoom you will go back to your program. A nice feature of this is that you can go right back to the same menu option by pressing zoom again. It looks like the idea of calling it zoom is that you are zooming into the ever present picture-in-picture (PIP). The biggest problem with zoom (after you figure out it replaces exit) is that it does not work on sub-menus. There you will need to pick a menu option like close or watch to get out.

There is one other problem with the loss of the exit button. There is no obvious way to clear the on screen info that is present in certain circumstance. For instance, if you pause the display to read some credits or see something else that is moving by, you cannot see anything behind the timeline display. On the old PVR you could just press the exit button and the overlaid information would be hidden. The only way I have found to do this now is press zoom then zoom again then wait for the program info bar to disappear. It works, but it is slow and awkward; a lot of work to read a Chuck Lorre vanity card.

There are some other changes on the remote like the back and next buttons for moving through help text or info. The biggest problem I have with the remote is that it seems ergonomically designed to make it easy for you to press fast forward, rewind, play, etc. rather than the arrow keys and okay. It feels good in your hand, but it positions my fingers in the wrong place for what I do most.

One of the first things I did after the basic setup was done was to setup my Harmony One remote. I have used this remote for some time and it makes a big improvement in ease of use for me. I even assigned the zoom feature to the exit button on the remote and it was just about like the old system operation. I can operate the old PVR and the new portal with just a quick push of a touch screen button and old and new works just about the same on the one remote.

I am starting to get the hang of some of the more complicated features like recording programs, but it does take some time to catch on to the idiosyncrasies. I suggest you double check that it was setup correctly. I had a program the other night record for the normal length when I thought I had set it up to record for an extra hour. It is easy enough to pick the options, but it is saving them is not as obvious as it should be.
Speaking of options, there is an option to change how many seconds the skip button jumps though a recording, but no option to change the replay duration. On the old PVR replay gave you 15 seconds, the new one is fixed to 7. I have not found an option to change the paging through a program. On the old PVR you could page back and forth 5 minutes at a time. The portal seems fixed to 15 minutes. That is not so helpful if you are trying to get to minute 25 on a 30 minute show.

There is one technology bonus I was not expecting. The portal can downscale HD programs. I found out about this totally by accident. I was actually thinking it would not be possible to watch HD channels or recordings on my old school TVs, but it works just fine because the portal does the necessary conversion to standard definition (480i) for you. This means I can start watching a HD recording in the basement HD theatre setup and continue it when I am in the kitchen on our old TV; bonus!

It was the problem we encountered with the install in the master bedroom that put me onto this feature. The installer had been unable to set the portal to the correct HD size 1080i, 720p, etc. He struggled with this for a little while then eventually said it must be working because we could see the HD channels. Apparently he was not aware the portal would downscale.

One more item that contributes to my okay rating is the periodic freezing of the skip feature. Sometimes skip does not go anywhere. You press the button, there is a small change to the timeline display to indicate a button was pressed, but it does not advance the program. This seems to clear up after a short time, but the other night I had to resort to FF to get the job done. Not only did I need to use FF, but I needed to press stop then go back and start playing again in order to move through the recording. This more extreme problem has only happen once, but I have no reason to believe it would not happen again.
Although not too important to me, the games are boring and ugly.

What Do You Get in the Box?

Arris Gateway MG5225G

The brains of the entertainment system and eventually everything else. It has a bunch of lights on the front, but no explanation of what they mean. Power is obvious, the other ones not so much. I noticed a red light came on today, but I do not now why.

The back of the unit has a number of connectors, but only two are used right now: power and coax. There is an eSata connector that I will use soon when I move my extender from the old PVR in a few weeks. There are also connectors that have no apparent use right now: two telephone (RJ11) jacks hidden behind a sticker, four Ethernet (RJ45) jacks, a USB jack on the front and one on the back, and something hidden behind a do not remove cover. There is also a reset button.

There is an IP address written on the back of the gateway, but it appears to be internal to the MoCA network. Although the Ethernet ports are electrically active, there does not seem to be anything else going on there.

A couple of manuals come in the box, but they are not much to write home about. The larger one is the list of safety warnings and license agreements related to the gateway. The other is entitled Shaw Gateway Experience: TV Like Never Before, Wow!. It is up to the normal Shaw standard of stating the obvious. Perhaps it is more useful to someone that has not owned a PVR before. There is on screen help built into the system, which says essentially the same thing.

Arris Media Player (Portal) MP2000

The client device that attaches to your TV. It is pretty basic on the outside and it is very nice and small. Not Apple TV small, but much smaller than the old digital terminals. There are a couple of lights on the front that seem pretty self explanatory: power and MoCA I presume. There is a menu button, arrow buttons, and ok. Their operation seems to mirror those on the remote. There is no power button and it seems the portals are never turned off. They do revert to a screen saver after a while.

There are all the basic ports on the back including composite and component video output. Analog audio RCA jacks along with digital coax and optical. HDMI is there, but S-VIDEO is not. Of course, power and coax are present. There is no power jack for plugging in other devices as was on the old digital terminals. There is a IR blaster port, which I have not tried. There are two USB ports, front and back, and an Ethernet port that are not in use. The installer told me the Ethernet port was working, but it is in the same state as on the gateway.

There is a reset button on the back, but it is not obvious if it works. I had occasion to try it recently when the portal lost the MoCA network the other day. I was forced to pull the plug and MoCA came back after a short while.

In addition to the missing the S-VIDEO jack, there is no RF-OUT jack (coax). I cannot imagine there are too many TVs out there that do not have a composite video input at a minimum, but if there are they will not work with the portal.

Perhaps the best part of the portal, or maybe a second to the size, is that there is no fan. I find the fan on the Motorola 6416 PVR pretty loud. The portal is totally silent.

[Update: Portal dimensions - 270 x 170 x 45mm]

Arris Telephony Modem TM602G

Not much to say about this device. It looks like it can do a lot more then is configured by Shaw, but it also seems to be a fairly old piece of equipment. I presume this will be replaced by the gateway at some point.

SMC Wireless Cable Modem Gateway SMCD3GN

Not much to say about this either. It is a piece of junk. Hopefully Shaw can replace it with a Motorola modem. Wireless coverage is not great compared to my D-Link, though I have not done any scientific testing; less bars. The administration user interface looks like it was designed 10 years ago and the features you can control are very limited.

Shaw Reps and the Call Back Blacklist

Perhaps for a last word, a comment on the various representatives from Shaw.
Shaw is yet another big utility company that holds a fair bit of control over our lives. As they expand beyond their traditional cable space other companies expand into that space. The days of them being a monopoly are gone. Clearly they need to do things that make them stand apart. There are two big items they push: customer service and no contracts.

On the customer service front they have done relatively well lately. I have spoken to many reps recently from the person that took my order, to the installer that came to the house, to the various technical support people that have helped me with my issues. Without exception they have been both friendly and appear genuinely interested to help. If they did not have an answer, they found it.

Obviously they still have a few issues. Clearly the installer could have used a bit more training on how some of the stuff works. The rep that could not explain the pricing probably was suffering from a bad job by the marketing department at Shaw. In general I give Shaw credit for better performance than similar companies.
The wait time is a bit of a problem these days. Perhaps it is the new gateway; maybe it is hard to find enough people; I do not know, but the wait times to get someone on the phone are very long. Most times I have called in the last few weeks the waits have been over an hour. I did get one that was less, but it was still over 30 minutes.

They have a call back option before you are put on hold. It works fairly well, but there seems to be a catch. I had to leave unexpectedly before my call back and there was no one at home to take the call. They tried three times before giving up as evidenced by my answering machine. However, now I never get a call back option. Have I been black listed? I can think of no other reason. Now I must listen to the horrible on hold music for an hour or more at a time. It is brutal punishment for one accidental lapse.

Conclusion

Would I do it again? YES

Do I wish it was better? YES

Do I expect Shaw to give me more in the near future? DEFINITELY

Managing Microstock

2011-10-07T22:48:00.000-07:00

Microstock is a method for selling photographs to a very large audience and get very small returns for every sale. The idea is that if you have enough images out there you can earn a worthwhile income. It is accessible to amateur photographers in a way that traditional stock photography simply is not. You need pro quality, just not pro credentials. With digital workflow the quality problem is much less than it was with film.

As far as money goes, I have not seen too much. I often hear that you need 1000 - 2000 images to realize a steady income; I have less than 100. Image royalties are often 25 cents, though sometimes they can be $20 - 40 USD. It depends on which microstock eCommerce site your photograph sells from, on the size of the image, whether or not the purchaser has a subscription for multiple purchases per day, and whether the licence is good for only adding the photos to a blog or if they will go on product packaging, etc.

I started with microstock in 2007 without much success. I had trouble getting my photos accepted on sites and they sure did not sell very much. The good news is I have figured out what images work and what do not; I know what sort of post production is necessary and I think I have figured out how to keyword. The bad news is that I post each photo to at least 10 different microstock sites.

When I started I was only on iStockphoto and Shutterstock. I used a spreadsheet to help me track what photos were accepted and which ones were rejected and why. It was pretty painful pretty quickly despite my small numbers. Then I found a tool called ProStockMaster. It handled the uploading and helped with the keywording and tracking. It is pretty quirky, but it was a huge improvement over the spreadsheet / manual approach.

I had noticed there were some cloud based services that did similar things to ProStockMaster. They did not support the sites I really cared about, so I did not look too deeply till recently. I was having problems with the ProStockMaster uploads yet again and I had just signed up with Veer which was not supported in ProStockMaster, so I thought it was time to have another look.

A Google search or two and a few blogs eventually got me to a site called PicWorkFlow. It is made specifically for microstock by a photographer that does microstock, so I thought I would give it a try as it supported Veer. You pay a penny per upload and you get a 100 penny credit when you sign up. Unlike ProStockMaster, the PicWorkFlow way uses your bandwidth only once to upload to PicWorkFlow and their bandwidth moves the files to the various microstock sites.

I have tried the tool with about 20 of my images in the last day or so. I am sold! While far from perfect it is a big improvement on ProStockMaster. I am a new contributor on Veer and DepositPhotos, so I have been uploading mostly to those site, but I have updated the tracking information for my other sites including Fotolia, Shutterstock, Dreamstime, iStockphoto, 123RF, CanStockPhoto, BigStockPhoto, and PantherMedia. Everything seems to work, but some of the site is a work in progress. It is not pulling sales data yet and some of the tracking must be done manually, but improvements are promised.

The guy that created the site, Robert Davies, says he is a workflow expert, but I think there is still some work to do in this area. When I want to put a photo on a microstock site, there are several things I want to do using PicWorkFlow. First I upload to their site, then I fix up metadata, then I decide where I want the image to go, once the images are sent I need to go to the microstock sites and finalize their submission requirements, and finally I want to track my sales. I can do most of that stuff with PicWorkFlow, but I need to manually remember or move my photos from one step to the next.

There is a grouping feature, but it is very cumbersome to use. I created groups called Prepare for upload, Ready for upload, Verify uploads, and Complete. I put the images in Prepare for upload first during the import process, so I can work on the metadata. When I have finished that step I remove the image from that group and add it to Ready for upload. I need to do this manual step, in and out, each time a photo moves along my workflow. Quirkiness continues when I finish all stages in the workflow and remove the last photo; the group disappears. There must be an easier way.

The tool also has a few quirks that take some getting used to. I am using Google Chrome browser, one of the recommended ones for this site. To start with, the top menu expands horizontally rather than vertically. It expands as the mouse hovers over the main option and disappears if you happen to move the mouse a little off the menu bar. Another strange thing is how you change the status on an image. You can set the status to untasked, pending authorization, review accepted, review rejected. It seems simple enough, but you need to click several different places to achieve the various combinations. It is not intuitive. These are just two of many. In general it seems the intent is to keep the user interface clean, and it is, but it should not be weird.

Bugs seem minimal. I have only noticed one so far. It was fairly innocuous. I was able to get the global change panel to repeat itself by clicking on it repeatedly. Other than that all has been good. I am going to stick with it and see how it progresses.

Quirks and all, PicWorkFlow is the most promising solution I have see for multisite image management and well worth the penny to costs for an upload.

A Story of an Incompetent Customer Service Organization

2011-08-24T16:11:00.000-07:00

I am sure I will not be the first one to say it, but Rogers Wireless is a tough company to deal with. I have been with Rogers since 1992. Back then I had a giant Motorola brick. It had a ten digit LED display for the phone number, a simple keyboard with large keys, and it was intrinsically safe (I guess so I could safely use it while pumping gas). This was a time when phones cost a bundle, wireless minutes cost a bundle, and Rogers was one of only two games in town. I do not remember how I chose them over the competition, but one thing I do remember was that I spent a fair bit of time on hold waiting for a customer service agent whenever I needed one.

Wind forward to 2011 and the world is a different place. My Blackberry Torch is a multi-function device with hundreds of megabytes of RAM, gigabytes of flash storage, a camera, barcode scanner (I could go on), and Rogers is a much larger company that has reduced my hold time to get a customer service agent to less than a minute. The problem is that they have settled for putting any human on the line regardless of their customer service skill and aptitude or their training and experience. Where I used to wait many minutes to get an agent that could solve my problem I now get an agent right away that will help me begin a long journey of pain and frustration.

During my latest ordeal I wanted to add my Blackberry Torch and phone number formerly from another account to my personal account. I do not want to go into all the details here, but it was three days of customer support hell. I went from agent to agent, department to department trying to get things working. I was exposed to what can only be described as extreme incompetence. I was offered things by one agent that I was told do not exist by another. I was sold a product that could not work with my equipment. Agents offered to transfer my calls to new departments and either I got the wrong department, the call was dropped (without a call back), or I reached the new department and had to start from square one with the new agent. One agent even coached me to act irate when she transferred me to the new department to ensure I got the service she thought I deserved.

All tolled, I spent no less than 5 hours talking to agents, waiting to be transferred, or waiting for the agent to ask someone (or something) how to solve my problem. Worst, this is not the first time I have had such problems. They occur to one degree or another every time I interact with Rogers. Switch phone companies you say; sounds easy enough. I spend well over $3000 per year with Rogers, so surely other companies would be interested in my business. Of course, every time you make a change to your plan you get locked into a new contract for a minimum of a year and it would cost at least $700 for me to leave with no guarantee that any of the other companies are any better; I guess I will stay for now; after all the actual product works fairly well.

Surely Rogers can do better! I know very little about the economics of wireless providers, but I must have sucked a fair bit of the margin from our relationship this year in these three days of frustration. I suspect the speed of Roger's growth, the numerous and ever changing products to remain competitive, and high agent turnover are probably all factors in the problems that I have suffered. I have worked as a lean management professional and it is clear that the ratio of waste to value added activity at Rogers is extreme and must be a negative factor to their bottom line. Revenue was up this last quarter, but income was down. Perhaps Roger's shareholders would appreciate a change as much as I would. Maybe their call centers are a minor cost in their operation and they really are not too concerned about all this, but they should not expect a positive net promoter score from this consumer. I would not wish Roger's customer service organization on my worst enemy.

Taking the Next Step for Great Fireworks Photos

2011-08-12T17:17:00.000-07:00

I was recently giving some advice to someone new at photographing fireworks. After he got the basics of a steady tripod and focusing in the dark figured out, he asked what he could do next.

I have taken photographs of fireworks since I was a kid, but I would not claim to have gotten photos I really loved all that time. Over the last decade I have thought a bit more about this and what it would take to get photos that might say "wow" to me.

While it is nice to get a bunch of fireworks in the frame it is often kind of boring.

Try to find an interesting foreground/background to go along with the show. It especially works well when the bursts light up or accentuate your subject.

Do not be afraid to get a bit more light onto the sensor. You do risk blowing out the colours, but if you can avoid that the results can be really nice.

Crop a bit tighter. You can do some of this in post, but you will reduce the pixel power of your camera when you do this. It is tricky, but try to guestimate the height of the bursts. I typically have a zoom lens on the camera and I play with my tripod positioning throughout the show.

Finally, a nice composite can often create the shot you wanted, but could never actually get. My tripod was in the same position for each of the shots, but I could have never exposed the couple correctly at the right moment and got the right number and type of bursts in the frame at the same time (though I did try). The final result combined several images in Photoshop using layer masks.

iPods and Incremental Innovation

2011-07-27T16:52:00.000-07:00

This is a republishing of a post I made on a company internal system last year May 3, but I thought it deserved a life on its own outside. It was a comment to a post called On Concept Cars, Software, and Gunter Dueck about innovation at SAP.

I was speaking to a group recently about capturing customer requirements in an effort to determine what the customers valued, then use that understanding of value to produce products and services. One of the attendees piped up that you would never invent an iPod like that; point taken. What customer would ask for an iPod when they have never seen the concept before? Kane Kramer, that’s who!

Kane Kramer is an inventor. He invented the iPod before anyone knew it would be called that. He also invented iTunes while he was at it. Unfortunately he was about 20 years ahead of his time. His pre-FLASH, pre-internet, pre-MP3 prototype could hold about 3.5 minutes of music and the users had to go into a record store where they could purchase and download the latest hits from a central server.

By the time of the turn of the millennium, digital music was the Wild West. Napster and similar file sharing systems were driving the record companies nuts. MP3 players were coming out from everywhere. For the most part they were clunky and tough for the non-techie to use. Steve Jobs decides he could do with digital music players what he did with the PC: make one that looks and feels good. So he did; nothing revolutionary though. You could certainly recognize that original unit as an iPod, but it was huge, expensive, had a very low resolution monochrome screen, and mechanical scroll wheel.

We look at the iPod now and say, “wow, who could have thought of that?” The reality is that it was nine years of innovating a line of products (19 in all, that’s about 2 per year) in an incremental fashion, each one building a little on the one before or addressing a little different market than the one before. It was not even an original idea! Not only had plenty of MP3 players come before, but Kane Kramer invented it all 20 years earlier.

I have never worked at Apple, but I can guess at a couple of things they did right to get to where they are today with the iPod:

I bet they have a formal product innovation process that has a big funnel to allow lots of little ideas to germinate.
Steve has a vision and he keeps the product moving along that vision. How many successful products before the iPod did not let you change the battery? Look at the current battle with Adobe and Macromedia’s FLASH technology. Steve does not need it for his vision, so it does not let it compromise his product.

Can we institutionalize the “process of innovation”? Short answer, of course, we must!

In order to work, an innovation engine must produce; frequently. We (SAP) are after all building software. As a software engineer I have no delays while my dies are getting formed at the tool makers so I can shoot my plastic parts as I would if I was making something hard. I do not have to wait to cut multiple iterations of circuit boards. I do not have to work with the packaging companies, physical distribution channels, etc. Anyway, you get the point. Nothing restrains software products from being created quickly except us.

If you want to get more out of the formal product funnel, you have got to put more in. In sales you do not get one lead and hope to get one sale from it. You need to get ten leads and try and push them through different stages of the funnel to get one sale. Same goes for innovation; let’s throw more pucks at the net (sorry, Stanley Cup playoff time). I do not know the conversation ratio, but it should not be too hard to figure out. We want X new product innovations a year, we need at least Y (Y > X) ideas in the front of the funnel.

Finally, do not be scared to innovate. We have lots of products in the market, so we know all the problems that new products are likely to cause. As a result we are very, very careful about how we get them to the market. I remember when I was on my own building software. I could get new stuff to my customers in no time. I was young and fearless (I hate to look back at how ugly that code was, but somehow it worked). Later, when I had some experience I knew what pain I would have when I made a mistake, so I avoided mistakes. This meant adding fewer features to my software less frequently. I crippled my ability to innovate. I let my company become structured in a way that had no tolerance for risk of any kind.

Mine was not the only company in this position. Before the mid 80’s, creating a new car at Porsche was a bet-the-business proposition. No company structured in such a way will ever have a process of innovation. It is too scary. Let’s institutionalize a process of innovation that balances risk and reward, allows the funnel to become full of ideas, and delivers the best ideas to market frequently.

Street Photography

2010-03-14T11:49:00.000-07:00

For me, the most fun I have with photography is finding a view of a subject that no one else sees. It could be a shot of something truly unique or more likely an unique look at something very commonplace. Candid photography can be a great source for that "unique look at something very commonplace".

Merriam-Webster says candid is all about photographing subjects that are acting naturally or spontaneously.
A facial expression, an awkward moment, and odd situation, or juxtaposition of interesting characters; these all scream to me to be photographed. There are plenty of great places to find that one-of-a-kind candid moment: family gatherings, sporting events, school, clubs, work, and the list goes on. Perhaps one of the most interesting and challenging is out on the public street.

Street photography is all about what is happening around the photographer on a city or town street or generally in public places with people present. It often has a documentary or journalistic feel to it. A random shot into a crowd probably will not have too much interest, but focusing on a lost child in the crowd looking for a parent tells a story. There is a nice definition on Wikipedia if you want to go deeper, but what I really want to talk about is technique.

It is not enough to say you are creating street photography if you are just standing on the sidewalk and taking pictures. These days you will see these photographers everywhere as it seems nearly everyone has a camera with them at all times. The chances that those photographs will amount to anything is pretty small if the photographer is not considering what goes into making a good image. Just taking a photograph does not make you a photographer.

First off there is your equipment. Because you are making candid shots you will need to prepare in advance. You will not have time to change lenses, setup a tripod, or play around with the exposure. If you can, pack light and keep things small. You need to be able to maneuver easily and not stick out like a sore thumb.

While it would be nice to have a lens for any circumstance you will probably not have time to change on the fly. Consider your circumstances, make a choice and stick with it. A small camera is better than a large one, but do not assume you can use one of the little point and shoot digitals. They are not well suited to street photography for many reasons. A small rangefinder would be ideal, an SLR would also work well, but they are a little larger.

Okay, ready to capture an image? Go ahead, point somewhere and push that little button on top. Chances are you just captured something no one, including you, will want to look at. Like all good photography you need an actual subject; something that draws the viewer's eye. You have only a single still frame to capture the viewers interest. The full story must be there. I am always looking for an interesting face or juxtaposition, great lighting, or best a combination of all.

What makes a good subject will vary depending on whether you go with a narrow or wide field of view. If you take a narrow field, go with a small depth of field. You can use a telephoto lens or a lens with a large aperture to achieve this. If you go wide you will likely have a large depth of field whether you want it or not. You will not be able to isolate your subject by focus alone.

Find the angle that no one else has. I often find myself getting on my knees or looking for a way to get up high. Adults look down at children; it is a simple fact of the physical proportions involved. Get down to eye level or lower for a different view. This goes for many other subjects and you can do it virtually anywhere (I assume your legs bend at the knees and you do not mind getting dirty or wet); dress appropriately. Up high can be tricky, but rewarding particularly if you have a wide field or a completely new and interesting perspective.

Night is a great time for street photography. In the day a city street can be a blandly lit place, but at night the artificial light brings it alive in new and interesting ways. You will need a camera that can take photographs in low light. A maximum ISO of 800 will not give you too much to work with. If your camera can create good images at 1600 or 3200 you will be much better off. Also, you will need a fast lens; f/2.8 might not be enough. Look for f/1.8 or better. One of my favorite lenses on the street is a speedy f/1.2. Shooting with a telephoto lens will probably be out of the question as you will likely use exposures of 1/60 of a second or longer. You just cannot hand hold a long lens at speeds like that. A newer model lens with vibration reducing capabilities will help, but you need to be steady and you need to practice.

When you shoot at night you have less flexibility in how you shoot in addition to just the low light capabilities of your camera. Not everything is well lit. You will want your subject positioned near a light source that is as good or better than the surroundings. Color balance will also be very tricky at night as your will find many different color temperatures in the same location. You will need to choose one and live with color problems elsewhere. Sodium vapor lamps are particularly tricky as they only give a narrow spectrum of light. Auto white balance will often fail you in these circumstances so make sure you can preset your camera accurately or make adjustments after the fact; shoot in RAW if you can. Black and white processing is always an option if you do not like what you can do with the light that is available.

One last thing about night street photography is the focus. Most cameras these days have an auto focus feature. While this is very convenient for many things it will frequently fail in low light. Time to practice manual focusing. This can be very tricky with modern cameras as they were never designed for manual focusing. Your auto focus lens may allow it, but not make it easy. It may be hard to distinguish in focus and out of focus in your viewfinder. Best thing you can do is practice. Getting a manual focus lens can also help. Some newer DSLRs can also display the image on the LCD screen while you are taking the picture. Focusing from the LCD can help, though I find it can also complicate and slow things down. If you are digital, fire a few times while moving the focus ring. You are liable to get one right.

Finally, you need to get the exposure right. I find this can be one of the most tricky parts of street photography. The exposure meter can be easily fooled when you are trying to grab that once chance at a great shot. This is easier for those with digital cameras as you can make a test shot in advance and check the image and histogram and make adjustments. Once I find the right exposure for a given setting I do all my shooting on manual. I will keep two or three light scenarios in my head according to what the situation is and set the exposure before I make the shot.

For instance, at night on a street with shops you may find that ISO 3200, f/1.2, 1/125 works well, but if you subject moves away from the shop under a street light you will have to switch to 1/60. Figure this out in advance otherwise the shot will be lost. If I need to quickly test a new situation I will just fire blindly, check the histogram, and make the adjustments accordingly. If your subject is a fleeting one, you may not have time for anything else. Practice with your camera when you are not in a rush. If you shoot RAW, you will have more options for making adjustments after the fact.

So what about those subjects that just do not want to be photographed? If they are in public view, they are probably fair game in most places. You could abstain or go incognito. There is always the ever popular "I know your sneaky photographer tricks and I can avoid you" shots that can be interesting.

Interaction with the photographer can produce some exciting results in street photography. You could argue that these shots are not candid, but if you capture a genuine reaction (good or bad) you have a candid shot. There is nothing wrong with getting into the action as one of the crowd that happens to have a camera. When you take a shot, no one thinks twice. Offer to email an image to your subject if they seem interested after you have taken their picture. Your might find some other interesting opportunities as a result. Better yet, get them to sign a model release and open some additional possibilities for you to use the image later.