Gepsio

New Home for the Blog

2013-08-24T11:37:00.002-05:00

The Gepsio blog has moved to http://gepsio.wordpress.com. See you there!

New Document Validation Design Shipping In Next CTP

2013-06-06T21:11:00.001-05:00

I have, after a long break, checked in a pretty significant changeset for Gepsio that will change how consumers check the validity of a loaded XBRL document. The idea was originally discussed in this post, and, after some work, this scheme has been put into place.

The XbrlException class will no longer be supported with this change, and code currently making use of this exception class will have to change to remove it. Instead of catching an XbrlException instance when a document is loaded, do the following:

Load an XBRL document as usual.
Check the state of the XbrlDocument’s IsValid property. The IsValid property will be true if the document is valid according to XBRL conformance rules and false if the document is not valid.
If the document is not valid, the XbrlDocument’s ValidationErrors collection will contain more information on the validation errors. This collection is a .NET collection of ValidationError objects, each of which contains a property called Message which describes the error.

Using C#, XBRL document loading and validation code will look something like this:

newXbrlDocument.Load(instanceXmlSourceFullPath);
if(newXbrlDocument.IsValid == false)
{
    foreach (var currentValidationError in xbrlDoc.ValidationErrors)
    {
        var validationMessage = currentValidationError.Message;
        // display message
    }
}

The ValidationError class is a base class, and, depending on the error, certain derived classes may contain more context to describe the error. For example, validation errors in calculation linkbases might reference the contributing facts and the summation fact involved in the failed calculation.

The original XbrlException scheme caused Gepsio to stop validating an XBRL document as soon as the first error was found. This new scheme allows Gepsio to completely validate an XBRL document and report on all of the errors in one pass.

This change will ship with the next CTP, which I hope to release soon. It’s been a while.

Sep 2012 CTP Released

2012-09-09T21:29:00.001-05:00

The Sep 2012 CTP release of Gepsio is now available! Download the latest binary from the project’s Download page.

Here’s a peek at the newest enhancements to the latest build of Gepsio:

Role Types

Role types found in schemas are now available from within Gepsio. The XbrlSchema class now contains a property called RoleTypes. This property is a collection of objects of a new class called RoleType. Objects of the RoleType class expose any role types defined in the schema.

HTTP-Based Schema Linkbase References

Gepsio now supports HTTP-based schema linkbase references. Previous releases assumed that all schema linkbase references were based on local filesystem paths.

CalculationLink.Linkbase Property

The CalculationLink class contains a new property called Linkbase, which references the LinkbaseDocument object containing the calculation link.

LinkbaseDocument.Schema Property

The LinkbaseDocument class contains a new property called Schema, which references the XbrlSchema class referencing the linkbase document.

SummationConcept.Link Property

The SummationConcept class contains a new property called Link, which references the CalculationLink object containing the summation concept.

Resolved Issues

Work Item 9401: Valid XBRL Doc Failing To Be Loaded

The latest Amazon quarterly XBRL filing passes other validation tests I used, however attempting to Load the doc in Gepsio throws an error. Since it is an Object Ref Not Set to Instance error, it is not entirely clear what caused it although the final line in the stack trace is JeffFerguson.Gepsio.QualifiedName.Equals(Object obj). I attach the filing for your ref. Any ideas on how to get around this?

A bug in the QualifiedName equality testing code failed to detect various null conditions. This has been fixed.

Work Item 7843: The given path's format is not supported

Fixed bug that allowed paths of the form "file:///C:/blah/blah/http://blah/blah.org" to be created in GetFullLinkbasePath when filings that reference remote documents are stored locally. This caused a NotSupportedException to be thrown. Supplied as a patch from Codeplex user matthewschrager.

Work Item 9465: WebException Objects Thrown During XbrlSchema Creation Are Not Wrapped in XbrlException Objects

The XbrlSchema constructor uses an XmlTextReader to read an XBRL schema. If the URI for the XBRL schema to be read is an HTTP-based URI, then the XmlTextReader will use the .NET Web stack to read the schema using HTTP. If something fails during that process, the .NET Web stack will throw a WebException. Thrown WebException objects were not wrapped in an XbrlException object and were consequently thrown back to the client as a WebException object.

To be clear that the issue is an XBRL issue caused by an HTTP failure, the XBRL schema creation code now creates an XbrlException object, stores the caught WebException as an inner exception to the XbrlException, and throws the XbrlException object back up to the client.

Work Item 9571: No Support for Taxonomy Role Types

Role types found in schemas are now available from within the object model. The XbrlSchema class now contains a property called RoleTypes. This property is a collection of objects of a new class called RoleType. Objects of the RoleType class expose any role types defined in the schema.

Gepsio on Twitter

2012-09-06T17:19:00.001-05:00

I’ve just opened a Twitter account for Gepsio announcements, so I can keep them separate from my main (unfocused) Twitter account. I’m planning on using Gepsio’s Twitter account in much the same way as I use Gepsio’s Facebook page (www.facebook.com/gepsio) in that I will be announcing blog postings, release notes, bug fixes, and answering your questions.
Follow Gepsio on Twitter at @GepsioXbrl … see you there!

Gepsio XBRL Validation Strategies: The Present and the Future

2012-09-03T10:34:00.001-05:00

I’ve just received the following email:

I’ve had a chance by now to examine a fair number of SEC Xbrl documents using Gepsio with my [code]. I’m consistently seeing around 9-10 % of the documents in a given processing session that fail to load …
In my project … I am processing a fixed number specific data points.
I only need a few data points with their durations, start and end dates, along with basic entity information and fiscal period focus info. But I need this for a large number of documents.
I am wondering if the careful and thorough validation done on document.Load, which is really necessary for many users of Gepsio, might be overkill for what I am doing.
Could some of this validation be preventing the load of documents that might be flawed in ways that would leave them still usable for my project?
I don’t know much about xml or xbrl but I get the sense that some of the validation occurring is relevant mainly to the re- presentation of the data, not so much putting the data into a db.

Does it make any sense to think about an optional overloaded version of the document.load method with parameters that could cause it to skip some validation of things that might be peripheral to processes like mine?

This is a very thought provoking approach, as it echoes a thought that I have had for some time. Let me begin by explaining Gepsio’s current approach to reporting on XBRL validation issues and follow that up with a possible design change.

Gepsio’s Current XBRL Validation Strategy

Today, Gepsio validates a loaded document against the XBRL specification after the document is loaded. An exception is thrown when the first XBRL specification violation is discovered. At a high level, the algorithm is as follows:

load document as XML document
if XML is invalid throw XML exception
read taxonomy schema references
read contexts
read units
read facts
read footnote links
validate context references
if validation violation found throw XBRL exception
validate unit references
if validation violation found throw XBRL exception
validate context time spans against period types
if validation violation found throw XBRL exception
validate footnote locations
if validation violation found throw XBRL exception
validate footnote arcs
if validation violation found throw XBRL exception
validate items
if validation violation found throw XBRL exception

Today’s pattern asks callers to write code like this:

var myDoc = new XbrlDocument();
try
{
    myDoc.Load("MyXbrlDocument.xml");
}
catch(XbrlException e)
{
    Console.WriteLine("Validation Failed!");
    Console.WriteLine(e.Message);
}

With today’s pattern, any exception thrown by the validation code forfeits the rest of the validation process. If a validation violation is found during the validation of unit references, for example, the context time spans, footnotes and items won’t even run through validation. Once an XBRL document is deemed invalid, then no more validation is even attempted. The analogy would be a source code compiler that stops the compilation process after the first error.

Gepsio’s Possible XBRL Validation Future

As an alternative, Gepsio could stop throwing exceptions on validation errors and simply build a list of validation errors that could be examined later. In the alternative design that I have considered (a design which is shamelessly “lifted” from the CSLA.NET business objects framework) an XBRL document loaded by Gepsio would maintain a Boolean property indicating its validity as well as a collection of validation errors. These could all be examined by the caller after Gepsio loads a document and could use this information to decide whether or not the caller can proceed with the planned operation against the loaded document. This design could make document loading look something like this:

var myDoc = new XbrlDocument();
myDoc.Load("MyXbrlDocument.xml");
if(myDoc.IsValid == false)
{
    Console.WriteLine("Validation Failed!");
    foreach(var validationError in myDoc.ValidationErrors)
    {
        Console.WriteLine(validationError);
    }
}

This design takes advantage of two hypothetical additions to Gepsio’s XbrlDocument class:

A read-only Boolean property called IsValid. Gepsio would set this value to true if all validation rules passed and false if at least one validation rule failed.
A collection of validation error objects, with each object describing information about the error. In the simple design above, these are simple strings; however, in an actual design these may be full Gepsio-namespaced objects that describe a validation error with as much fidelity as possible to identify the root of the problem as well as the error category. Callers may want to distinguish between a unit reference error and a calculation summation error, and a simple string would not give that kind of fidelity.

In this design, no exceptions are thrown, allowing Gepsio to perform as much validation as possible.

Feedback Welcome

Your feedback is welcome. Do you like the current exception-based validation scheme? Do you favor the validity design shown above? Or do you have another idea? Add your feedback to a comment on this post or post a message to the Gepsio Facebook page at http://www.facebook.com/gepsio.

Importing XBRL Fact Values Into SQL Server Using PowerShell and Gepsio

2012-08-27T22:06:00.001-05:00

A Gepsio user has just written in with a success story on importing XBRL data into a SQL Server database using SQL Server Integration Services (SSIS), PowerShell and Gepsio. The solution was too good not to share, and, with the user's permission, this blog post will describe that solution.

The user was tasked with importing some XBRL data into a SQL Server database instance. The overall architectural idea was to use an SSIS package to run a PowerShell script which would create a comma separated values (CSV) file from a raw XBRL document, and the values in the CSV would be used to import data into the database.

The first trick, which isn't really XBRL-specific but still worth mentioning, is to get SSIS to run a PowerShell script. Basically, PowerShell scripts are run from within an SSIS package by adding an "Execute Process Task" and using a command line in the following form:

C:\[PATHTOPOWERSHELL]\PowerShell.exe -ExecutionPolicy ByPass -command ". 'C:\SCRIPTPATH\MyScript.ps1' 'param1' 'param2'"

Read the forum discussion at http://social.msdn.microsoft.com/Forums/en-NZ/sqlintegrationservices/thread/216d2ee6-0f04-480f-808d-8241bc4a8d18 for more information about this process.

The next trick, of course, is creating the CSV file from the raw XBRL document. The user turned to PowerShell and Gepsio for this work, and life became a lot easier. Here is the example PowerShell script that can do this work:

param([string]$instFile = "C:\XBRLDOCPATH\XbrlDocument.xml")

#load Gepsio
Add-Type -Path "C:\GEPSIOPATH\JeffFerguson.Gepsio.dll"
$XbrlDoc = New-Object -TypeName JeffFerguson.Gepsio.XbrlDocument
$XbrlDoc.Load($instFile)
$instCSV = "C:\OUTPUTPATH\Allinfo.csv"
    
New-Item $instCSV -type "file" -force -value "EntityRegName,EntityFilerCat,FactName,Value`r`n"    
$stream = new-object system.IO.StreamWriter($instCSV,$true)

[string] $script:Entity =""
try
{
    foreach($CurrentFragment in $XbrlDoc.XbrlFragments)
    {
        GetEntityInfo $Currentfragment
        try
        {
            WriteItemCSV $CurrentFragment "EarningsPerShareBasic"
        }
        catch
        {
            Write-Error("us-gaap_EarningsPerShareBasic: " +$_ )
        }
        try
        {
            WriteItemCSV $CurrentFragment "NetIncomeLoss"
        }
        catch
        {
            Write-Error(":us-gaap_NetIncomeLoss " +$_ )
        }
    }
}
catch
{
    Write-Error("main foreach writeloop: " +$_ )
}
finally
{
    $stream.close()
    $stream.Dispose()
}

Function GetEntityInfo
{
    param($fragment)
    $script:Entity = ""
    $entr = $fragment.Facts | Where-Object {$_.Name -eq "EntityRegistrantName"}
    if(!$entr)
    {
        $entr = ""
    }
    $efc = $fragment.Facts | Where-Object {$_.Name -eq "EntityFilerCategory"}
    if(!$efc)
    {
        $efc = ""
    }
    $script:Entity = "`"" + $entr.Value + "`",`""  + $efc.Value +  "`","
}

Function WriteItemCSV
{
    param($fragment, $ElId)
    $Ff = $fragment.Facts | Where-Object {$_.Name -eq $ElId}
    if($Ff)
    {
        [string]$S = $script:Entity
        if ($Ff.GetType().fullname -eq "JeffFerguson.Gepsio.Item")
        {
            [string]$S = $script:Entity
            if( $Ff.Name)
            {
                $S = $S +  "`"" + $Ff.Name + "`","
            }
            else
            {
                $S = $S +  ""   +","
            }
            if( $Ff.Value)
            {
                $S = $S +  $Ff.Value + ","
            }
            else
            {
                $S = $S +  ""   +","
            }
            $stream.WriteLine($S)
        }
        if ($Ff.GetType().fullname -eq "System.Object[]")
        {
            foreach($i in $Ff)
            {
                [string]$S = $script:Entity
                if( $i.Name)
                {
                    $S = $S +  "`"" + $i.Name + "`","
                }
                else
                {
                    $S = $S +  ""   +","
                }
                if( $i.Value)
                {
                    $S = $S +  $i.Value
                }
                else
                {
                    $S = $S +  ""
                }
                $stream.WriteLine($S)
            }
        }
    }
}

Let's take a look at this PowerShell script in more detail.

The opening statements adds the Gepsio types to the PowerShell script and loads the XBRL document named in a PowerShell script command line parameter into a new Gepsio XbrlDocument instance:

param([string]$instFile = "C:\XBRLDOCPATH\XbrlDocument.xml")

#load Gepsio
Add-Type -Path "C:\GEPSIOPATH\JeffFerguson.Gepsio.dll"
$XbrlDoc = New-Object -TypeName JeffFerguson.Gepsio.XbrlDocument
$XbrlDoc.Load($instFile)

At this point, the PowerShell script maintains a variable called $XbrlDoc which contains all of Gepsio's knowledge about the loaded XBRL document. Loading and validating an XBRL document can't get much easier.

Once the XBRL document is loaded, then the output CSV file is created:

New-Item $instCSV -type "file" -force -value "EntityRegName,EntityFilerCat,FactName,Value`r`n"    
$stream = new-object system.IO.StreamWriter($instCSV,$true)

Here, a new file is created and the CSV column header row is written to the new file. The CSV is set up to capture the following values found in the XBRL document:

Entity Registrant Name
Entity Filer Category

A .NET StreamWriter object is created to reference the newly created CSV file and is available from within a PowerShell script variable called $stream.

Once the CSV is available, the PowerShell script iterates through each of the XBRL fragments found in the XBRL document loaded by Gepsio:

try
{
    foreach($CurrentFragment in $XbrlDoc.XbrlFragments)
    {
        GetEntityInfo $Currentfragment
        try
        {
            WriteItemCSV $CurrentFragment "EarningsPerShareBasic"
        }
        catch
        {
            Write-Error("us-gaap_EarningsPerShareBasic: " +$_ )
        }
        try
        {
            WriteItemCSV $CurrentFragment "NetIncomeLoss"
        }
        catch
        {
            Write-Error(":us-gaap_NetIncomeLoss " +$_ )
        }
    }
}
catch
{
    Write-Error("main foreach writeloop: " +$_ )
}
finally
{
    $stream.close()
    $stream.Dispose()
}

For each XBRL fragment found in the Gepsio document instance, entity information is read and written into the CSV file. Entity information is read from the fragment, and then values are written to the CSV. These operations are performed by functions in the PowerShell script called GetEntityInfo and WriteItemCSV, respectively.

Let's take a look at the GetEntityInfo function. It is defined as follows:

Function GetEntityInfo
{
    param($fragment)
    $script:Entity = ""
    $entr = $fragment.Facts | Where-Object {$_.Name -eq "EntityRegistrantName"}
    if(!$entr)
    {
        $entr = ""
    }
    $efc = $fragment.Facts | Where-Object {$_.Name -eq "EntityFilerCategory"}
    if(!$efc)
    {
        $efc = ""
    }
    $script:Entity = "`"" + $entr.Value + "`",`""  + $efc.Value +  "`","
}

This function, which accepts a Gepsio XbrlFragment object as a parameter, searches through each of the fragment's facts, looking for a fact whose name is "EntityRegistrantName" or "EntityFilerCategory". If they are found, the Gepsio Fact objects are stored in local script variables -- $entr and $efc, respectively. Once the search is complete, the Fact object's values are stored as a string in a global-level script variable called $script:Entity.

The other interesting function in the PowerShell script is the WriteItemCSV function:

Function WriteItemCSV
{
    param($fragment, $ElId)
    $Ff = $fragment.Facts | Where-Object {$_.Name -eq $ElId}
    if($Ff)
    {
        [string]$S = $script:Entity
        if ($Ff.GetType().fullname -eq "JeffFerguson.Gepsio.Item")
        {
            [string]$S = $script:Entity
            if( $Ff.Name)
            {
                $S = $S +  "`"" + $Ff.Name + "`","
            }
            else
            {
                $S = $S +  ""   +","
            }
            if( $Ff.Value)
            {
                $S = $S +  $Ff.Value + ","
            }
            else
            {
                $S = $S +  ""   +","
            }
            $stream.WriteLine($S)
        }
        if ($Ff.GetType().fullname -eq "System.Object[]")
        {
            foreach($i in $Ff)
            {
                [string]$S = $script:Entity
                if( $i.Name)
                {
                    $S = $S +  "`"" + $i.Name + "`","
                }
                else
                {
                    $S = $S +  ""   +","
                }
                if( $i.Value)
                {
                    $S = $S +  $i.Value
                }
                else
                {
                    $S = $S +  ""
                }
                $stream.WriteLine($S)
            }
        }
    }
}

This function accepts both an Gepsio XBRL Fragment object and a fact name as parameters. It begins by searching through each of the supplied fragment's facts, looking for a fact whose name matches the supplied name. If it is found, the Gepsio Fact objects are stored in a local script variable called $Ff. If the fact is found, its name and values are appended to the global script variable $script:Entity, after which the entire line built up in the $script:Entity variable is written out to the CSV stream.

[One quick note about the placement of the functions in the PowerShell script: When the script is executed from the PowerShell command line, the functions can be found anywhere in the script. SSIS, however, may need to have the functions defined before they are used in the script. If SSIS does not execute the script as written, the functions may need to be moved forward in the script so that they are defined before they are used.]

In the end, the PowerShell script will produce a CSV file that looks something like this:

EntityRegName,EntityFilerCat,FactName,Value
"COCA COLA CO","Large Accelerated Filer","EarningsPerShareBasic",1.22
"COCA COLA CO","Large Accelerated Filer","EarningsPerShareBasic",2.05
"COCA COLA CO","Large Accelerated Filer","EarningsPerShareBasic",2.14
"COCA COLA CO","Large Accelerated Filer","EarningsPerShareBasic",1.24
"COCA COLA CO","Large Accelerated Filer","NetIncomeLoss",4703000000
"COCA COLA CO","Large Accelerated Filer","NetIncomeLoss",2788000000
"COCA COLA CO","Large Accelerated Filer","NetIncomeLoss",4842000000
"COCA COLA CO","Large Accelerated Filer","NetIncomeLoss",2800000000

Once this script is in place, the SSIS package to actually import the data becomes an easy set of two simple tasks:

execute PowerShell script to create CSV from XBRL
import CSV data into a database table

For the sake of brevity, the PowerShell sample shown here loads only a few XBRL values into the CSV file. In practice, however, there is no limit to the amount of data that could be loaded into the CSV.

Once all of this was working, the user sent in a testimonial that read:

"Thank you for making Gepsio available. It saved me an awful lot of work (xpath, YIKES)!".

This is gratifying, as this is what Gepsio is all about: making access to XBRL data easier without resorting to complicated XPath queries or understanding a lot about the rules of XBRL validation or technical syntax. Gepsio provides access to XBRL data without needing to worry about XBRL itself, which frees developers to work on the more important problem of building added value on top of the data.

Many thanks go out to the Gepsio user who provided this solution.

WebException Objects and XBRL Schema Object Creation

2012-06-22T23:08:00.001-05:00

The other day, I was doing some performance testing in Gepsio and noticed that HTTP errors found during the XBRL schema object creation process were not reported in the same way as other exceptions thrown by Gepsio.

The constructor of Gepsio’s XbrlSchema class uses an XmlTextReader to read an XBRL schema. If the URI for the XBRL schema to be read is an HTTP-based URI, then the XmlTextReader will use the .NET Web stack to read the schema using HTTP. If something fails during that process, the .NET Web stack will throw a WebException.

In the Nov 2011 CTP (and, in fact, all Gepsio CTPs to this point), thrown WebException objects were not wrapped in an XbrlException object and were consequently thrown back to the client as a WebException object.

I have just checked in a fix for the issue. The fix will be available in the next release (currently planned for Jul 2012). If you want to grab the code ahead of time, feel free to grab the latest source code here.

Unsupported Path Bug Fixed in Linkbase Document Class

2012-06-19T17:33:00.001-05:00

Codeplex (and Gepsio) user matthewschrager recently submitted a patch for a rather old issue that caused paths of the form "file:///C:/blah/blah/http://blah/blah.org" to be created in the LinkbaseDocument class’ GetFullLinkbasePath() method when filings that reference remote documents are stored locally. This caused a NotSupportedException to be thrown.

I have just applied this patch and have checked in a fix for the issue. The fix will be available in the next release (currently planned for Jul 2012). If you want to grab the code ahead of time, feel free to grab the latest source code here.

Many thanks to matthewschrager for sending in a patch!

Null Reference Bug Fixed in QualifiedName Class

2012-06-18T21:41:00.001-05:00

Codeplex (and Gepsio) user jwokelly recently filed a bug noting that the Gepsio Nov 2011 CTP was failing to parse the XBRL documents created by Amazon to report their financial information for Q1 2012. More specifically, Gepsio was throwing a NullReferenceException while attempting to perform equality comparisons in the qualified name code.

I have just fixed this bug and have checked in a fix. The fix will be available in the next release (currently planned for Jul 2012). If you want to grab the code ahead of time, feel free to grab the latest source code here.

Many thanks to jwokelly for reporting a bug and for sending along the offending documents in a ZIP file. Gepsio now includes a unit test that parses these documents and ensures that no exceptions are thrown.

Full Documentation Available for the Nov 2011 CTP

2012-06-17T09:58:00.001-05:00

The first release of help file-style documentation for Gepsio is now available for the Nov 2011 CTP! To grab your copy, go to the Nov 2011 CTP Release page here and download the JeffFerguson.Gepsio.chm file.

If you download the file directly, and save it to a location such as your Desktop, you may get an “Navigation to the webpage was canceled” message when you open it:

To fix this, close the CHM file.Then find the downloaded CHM file, right click it, and select “Properties” from the context menu. You will see the file’s Properties dialog:

If you see the message at the bottom that reads “This file came from another computer and might be blocked to help protect this computer”, click the “Unblock” button. Click the “OK” button to close the Properties dialog, and reopen the CHM file.

Voila!

This online documentation is generated automatically from comments I embed in the source code. The nice part about that is that, as I update the source code comments and produce new builds, the online documentation will be updated and synchronized with the latest information available in the Gepsio object model.

Gepsio Reviewed by HereBeDragon

2012-06-14T20:20:00.001-05:00

I found a very nice review of Gepsio today on a blog at herebedragon.blogspot.com. Here’s what the author had to say:

My first stop was Codeplex, where I searched for the term 'XBRL' and found 3 projects. That's too few. Of these, one particularly caught my eye (in fact I have followed this project for a few months) and that is 'Gepsio'.

Gepsio is a .NET based document object model for XBRL that can load and validate XBRL documents. It's a 'strict' processor for XBRL that throws exceptions when the loaded document is not valid. Perhaps it's possible to make changes to and save XBRL files using Gepsio as well, but frankly I could not look into it in depth because of the exceptions and because I was short of time.

I did give the source code of the XBRL Reader part of Gepsio a good look though. I was impressed to see sustained work, focus on meeting XBRL conformance requirements and a clean, well organized codebase. Gepsio is a good option to evaluate if you are looking for open source XBRL software.

The Gepsio XBRL code is licensed under Microsoft Public License (Ms-PL), which allows you to use the code in a commercial application.

Find the author’s review of Gepsio, and many other open source XBRL tools, through this link.

Thanks to HereBeDragon for the review!

Gepsio on Facebook

2012-06-11T19:30:00.001-05:00

I have just created a page for Gepsio on Facebook. “Like” Gepsio at http://www.facebook.com/gepsio to get information on releases, blog posts, documentation, samples and questions and answers. Looking forward to seeing you there!

Documentation Work in Progress

2012-06-09T12:45:00.001-05:00

I am currently working on providing some online help for Gepsio:

I am currently adding XML comments to all of the public classes, properties and methods so that I can use Sandcastle to generate CHM-style documentation. The XML comments should also aid in providing documentation to Visual Studio’s Intellisense infrastructure.

I realize that Gepsio’s growing array of public classes and properties can be daunting and I hope that this documentation will go a long away towards making its public interface understandable.

Using Gepsio From C#

2012-03-24T10:41:00.001-05:00

A developer new to Gepsio asked for a C#-based sample of using Gepsio. I have built a simple C# console app using Visual Studio 2010 and the Gepsio Nov 2011 CTP. I’ll show the code first, and then I will discuss some key points after the code is shown:

using JeffFerguson.Gepsio;
using System;

namespace GepsioConsole
{
    class Program
    {
        /// <summary>
        /// The magic begins here.
        /// </summary>
        /// <param name="args">
        /// A collection of program arguments. One argument should be supplied: the address of the XBRL document
        /// to load.
        /// </param>
        static void Main(string[] args)
        {
            if (args.Length != 1)
            {
                Console.WriteLine("usage: GepsioConsole [XBRL document]");
                return;
            }
            ProcessXbrlWithGepsio(args[0]);
        }

        /// <summary>
        /// Process a named XBRL document.
        /// </summary>
        /// <param name="xbrlFile">
        /// The address of the XBRL document to process.
        /// </param>
        private static void ProcessXbrlWithGepsio(string xbrlFile)
        {
            try
            {
                var xbrlDoc = new XbrlDocument();
                xbrlDoc.Load(xbrlFile);
                foreach (var currentFragment in xbrlDoc.XbrlFragments)
                {
                    DisplayFragmentStatistics(currentFragment);
                    WriteFactValue(currentFragment, "EntityRegistrantName");
                    WriteFactValue(currentFragment, "DocumentPeriodEndDate");
                }
            }
            catch (XbrlException xbrle)
            {
                Console.WriteLine("ERROR: {0}", xbrle.Message);
            }
        }

        /// <summary>
        /// Display statistics relating to the loaded document fragment.
        /// </summary>
        /// <param name="currentFragment">
        /// The XBRL fragment whose statistics should be published.
        /// </param>
        private static void DisplayFragmentStatistics(XbrlFragment currentFragment)
        {
            var factsCollection = currentFragment.Facts;
            Console.WriteLine("Number of facts...: {0}", factsCollection.Count);
            var unitsCollection = currentFragment.Units;
            Console.WriteLine("Number of units...: {0}", unitsCollection.Count);
            var contextsCollection = currentFragment.Contexts;
            Console.WriteLine("Number of contexts: {0}", contextsCollection.Count);
        }

        /// <summary>
        /// Display the value of a fact pulled from an XBRL fragment.
        /// </summary>
        /// <param name="currentFragment">
        /// The fragment containing the fact to be found.
        /// </param>
        /// <param name="factName">
        /// The name of the fact to be found.
        /// </param>
        private static void WriteFactValue(XbrlFragment currentFragment, string factName)
        {
            foreach (var currentFact in currentFragment.Facts)
            {
                if (currentFact.Name.Equals(factName) == true)
                {
                    var currentFactAsItem = currentFact as Item;
                    Console.WriteLine("{0}: {1}", factName, currentFactAsItem.Value);
                    return;
                }
            }
        }
    }
}

I wanted to offer a few notes regarding this code:

This is a console application, and is intended to be run with a command line argument specifying the address of the XBRL document to be loaded, as in GepsioConsole http://www.sec.gov/Archives/edgar/data/21344/000104746911006790/ko-20110701.xml. Gepsio can work with documents stored on the Web, so HTTP-based document addresses are valid.

From Gepsio’s point of view, an XBRL document is a collection of fragments. The fragments idea was originally designed to support the notion of Inline XBRL, where a document may consist of multiple XBRL fragments. For standard XBRL documents, however, the entire document is an XBRL document, which makes up one “fragment”. This explains the “for each fragment in document” code in the ProcessCodeWithGepsio() method.

The WriteFactValue() method looks for a fact in the fragment’s collection of facts. Remember that, from an XBRL point of view, a fact is a type of item. In XBRL parlance, items can be facts, which have a single value, or tuples, which can have more than one value. Gepsio models this fact by defining a base class called Item and then deriving both Fact and Tuple from Item. This explains the cast back to Item in the code for WriteFactValue(). The cast may look a bit strange … perhaps I’ll revisit this in a later CTP.

As I developed this sample, I noticed that Gepsio performs as intended but, for larger documents, its performance can be improved. I’ll be addressing this shortly. Look for a future blog post where I discuss performance, where it could be improved, and how the improvements will be implemented. I will be using the code in this blog post as the showcase for the performance improvements, so you will be seeing this code again.

Gepsio Entered Into XBRL US Challenge

2012-02-23T08:00:00.004-06:00

The folks over at XBRL US have launched a contest called the XBRL Challenge, which they have billed as "a contest that invites participants to contribute open source analytical applications for investors that leverage corporate XBRL data". I have entered Gepsio into the contest. I really don't think it has a chance to win, since it's more of an "engine" or "framework" than a full ready-to-use application, but I figured that the exposure might be a good thing.

The judges have asked all of us entrants to answer a couple of questions about our perception of our entries to the contest, and I thought that I would share my answers here:
What deficiency or opportunity did you see - with respect to published financial data analysis - that drove your idea for an application using XBRL?The relatively high barrier to entry. Building an XBRL-aware application today means that XML parsing code must be written, after which the XBRL validation rules must be encoded and tested. The opportunity I see with Gepsio is one in which the parsing and validation logic is packaged in a ready-to-use assembly that can be picked up and used by others to build application and value-added logic, thereby dramatically lowering the barrier to entry for XBRL developers. It also opens the door for PowerShell users to write “XBRL scripts” without ever encoding low-level XML and XBRL details.

What was the moment you enjoyed most during this competition?I have enjoyed seeing Gepsio used and embraced by others. I put Gepsio up on Codeplex to little fanfare, unsure of what might become of it. To see it used by others – to see the project followed by other users on Codeplex; to see Gepsio mentioned in XBRL answers on the Stack Overflow Web site; to get emails from people using it – all of it makes the effort worthwhile. It’s validation that what Gepsio offers fills a need as yet untapped by the current developer tool space, and that is very gratifying.
The winner of the contest is judged by a panel of judges, although a People's Choice vote is also being held for another week or so. If you feel compelled to vote for Gepsio in this People's Choice competition, head on over to the XBRL US Challenge page and click on the "Vote for the Best XBRL App" link in the upper right-hand section titled "XBRL Challenge Judges".

Good luck to all Challenge participats!

Gepsio Nov 2011 CTP and iXBRL

2012-01-06T08:51:00.001-06:00

In the discussion located here, a gentleman asked about Gepsio's support for iXBRL after noting that iXBRL is coming up to speed in the UK. I mentioned that I would look into it, noting that I had structured Gepsio in such a way that a document was a collection of XBRL fragments, and that the design was put in place specifically to handle multiple XBRL fragments in a document, thereby supporting the iXBRL model of XBRL embedded within a larger XML document whose root was not necessarily an XBRL root.

The gentleman with whom I had the conversation was very generous in supplying some iXBRL documents with which I could test Gepsio, and I used the following PowerShell script to load the sample and see what I could find out at a high level:

Add-Type -Path "C:\MyPathToGepsio\JeffFerguson.Gepsio.dll"

$XbrlDoc = New-Object -TypeName JeffFerguson.Gepsio.XbrlDocument
$DocumentLocation = "sample-ixbrl.xml"

Write-Host "Loading and validating" $DocumentLocation"..."
$XbrlDoc.Load($DocumentLocation)
Write-Host "Loaded." $XbrlDoc.XbrlFragments.Count "fragments in document."

foreach($XbrlFragment in $XbrlDoc.XbrlFragments)
{
    Write-Host $XbrlFragment.Facts.Count "facts in fragment."
    Write-Host $XbrlFragment.Units.Count "units in fragment."
    Write-Host $XbrlFragment.Contexts.Count "contexts in fragment."
    WriteItemAndValue $XbrlFragment "EntityRegistrantName"
    WriteItemAndValue $XbrlFragment "TradingSymbol"
    WriteItemAndValue $XbrlFragment "DocumentPeriodEndDate"
}

Function WriteItemAndValue
{
    param($fragment, $name)
    $FoundItem = $fragment.Facts | Where-Object { $_.Name -eq $name }
    Write-Host $name ": " $FoundItem.Value
}
As it turns out, I have good news and bad news:

The good news is that Gepsio parses the ful document at the XML level without errors, even when the root of the XML document is not an <xbrl> root.
The bad news is that Gepsio didn't find any XBRL fragments in the supplied document, despite the document's valid iXBRL tagging.

Gepsio makes an assumption that iXBRL fragments are marked up using the same schema as a full XBRL document, which is not the case. The iXBRL tagging structure, documented here, is not supported by Gepsio in the Nov 2011 CTP, and it will be my job to correct that flaw.

My plan is to work on Gepsio's iXBRL support this month to see how far I get. I hope to report back on my progress soon. In my ideal world, I would like to release a new CTP of Gepsio this month that includes iXBRL support.

Wish me luck!

Easy XBRL Scripting with Gepsio and PowerShell

2011-11-17T21:54:00.001-06:00

Microsoft’s PowerShell scripting environment provides full support for using .NET objects, which makes it an ideal scripting language for use with Gepsio. PowerShell and Gepsio can work together to make working with XBRL data very easy.

Here is a simple PowerShell script that pulls an XBRL document from the SEC site and displays some simple information about the document:

Add-Type -Path "C:\GepsioPath\JeffFerguson.Gepsio.dll"

$XbrlDoc = New-Object -TypeName JeffFerguson.Gepsio.XbrlDocument

$DocumentLocation = "http://www.sec.gov/Archives/edgar/data/21344/000104746911006790/ko-20110701.xml"
Write-Host "Loading and validating" $DocumentLocation"..."
$XbrlDoc.Load($DocumentLocation)

foreach($CurrentFragment in $XbrlDoc.XbrlFragments)
{
    Write-Host $CurrentFragment.Facts.Count "facts in fragment."
    Write-Host $CurrentFragment.Units.Count "units in fragment."
    Write-Host $CurrentFragment.Contexts.Count "contexts in fragment."
}

This script gives the following output:

Loading and validating http://www.sec.gov/Archives/edgar/data/21344/000104746911006790/ko-20110701.xml...
1081 facts in fragment.
4 units in fragment.
281 contexts in fragment.

Let’s take a look at the script in more detail. It begins by adding the Gepsio runtime into the PowerShell environment with a call to the Add-Type command:

Add-Type -Path "C:\GepsioPath\JeffFerguson.Gepsio.dll"

Every PowerShell script that uses Gepsio will need to use this command so that the Gepsio runtime can be used by the PowerShell script.

Once Gepsio is loaded, a new Gepsio XbrlDocument object is created and saved in a variable:

$XbrlDoc = New-Object -TypeName JeffFerguson.Gepsio.XbrlDocument

This statement states that a new .NET object of type JeffFerguson.Gepsio.XbrlDocument, which is the Gepsio type that represents an XBRL document, should be created and be made available in a PowerShell script variable called $XbrlDoc.

One the new XbrlDocument object is created, another PowerShell variable is created to describe the address of the document to be loaded:

$DocumentLocation = "http://www.sec.gov/Archives/edgar/data/21344/000104746911006790/ko-20110701.xml"

This statement creates a new PowerShell script variable called $DocumentLocation and assigns it a string of “http://www.sec.gov/Archives/edgar/data/21344/000104746911006790/ko-20110701.xml”. This is the address of the XBRL document that the script should load.

Once the address of the XBRL document is made available, it is loaded into the Gepsio XbrlDocument object through a call to the XbrlDocument.Load() method:

$XbrlDoc.Load($DocumentLocation)

Gepsio treats an XbrlDocument as a collection of XbrlFragment objects. Each XBRL fragment is XML data having the <xbrl> tag as its root. Generally, an XBRL document will have only one XbrlFragment, although Gepsio supports documents that may have more than one fragment. The script iterates through each fragment in the document and examines the Xbrlfragment object’s three main collections: Facts, Units and Contexts. Each collection has a property called Count, and each of these counts are displayed as script output:

foreach($CurrentFragment in $XbrlDoc.XbrlFragments)
{
    Write-Host $CurrentFragment.Facts.Count "facts in fragment."
    Write-Host $CurrentFragment.Units.Count "units in fragment."
    Write-Host $CurrentFragment.Contexts.Count "contexts in fragment."
}

Future blog posts will explore PowerShell’s access to the Gepsio runtime in more detail.

Gepsio Available as a NuGet Package

2011-11-07T17:39:00.001-06:00

Beginning with the recently-released Nov 2011 CTP, Gepsio is now available via NuGet as well as from the project Web site. The name of the NuGet package is JeffFerguson.Gepsio, so, from within the Package Manager Console in Visual Studio 2010, you can execute the following command:

PM> Install-Package JeffFerguson.Gepsio

You will also find it from the “Manage NuGet Packages” submenu item from the “Library Package Manager” menu item on the Visual Studio 2010 “Tools” menu. Just search for Gepsio:

Nov 2011 CTP Released

2011-11-01T19:35:00.001-05:00

I am pleased to announce the Nov 2011 CTP release of Gepsio. This release passes 191 of the tests in the XBRL-CONF-CR3-2007-03-05 conformance suite (which is 32 more than the 159 tests that passed in the last CTP release). This release builds on the previous CTP release in the following ways:

First Class Support For Items and Tuples. Previous releases of Gepsio used a single class called Fact to represent a fact in an XBRL fragment. Starting with this release, the Fact class is a base class for two derived classes: one called Item and the other called Tuple. The Facts collection available from an XbrlFragment object is a collection of Fact objects, and each Fact in the collection could, in fact, be either an Item object or a Tuple object. This work brings the XBRL concept of tuples in as a first-class citizen of the Gepsio object model.
Complete Support For Essence Aliases. This release of Gepsio passes all of the conformance suite tests relating to the concept of essence aliases.
Better Support For Calculation Arc Validation. Gepsio now has a better validation engine for calculation arcs. This release now includes better handling of context-equals and unit-equals items, items with a nil value, and better support for contributing concept items found in tuples.

Download this latest version (numbered 2.1.0.5) at http://gepsio.codeplex.com and click the “Downloads” tab to grab the latest release.

Support Added for Calculation Arcs with Destination Labels Referencing Multiple Locators

2011-10-20T09:37:00.001-05:00

I have just checked in a change to Gepsio that correctly validates calculation arcs that use destination labels that reference more than one locator.

Take a look at the following calculation link:

<calculationLink xlink:type="extended" xlink:role="http://www.xbrl.org/2003/role/link">

    <loc xlink:type="locator" xlink:href="397-ABC.xsd#A" xlink:label="summationItem" />

    <loc xlink:type="locator" xlink:href="397-ABC.xsd#B" xlink:label="contributingItem" />

    <loc xlink:type="locator" xlink:href="397-ABC.xsd#C" xlink:label="contributingItem" />

    <calculationArc
        xlink:type="arc"
        xlink:arcrole="http://www.xbrl.org/2003/arcrole/summation-item"
        xlink:from="summationItem"
        xlink:to="contributingItem" weight="1"
    />

</calculationLink>

This example, taken from the 397.00 conformance test in the XBRL-CONF-CR3-2007-03-05 conformance suite, defines a calculation arc that arcs from “summationItem” to “contributingItem”. The issue here is that there are two locators that use the “contributingItem” label:

397-ABC.xsd#B

397-ABC.xsd#C

Both of these locators must participate in the calculation arc. In the current CTP of Gepsio, only the fact referenced by the first locator is used to validate the calculation arc. With the new code, both facts referenced by both locators are used to validate the calculation arc.

This improved functionality will be available in the next CTP to released (which I am currently targeting for release in the next week or so).

Calculation Validation Location Bug Fixed

2011-10-11T14:28:00.001-05:00

I have checked in code that fixes a bug in the validation of calculation arc values in Gepsio.

In the current CTP, the value of a xlink:href attribute of a locator element for a calculation arc is used to find the appropriate contributing concept. This is in error; in fact, the xlink:label attribute must be used instead. This bug hasn’t been caught until now because many of the XBRL documents in XBRL-CONF-CR3-2007-03-05 conformance suite use the same value for the xlink:href and label attributes. The 305.07 test, for example, references a calculation linkbase document that includes markup as follows:

<loc
    xlink:type="locator"
    xlink:href="305_07_decimals_test.xsd#decimals_Land"
    xlink:label="decimals_Land"
/>

As you can see, the resource ID of the value of the xlink:href attribute (decimals_Land) matches the value of the xlink:label attribute (decimals_Land), so it didn’t matter which value Gepsio used to find the correct element.

The 395.01 test, however, uses different location markup:

<loc
    xlink:type="locator"
    xlink:href="SummationItem.xsd#CurrentAsset"
    xlink:label="labelCurrentAsset"
    xlink:title="CurrentAsset"
/>

In this markup, the resource ID of the value of the xlink:href attribute (CurrentAsset) does not match the value of the xlink:label attribute (labelCurrentAsset) and Gepsio’s use of the resource ID is in error. Gepsio must use the value of the xlink:label attribute to find the correct element.

This bug has been fixed and will be available in the next CTP of Gepsio (which I am currently planning on releasing on Nov 01 2011).

Essence Alias Support Complete

2011-10-07T10:29:00.001-05:00

I have just checked in code that completes Gepsio’s support of the XBRL essence alias concept. All of the essence alias conformance tests found in the XBRL-CONF-CR3-2007-03-05 conformance test suite (numbered with the 392 prefix) now behave as expected when parsed by Gepsio.

The next CTP of Gepsio will include all of this work and will correctly validate essence aliases found in XBRL documents.

Dates Available For Instant Period Contexts

2011-10-06T14:04:00.001-05:00

I have just checked in a change to the Context class that will appear in the next CTP. The Context class now includes a public property called InstantDate, which exposes a DateTime value. The value of the InstantDate property for a Context reflects the date given for an instant period. This value is valid only when the Context reflects an Instant period. You will be able to write code like the following:

if(CurrentContext.InstantPeriod == true)
{
    var ContextDate = CurrentContext.InstantDate;
}

Facts, Items and Tuples

2011-09-30T15:12:00.001-05:00

The current CTP of Gepsio (which, as of this writing, is the Feb 2011 CTP) exposes a collection of Fact objects as a property in an XbrlFragment. With this arrangement, you can write code like this:

var MyXbrlDocument = new XbrlDocument();
MyXbrlDocument.Load("xbrldata.xml");
foreach(var CurrentFragment in MyXbrlDocument.XbrlFragments)
{
    foreach(var CurrentFact in CurrentFragment.Facts)
    {
        // Ooh! An XBRL fact!
    }
}

I have been working on adding support for tuples in the next CTP, and, in doing so, have come to realize that this scenario is incomplete.

The Fact class in the current CTP is a standalone class. It doesn’t inherit from any other class, nor is it a base class for a derived class. The issue with the current CTP is that the only facts that it reads and exposes are single-value facts. Any tuples in the XBRL document are ignored and not included as a Fact instance.

In actuality, the XBRL specification supports two types of facts:

single-value facts, called items

multiple-value facts, called tuples

The next CTP will provide support for tuples, and, as such, Gepsio’s understanding of a fact will be more complete. In the next CTP, the Fact class will remain, but it will serve as a base class for a new class called Item and another new class called Tuple:

public class Fact
{
    // magic happens here
}

public class Item : Fact
{
    // more magic happens here
}

public class Tuple : Fact
{
    // even more magic happens here
}

Fact collections will remain in place, but Fact items can be examined to determine whether they are, in fact, items or tuples. This updated design will allow Gepsio to detect tuples in loaded XBRL documents and perform the appropriate work needed to expose tuples to Gepsio clients.

Rounded Values of Essence Aliased Facts Now Checked

2011-09-29T10:31:00.001-05:00

I have just checked in code that will be released with the next CTP of Gepsio. The code validates the rounded values (that is, fact values with any applicable rounding and truncation applied) of essence aliased facts and throws an exception if those values do not match.

For example, consider the following XBRL document:

<xbrl xmlns="http://www.xbrl.org/2003/instance" xmlns:link="http://www.xbrl.org/2003/linkbase" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:example="http://example.com/xbrl/taxonomy/EssenceAlias" xmlns:iso4217="http://www.xbrl.org/2003/iso4217" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://example.com/xbrl/taxonomy/EssenceAlias EssenceAlias.xsd">
    <link:schemaRef xlink:href="EssenceAlias.xsd" xlink:type="simple"/>
    <unit id="u1">
        <measure>iso4217:USD</measure>
    </unit>
    <context id="c1">
        <entity>
            <identifier scheme="www.example.com">example</identifier>
        </entity>
        <period>
            <instant>2003-03-31</instant>
        </period>
    </context>
    <example:CurrentDeferredIncomeTaxExpense contextRef="c1" unitRef="u1" precision="4">100.0</example:CurrentDeferredIncomeTaxExpense>
    <example:ForeignDomesticIncomeTaxExpense contextRef="c1" unitRef="u1" precision="3">100</example:ForeignDomesticIncomeTaxExpense>
    <example:TaxExpense contextRef="c1" unitRef="u1" precision="3">200</example:TaxExpense>
</xbrl>

Now suppose that the CurrentDeferredIncomeTaxExpense and TaxExpense facts are paired in an essence alias relationship, as shown in the following linkbase document:

<?xml version="1.0" encoding="UTF-8"?>
<linkbase xmlns="http://www.xbrl.org/2003/linkbase" xmlns:xbrll="http://www.xbrl.org/2003/linkbase" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.xbrl.org/2003/linkbase ../lib/xbrl-linkbase-2003-12-31.xsd">
    <definitionLink xlink:type="extended" xlink:role="http://www.xbrl.org/2003/role/link">
        <loc xlink:type="locator" xlink:href="EssenceAlias.xsd#TaxExpense" xlink:label="labelTaxExpense" xlink:title="TaxExpense"/>
        <loc xlink:type="locator" xlink:href="EssenceAlias.xsd#ForeignDomesticIncomeTaxExpense" xlink:label="labelForeignDomesticIncomeTaxExpense" xlink:title="ForeignDomesticIncomeTaxExpense"/>
        <loc xlink:type="locator" xlink:href="EssenceAlias.xsd#CurrentDeferredIncomeTaxExpense" xlink:label="labelCurrentDeferredIncomeTaxExpense" xlink:title="CurrentDeferredIncomeTaxExpense"/>
        <definitionArc xlink:type="arc" xlink:arcrole="http://www.xbrl.org/2003/arcrole/essence-alias" xlink:from="labelTaxExpense" xlink:to="labelCurrentDeferredIncomeTaxExpense" priority="0"/>
        <definitionArc xlink:type="arc" xlink:arcrole="http://www.xbrl.org/2003/arcrole/essence-alias" xlink:from="labelTaxExpense" xlink:to="labelForeignDomesticIncomeTaxExpense" priority="0"/>
    </definitionLink>
</linkbase>

The XBRL document is invalid, because the two facts are in an invalid essence alias relationship. The relationship is invalid because the rounded values of the facts differ. In the current CTP, this issue is not reported by Gepsio. Starting with the next CTP, this issue will be reported during validation through an XbrlException thrown back to the caller. The exception will contain a message describing the mismatched values. For the document above, Gepsio will provide a message that reads as follows:

Facts named TaxExpense are defined as being in an essence alias relationship with facts named CurrentDeferredIncomeTaxExpense. However, the fact with ID has a rounded value of 200, which differs from the fact with ID , which has a rounded value of 100. These two facts are therefore not in a valid essence alias relationship.

(Gepsio will include the fact IDs in the message. The document shown above has no IDs for the facts, so that part of the message is empty.)