Of course, http://hitachidatainteractive.com/ remains at the center of the conversation — all the technology in the world is no substitute for a thoughtful editor.

Personally, I strongly believe XBRL will improve the transparency of financial reporting, which I believe should be the #1 goal of all of us in the financial reporting supply chain. In my opinion, the real debate seems to be in how to get there most efficiently.

Needless to say, being the first company found, as Michael says, “to have an incorrectly tagged XBRL document render to look identical to the paper document, but in fact distribute the data in a[] misleading way” would be bad. Very very very bad. Given the attention paid to large cap company financial statements, even in dead tree format, I doubt the big firms that represent the vast majority of public shareholder investment could get away with it. By the time smaller companies, which are a relatively tiny share of total market cap, are subject to the mandate, that software the SEC is looking to buy should be upgraded a few times. If we’re smart, technology to produce more reliable financial statements for smaller companies will soon be in place too, meaning the smaller companies, which produce relatively more new jobs, will have access to a fairer share of public capital, and therefore XBRL will help reduce the unemployment rate.

I’ve never sold short, but if I happened to have an analyst or two with extra time on their hands who could write software to help identify serious tagging problems before the rest of the market and the SEC, it might be nice to short companies with significant tagging problems ahead of their public disclosure. (Whether the SEC or the private sector is the first to publicize such discrepancies will be interesting to see.)

Finally, since the third party tagging system survives while the mandate only applies to companies with more than $5 billion in market cap, and there’s potentially a 30-day gap between the dead tree version and the XBRL version for first time filers, I expect some dead tree format financial statements will end up being tagged both by the company and by third party taggers. That could make make for some interesting comparisons between what investors have long thought companies meant by particular financial statement captions and what companies in fact mean when XBRL forces them to use judgment to tie their caption to a specific provision of GAAP. Of course, Michael’s uncertainty about cost effectiveness and sufficient concern to do such work apply here, too.

It will be interesting to look back in a few years to see what transparency hath and hath not wrought.

BOB: “Doesn’t capturing key data points from text-based documents like press releases and prospectuses present the same sort of difficulties that have slowed tagging of financial statement footnotes?”

MAX: I’m not sure I completely understand all the difficulties for footnotes, but yes, the entire problem in corporate actions is just that: “capturing the data points from text-based documents.” I’m sure we’ll have to be flexible and open in designing the solution. This is especially so because announcement of a corporate event is not a singularity, but a stream of updates from the first, early notification to the final terms. If you think of any big merger, you can imagine the complexities from the first news through anti-trust review, through shareholders’ meeting (think of Oracle-Peoplesoft or Microsoft-Yahoo).

The really big point is that the very difficulty you describe — i.e., teasing out the correct data from a 2 paragraph press release or a 257 page prospectus — is currently being done redundantly by dozens of firms, thus introducing significant opportunity for differences. These differences emerge when the standard messages that flow downstream through financial servicers to the investment firms whose holdings are affected. A “fuzzy” announcement, with room for interpretation, guarantees significant effort to manually reconcile differences, at significant cost. So no matter if this approach is fraught with technical challenges, the benefits of the solution make it all worthwhile.

BOB: “And with an overhaul of financial regulation in view in the US and other jurisdictions, won’t corporate action taxonomy maintenance be unusually difficult for the next few years?”

MAX: The corporate action taxonomy is not going to be driven by financial regulation, but rather by the messaging standard for corporate actions, ISO 20022, which is already quite mature. We don’t, at this point in time, see huge changes coming in content of disclosure and we actually hope for a rule to add XBRL to the existing disclosure requirements. We are taking the position that the corporate action information from issuers is generally adequate, but the method (pure text) is inadequate because it injects the opportunity for faulty data systemically into the notifications that reach the investors.

Another way of saying this is: “the data is there, it’s just darn hard find.” This obscurity is due to the legal wrapping that is required. Remember that this is all in order for a corporation to offer or announce actions that will affect their investors’ holdings. The legal framing and conditions and terms and exclusions really cannot be mucked up. All that necessary verbiage is intended to deal with litigation risk. But buried therein is a lot of data that will operationally generate changes to millions of accounts multiplied by hundreds of servicing accounts multiplied by multiples of amendments per announcement multiplied by 400,000 unique events per year (in the US).

So this is a very fundamental problem and is not significantly confounded by regulatory changes. The standard is able to accommodate everything from a huge merger deal that lasts for many months to a simple dividend. Even then, XBRL will be a progressive step forward for the corporate actions community because if a regulatory change does impact an announcement, an extension can immediately capture the metadata of a change and serve to feed back into both the standard and the taxonomy. If we can close the loop in this system, then changes can be integrated much more efficiently.

When input data itself is audited, wont the output be called audited also?

What ever data that is signed by auditors, the same is going to be used for filing interactive data.

Why is then confusion. Whats the need of fuss, for auditing interactive data?

It is true that one reason for the mandate was to give investors better information, which presumably they want. But because there were other reasons, the “contradiction” isn’t quite that. Other reasons included that tagging by people who know what they mean presumably results in better information than tagging by third parties, which continues to be the case for companies not yet reporting in XBRL. Establishing a sturdy, more useful, more widely used common computer standard in the world of public company disclosure has many benefits beyond improving the data that investors already use.

Computer standards competition can go on forever, and left to its own devices typically does. But until someone steps up to the plate and finally says, “enough,” the benefits of standards are limited. HTML got where it got mostly because it was the only game in town and Web browsers to use it were freely available. U.S. GAAP, on the other hand, because of the challenge of making it reliably machine readable in HTML or plain XML form, required a more sophisticated computer language — and the economic incentives to support the development of less expensive tools to use it. The mandate was intended to drive that development, although the leisurely phase-in has resulted in similarly leisurely software development. (Wealthy investors have long had access to costly XBRL tools to analyze financial statements tagged by third parties.) The facts that XBRL is being used in China, Australia, the UK, the Netherlands, Spain, and in many other jurisdictions for various data in the information supply chain, and the ultimate benefits of global business communication, were added features.

It’s also important to note that it’s the XBRL spec that is now at version 2.1, not any particular taxonomy. With respect to the U.S. GAAP taxonomy, we’re now at 2009. 2010 should be done early next year. One more reason investors and preparers might like XBRL: Their current software will work with updated taxonomies, as long as the spec remains the same. And since disclosure costs are paid from profits that would otherwise belong to investors, and investors benefit when the companies in which they invest can attract enough capital to expand their profits, long-term investors should like it when technology facilitates additional automation of financial statement preparation and auditing. After all, it’s a lot less expensive to build a computer program that can perform the same task repeatedly at zero marginal cost than it is to pay a person to do the same task repeatedly.

It’s the human controls that are most expensive and least reliable when it comes to SOX. Now that it’s possible to use technology to verify that data hasn’t been altered between the source and the output, auditors who take advantage of such tools should be able to bill their clients fewer hours and provide stronger assurances. The question isn’t whether investors will demand XBRL — you’d be pretty strange to have an abstract attraction to something with such a strange name — but whether management will demand better auditing using modern technology at lower prices on behalf of their investors.

A focus on human auditing of XBRL DOCUMENTS could, indeed, be expensive. That’s why machine auditing of XBRL DATA is important. If investors should demand anything, it should be that XBRL improves the business process of auditing. When the SEC Chief Accountant talked about faster, better, and less expensive financial information from XBRL back in 2005, he wasn’t talking about it for his own health. He was talking about it for the health of U.S. capital markets. Much of the newspaper profession didn’t see what the HTML standard would mean to them until it was too late for them to do anything about it. Auditors still have a chance to adopt to the XBRL standard before the potential onslaught of tools that will involuntarily change their work. Perhaps the clock is ticking before technology starts to make old fashioned financial auditing practices seem obsolete to many investors, just like technology has made old fashioned newspaper reporting practices seem obsolete to many readers.

The neat thing about predicting the future is that it’s tough to predict exactly how things will change. But it’s a pretty sure bet that things will change in ways we don’t expect. It seems like a safe prediction that a standard global computer language capable of supporting a revolution in the entire business information supply chain could result in things that even the experts who read and write this blog can’t anticipate. The best one can often do is follow the old Boy Scout motto: Be prepared.

Perhaps I am misinterpreting what you mean by “fold up”, but context element’s @id is ID, while @contextRef on facts is IDREF.

“Similarly, linkbases, one of the central characteristics of XBRL, intermix a huge amount of information that needs to be processed and stored in memory in a far more efficient form, typically one that involves resolving three to four orders of links. Yet for some areas such as labels, the most likely putative cases for having more than one label per schema, localization, is generally better solved by having multiple localization files, each of a different language, with the mappings contained in a direct one-to-one manner between schema name and label all the terms in that language. When multiple instances do arise, the XML can be modified accordingly with secondary attributes to discern this information.”

Language is not the only variation of labels for the same element. Another factor is label role. For example, for a “Profit or Loss” element you can have a “positive” label (”Profit”) and a negative label (”Loss”). You can use terse labels for display on mobile applications. You can even create your own label roles. And even switching display between languages dynamically (without having to re-request or re-process anything) is easier achieved with a single label linkbase. Nothing prevents you from creating a separate label linkbase file per language, role, or their combination, and only including those you are interested in into the discoverable taxonomy set.

As far as other types of linkbases go, they are not necessarily restricted to linked lists (trees), as some allow cycles. Generic linkbase allows you to reference a lot more than just concepts and resources. It is used, for example, for Formula linkbase, where you can find references for filters, variables, variable sets, etc.

Of course, XBRL files that make up taxonomies and instances can be processed in XSLT and at times it is very useful. However, there are many other kinds of transformations or queries that cannot be done in XSLT. Standards for expressing business and workflow rules in XML are only beginning to emerge. Of course, there can be a simplified XBRL model, more suitable for XSLT or for running XQuery on, but I doubt it would be the canonical model (that is, roundtrip-transformable).

Let’s begin by noting two distinct use cases – one for data points and one for formatted disclosures. If you think about your hypothetical request to the SEC, you are asking for data, an arbitrary set of data. Responding with whole documents is probably not the right way to answer this query! You’ve got a nice WHERE clause, but your SELECT needs work. For example, look at the posts by Peter Haynesworth on xbrl-public. He’s the kind of financial analyst you are thinking of, and all he wants is five or six data points on every Rhode Island based company. (I’m not sure the SEC will ever have a web service like this, but other data intermediaries will.)

The other use case is for specific disclosures – data with structure and presentational qualities. For a number of netbook and web-phone uses, I do not think that client-side mashup is the right way to structure this, it should be formatted on the server.

You could possibly satisfy both of these “last mile” use cases with Inline XBRL – XBRL instance data wrapped in XHTML. The server side app is not going to send either user the entire US GAAP Taxonomy in one go, no more than any AJAX app dumps an entire database onto the client. The trick is to know how to chunk the data. The server side of “HyperAnalyst” is going to intermediate between the web clients and the regulatory filings. It has to, because most queries are going to ask for comparative data across several filings from different companies.

It’s not a question of “Is XBRL lightweight enough?” It is a question of “How do I store XBRL in a RDBMS or CMS for most effective satisfaction of the expected queries?”

Can you substitute JSON or YAML data structures for the XBRL? Sure, but then you are erecting a semantic firewall between the user and the source data.

The somewhat more substantive issue underneath all of this is whether XBRL needs to shift from a syntax based standard to an infoset/model based standard with official implementations of that model in different syntaxes. Several years ago I argued against such a move, but it is a question that should be periodically re-examined.

http://www.thenationaldialogue.org/ideas/data-quality-and-data-standards-are-key-to-gaining-public-trust

One clarification: By “pro-sumer,” I mean the Toffler concept: http://en.wikipedia.org/wiki/Prosumer.

If David Weinberger is correct that “Everything is Miscellaneous,” my attempt to apply a taxonomy to data users may be misguided. In any case, I don’t think DVK is saying Web technical users are unimportant. David? I agree Web technical users are vital — and must also have standards available to them so they can make it as easy as possible for non-technical and deep-technical users alike to move from documents to data. Of course, standards should be as light as possible, but no lighter. I look forward to witnessing progress along all of these lines at the Santa Clara conference, July 28-30 (http://xbrl.us/News/Pages/20090629.aspx).

Today’s unemployment data underscore the importance of fast action to improve market trust and confidence — and bring to mind Elvis:

A little less conversation, a little more action please
All this aggravation aint satisfactioning me
A little more bite and a little less bark
A little less fight and a little more spark

Or, to paraphrase another thinker, it’s not important how it happens, just that it happens — it being better information for economic growth. Notwithstanding Elvis, whatever David, Kurt, and others can do to break down data standard silos can only expedite progress. That requires conversations like yours. Keep up the good work!

This is turning into a very interesting discussion, though I’d like to see it move away from being confrontational (and will endeavor to do the same on my end). To make it a little easier to address points, I’m going to number them, in response to both comments.

1. The Simplicity Principle. I think that Paul gives a couple of very good examples, perhaps not intentionally, about the simplicity aspect. I know Tim Bray (have even had the rare privilege of his kids and mine playing together once or twice) and while I can’t speak for him, I suspect that what he’s saying actually is quite applicable. Paul lays out the distinction for the specification as non-expert consumers (lay people), expert consumers (analysts), and pro-sumers (regulators & Fortune 500/1000 accountants). While I think this is valid division, I’d also recognize a second dimension – non-technical, deep technical users, and web technical users, which I’d define as follows:

Non-Technical Users. These are people that may have some accounting, finance or investing skill, but that do not have the technical ability to build applications that can utilize XBRL in any manner, and as such, are dependent upon tools that will produce meaningful results from XBRL resources.
Deep-Technical Users. These are application developers that are deeply skilled with XBRL, have the technical acumen to integrate these into commercial applications or pipeline processing applications (such as BPEL), and are usually employed by vendors developing tools in this space.
Web Technical Users. These are people who work with data streams and data processors to build what are called “mashups” in the vernacular, though I think a better term for these people are integration developers.

David’s contention is that this third group of people (if I’m reading your last post correctly) are not that important to the evolution of XBRL. My contention is that they are in fact critical, which is in fact the underlying thesis of the article that I originally wrote.

There are four primary trends going on in application development circles at this stage, and each of these has some impact upon XBRL. The first is the rise of RESTful Web Services. What this means in practical terms is that we are moving to a deployment model in which resource URLs act as front ends to databases, HTTP GET, POST, PUT and DELETE become primary mechanisms for interacting with databases, and parameterizing these URLs (essentially binding them to queries within the database) can be used to return feeds of content that are custom to a particular goal.

For instance, I should be able to point to the SEC’s website relatively soon with a URL of the form

http://www.sec.gov/xbrls?q=total-assets:1.2B,5.5B;reporting-quarters:1Q08-3Q09;industy:pharmaceuticals

and get a feed (either RSS, Atom or possible plain old XML) which contains links to all XBRL filings that satisfy the above constraints (the specific URL is fictional, just used for illustration purposes).

The links in turn would return the XBRL documents in forms that could be utilized by the lay consumer (from simple HTML based report listings for that XBRL) to the analyst (forms that provide graphical analysis of all pharmacuticals in that domain) and to the regulator (showing trend analysis for company X compared to the industry overall). The mechanism for generating these different faces come about via XQuery or XSLT Transformations, or via jQuery or similar components.

Similarly, I should be able to post an XBRL document to the SEC, probably along a secure HTTPS conduit, which would then be validated, processed and made available as part of the feeds described above. What’s more, this same kind of mechanism also allows for the development of web enabled tools (XForms, XBL based or one of a couple of dozen AJAX toolkits from TIBCO to jQuery) to allow for the creation of XBRL content, right from your browser.

Such information, moreover, can be processed in such ways as to provide a certain degree of opacity as necessary for the companies in question in order to prevent exposure of sensitive content – through role-based filters or similar mechanisms. This type of mechanism is one of the key characteristics of medical record systems (which I have worked on, in the HL7 space) and is instrumental to being able to provide adequate adoption of records in that space.

This kind of architecture is now becoming prevalent in computing circles, and similar architectures exist for working with such content in XQuery, Ruby, Python, RESTful Java Servlets and .NET, among others.

This technology is here. It has proven itself on smaller scale implementations, and is now scaling its way up to domain content in areas as intractible as health records, government specifications and, yes, business reporting formats. Thus, when I’m talking about the kid building mashups, keep in mind that this kid is a programmer working for the SEC or fill in the blank Fortune 1000 company.

The second trend that’s occurring at this stage is the evolution of document-centric (and property centric) databases. When XBRL was initially designed, most of the databases that were available to handle processing were relational SQL databases, and it shows clearly in the underlying design of XBRL, in which structures are maintained in as flat a manner as possible in order to better handle such databases.

However, since 2007, there has been a profound shift in the market towards the emergence of XML-capable databases, most of them employing XQuery. In mid-2009, as I write this, every major database has an XML database offering, often integrated directly into their next generation product line. Companies also have dedicated XML databases that are completely optimized for XML querying, able to serve up XML content in a small fraction of the time of established vendor systems, and there are even a number of very capable XML databases such as eXist-db available in the open source realm.

Most of my comments about XBRL’s lack of optimization come from the fact that the linkbase structures that XBRL employs are the least efficient way of being able to pull together relevant data within the technologies employed by these databases, decreasing performance in some cases by an order of magnitude or more compared to document structures. I know – I’ve run a number of benchmarks comparing multiply linked flat content compared to document-centric content (XBRL documents, specifically from the SEC), and the performance from such unoptimized XBRL frankly sucks. The most efficient solution is, upon importing such content from the SEC feed, to actually reify it into folded document structures and use that, which makes the accessibility of the documents reasonable for queries. This is my central argument, that if canonical XBRL is in fact this inefficient, perhaps an alternate format that is more document-like may be more amenable to processing.

The next trend that is occurring is the move towards client modularization. We’re moving away from storing HTML content and moving towards streaming of structured data content (mostly XML and JSON, but occasionally YAML or similar formats) to web-enabled components that can then handle conversion of this structured data into an appropriate output format. In this case, small and lightweight matter.

To give you an example of this, consider the case of a hypothetical tool called HyperAnalyst. It’s an extension to Firefox or (soon) Google Chrome that will query a repository of financial reports in XBRL* as illustrated above, and receive a feed to the relevant reports through an appropriate web or web-like gui. This component may be a browser extension. It may be an inline component. It may be a stand-alone widget built on Webkit or similar application, or even an iPhone app, but the key is that it will be accessible at a moment’s notice, and it will allow for fairly sophisticated metrics and similar comparisons through services. The tool may also grab FinancialXML and UML or custom XML or JSON from the NYSE or Bourse.

The question I’d raise is whether XBRL, as it exists now, is lightweight enough to work within such an environment. Your response time is no longer going to be acceptable if it takes several minutes to download a set of five or six large XML files on even a high bandwidth system, and if that code also needs to renormalize the internal references and bind that to the appropriate presentation layers for validation, this application is going to be a dog performance-wise.

Is that analyst not in your use case for XBRL? Most financial analysts and financial reporters that I know are as likely to be watching the market over their iPhone or Netbook while talking with clients or investigating trends while sitting at Starbucks. Those are tomorrow’s platforms (and rapidly are becoming today’s platforms).

The final trend is the shift from static to streaming content. The scenario that I see here is one that I expect is going to be extraordinary common for areas such as carbon markets (because they are essentially run by by the same young turks that are writing those widgets), but will eventually shift to most equity and commodity markets. In this particular case, what you see within a company is a RESTful XBRL server that is tied into the financial operating system of a company. What this means is that, far from reporting information on an annual or quarterly basis, this information is being updated in real time – the financials of a given company are essentially up to date as of the last day, or hour, or even minute.

This scenario is of course one of the holy grails of XBRL (and should be if it’s not) – real time financial reporting is what makes this technology so compelling. Do I think that banks and big multinationals are going to move to this? No (at least not immediately), but I also think that banks and big multinationals actually represent only a small portion of the overall potential for XBRL as a technology. This technology will become dominant with new companies that are going to come out of the current economic carnage, because they will see it as a selling point – their financials are transparent, people can trust them. Again, I think that if such a system becomes reliant upon a complex format that doesn’t integrate cleanly with their XML pipelines, that doesn’t respect the idea of services, and that isn’t performant, then they will seek solutions that are – and XBRL will become an also-ran technology.

This is the case to be made for lightweight protocols that Tim Bray, Jon Udell and others are advocating. Both Tim and Jon and a number of others that I know in the technology community that are following emerging technologies (and in some cases creating those same technologies) understand that the above scenario is happening, and recognize that simply having a common accounting language is not enough. Some concession needs to be made to its use within the web in order for that technology to succeed.

BTW, there are certainly precedents for this. The Geographical Markup Language was formulated by the OGC at about the same time as XBRL, and shares certain similarities in terms of complexity and weight. As more and more people used GML, they came to realize that it was not sufficiently performant for web use, and two different standards – one based upon Google’s KML format for GoogleEarth, the second, geoRSS, based upon the Atom specification, were developed to provide top-level metadata and a packaging mechanism over syndication channels. These proved so successful that the OGC incorporated them as part of the GML specification, essentially as wire formats.

This is what I was referring to in my original posting that prompted this discussion. There is in fact no need to radically change the underlying XBRL model, but such wire formats would automatically perform the task that currently needs to be handled programmatically – denormalize the contexts, associate labels, calculations, and reference bindings to property nodes as attributes, consolidate the XSD schema, and bind them together as a single XML document, identifying contexts so that in those cases where you have one-to-many associations you can create idrefs back to the initial id.

2. The Semantic Web. I endorse both document-centric development and semantic web structures for a very simple reason – they do very different things. XML is relatively lightweight, can be quickly indexed and queried and can be transformed readily into different formats, making it ideal for both web-centric and pipeline oriented applications (which are, in my experience, converging to the same thing). XML is not good for imputing relationships, because it is node-centric rather than link-centric, and as such, it has less utility for performing inferential analysis with other data that may not necessarily fit within its schema.

RDF/OWL, on the other hand, is edge-centric, and its primary mechanism is to create relational graphs. XML lets you compare apples and apples, and for many, many tasks, that is primarily what is needed. RDF/OWL lets you compare apples and oranges, and infer from that that both are fruit, both are roughly round shape, that they originate from different parts of the world and that one happens to be earning more in the Chicago commodities exchange.

As such, RDF/OWL requires both a different set of computational mechanisms and a different way of storing and working with that information. It is also, generally, a couple of magnitudes slower in processing time than XML because what you are doing with inferencing is creating networks of informational content on the fly – a significantly more complicated task than traversing trees. As such, SW (I don’t like typing RDF/OWL all the time, so will use this as an abbreviation for Semantic Web) has different types of applicability than XML.

The fundamental challenge that XBRL has in the Semantic world is that the linkage architecture that comes from link bases is very primitive in comparison to what SW is able to bring to bear, from my understanding of both. There’s a certain degree of inference which needs to be made in order to extract enough information out of an XBRL document to make it worthwhile. That doesn’t mean its not worth the effort, it just means that it is not a trivial operation.

One of the things I’ve come to realize with regard to SW is that it really requires the advent of a powerful triples store database in order to gain significant benefit from it – just putting an XBRL document into an RDF or Turtle format doesn’t gain you that much advantage, save that it makes it easier for such a database to consume the XBRL and build the triples linkages from it.

I think that this is an effort that should be seen as another “format” for XBRL, one for which you can write a single centralized transformation (such as being worked on now by Dave Raggett of the W3C). Its utility comes from the fact that once you have an XBRL document in RDF, it can be consumed by a triples database (and converted, as a consequence, into a set of triples) which in turn handles the associated mappings and bindings for creating appropriate “lenses” to view the data. This means that you can do such things as described above – being able to see how the R&D investment vs. earnings ratios for those same pharmaceuticals correlate to the efficacy of treating depression pharmecologically, something that would be difficult to do with XML because what’s being explored here are relationships, not nodal content.

Again, and I think this is an important point, what is being explored here are ways of utilizing the information of the models in formats that are appropriate to their respective needs. The XBRL format that was developed over the last decade did so largely as an institution-to-institution mechanism, and as such evolved to fill that niche. I believe it is a mistake to confuse the ontological design with the delivery format, and to recognize the benefit of having expressions of the language that can readily (and directly) work with other technologies such as the Semantic Web.

3. Document Enrichment and Documentation. There is yet another middle ground that is being explored by the XBRL organization, and that is the notion of RDFa in the context of document enrichment. Enrichment’s an interesting concept – by the use of attribute bindings you can associate information in a document with a particular conceptual namespace. There are two approaches that can be employed here. The first approach is that an accountant, probably via tools, can embed within an annual report written in HTML, DITA, ODF or WORDML, the relevant associations to property bindings against an XBRL ontology. The relationships so embedded can then be parsed by a GRDDL processor to generate RDF, which can then be consumed by a triples store.

The second approach is perhaps more intriguing, and that is to utilize this process in reverse to generate reports. In this case, the report writer establishes the RDFa elements in a document template, and, by running these against either an XQuery or SPARQL transformation (or an XSLT, for that matter), this populates the template with the appropriate values for the given context or operation. I see it being employed much more extensively now that the W3C Semantic Web Group has stepped up their XBRL Activity.

By the way, you also asked about the role that DITA plays, vis-a-vis XBRL, as well as it’s “goodness”. Again, my focus is essentially on utility. DITA is a topic-oriented documentation system, and, while it’s something of a pain in the butt to set up, is very effective at being able to create discrete topical blocks that can, nonetheless, be compiled together via DITA maps into localized help and information context systems. In that regard, I think that XBRL and DITA are ideally suited for one another, especially as you see XBRL move from being a static format to being more of a services oriented technology. DITA can be transformed into HTML content, can embed XBRL/RDFa content and contexts, and can similarly be compiled into more extensive documentation, from CFM help files to large scale living annual reports.

4. NIEM. There were a group of about five senior ontologists and information architects that collectively worked on the NIEM architecture from its origins in the DoJ to its current “wildfire” growth through much of the Federal government. I know two of them personally, and they are easily two of the best data architects I’ve ever met, globally, so I’d be careful about dismissing NIEM out of hand. More to the point, they begun this effort largely under the radar of the Bush administration, and as such, the level of politicization was consequently very low.

I brought up NIEM to illustrate differences in architectural approach, not necessarily to endorse it as a better solution (though I think, obviously, that it is a good one). More to the point, I also think that it has the potential to be a competitive solution to XBRL within other governmental agencies, especially in areas such as procurement, resource accounting, financial contract management, and energy systems management. Without Chris Cox championing the standard at the SEC, and Mary Schapiro singularly cool about XBRL, I think that understanding specifications such as NIEM and how XBRL could be adapted to work in an IEP model might prove to be most prudent.
specification

5. Namespaces. Again, this is a difference in design principles. NIEM utilizes namespaces as the cornerstone of a document design architecture – in essence, there’s a one-to-one correlation between a namespace and an IEP (not strictly true in all cases, but a good general rule). XBRL utilizes namespaces to identify authority rather than topicality, as far as I can tell. Both are valid approaches, and there are somecorrelation between deep debates within the XML community about the relationship between namespaces and object classification. Personal preference, I’ve come to align toward namespaces establishing a topical reference rather than just an authoritative one as this seems to result in cleaner ontologies, but I’d also concede that large scale data design is still more art than science, and as such is subject to interpretation.

6. External citations. This may be perhaps a misunderstanding of mine with regard to XBRL. External citations in this regard refered to Reference Linkbases, which serves, I assume, to establish equivalency of terminology (if term A in XBRL document A and term B in XBRL document B both reference the same reference citation under the same arc role, then this should imply that there is a semantic equivalency between A and B). However, I’ve noticed that this isn’t always satisfied properly in XBRL documents that I’ve read, so going by real world examples, I may be missing something.

7. XLink. XLink has been somewhat contentious for awhile, largely because as a standard it has needed to satisfy the concept of linkages in a number of different contexts. The RDF/OWL community would argue that the Semantic Web effectively obviates the need for XLink. The HTML community gets twitchy about anything but simple links, the XML community has perhaps been deliberately vague with regard to this specification, and its adoption in other areas tends to be spotty – SVG, XForms utilize it, XHTML theoretically does but only XML purists use xlink, DocBook (which isn’t a W3C spec) uses it, but other technologies don’t. XInclude theoretically does use it, but XInclude seems to scare the heck out of the HTML Working Group (of course, anything even vaguely XMLish scares at least some members of that group, but that’s not a discussion for this forum).

David, I personally agree with you on XLink. I’d like to see the W3C come to some formal decision as to what actually it is, does, and where it’s scope lies, and would prefer to see it universally required in the underlying schemas, but I’m probably one of the few people who feels that way.

In summary, my suggestions with regard to XBRL involve examining that analyst use case, as ultimately I think it has a great deal of bearing upon the further evolution of the language and its adoption. I am not suggesting that XBRL 2.1 change in any way – it solves handily the use cases that it was initially designed for, it has widespread adoption in the institution-to-institution space, and it effectively does so in a manner that is consonent with the existing participants. However, I do feel that this optimization leaves it suboptimal in other areas for which there are relatively simple fixes – making it more streamlined and better able to work in a services environment, providing an accepted mechanism for mapping XBRL into Semantic Web terms for consumption in inference engines, establishing ways that the language can co-exist in mission critical arenas with other specifications, from NIEM to DITA to RDFa. None of these reduce the importance of the language, its design or its goals, and I believe that all of these may actually encourage wider-spread adoption of the language outside of its initial area of regulatory reporting.

I agree that ontologies need to be designed and modeled carefully, and independent of process if we intend them to have applicability past the original process definition. This is the lesson of relational vs hierarchical databases. George Santayana had something to say about this.

The idea that XBRL is not an efficient representation for XML processing systems is a red herring. As you have stated before, we are always “just a stylesheet away” from a more efficient processing format. As you have stated before, storage, network speeds and processing speeds constantly make these issues less important. The main point is not to judge a transfer format as a processing format. There is this concept of normalization and denormalization in the implementation of relational databases that is very apropos here. (Santayana, ibid.)

I’m glad you are not trying to credential yourself as a language designer on the basis of your writing experience. I think your point that there have been advances in the last ten years that XBRL should take note of in any future refactoring or redevelopment of the core standards is very important and valid. There are also things that haven’t been done in that time period (by the W3C, for example) that are even more important than RESTful architecture, AJAX, and Drupal, if that is conceivable. For example, SQL, Java, UML, and XML Schema all have incompatibilities in their data typing systems. Wouldn’t it be nice if there were no corner cases to code for because of these differences?

With respect to the NIEM, I have to admit to an initial skepticism about government developed designs or methodologies. Remember Ada? However, given NIEM’s focus on issues of criminal justice and homeland security, it is inevitable that NIEM and XBRL are going to intersect in their concerns over the area of forensic accounting, fraud, money laundering and general “follow the money” issues.

I agree with your position that XBRL is closer to RDF/OWL than may appear on the surface. This is in part because of the reification of multiple structures in XBRL (those pesky but oh so useful linkbases!).

If the bottom-line benefit of IEP, etc is that it is a document centric model, I will agree that there is a key philosophical divide about designing data centric vs document centric languages. I will happily admit that XBRL is designed to be data centric, but can fit into a document centric model. Just as NIEM has a data dictionary schema that it mines for elements for its packages and models, XBRL taxonomies serve the same purpose. You can even do document centric modeling with XBRL, to some degree, using tuples. I’d be very happy to explore mixed NIEM-XBRL applications.

I’m not sure why you say XBRL avoids namespaces. XBRL uses namespaces just as much as other XML Schema based XML designs. Can you give an example of what you mean? I’m also not sure what kind of ‘external citations’ you mean, and why you think they have something to do with equivalency. Let’s say taxonomy author Alice creates an element in the substitution group of us-gaap:assets, and includes in her taxonomy a reference linkbase with a reference arc from alice:A-assets to the same definition of assets in the FASB Codification as was used in the US GAAP Taxonomy. Taxonomy author Bob independently does the same thing in his taxonomy for bob:B-assets. Instance author Charles uses both taxonomies. Nothing in the above guarantees that alice:A-assets = bob:B-assets in an instance in the same context.

On the subject of DITA, since DITA is being used by the FASB for the new Codification, there are plenty of opportunities for XBRL-DITA interaction. Where does DITA sit in your scale of design goodness?

XML is semantically neutral. That neutrality guarantees that data centric designs are as ‘valid’ as document centric designs. If tools are not neutral, that is not something that can or should constrain design. Tools can change faster than language designs. EDI/X12 is document centric. (Santayana, ibid.)

I’m not sure why you would say that XBRL is or wants or tries to be a third alternative to SW or XML designs. XBRL is nodes and edges, graphs of data and metadata. XBRL instances and schemas are lists of nodes, XBRL linkbases are lists of edges. It is that simple.

Web apps may become a flood, does that mean they won’t use databases? I agree that an AJAX app should not be flipping XBRL back and forth between the browser client and a server. Why would you even start down that path? I think you’ve seriously misunderstood the use cases and actors if you think an investment analyst or fund manager needs an XBRL parser written Drupal.

I’m having trouble reconciling your enthusiasm for NIEM document centric design with your enthusiasm for SemWeb designs using RDF/OWL etc. As you’ve said, XBRL (under the surface) is close to the “bag of triples” designs of the Semantic Web. Do you not agree that there is a tension between these enthusiasms or do you feel that XBRL is somehow exempt from the way you resolve that tension?

Of all the W3C technologies upon which XBRL is based, I would hold that only XLink is seriously in trouble (out of date and largely abandoned). XML Schema, Namespaces, XML Base, XPath – this criticism does not apply. Do you think there should be a standard XML hyperlinking language? I do, and I would advocate reinvigorating XLink in order to satisfy this problem.

As a conversational gambit “I’m a humble writer” somewhat undercuts your previous “I’m a credible XML expert” position. I absolutely disagree that XBRL’s success or failure rests on its appeal to developers. Tim Bray has made similar comments that XBRL needs to be a wildfire, up-from-the-coder success to truly be a winning technology. That is making a mistake of the superficial (XBRL is XML and must follow the path of RSS.) with the essential (XBRL serves the needs of corporate data transfer.).

That is an excellent point worth repeating. It goes to the heart of the role of information in the marketplace.

Some information is more likely to be used by “expert consumers,” other information by “pro-sumers,” and still other information by “non-expert consumers.” A remarkable thing about XBRL is its robust capability to support a variety of uses. (As for admitting “the obvious,” note that XBRL US has called itself “the national consortium for XML business reporting standards” on its Web site for some time.)

The more information in the marketplace available to information consumers at various levels, the better the price system works — whether it’s the price of gasoline, the price of food, the price of stocks, or some other price. In the case of gasoline, experts can tell you what octane rating is better for your car; a pro-sumer might account for the car wash discount at a particular station; you can balance the distance to a particular gas station against its lower price. (Unfortunately, it remains difficult to raise capital to compete against oil companies because it’s costly and relatively complex for small companies to live in the relevant information eco-system — perhaps XML and XBRL can help with that, too.)

What should “drive design and technology choices for XBRL” is what gets the most information into the marketplace in the most timely manner so that information consumers at various levels can use the information, directly or in collaboration with other information consumers with varying degrees of expertise.

Here’s one example of why getting information into the marketplace sooner could be more important than delaying things for “non-expert consumers.” While nothing in life is guaranteed, had information about asset backed securities been available to the marketplace in more useful structured format rather than “Web-friendly” html format, the odds of an expert or an academic or a pro-sumer discovering the underlying defects in the asset class sooner would have been higher. Html was used for ABS because html, like the thousands of pages of paper underlying ABS, is “document-centric.” Alas, that didn’t turn out well at all.

(Many of us know people who don’t qualify as the “sophisticated” investors deemed smart enough or wealthy enough to trade ABS, but who might have noticed ABS defects sooner had they been able to analyze structured disclosure. Now, thanks to opacity, we all own document-centric ABS, or at least its wreckage.)

As it happened, thanks to a private implementation of XBRL, expert short-sellers discovered the ABS problems rather late, but before the rest of the marketplace, meaning they likely profited much more handsomely than would have been the case had the defects been discovered via disclosure of structured information to the larger marketplace. (See http://www.wired.com/techbiz/it/magazine/17-03/wp_reboot.) Well-structured GAAP reporting helped end the Great Depression by restoring trust in public companies; well-structured ABS reporting could do the same by restoring trust in ABS. It certainly couldn’t be worse than unstructured or semi-structured ABS “information” – information that was, in fact, posted on EDGAR in “Web-friendly,” document-centric, html format! Neither html nor the documents it represented were up to the task of fully describing complex securities big enough to create systemic risk.

A simpler example is that I’m not a CPA. I rely on the market to value public companies according to their financial statements and am thrilled the market will be able to do so more efficiently thanks to XBRL. That means my key investment criteria, such as business strategy and product quality, will become relatively more important. Alas, there aren’t mandatory XBRL tags for those criteria — yet. But it sure will be nice when, someday, I won’t need to wade through the text of hundreds of 10-Ks and 10-Qs to locate and compare what every company says about its own strategy and product quality. And if people who are better at accounting than I am can mash-up strategy, quality and accounting, because they’re reported in a compatible widely-accepted format that they had to learn to be able to use modern financial statements, more power to them. That simply means better allocation of capital and a rising tide for all.

That is all to make two main points:

First, making the perfect the enemy of the good can have significant consequences, and good people like David who have dedicated countless hours to XBRL have experience that can do for business reporting in the 21st century what GAAP did for business reporting in the 20th century.

Second, while GAAP is the most famous use case for XBRL so far, because the XBRL standard is applicable to all business information it’s difficult to predict future benefits. We do know this: Better information makes better markets, and the more applications for which a standard is used, the better the tools to use the standard are likely to become. To the extent XBRL can expedite the disclosure and analysis of information that market participants may find important, and to the extent its application to other purposes can lead to the creation of even more useful tools, full speed ahead. (Since executive compensation disclosure relies on GAAP for part of its content and XBRL data tags already exist, that might be a logical next step.)

Disclosure: Long EDGR and BR, both of which include XBRL in their strategy.

I figured that my post might be somewhat controversial, and am honored that you chose to respond to it as you have.

We’re entering an interesting age. Over the last ten to fifteen years, there has been a steady ramp of complexity of data models and modeling mechanisms. I’ve been following a fairly fundamental sea-change in the move towards RESTful architectures, a direct consequence of the understanding that data models and ontologies need to be understood separate from process. Especially when you move into large ontologies (e.g., financial reporting, health level services, aircraft component specifications), the challenge that people face is how best to model them, not just in terms of the ontology but in terms of the language formalism of the ontology.

Moreover, once a modeling formalism has been determined, the question comes back to whether such a formalism can in fact be expressed in a language that is efficient for all the potential uses, or whether there are in fact multiple representations that may be more efficient for certain types of applications rather than others. This is the heart of my contention, more than anything – that whereas XBRL has certain advantages for certain types of implementations (many deriving either from a desire to target spreadsheet or SQL data stores) and implementors (financial services companies) it is perhaps not as efficient a representation (largely owing to the need to rectify links) for use within XML processing systems.

I am not an accountant. I have, through the last couple of years, learned enough of the terminology to understand most of what’s needed from an implementation designer’s standpoint, but have neither the talent nor the inclination to be a CPA. I am, however, an XML developer, and have fourteen books to my credit as well as a couple of hundred articles, white papers, seminar presentations and the like within the XML and web development space. The upshot of this is not to play a game of my skills are better than yours are, but rather to point out that I have the experience to see that many of the issues that XBRL has spent years on are issues that could readily be resolved much more cleanly by the tools (and perhaps more importantly, methodologies) that have emerged in the last decade on the XML side.

Consider, for instance, the model currently deployed by the National Information Exchange Model XML (NIEM) body as part of dealing with complex ontologies in the US Federal information space. The NIEM model is intriguing – the use of information exchange packages and packaged documentation (known as IEPs and IEPDs, respectively) provide a way for a number of federal agencies, each dealing with their own particular domain needs, to nonetheless share a common set of ontologies.

Certainly these domains are no less complex in their own right than accounting standards are. They do deploy differing namespaces to help manage the complexity, because namespaces establish the boundaries of domains, and yet those namespaces do not materially contribute to the complexity of the specifications – indeed, they actually work well in simplifying application development around instances of these standards. What’s more, because there is in fact a central clearing house for NIEM IEPDs – for the schemas and not the instances – what this does is allow for a good middle ground in which common entities can be defined, yet that also provides room for the evolution of new, distinct objects that can either add to or subclass existing IEPs.

In essence, there is a distinct difference in philosophy between NIEM and XBRL. XBRL works upon a property-centric ideal (something that again works better in relational databases or spreadsheets than in XML per se) in which a given report is a bag of properties, the role of the schema is only to identify those properties, and structure exists only by dint of contexts. It is a model that is, ironically, much more similar in approach to RDF/OWL than may be obvious on the surface, though the latter technologies, like XBRL, tends to suffer from performance because all structure has to be imputed.

The NIEM architecture, on the other hand, is document-centric. Each IEP effectively defines a set of structural constaints, either through XSD or through other schematic languages (such as Schematron, which could go a long way to solving the validation complexities associated with equations or functions within XBRL if expressed in a document-centric approach). Presentation into XHTML, DocBook, OOXML, etc. can be easily accomplished in XSLT (and more so in XSLT2). IEP document instances can readily be stored in XML databases, and queried via XQuery in order to form reports and analyses.

Because of the document-centric nature of IEPs, development of web based tools for both viewing and editing such reports online through XForms, Air apps, .NET or Java clients and so forth is made much simpler as well; I can point together an online application for viewing and editing police reports or similar IEPD apps in a few hours, and because of the underlying standardization that’s involved, I also feel reasonably comfortable in asserting that police reports that I create for use in Washington state could just as readily be read and understood in Texas, Virginia or the US Dept. of Justice.

In the interest of avoiding namespaces and structures (even if moderated), XBRL relies upon using external citations as a mechanism for equivalency (a namespace would have done the job just as well, by encoding the equivalency as a citation reference within an <appinfo> field within a schema definition, and would have been far more efficient in terms of processing). Labels similarly would have resided within structures, or would have been referenced en mass by the schema in question in an external document (perhaps tying in DITA, which would have allowed you to maintain all of this information in a distributed topical system).

Put another way, rather than relying upon a plethora of perfectly good standards, XBRL has reinvented most of them under the justification that Accounting is different than any other form of human intellectual endeavor. This is perhaps the most egregious problem that I see with XBRL. XML is defined to be a semantically neutral language, and is designed so for a number of very good reasons. As a semantically neutral language (the definition of the concept of an infoset along with a few syntactical rules for constructing representations of such infosets) it is independent of transport mechanism, it can be transformed and processed in any number of different ways with any number of tools, it can be queried in a manner that is independent of the underlying semantics, and you can mix and match multiple conceptual namespaces as appropriate without fear of collision. These take place because at their core, XML tools are very document-centric.

XBRL’s property-centric approach could work well in a semantic environment … and contrary to your assertion, Semantic web tools such as SPARQL, RDFa, GRDDL, and RIF are very solid, there are a number of vendors providing such tools, and those tools do everything that XBRL can do, from buidling associations and referencing citations to performing inference analysis. SW differs from XML in that, while both work upon graphs, SW’s focus is edge-centric rather than the node-specific mode that XML employs. XBRL is not some third alternative – a graph has edges and nodes, it doesn’t have (nor does it need) accounting.

Ultimately, my contentions here are that XBRL, as it stands now, neither has the comparatively lightweight structure that makes it attractive as a web technology, nor does it have the rigorous formalism that makes it attractive as a semantic technology. This means that while XBRL can be utilized from within stand-alone applications that have very dedicated functionality, it will be harder and harder to justify this as the migration to web applications become a flood. You will have few people that will write XBRL extensions in Firefox or XBRL parsers in Drupal, perhaps not an issue if you are a company reporting earnings to the SEC, but a definite issue if you are an analyst or fund manager.

It means that it becomes a harder sell for XBRL as an accounting standard outside of the immediate accounting circles, because NIEM type models are gaining adherents among agencies that need to work with more than just accounting information (it may also be part of the reason for the foot-dragging of the SEC in regards to XBRL adoption, as they have a chance to evaluate alternative models).

I am not advocating that XBRL abandon the standard that they have worked diligently for more than a dozen years. In the domain where it grew up, the tools are adequate for the needs asked of it, and changing standards is always a difficult process. What I am advocating, however, is that some serious thought be given to an XBRL 3.0 that is able to be expressed canonically in an efficient XML representation and in OWL, that can be effectively packaged (and there should be a package mechanism), that can work well both for the Fortune 500 company and for the SME market. Perhaps it is time that a re-examination of XBRL in light of NIEM, OWL, the XQuery stack and similar architectures be made – the technology has changed profoundly in the last decade, and to insist upon using out of date and largely abandoned technology simply because it is too difficult to make the social adoption changes seems disingenuous to me.

One final point in that regard. I am simply a writer. What I say will not matter to my company one way or the other – I cover the XML industry as an analyst, nothing more. What will matter at the end of the day is whether developers adopt this technology – not CEOs, not CFOs. If the technology does not provide a compelling benefit in terms of what it can do in terms of the existing pipelines and toolsets that they have to work with, then you will not see wide-spread adoption of the technology beyond the absolute minimum necessary to comply to state mandates. I was originally very positive about XBRL because I believed that the goals that it was designed to solve were noble and necessary. Yet the more time I’ve spent working with it, the more I have to wonder whether it will in fact solve the problems it was intended to solve. Perhaps I am wrong, and my own perspective blinds me to the brilliance of the language, but overall I see a language that attempts to optimize on constraints that are no longer important while missing the constraints that are.

Thank you for the engaging conversation.

Kurt