Dr. Macro's XML Rants

Why DITA Requires Topic IDs (And Why Their Values Don't Matter)

2009-05-16T06:56:00.002-05:00

The DITA standard requires all topics to have an id= attribute.

Why?

The reason is simple: so you can point to elements within topics. And for
no other reason.

Surprised?

Most people seem to assume that topics are required to have IDs so you can
point to the topics. And they further seem to assume that topic IDs need to
be unique within some fairly wide scope (e.g., within their local topic
repository).

But that's not the case at all.

For the case of topics that are the root elements of their containing XML
documents and that contain no elements that themselves have IDs, the topic
ID isn't needed at all. In this case the topic can be unambiguously
addressed by the location of the containing XML document (e.g.,
"mytopic.xml").

In the case of topics that are not root elements and that are not themselves
pointed to and that do not contain any elements with IDs, again the ID is
not needed (because nothing points at the topic or its elements).

So why does the DITA standard require topics to have IDs?

It is because topics establish the addressing scope for their
direct-descendant non-topic elements.

By the DITA spec, to point to an element that is not a topic you use a
two-part pointer: {topicid}/{elementid}.

Without a topic ID it would be impossible to point to a non-topic element.

By requiring all topics to have some ID, it ensures that any non-topic
elements with IDs are immediately addressable without the need to also add
an ID to their containing topic.

In general by normal DITA practice, non-topic elements are given IDs only
when they are intended to be either used by conref or be the target of a
cross reference. Both of these tend to be carefully considered decisions
driven by editorial and business rules, not arbitrary author decision. Which
means you would tend to know, in advance of creation of a given element,
that it is a candidate for conref use or xref use, which means you know to
give it an ID at the time you create it.

By requiring that topics always have IDs it means that authors don't have to
worry about adding IDs to topics just because they also happened to put an
ID on an element. [In normal XML practice, elements are addressed directly
by ID within their containing document, which means it is sufficient to
simply put an ID on the element with no other dependencies. That is not the
case in DITA, which defines its own unique syntax for non-topic element
addressing.]

Because topic IDs are XML IDs (as opposed to non-topic-element IDs, which
are just name tokens and have no special XML-defined rules), any XML editor
will both require topics to have IDs and ensure that topic IDs are unique
within the scope of their containing document.

If topic IDs were not required, DITA-aware editors would have to have
special rules to know to require topic IDs whenever non-topic elements got
IDs and it would mean that generic XML editors would not ensure that this
important DITA rule was met (topics with elements with IDs must themselves
have IDs).

So the DITA spec requires that all topics have IDs.

But the fact that topics must have IDs does not imply that topic IDs need to
be either descriptive or unique within any scope wider than the XML
documents that contain them.

In the case where every topic is the root of its own document, the topic ID
can be the same *for every topic*. To make this point I have standard
practice of using the value "topicid" for the IDs of all my root topics.
There is absolutely no need to generate unique topic IDs for document-root
topics as a matter of standard practice.

The only other case is ditabase documents.

If you are using ditabase documents, stop.

Sorry.

There are some legitimate uses of ditabase documents, for example, as a
first-pass target for data conversions and as a way to hold otherwise
unrelated topics that need to be managed as a single unit of storage, such
as topics that exist only to hold reusable elements.

[NOTE: Using ditabase simply to allow the mixing of different topic types in
a single document during authoring* is the wrong thing to do. You should
have already created local shell DTDs and within those shells you can allow
whatever topic type mixing is appropriate for your local environment. There
is no need to use ditabase in that case and many reasons not to. See my many
other posts about why you should always create local shell DTDs as the first
step in setting up a production use of DITA.]

In that case, the topic IDs must be unique within the scope of the ditabase
element, simply because XML rules demand it. But the IDs need not be unique
beyond that scope and they need not be meaningful.

One of the implications of this is that if you always edit topics as
individual documents and never have nested topics you never have to think
about topic IDs. Your topic document template should already have an ID
value and it can be something like "topicid" and there is no reason
whatsoever for that ID to ever be changed.

In the case where you do edit topics with nested topics (for example, you're
authoring more or less narrative documents or you've designed some topics
types that need nested topics to allow a bit of hierarchy where the nested
topics would never be meaningful in isolation) then you either have to
configure your editor to assign IDs to the nested topics for you (if your
document template doesn't already have the subtopics with IDs assigned) or
you have to think about it. But even in that case, the IDs can be pretty
generic, e.g. "st1", "st2", etc. The IDs in that case still don't need to be
unique beyond the scope of the containing document.

Help Me Learn: How to Design a Solar Charging CIrcuit

2008-05-06T10:21:00.002-05:00

I have an general interest in sustainable power systems (my home has a 5-star rating and a 3Kw PV system) but I am not an electrical engineer and have no useful understanding of electrical circuit design beyond very basic stuff (I know what a resister is but I couldn't reliably tell you how resistance relates to current and voltage).

I have two projects in mind that I'd like to pursue, both of which require a bit more knowledge than I have and I have no idea where to go to get the knowledge--all of my resources are focused on small-scale electronics (digital circuits, basic occilators, etc.).

My first project is to create a water feature that is variously powered by wind, solar, humans, etc. where all the different power sources contribute to charging a storage system which then drives an electrically-powered pump of some sort. I'm thinking of something like a 6-volt marine battery, something that can hold a good charge and produce enough current to drive a beefy motor.

What I don't know is how to design a charging circuit that will feed the battery from multiple input sources.

The other project I'm thinking about is modifying an RV to be electrically driven so that it could be, as much as possible, solar powered (e.g., for traveling about the American West during summer). That is, building an electric RV that would run off batteries for cruising and be recharged by a combination of solar, auxiliary generator (presumably a diesel engine that could run on waste vegetable oil or the most ecologically sound fuel available at the moment), or grid connection when parked.

It would need to enable a 200- to 300-mile range on a single charge to account for the (almost) worst case where you have no solar input and must recharge overnight from a campground. The worst case is no solar input and no grid access, so you'd have to run the generator in order to get to the nearest power source (or wait out the clouds without your beer getting too warm).

Some obvious questions are:

- Assuming an Airstream RV (chosen to minimize drag, even though they're frightfully expensive), how much energy would be required to provide a 200-mile range at 55 MPH?

- Assuming a more affordable typical RV, what would the cost from drag be?

- Given current solar panel technology, what output could be expected from the maximum area one could reasonably attach to an Airstream? Would it make sense to include some sort of fold-out panel system for use when parked (e.g., you're stopped for the afternoon at some tourist spot)?

- Assuming worst case of no solar input and no access to the grid, what size of generator would be needed to enable direct operation of the vehicle at say 40 MPH?

All of this would go to answering the first question, which is "is this even practical with today's generally-available and affordable technology?" If the answer to that is "no", then what advances would be required to make it affordable?

We could start with the presumption of a 50,000 USD budget, which is about what it costs to buy a full-sized conventional RV. So if I bought a used one and refit it, could that even be done for that budget?

Another consideration is the value of not buying fuel. With gasoline pushing 4.00USD a gallon and diesel already over that as of May 2008, a 3000-mile trip at say 8 MPG starts to add up pretty fast. That's roughly 1500 USD in fuel costs for that trip. At 4 dollars a gallon, I can recoup 15,000 USD in investment in 10 years of driving vacations. If fuel was at European rates that payback would of course be much higher (and it seems reasonable to expect that U.S. fuel will climb to approach European rates over the next 10 years simply because of both market pressures and increasing social acceptance of the true cost of our life styles in the face of global warming).

So I'm wondering if anyone can provide pointers to resources, online or otherwise, where I could start developing the necessary knowledge to start answering these questions?

I don't think any of this is particularly challenging from either a design or implementation aspect, I just have no idea how to go about learning about it efficiently....

Choosing an XML Schema: DocBook or DITA?

2008-04-18T09:45:00.003-05:00

Richard Hamilton has presented a thoughtful analysis of when to choose DocBook or DITA, published on the Content Wrangler blog here: http://www.thecontentwrangler.com/article_comments/ choosing_an_xml_schema_docbook_or_dita/

I started to post the following as a comment to that post but it got long enough that I thought it better to post my full response here.

I generally agree with Richard's analysis as far as it goes, but I think it misses several important points that I assert tip the scales significantly in favor of DITA over DocBook.

If you are looking for a documentation schema that you can just pick up and use and you don't need the modularity features of DITA (that is, you don't need the functionality of DITA maps) then DocBook probably makes the most sense for the reasons Richard cites, namely that there are more element types of likely utility out of the box and the processing infrastructure is more mature and better documented.

However, if you know you need to add markup for your specific requirements or are developing a new XML application where things like markup tailored for local users or requirements is important or modularity is important, then DITA has a very clear advantage because it is so much easier to develop and extend custom document types from a DITA base than from a DocBook base.

The reason is very simple: DITA's specialization mechanism, coupled with the declaration set design patterns defined by the DITA architecture, make it as easy as it could possibly be to develop new markup structures. In particular, having defined specializations you may need to do nothing more in order to have documents that use those new types work with existing DITA processors, editors, CMS systems, etc.

DocBook cannot have this characteristic until such time as it either adopts the DITA specialization mechanism (which it could easily do--I worked hard to have the specialization aspects of DITA defined as distinct from the DITA element types specifically so that it could be adopted by other XML applications with a minimum of fuss) or adds the equivalent functionality using some other syntax [one limitation in the current DITA specialization mechanism is no good way to support namespaced elements--that will be fixed in DITA 2.0 but nobody has yet started to work in earnest on what that might be--this could be an opportunity for DocBook to take the lead since DocBook definitely has a namespace requirement.]

With any DocBook application, if you define new element types, there is no defined way to map those back to existing types and DocBook processors are not designed to handle new types by processing them in terms of some base type. That means that if you define new element types in a DocBook context you must update all processors that need to act with those documents even if all they need to do is nothing with those elements.

On the subject of narrative documents, there is essentially no practical difference between DITA and DocBook in their ability to support the creation of single-instance documents of arbitrary depth. This is obvious for DocBook (because that's what it was designed for), not so obvious for DITA (because it was designed for the opposite).

But with DITA all you need to do is configure your local doctypes ("shells" in DITA parlance) to allow topics to nest. For example, the simplest case is to simply allow generic topic to test. With that you can represent any possible narrative document structurally.

The only meaningful difference in this scenario between DITA and DocBook is that DITA requires the body of a section to be wrapped in a container (the topic body), while DocBook does not provide such a container (or at least it didn't last time I looked).

This is really a trivial difference.

For several clients who are doing publishing rather than technical documentation I have developed essentially trivial specializations that provide generic topics distinguished only by their topic type names but using otherwise generic DITA elements for content. I usually define a specialized topic called "subsection" that can nest to any depth. With that model you can represent documents as well as or better than you can with DocBook and you get all the other DITA goodness as well.

Finally, there is a free DITA-to-DocBook transform that is part of the free DITA Open Toolkit that allows you to use all the DocBook processing infrastructure with DITA-based content. This is used, for example, to use non-DITA-aware composition systems like XPP with DITA-based content.

Because DITA offers a number of very important features that DocBook does not, in particular specialization, modularity, and external links (relationship tables), and because DITA can be configured to work as well for non-modular documents as DocBook can, and because DITA lowers the cost of developing new element types as low as it could possibly be, I've come to the conclusion that DITA is the best answer for any XML-based document-centric application I've seen.

Just the fact you can get OxygenXML for almost nothing, define a completely new DITA specialization, deploy it to your local Toolkit as a plugin (a very easy operation once you know what to do, something I need to write a tutorial for), you can then edit documents using that specialization in a full-featured graphical, tags off editor with no additional work of any sort is pretty powerful. DocBook simply cannot enable that because it doesn't have DITA's specialization feature.

If DocBook adopted DITA's specialization mechanisms then this discussion wouldn't even be meaningful because DocBook would get all the value that specialization accrues to DITA and would still have the value of being a conceptually simpler model for documents.

Which raises the question: why doesn't DocBook simply adopt DITA's specialization mechanism? It would cost DocBook almost nothing to add and add tremendous value. It would not require DocBook changing anything about its current markup design, except to possibly back-form some base types that are currently not explicit in DocBook but would be useful as a specialization base. But that would only make DocBook cleaner.

XML is 10

2008-02-17T09:58:00.003-06:00

The XML Recommendation is celebrating its 10-year anniversary, that is, the anniversary of the official publication of the Recommendation on 10 February 1998. However I think of XML as really starting in 1996, when the activity was revealed publicly for the first time at the SGML 2006 conference. I wrote about XML and its development at that anniversary here: Dr. Macro's XML Rants: XML: Ten Year Aniversary (And I discovered this post, which I had totally forgotten about, when I googled "sgml 1996 conference" in order to verify that my memory of the dates was correct. How sad is that? [or conversely, how cool is that?--you choose.]).

I will re-iterate what I said two years ago: while Tim Bray and Michael Sperberg-McQueen, as the editors of the XML 1.0 Recommendation, are most publicly associated with the XML it was Jon Bosak who made XML happen. It was Jon who put the "SGML on the Web" working group together, personally invited all the initial members, set the working rules that allowed us to work quickly and productively, and managed the political and procedural process of getting XML through the W3C. Jon knew what he wanted and knew the ingredients that were needed and knew how to put them together in a way that would most likely produce the desired result. In that sense he was like a chef producing a dish dependent on the complex interactions of different ingredients, a dish that is not a simple assembly task but one that involves carefully managed reactions and cooking times applied to a variety of ingredients where quality was a key determining factor.

Without Jon's drive, judgment, and leadership, the XML development process could have easily bogged down or been derailed in any number of ways. It would have taken only one spoiler or resistance from inside the W3C or simple poor management of the process to kill or delay the whole thing.

It's also important to remember that what we developed as XML represents absolutely no technical innovation. There is nothing in the XML 1 Recommendation that isn't in SGML, with the possible exception of well-formedness being sufficient (since SGML required the use of DTDs with document instances). The genius of XML, and the challenge in developing the spec, was figuring out what to leave out of XML. Each of us on the Working Group had our pet features, without which we felt XML would be at best crippled, at worst useless. I think we did a remarkably good job of not including features that were not essential.

In retrospect, I wish we had gone farther and left out DTDs and entities entirely, but of course that would not have been politically acceptable at the time and there would have been nothing to replace DTDs with (in fact, I still find it amazing that the XSD spec was ever finished given the challenge inherent in developing that specification given the wide range of requirements and constituencies driving it).

I think it's also fair to say that XML has succeeded far beyond any of our initial expectations. All we really wanted was a way to publish SGML data using Web technology. It never occurred to us that it would be embraced as a general-purpose data structuring and program-to-program communication format (for good or ill). I've always found it a little annoying that the vast majority of data using XML has nothing or little to do with documents in the sense of information intended primarily for human consumption. Whatever.

I suppose prognostication is expected at this point.

Where do I see XML going in the next 10 years?

I think it's fair to say that XML is entrenched and unlikely to be replaced any time soon. It's hard to imagine that any group would have the motivation and resources to build a general-purpose XML alternative given XML works more than well enough for most of the applications to which it is put. From an engineering standpoint, it would be a case of overoptimization.

In the domain of structured documentation I think that the DITA standard in particular will accelerate the adoption of XML for docment representation. The values have been well understood for decades and they aren't going to change. Because DITA, leveraging XML's deep and ubiquitous infrastructure, lowers the cost of entry of using XML for sophisticated document representation it can only serve to bring more enterprises and users to XML, users for whom in the past an SGML or even XML solution would have been prohibitively expensive. I find that very exciting. I don't remember well enough to know if that particular effect of XML was envisioned or even hoped for, but I think we all, even at that time, understood to some degree the power that Web technology had in general to make things easier and cheaper. But certainly lowering the cost of building XML parsers was a primary design driver, our mythical "graduate student with a weekend" to build a parser. That vision has definitely been realized.

In the domain of program-to-program communication it would not surprise me if something specifically designed for that task supplants XML, something like JSON. This is a domain where, because there is no particular great body of data, but only processing code, APIs, and support libraries, the engineering equation would make optimization more attractive: there's no question that XML is not the best solution for character-based serialization of arbitrary objects and data structures. I certainly wouldn't object to proposals to replace XML with JSON for those applications. The key is to understand that XML is still the best available solution for persistent data. I think a lot of people who use XML day to day forget (or never were told) that XML, via SGML, was originally designed to facilitate search and long-term, application-independent archiving of data. It is almost coincidence that makes that same application-independence useful for communication of transient data. Convenient but not optimal.

I fully expect to be able to do more or less what I'm doing now ten or twenty years from now. Whether I will be is another question, but so far, just when I thought I was completely bored of it, something new in the XML world has come along to re-energize my interest. And we're still struggling to build truly useful XML-aware hyperdocument management systems. Hopefully that won't be the case in 2018.

And lets not forget Dr. Charles Goldfarb, who's own singleminded passion, drive, and leadership produced SGML, without which XML (and HTML, for that matter) would never have happened. SGML turned 20 in 2006. It's largely now forgotten except by a few early adopters who have been using their SGML-based systems productively for ten or fifteen years now and had no compelling business reason to move to XML. But I remember.

Kids today....

FASB ASC U.S. GAAP DITA Application Is Live

2008-01-23T11:10:00.000-06:00

[How many initialisms can I get in one post title?]

For the last year or so I've been working as part of a larger team at the Financial Accounting Standards Board (FASB), helping with the implementation of a DITA-based system to support authoring and delivery of the newly-codified U.S. Generally Accepted Accounting Principles (GAAP), the Accounting Stanards Codification (ASC).

I contributed design of the DITA topic and map specializations used for the codified content and also implemented automated data conversion from an earlier XML format used in the initial codification editorial process.

The live Web site is here: http://asc.fasb.org/home

I've posted some details about the project on the Really Strategies blog: http://blog.reallysi.com/2008/01/live-dita-appli.html

Update: I should have mentioned (but wasn't 100% sure of the details) that the CMS system was built on the empolis e:CLS product and the Web delivery platform uses the empolis e:IAS search platform. The empolis Web site is www.empolis.com. I was not personally involved with that aspect of the system and was not involved with FASB's technology selection process (I was brought onto the project after they had selected their core techology). The system integration and development work was done by Ovitas, empolis' chief North American integrator.

I For One Welcome Our Cleaning Robot Overlords

2008-01-01T12:48:00.000-06:00

As long as they keep the house clean.

My main gift this holiday season was a new Roomba 560 floor cleaning robot. This brings the cleaning robot population of Chez Kimber/Woods to 3, including our original Roomba 300 series and the Scooba.

I wanted the new Roomba because I found that, with a two-story house, it was just inconvenient enough to move the one roomba between floors that I was less likely to go to the trouble to run it at all. We also found that the 300 series couldn't really deal with the area rug in our livingroom (we have concrete floors with one big run in the livingroom) and that the noise of running it on the concrete floors was just a little too annoying. So in short, the robot was underused and the house tended to be not as clean as we would like (but couldn't actually be bothered to clean ourselves, not being what you would call obsessive house cleaners).

The 500 series promised to address all those problems with improved tolerance of things that would stop the 300 series (such as cords, furniture it tended to get trapped under, and the edge of the rug), reduced noise levels, and more effective capturing of pet hair (the 300 tended to just push around big clots of pet hair rather than sucking it up).

So far I have been very pleased with the 500 series--if anything it exceeded my expectations. It is significantly quieter, handles the rug just fine, doesn't get trapped where the 300 did (we have one big sideboard with these decorative bits at the base that the 300 would tend to get wedged under, the 500 never does) and seems to have a longer battery life.

So now the old 300 lives upstairs where it can focus on keeping our master bedroom clean and the 500 takes care of the downstairs.

As I said in my report on the original Roomba, these are amazingly well-engineered products that can serve as models and inspirations for all of us that build things for other people to use. Compared to the 300 the 500 is not signficantly different but there are a number of minor but important refinements that add up to a much improved user experience, from the simplified controls (got rid of one button that wasn't of much use) to the better brushes to the easier-to-empty dirt chamber. And all at a reasonable price.

And it makes cleaning the house fun.

Loopwing Wind Generator

2008-01-01T12:28:00.001-06:00

IMG_4216

One of my best Christmas gifts this year was this model of a loopwing wind generator from Tamiya (http://www.tamiya.com/english/products/75021loopwing)

I was unaware of this particular wind generation technology but it seems quite intriguing in that it claims to be better able to extract energy from light winds and takes less vertical space (and presumably is less dangerous to birds) than straight-wing wind turbines. This means you could have one on your house in your back yard and maybe not put the entire neighborhood in danger or violate local noise ordinances.

The kit itself went together quite quickly, the hardest part being cutting out the wings themselves, which actually required a little skill and care rather than just screwing the parts together (there's no gluing or anything).

The turbine drives a generator that then charges a little model car that plugs onto the top of the generator body. The energy is collected in a super capacitor that can then run the car for about 3 minutes on a full charge.

The connector to the car appears to be standard connector so it ought to be easy to build other things that can charged. I was thinking a little LED display that indicates the level of output or something or maybe something decorative. It would certainly be easy to adapt it to charging solarengine BEAM robots.

The generator doesn't swivel to face the wind but it would easy enough to mount it on a turntable with a wind vane if you really cared. I've got it mounted on a pipe that rises to about 5 feet and stands where the north side of our house forms a little wind tunnel that catches the northwest wind that tends to blow this time of year.

I find the prospect of having a home-sized loopwing generator interesting. We already have a 3K watt PV system on our house--it couldn't be that hard to add in the output from a small turbine, such as described here: http://www.treehugger.com/files/2006/11/loopwing_wind_t.php

Where we are in Central Texas we have a pretty reliable 5-10 MPH breeze most of the time and quite often stronger winds, especially in the spring and fall.

I think this year will start to see some interesting developments in alternative energy generation. Austin will be home to a new thin-film solar cell factory and is already home to a company trying to make high-capacity capacitors usable in electric vehicles. Taken together those technologies could make electric and solar-electric vehicles much more attractive in cost and range, not to mention the possibilities for home energy.

For example, imagine having a bank of capacitors that could provide the same power output as the little gas motors in all the three-wheel taxis in all of Asia and that can fit in the space currently used by the fuel tanks those vehicles carry (or otherwise fitted into available space).

Now imagine putting low-cost, flexible solar panels on the top of each of those three-wheels (they all have some sort of canopy on them) as well as on taxi stand shelters scattered around a typical Asian city. If most of those three-wheelers spend most of their time waiting for a fare, it seems reasonable to think that they could be mostly or entirely charged by their solar panels, taking from the main grid or a taxi-stand battery or capacitor bank only during peak times (e.g., morning and evening rush hour). Or maybe they could use one of those small fuel cells the Japanese are selling for home power use for peak-time charging where the grid is not reliable (or where natural gas is inexpensive).

The effect of such a change would be dramatic: a significant source of air polution would be eliminated, the need for fossil fuel would be significantly reduced in a part of the world where oil demand is rising much too sharply, and the operating cost of the taxis themselves would be reduced (assuming both that electricity costs per kilometer would be lower than fuel costs and that much of the operating energy would be from the vehicles' own solar panels).

With current battery technology, batteries could never be used to realize this vision: they're too expensive and too toxic and have too little energy capacity. But capacitors, if the current claims of orders of magnitude improved capacity prove out, could, because they have both a much higher energy density and lower toxicity (at least I assume they do) and they can charge very quickly, meaning that a taxi could do a 15 or 20 minute fare and then recharge in minutes at a recharging station or charge over say an hour using its own solar cells. That means a three-wheel taxi doesn't need to carry as much on-board energy capacity as it would for a battery solution.

Assuming the technology were there, what would it cost to, for example, provide a retrofit kit to every tuk-tuk operator in, for example, the Philippines? It would be several hundred million dollars at least (e.g., say $500.00 per vehicle) and as difficult to administer fairly and efficiently as any other aid project, but I would think that there would be lots of incentive from many parties to make something like that happen. And once the local population got used to the technology and had access to spares and second-hand parts so forth, the technology would be applied in many other creative ways. And at some point you'd hope it would be good enough to, for example, allow Philippine jeepneys to be retrofitted for electric power.

And cities like Manila and Columbo and New Deli would be much much quieter, with all those two-cycle motors replaced with electric drives.

Of course, the possibilities for other transforming uses of low-cost, physically flexible (that is, bendable) solar panels in developing and third-world countries are quite exciting. It will be interesting to see how the technology develops in terms of its economics and manufacturing environmental costs.

While there's no obvious direct connection between XML and alternative energy we, as an industry and as a society of large-scale computer system users are starting to realize that the collective cost of computing equipment does represent a significant fraction of our total societal energy draw. So the degree to which a technology like XML enables more people to do more with system, the greater the power such use will draw.

As a I write this, I'm sitting in a room with three computers running, drawing a couple hundred watts, as well as using Google and Yahoo, backed by massive data centers drawing terawatts of largely coal-produced electricity (except for those data centers built in Central Washington to take advantage of the cheap hydropower provided by salmon-habitat-destroying dams on the Columbia river and its tributaries). I'd feel a little better about that if I could at least make my urban house electricity self sufficient without spending too much more than I already have on alternative energy systems that make little economic sense under current U.S., Texas, and Austin energy policy (in particular, that, unlike Europe, utilities can buy back excess power at a steep discount from market rates, making the payback on my solar PV system 15 years or more *after* having half the initial cost rebated by the city and federal tax credits). Obviously we did it because we felt it was the right thing do and we could afford it, not because we had any financial incentive to do so).

Anyway, that's a long way from a cool toy that I got for Christmas....

I'm Bein' Macified

2007-10-12T07:00:00.000-05:00

Through a series of more or less accidents I came to have physical possession of Really Strategies' one and only MacBook, purchased in order to support testing and delivery of software to a Mac-based client (which, considering that most of our clients are publishers should be most of them, but apparently hasn't been to date).

After some soul searching I have decided to make this Mac my primary development machine, giving up my oh-so-familiar Dell Windows-XP-based laptop.

We'll see how it goes. I must say that it's been quite an adjustment for me, somebody with nearly 20 years of Windows brain damage, to move to a Mac.

Of course it helps that most of the development tools I use are completely cross platform: Eclipse, Java, OxygenXML, Syntext Serna. It also helps that OS X is an *nx-based system under the covers, so I can get a command line that is familiar, although the configuration details are not (I've been using Debian-based distributions for most of the time I've used Linux). And other key tools have solid Mac versions (e.g., all the Adobe products).

I will even be able to get an RSuite server running on this machine, using an unsupported OS X build of MarkLogic.

I'm even starting to get used to the bizare control key mechanism, although it's still a struggle--it feels like trying to learn a new musical instrument that is just enough different from one you know to really hose you up.

I'm even writing this post using Safari, rather than Firefox, which I would normally use, but it's acting up this morning.

So wish me luck as I start on this new adventure in computing....

Automatic Handling of DITA Docs In XML Editors

2007-10-01T07:00:00.000-05:00

I'm in demo prep heck at the moment, trying to get some real DITA functionality built on top of Really Strategies' RSuite CMS product. One of the key challenges here is integrating XML editors to handle this use case:

Initial state: You are presented with some valid, conforming DITA documents in some locally configured and/or specialized document type, organized by one or more maps. You (and your repository and supporting tools) have never seen this particular set of documents or their DTDs before.

Step 1. Import map and all dependencies (including its DTD) into the repository

Step 2. Within the repository, find a topic to edit and push the "Edit with {name of integrated editor}" button in the repository UI.

Step 3. Editor opens with document, with all DITA support features applied.

It is that step 3 that is currently causing me a bit of pain. And it shouldn't.

The reason it's causing me pain is because every graphical XML editor has been built on the presumption that document types are relatively static and that some XML specialist will develop lots of doctype-specific setup and then deploy that setup once, followed by a long time with no changes to that setup.

Thus, if you're presented with new documents in a heretofore unseen DTD, they're not going to work in the editor until you go through the setup and configuration process for the new document types [And remember that DITA 1.1 requires at least six distinct shell types: map, concept, reference, task, glossentry, and dita, plus any additional specialized map or topic types you might have--that's a lot of DTD-specific configurations to set up, even if most of that effort is just copy and paste, it's still tedious and prone to the usual errors of catalog misconfiguration, filename misspelling, and so on.]

However, DITA totally chunks the assumption of static, well-known doctypes out the window. DITA says "hey, every shell is different, specialize away, apply agile approaches to developing and refining your local DITA-based DTDs, combine topics from everywhere willy-nilly, go nuts, have fun".

To support this DITA does something very important: it enables reliable auto-recognition of DITA documents, regardless of the details of the local configuration or the use of specialization.

DITA must have this mechanism because the specialization feature allows generic DITA processing to be reliably applied to any conforming DITA document. Because it can be, it should be.

For the DITA Open Toolkit this means applying default processing (transforms, filtering, etc.).

For editors it means applying default editing style sheets, enabling DITA-specific user interface components (e.g., "Insert topicref"), etc., if no more specific configuration already exists for the document or its shell doctype.

And there's no reason for any DITA-aware editor not to, except that, without exception that I can find, they've all implemented their document-to-functionality mapping in a way that doesn't enable this sort of dynamic association. The closest I've found so far is Syntext's Serna editor, which while it doesn't recognize specialized topics as DITA topics and apply its (very nice) built-in DITA support, it does make it a two-click process to manually apply their built-in DITA support. So kudos to Syntext. But it should be a zero-click process.

For this automatic process to work processors have to be able to examine any document they're presented with and reliably determine whether or not the document is or is not DITA-based. Note that the Open Toolkit presumes that what it's given is DITA-based because that's the only thing it is designed to process. But things like editors and CMS systems are, for the most part, completely generic and designed to handle any XML at all. So they cannot presume (or at least they should not presume).

The recognition of DITA documents cannot be based on the use of any particular DTD's system or public ID, because they'll all be different. You can't look for a particular well-known element type because the element types could be completely different from anything previously seen (let's imagine a specialization where all the element type names are in Chinese--there's nothing that prevents it and if I was a native reader of Chinese and wanted to create tech docs I'd probably do just that).

That means you've got to go by something invariant that is reliably in every document. In XML that really means the use of a particular well-known namespace. However, DITA element types cannot be in namespaces because the current DITA class mechanism syntax cannot support namespace-qualified names. Knowing that about DITA you might think "well what to do then?"

However, just because elements can't be in a namespace, it doesn't mean attributes can't be. And that's the trick DITA uses in DITA 1.1 to enable autorecognition of DITA documents, regardless of any other aspects of the DTD (it's public or system IDs, the element type names used, etc.).

This trick is the DITAArchVersion attribute. This attribute is in the namespace "http://dita.oasis-open.org/architecture/2005/". Any document that includes this namespace is almost certainly a DITA document, especially if the namespace qualifies an attribute named "DITAArchVersion" and the element on which that attribute occurs has a class= attribute conforming to the DITA class attribute syntax.

This means that regardless of the actual DTD or schema a DITA document uses, it can be recognized as being a DITA document. That means that you can then reliably and usefully apply default DITA processing to the document without having specifically configured its particular DTD or schema as being a DITA schema.

That is, the behavior I expect from any editor that claims to be DITA-aware is that if I open any conforming DITA document, regardless of what declaration set it happens to use, I should get all the default DITA-specific stuff automatically.

While the most robust implementation of this behavior would make all the checks described above, it is probably sufficient to assume that if a document's root element has a DITAArchVersion attribute or if the root element is named "dita" and any of its children have a DITAArchVersion attribute, then the document is a DITA document.

The DITA spec only really recognizes three possible configurations of elements in a conforming DITA document: root of base type "map", root of base type "topic", or root of type "dita" [the dita element is not specializable in DITA 1.1] where its direct child elements are of base type "topic"--anything else is not a conforming DITA document (although it may contain individually conforming topics or maps) and you have no obligation to apply DITA-specific features to it (although you could if you wanted to).

That's by way of saying it's probably good enough to just look for the DITA namespace anywhere in the document and go by that, but it could lead to false positives in cases where the document is not strictly a conforming DITA document.

And it would be really cool if editors provided defined extension points by which this type of recognition could be added to doctypes as plug-ins to the editor.

It Just Works: Not So Much

2007-09-23T15:25:00.001-05:00

When I started my new job I bought an inexpensive desktop to have as a home development machine (my last desktop had long since died and I'd not had justification or need to replace it). The machine, a Gateway Dual Core Pentium, came preloaded with Vista Business, which would not have been my first choice but there didn't seem to be a lot of options. After putting in another Gig of RAM (having discovered that Vista at idle takes about 950 Meg--yow) Vista ran OK but it just seemed to be stupidly slow, especially Windows Explorer. I really don't know anything about Vista nor do I care to, but at Windows XP seemed to work reasonably well. Vista, running on twice the hardware seems to perform half as fast. Why? Microsoft is certainly capable of producing good software.

I more and more got tired of having to wait for Explorer to respond and the inexplicably show unzip times and stuff and came close to putting on Ubuntu a couple of times, even though I don't really have time to spend on it. But finally I ran into a problem where I couldn't unzip the Web Tools distributions of Eclipse because of some filenames in the package that are too long for Windows to handle. WTF?

So I found some very clear instructions for installing Ubuntu alongside an existing Vista installation, resized my Windows partition and installed Ubuntu. That went quite smoothly, which I've come to expect from Ubuntu. So far so good.

But then things went down hill. First, I couldn't get the sound to work and what I could find in some frantic Googling suggested that others had had the same problem and had not had much luck fixing it. How can that be? This machine should be a pretty standard setup--it's clearly all-Intel motherboard with built-in sound and video.

I could live without sound (although I wouldn't like it--my desktop is also my office music system) but then the video stopped working and I can't get it back. The machine is attached to a Westinghouse 22in display. On install the desktop was using the standard 1280 x 1024 resolution. Once I got things otherwise working I turned my attention to the display. I reran the X-config application, chose the appropriate resolutions and so forth, and restarted the X server.

Boom "Out of Range". So far nothing I've been able to do to my X configuration file has made a difference. I was able to track down the manual for the display to get the actual horizontal and vertical sync frequencies but specifying those didn't work. I was not able to find any particularly useful guideance online beyond "rerun the configuration utility" or "specify the right sync values in the config files". The log-in screen is clearly running at the appropriate resolution for the display but why does it stop working when I log in? It's quite maddening.

But the real problem is that this just should not happen. It should not be possible for the windowing system to not work. As long as this is the case, as long as the first and only way to approach this failure is to open a terminal window and use sudo to edit a configuration file, it's a non-starter for anyone except the geekiest of geeks. If there's anything that should always work it's the display, even if it's to fall back to a lesser resolution. But I'm not even getting that.

I would really like some distribution of Linux to be a viable alternative to Windows for non-geeks but Ubuntu has definitely failed this test. I'm a geek. I've installed Linux countless times and gotten it working on a variety of machines. I should be able to make it work. Granted I was doing this while trying to do other stuff and didn't have endless hours to solve this problem, but that's the point: I shouldn't have to have endless hours or only expect Ubuntu to work on ancient machines (which is where I've been most successful). Ubuntu advertises itself as the Linux that just works. So it should just work. Certainly it should work on a commercial machine from a major vendor using generic Intel hardware.

I'm not sure if the problem is a technical one or a cultural one or an economic one. I recognize that hardware manufacturers don't have a huge motivation to write Linux drivers and that a volunteer-based project is dependent on people actually doing what's needed and maybe they will and maybe they won't. But there also seems to be a cultural component of "well, if you can't edit a configuration file you really don't belong in our club". And of course, since there's no single enterprise with economic incentive to make Ubuntu or any Linux distribution work as smoothly as Windows or OS X there's no reason to expect it will.

But that doesn't make it any less frustrating. It doesn't help that I've recently had to use a Mac for work, which makes it clear that there's no reason a *nx-based system couldn't be as smooth as the commercial systems are, but clearly it helps to have a multi-billion dollar company driving the activity.

Anyway, it's very frustrating since I had to boot back to Vista, I still can't install the Eclipse Web tools (and don't have any more time to spend working out that problem than I do on my Ubuntu problem), and now half my hard drive is tied up with a a useless operating system.

I know I can get back to a minimally working system by re-installing Ubuntu, but I have no confidence that I'll have any better luck getting the display to work.

Hmph.

InDesign CS3 and XML Authoring: Could be Good

2007-07-20T06:05:00.000-05:00

In my new job at Really Strategies I have started digging pretty deeply into how to get XML into and out of Adobe InDesign CS3. This has turned out to be pretty interesting.

In InDesign CS2 the XML support was somewhat weak. While you could import an XML document into InDesign and then associate styling with it, it was very simplistic in that you had no direct way to do context-based associations and no easy way to script it, either on import or inside the editor.

In CS3 that has largely changed. CS3 adds several new XML support features that appear to serve to make InDesign a quite powerful XML rendering tool that could be integrated loosely or tightly with any other XML authoring tool to create an interesting environment. (You could, in theory use InDesign to author the XML as well but it wasn't really designed for that and I don't think it's a good use of resources to try to make it an XML editor, not when the process I outline here is so easy to implement.)

Here's the general mechanism I'm working toward:

1. Using InDesign, you create a template document that will accept your XML. This requires setting up all the usual styling stuff (page masters, frames, named styles) as well as creating instances of the markup structures that will populate different text frames (InDesign's XML import works by matching imported elements to existing elements and replacing the existing ones with matching structures on import, more or less).

2. You create an XSLT that takes your XML source and "augments" it with Adobe-specific attributes that specify the per-element-instance mapping to InDesign paragraph and characters, as well as generated elements for any generated text that needs to be rendered as a separate paragraph (analogous to the gentext psuedo elements Arbortext Editor uses to manage generated text display).

This XSLT can be pretty simple--it's just an identity transform with a little bit of per-element-type logic to define the mapping (and it could be further parameterized through some sort of more direct mapping specification, although I'm not sure it's worth the effort). This script could also re-order things as needed, generate TOCs, etc. But the minimum required is pretty small. There're a few more things you need to handle, but they can be generalized easily enough.

The main gotcha here is that InDesign is sensitive to newlines in the XML data, because newlines trigger the application of paragraph styles. What I've found so far is that you have to manage the text content very carefully so that you only emit newlines at true paragraph boundaries. This also means that you only apply paragraph styles to the lowest-level elements that will become paragraphs in InDesign--you can't just blindly apply styles at higher levels in the XML hierarchy (InDesign is not XSL-FO).

3. You run this transform outside of InDesign. InDesign lets you apply a transform as part of the import process, but we don't want to do that for reasons that will become clear in a moment (unless I've missed a feature of InDesign, which is quite possible--I'm still coming up to speed on its intricacies).

I use OxygenXML for most XML editing and it provides a very convenient mechanism for applying a transform to a document and saving the result wherever you want. But any good XML editor should provide a way to do this so that you have some sort of "run the transform" button or menu item. The key is that the result (the XML with the InDesign augmentations) is always put to some consistent place.

4. Import the augmented XML (not the XML you're authoring in your XML editor) into InDesign using InDesign's XML import (without applying the XSLT) but being sure to check the "link to XML" check box and select "merge" not "append"--this is the key.

5. Go back to your XML editor, make changes to the XML and push the "transform" button again.

6. Switch to InDesign and bring up the Links pallet. In that you'll find your XML document listed. Select it and click the "update link" button. Magically, your XML changes are re-imported into InDesign and the styles applied.

Hey presto! Immediate, easy, convenient pagination of XML using InDesign. Something that was not immediate, easy, or convenient with CS2.

I haven't looked into it but it should be possible to script the triggering of the link update as well, although that might require a little C code, I'm not sure. But it's clear that by this mechanism you can use InDesign as a "page preview" mechanism from any XML editor with very little work.

Beyond the simple element-to-style mapping you can do on import, CS3 also provides scripting support for working with XML in the form of XPath-based functions that allow you to easily apply any script to elements in context. I haven't used this yet but a brief look at the docs suggests that it's just the thing to take your XML to the next level.

It's still not going to give you what products like Typefi give you, which is complete complex layout heuristics, but it should be sufficient for relatively simple layouts such as typify technical documentation. It occurred to me, for example, that it wouldn't be very hard to create a process that would allow you to use InDesign to create nice books from DITA source using this mechanism. Hmmm

Note that you can download a one-month eval of InDesign from Adobe's Web site.

And Now for Something Completely Different....

2007-06-29T14:14:00.000-05:00

Well, not that different actually.

As of Monday 2 July I will be moving from Innodata Isogen (www.innodata-isogen.com) to Really Strategies (www.reallysi.com). This move comes after 11 years at ISOGEN in its various incarnations, from a 20-person consultancy to a part of DataChannel (riding that Internet wave) to part of Innodata. It was a great period in my career--I got to do lots of interesting things and work with really great people, but it was time for a change, time to see what else was out there.

One big change with this job move is that I will, for the first time since becoming a Standards Maven, work for a company with a product. I have until now been a fiercely product-independent integrator, working for companies that had no preferential agreements with any vendors, partners with all.

Really Strategies has in the last year developed and marketed its RSuite Content Management System, targeted primarily at publishers.

How do I justify this change as still being consistent with my values and principles as a standards-championing consultant? Am I simply throwing away the reputation I've built as a person who hates all products, for the reasons outlined in the first posts to this blog? I hope not. I'm sure people will tell me if I have.

My justification is driven by the following observations:

1. My personal situation has changed over the last two years in a way that has forced me to realign my priorities, in particular, having started a family and built a house, I have to give more weight to compensation than I ever had to in the past. In short, I've gone from living well below my means to living just beyond them. And I've got to think about things like gymnastics class and preschool tuition and what it costs to travel with a child and so on. So yes, my convictions have been moderated just a little by demon cash.

2. For financial and other reasons, I'm not ready to hang up a shingle. It's just too much risk right now. Maybe in a few years.

3. There are only a few companies that do independent XML-related systems integration in North America and even fewer who could meet my outrageous compensation requirements or otherwise integrate me into their business.

4. Really Strategies is still first and foremost an integration consultancy that happens to have a product, rather than a product company that has an unavoidable services group. So far they've found that having the product is resulting in much more services work, mostly unrelated to the product. That is, the product is serving as much as a marketing tool as it is as a revenue source (although I understand we're selling a few licenses too, which isn't bad). That means that it's not all about the product, but still about solving client problems.

5. As ISOGEN was and is, Really Strategies is still about standards and that is reflected in their products as well as their services work (and if it's not, you can be sure I'll have something to say about it, and maybe even something to do about it).

6. Part of my job responsibility will be contributing to the architectural definition of the RSuite product. Because it's a very new product, even if it turns out to be heinous (which I don't think it is but I haven't yet had a chance to look under the hood), there's lots of opportunity to correct it. In addition, it's built on top of MarkLogic, for which I have a great deal of respect, both in terms of its engineering quality and in terms of its respect for and adherence to the standards it implement Just the fact that the RSuite engineers made that choice is a good sign (although it's no guarantee of anything, but there are only a couple of correct choices for a CMS base and a whole lot of wrong ones, and MarkLogic is definitely one of the correct choices, in my opinion).

Thus, while I will be working for a product company, it's in a situation where standards are still paramount and where I'll personally have an opportunity to express my principles and thoughts and ideas about how a product like RSuite should serve its users and not the other way around. And I suspect that most of my work as an integrator will not be related to RSuite at all, simply because of the nature of the types of projects that Really Strategies tends to sell.

Time, of course, will tell.

In any case, I'm very much looking forward to this opportunity to have early and direct influence on a product and not just via bug reports and feature requests and whining at the engineers I happen to know inside the company. I'll continue to do that of course, but now I'll be able to whine at engineers I actually work with.

I will also continue to be involved in standardization activities as much as I can, continuing the work I've been doing with the DITA and XSL-FO standards and probably contributing to other standards that are more relevant to Publishers rather than technical documentors, which has been my primary focus to date.

And it shouldn't surprise anyone if you start to see stuff about DITA-specific features in RSuite. Just saying....

Comments: My Bad

2007-06-29T14:13:00.001-05:00

I just realized that since I turned on comment moderation to reduce comment spam, I actually have to go in and moderate comments. Doh!

My apologies to those who waited a month or more to see their comments approved. I'll try not to let it happen again.

A Message To Warm a Taghead's Heart

2007-06-01T09:43:00.001-05:00

no-xml-response

Trying to connect to my Yahoo! Mail account, I got this popup box. I was quite pleased to see that it was upset about not getting any XML in the response.

DITA Standard Practice: Always Make Local Shells

2007-04-24T07:35:00.000-05:00

As I started doing serious work with DITA and in particular implementing specializations, it became clear to me that the first thing anyone using DITA should do is make local copies of all the DITA-provided shell DTDs or schemas. This should just be automatic. [Note: I'm going to use the term "DTD" to mean "DTD or schema" from now on.]

Why?

DITA, like DocBook, is a generic standard designed specifically to allow controlled local configuration and extension. Anyone who uses DITA in any non-trivial way will need to do at least some configuration, if not specialization, before too long. It is the rare user of DITA who genuinely needs to use all of the topic types and domains reflected in the base DITA distribution. Even if you're not using any specializations you probably only need some of the domains DITA provides out of the box.

By "configuration" I mean adjusting the set of topic types and domains that are or are not used for a given document set. For example, if you're not documenting interactive software you probably have no use for the User Interface domain and would just as soon not have those element types available to authors. Turning off that domain for your authors is "configuration".

By "specialization" I mean new domains or topic types derived from the base DITA-defined types (see my specialization tutorial for a deeper discussion). Even if you don't develop your own specializations, it is likely that you will use specializations developed by others. This will be increasingly likely as the DITA community begins to develop more and more special-purpose specializations--this is one of the really cool things about DITA--it enables and rewards the creation of "plug-ins" that are relatively easy to create, distribute, and integrate with the base DITA document types and supporting infrastructure.

In order to do either configuration or use specializations you must create local shell DTDs that reflect the local configuration or integrate the specializations.

Since you're going to do it sooner or later, you might as well start your DITA life there and be prepared. Eat the (minor) pain up front of configuring your local environment to use your local shells and then you're set to go.

If you set up your local shells first, then as you add new DITA-aware tools to your system, you can simply configure them to use your shells from the get-go, rather than building a system of tools and a set of documents that then have to all be reconfigured later when you finally do implement local shells (or worse, you discover that your system has become such a lava flow that you can't reconfigure them meaning that you can't do any configuration or use new specializations because the cost of reconfiguration would be too high or too risky).

NOTE: when you create local shells you must give them unique global identifiers (URIs or PUBLIC IDs). You must not refer to them by the DITA-defined URIs or PUBLIC IDs. Local shells are just that, local. You create them, you own them, you name them. You should consider the DITA-defined shells and attendant module and entity files to be invariant, meaning that you should never ever modify them directly, but only use them by reference, configured using the DITA-defined configuration mechanisms (parameter entities for DTDs, named groups for schemas).

All DITA-capable tools should (dare I say "must"?) be capable of using local shells, otherwise they aren't DITA-capable, QED. Probably the biggest potential problem tool is FrameMaker, but then FrameMaker is something of a special case because it's not a true XML tool and it's design makes reconfiguration much more expensive than it is with any other XML editor you're likely to use. I'm sure it can be done but I wouldn't want to have to do it (of course, as systems integrator I might be asked to and of course I would do it but that doesn't mean I'd have to like it).

For example, I've just gone through the exercise of setting up Arbortext Editor 5.3 to support editing of heavily specialized topic types. Once you know what to do it's not too hard and is reasonably well documented in the online help. The basic process is:

Put each shell DTD in its own directory, named the same as the DTD or schema file. This organization is at least suggested, if not required, by Arbortext, but it's pretty good general practice anyway (even though the base DITA distribution doesn't organize things this way, there's no reason it couldn't and I've suggested that maybe it should, just on general principles of neatness).
Create an old-style (non-XML) OASIS entity catalog for mapping the URIs of your local shell DTDs to their local location. (Arbortext Editor 5.3 doesn't support XML-syntax catalogs.)
For each topic or map type shell, copy the Arbortext-specific configuration and style files from the Arbortext-supplied DITA doctypes that are the closest match to your local shells. Rename as necessary per the Arbortext naming conventions.
Edit the configuration files to reflect the details of your shells. This is stuff like setting the name used in the New file dialog, pointing to templates and samples, and so on. For specializations you'll need to account for new element types in the editor configuration, style sheets, and whatnot, if they require special handling.
Update the Arbortext Editor catalog path to include your catalog so it can resolve the references to the DTDs.

That's it. I would expect other XML editors to require a similar process (I haven't tried setting up XMetal for these specializations yet so I don't know what its configuration details would be).

Note too that as long as you are putting your shells in your own directory structures, and not in the dtd/ directory of the DITA Open Toolkit (which you should never do), it doesn't matter what you call your shell DTDs. That is, there's no particular reason not to call your local configuration of the concept shell DTD "concept.dtd".

So if you are a new user of DITA (by which I mean somebody setting up the DITA environment for a defined set of users, not an individual author [unless you are a writing team of one, in which case you are performing both roles]) I strongly urge you to create your own local shell DTDs right now if you haven't done so already.

Comment Spam Continued

2007-04-24T06:27:00.000-05:00

So after turning on comment moderation, I still got two or three spam comments from Blogger.com members (which are not moderated) and blocked two from non-Blogger members. Which means that comment moderation is not very useful. At least the spam just inane and not particularly offensive.

My initial take was that it must be humans doing the spamming, but googling on "captcha bypass" quickly leads to information indicating that picture-based captcha can be cracked with 80 to 100% accuracy.

So I guess there's not much I can do about the spam.

Hmph.

It does lead to the idle thought that maybe it will be the spammers who first develop true AI in their quest to win the humans vs bots arms race....

Moderating Comments

2007-04-17T16:58:00.000-05:00

Either spambots have cracked the comment captcha mechanism or humans are being paid to leave comments. In any case, I've turned on comment moderation to try to turn off the comment spam.

XML Documents vs XML Data Packages

2007-04-11T07:58:00.000-05:00

Both James Clark's recent posts on XML and JSON as well as some recent attempts I've made to describe what I do professionally with respect to XML led me to realize that there doesn't seem to be an easy way to distinguish XML documents that are intended primarily to produce human-consumed results (e.g, published books, Web pages, online help, whatever) and XML documents that are purely for program-to-program communication (e.g., the use case things like JSON are trying to address more effectively than XML necessarily does).

Also, there's a part of me that wouldn't mind XML being returned to it's more or less strictly document-centered use rather than being the all-purpose data serialization and communication language it's become. Of course that's not really a productive line of thought.

But it did make me start to think that the stuff that James and others are starting to think about reflects the historical accident that the world, and in particular the Web-based world, needed a more transparent data communication mechanism than things like CORBA and DCOM provided just when XML appeared. People with the requirement saw XML as a way to do what they needed without spending too much time thinking about whether or not it was optimal--it was there and it would work well enough and here we are.

But it leads me to think that I agree with what I think James is saying: that it's probably not a bad thing to start designing serialization languages that are optimized for the specific tasks of program-to-program communication.

The existence of such languages would not in any way threaten the status of XML as a language for (human readable) document representation.

One thing that XML has done is embedded a number of key concepts and practices into the general programming world, such as making a clearer distinction between syntax and abstraction, which sets the base for realizing that once you have the abstraction, the original syntax doesn't matter, which means you can have multiple useful syntaxes for the same abstraction. It has made the general notion of serialization to and from abstract data structures via a transparent, human-readable syntax a fundamental aspect of data processing and communication infrastructures.

I think this means that we are now at a place where the community at large can see how you could refactor the syntax part of the system without the immediate need to refactor the abstractions (which is where most of the code is bound, that is, code that operates on DOM nodes rather than code that operates on SAX events or directly on XML byte sequences).

But it seems reasonable to me to at least start planning this refactor simply in the name of system optimization. It will probably take 20 years (it took 20 years to go from SGML as published in 1986 to today, when we can clearly understand why XML isn't the best solution for some applications) but it seems doable.

While the infrastructure for XML is widely deployed and ubiquitous, we also have the advantage that that infrastructure is by and large modular (in the sense that it's provided by more or less pluggable libraries in Java, .NET, C++, and so on) and in languages that are themselves ubiquitous.

For example, if Java or .NET released core libraries with base support for something like JSON it would not be hard for application programmers to start refactoring their systems to move from using XML for data packages to using JSON. Of course the systems using heavyweight things like SOAP would have a harder row to hoe.

If we take the Flickr API as an example of an XML-based API where something like JSON might be a better fit (or at least simplify or optimize the serialization/deserialization process), it would take a few person months on the Flickr end to provide an JSON version of the API (which would have to live along side the XML version) and a few person days or weeks for each of the language-specific client-side bindings for the Flickr API to use the JSON version instead of the XML version. At some point, say in a couple of years, the XML version of the API could be retired. That seems like a reasonable refactor cost if the value of using something like JSON is non trivial (I don't have an opinion on the value of something like JSON in this case--I just don't care enough and I've never had too much patience for "but this is more elegant than that" arguments if that's your *only* argument).

The Flickr API may be a poor example only in that the data structures communicated are fairly simple, mostly just sets of name/value pairs (metadata on photos) or lists of pictures or sets or users or tags. In that use case, XML works as well as anything else.

But in a more complex use case, where the data structures serialized are more complicated, in the way James talks about, with non-trivial data types and complex composite object structures and whatnot, I can definitely see a purpose built language having real value, primarily in the ease with which programmers doing the serialization/deserialization can both design and understand the mapping from the objects to the serialized form.

I spent some time working with the STEP standard (ISO 10303), a standard for generic representation of complex data structures, originally designed for interchange of CAD drawings and 3-D models and eventually generalized into a language for general product data interchange. It provides a sophisticated data modeling language. I was involved in the subgroup that was trying to define the XML interchange representation of STEP models. This turned out to be a really hard problem precisely because of the mismatch between XML data structures and data types (String at the time) and the sophisticated STEP models. It confirmed what I already knew, which was that mapping abstract data structures to efficient and complete XML representations is hard and naive approaches based on simple samples will not work.

That means that a comparable interchange syntax that is a better match for complex data structures will have value simply by making the conceptual task easier, so that designing and understanding serialization forms is easy (or at least easier) than it is using XML.

And then I can have my XML all to myself for creating "real" documents....

Help Me Obi-Wan: Java and Encodings and XML

2007-04-10T16:58:00.000-05:00

While I consider myself a pretty good Java programmer I don't actually do that much processing of XML with Java and so I've never fully internalized the details of SAX and JAXP and all that. Pretty much I just crib code that will get me a DOM and hope it works or get someone else to implement the fiddly bits.

But today I ran into a wall and all my fiddly bit colleagues are elsewhere so I thought I would ask my readers for help.

Here's what I'm trying to do:

I have XML documents with Arabic content. I read these documents into an internal data structure, do stuff, and write the result out as different XML. Should be easy.

However, I'm finding several odd things that I don't quite understand:

1. Text.getData() is *not* returning a sequence of Unicode characters, it is returning a sequence of characters that correspond one-to-one to the bytes of the UTF-8 encoding of the original Unicode characters.

That threw me because I thought XML data *was* Unicode and therefore Text.getData() should return Unicode characters, not a sequence of single-byte chars. Or have I totally misunderstood how Java manages Strings (I don't think so)?

This is solved by getting the bytes from the string returned by Text.getData() and reinterpreting them using an InputStreamReader with the encoding set to "utf-8". (Is there a better way? Have I again missed something obvious?)

2. When I save the same document as UTF-16 the DOM construction process fails with "Content not allowed in prolog", which doesn't compute because it's not conceivable that any non-trivial XML parser wouldn't handle UTF-16 correctly.

3. When re-interpreting the UTF-8 bytes into characters, it mostly works, except that at least one character, \uFE8D (Arabic Letter Alef Isolated Form), whose UTF-8 byte sequence is EF BA 8D, is reported as EF BA EF, which is not a Unicode character and is converted to \uFFFD and "?" by the input stream reader.

WTF?

I suspect that I am in fact using a crappy parser but there is so much indirection and layers and IDEs and stuff that it's very difficult, at least for me, to determine which parser I'm using, much less how to control the parser I want to use. I'm developing my code using Eclipse 3.2. I've tried setting my project to both Java 1.4 and 5.0 with no change in behavior.

For this project I have the Xerces 2.9.0 library (as reported by org.apache.xerces.impl.Version) in my classpath.

Does anyone have any idea what might be going on here?

Any help or pointers on what I might be doing wrong or how to fix it?

James Clark in the House

2007-04-06T10:40:00.000-05:00

Norm Walsh reports that James Clark has entered the blogosphere: http://blog.jclark.com/.

Let me add my welcome to Norm's. You can bet that I'm subscribed....

DITA Specialization Tutorial Now on Xiruss.org

2007-04-06T10:07:00.000-05:00

I have started writing a more complete DITA specialization tutorial, which will eventually cover all aspects of DITA specialization and likely lead to additional tutorials on other aspects of using DITA (using the DITA Open Toolkit, writing a Toolkit plug-in, etc.).

The tutorial itself is published on my xiruss.org site here: http://www.xiruss.org/tutorials/dita-specialization/, including a package with all the source materials as well as the generated HTML version.

The source materials are managed for development in the XIRUSS Subversion repository here: http://xiruss-t.svn.sourceforge.net/viewvc/xiruss-t/specialization_tutorial/ should you for some reason want to track the development of the files or get the very latest stuff (can't imagine why but who knows?) or just get a particular file without downloading the whole package.

The tutorial includes an improved version of the DITA attribute domain specialization tutorial I posted here a while back.

It is of course written as a set of DITA topics, which is interesting in and of itself because a tutorial is a type of document for which the DITA concept/task/reference and highly fragmented presentation paradigms are not necessarily a good match. For example, I discovered that the only way to get prev/next links from one topic to the next within a logical narrative sequence of topics is to set their parent container in the organizing map to "sequence". However, this has the effect of numbering each topic in the sequence, which makes sense for the topics that represent a logical sequence of steps within the tutorial, but not for the purely conceptual overview of what DITA specialization is. (This is what the DITA Open Toolkit does today--whether this behavior is required by the DITA spec is a more subtle question.)

So it raises some issues, like do we need a tutorial-specific set of specializations and corresponding rendering customizations to get the effects I want as a tutorial author, or does the DITA spec need to be refined to reflect these sorts of more subtle rhetorical distinctions? Are my topics that describe a sequence of steps to be performed really task or concept topics (I've coded them as concepts because even in DITA 1.1, the task topic type is too restrictive in the way it represents sequences of steps)?

This makes the activity more fun than it would otherwise be--I always like it when the things I do result in both concrete products (a useful tutorial) and help to advance the state of our understanding and, hopefully, the supporting infrastructure, in this case, by serving as an experiment in applying DITA to a type of information for which it was not directly designed (not that I'm the first to create tutorials in DITA, or even the first to think about it--see discussion around this on the DITA Users Yahoo group--but as an informal, spare-time activity, this tutorial provides more opportunity for both introspection about the process and methods and, because it's public, more opportunity for community involvement).

I've also learned a lot about using DITA and hacking the Toolkit and stuff, which makes it fun.

Now if I could just stop waking up at 5:30 a.m. to work on the thing (It's not that I want to wake up at 5:30, it's just that once I am awake and my brain starts spinning I can't go back to sleep, so I am compelled to start working. Good for productivity, bad for physical and mental health.)

CMS Requirements for DITA

2007-03-22T10:53:00.000-05:00

At last night's Central Texas DITA User's Group we had a nice presentation from France Baril of Ixiasoft on some of the challenges that authoring DITA documents can pose, in particular the need to be able to find topics and know what the dependencies among topics are as you revise the topics through the life cycle of the documentation set.

This sparked a discussion on some basic requirements on CMS systems that provide DITA-specific features. In addition, one of my colleagues is doing a DITA CMS project for one of our clients and he and I got to talking about what the CMS they're implementing did and didn't do with the DITA data, which revealed that the CMS vendor was perhaps not displaying as much insight into and imagination about how DITA should be managed as it could be.

So I thought I would try to outline what I think the key DITA non-obvious content management features are that any CMS that claims to provide DITA support should provide. I will not state what should be obvious requirements related to the creation and management of links, the ability to search on content and metadata, and so on.

See my earlier posts tagged XCMTDMW for a discussion of general XML content management requirements. Those requirements are the base from which these DITA requirements start. Therefore I won't state obvious things like XML-aware query, basic link management, and so on.

Map-Related Requirements

Maps are a key feature of DITA and the management of maps is essential to productive use of DITA. Key map-related features are:

M.1 - Import of an entire map as a single action. Given a map, the system should be able to import the map and all maps and topics it directly or indirectly links to as a single action. The system should provide options for how imported maps and topics are organized into whatever the CMS' organization mechanism is (folders, cabinets, whatever). The system should provide options for how to handle the import of link targets that are not of scope "local", including the creation of proxy topics for locally unavailable targets. Following import, all links in the imported content must resolve correctly to their targets as imported.

M.2 - Export of an entire map as a single action. Given a map in CMS, the system should be able export to some location outside the CMS (the filesystem, a Zip file, a WebDAV repository, etc.) the map and all of its direct or indirect dependencies. Following import all the links in the exported should resolve correctly to their targets as exported.

M.3 - Map-based views. All CMS operations that involve access to topics should require or allow the selection of a map that establishes the "map context" for the operation such that the operation only reflects those subordinate maps and topics that are within the direct or indirect scope of the map. For example, you should be able to select a map and then do queries that only return topics in the map's scope or, when creating direct links, only provide as candidate targets those topics that are in the scope of the map.

M.4 - Map view of everything. The CMS should be closed over maps such that all DITA-related content is presented in a map (for example, the system synthesizes a map that includes all topics in the repository or all topics within the scope of a particular CMS-specific organizing structure). By the same token, the results of queries should be viewable as DITA maps (that is, given a query result, it is either literally returned as a map to which the normal CMS map functionality is applied or the CMS provides a "save as map" option).

M.5 - Support for compound maps. CMS must support the use of topicref format="ditamap" to construct "compound maps". Any CMS functionality that modifies existing maps must preserve any pre-existing map-to-map relationships. Any CMS functionality that creates new maps should provide features for creating subordinate maps.

Specialization-Related Features

S.1 - All processing applied to specializations automatically. Any CMS functionality that is specific to a DITA-defined base type should be automatically applied to specialized elements. For example, if the CMS provides a feature for importing maps, it should automatically provide that feature for importing any specialization of map. If the CMS provides some form of configuration for mapping from specific element types to generic CMS functionality (for example, indicating which elements are links), such a mapping defined for the DITA base types should automatically be applied to all specialized documents without additional user or configuration effort.

That is, one of the main points of specialization is that DITA-specific processing just works for any specialized element. This is how the DITA Open Toolkit works and all other DITA-aware tools should too.

Or said another way, DITA awareness means "specialization awareness" in addition to whatever else it might mean.

S.2 - Capture and maintain the dependency relationships among shell document types and the base and specialized modules they use. For example, if I import a shell document type that includes a specialization module I created, the CMS should capture the dependency between the shell and the specialization module as well as the dependencies from the specialization module to the base DITA-provided modules.

S.3 - Specialization project management. System should provide features for managing the components of specialization modules as "projects" such that there is clear binding between the specialization module name and the specific implementation components that make it up. This project manager should reflect and, as appropriate, enforce (or at least encourage and reward) the implementation design patterns defined by the DITA architecture. This management should include tracking dependencies among specialization schema components (that is, from local specializations to the DITA-provided modules they depend on).

S.4 - Generalized views. System should provide ability to see, on demand, a generalized view of a given map or topic. It should provide a way to select the level of generalization desired. This view should be read-only by default but should allow for saving the generalized view as a new object.

Tutorial: Specializing DITA Conditional Attributes

2007-03-15T11:59:00.000-05:00

[5 April 2007: This tutorial has been incorporated in my more complete and formal DITA Specialization Tutorial hosted here: http://www.xiruss.org/tutorials/dita-specialization/.]

A new feature in DITA 1.1 is the ability to specialize from the base= and props= attributes. For conditional processing, this lets you add your own attributes rather than using otherprops=, which can be clearer to authors and implementors. [NOTE: at the time of writing the DITA Open Toolkit does not implement support for specializations of props=, but it should be added soon.]

This form of specialization is fairly easy to implement. This tutorial shows you how to do it using DTDs (the mechanism using Schemas is essentially the same and if you've stepped up to using the DITA 1.1 schemas I'm going presume you can figure this out on your own).

The specialization requires two things:

1. Modification of any shell DTDs that need to reflect the specialized attribute (e.g., topic.dtd, reference.dtd, or your own specialized topic types' shell DTDs). You integrate the specialization attribute domain through the shell DTDs.

2. For each specialization of props=, a .ent declaration set that defines the attribute and a corresponding domain declaration. This is the "attribute domain specialization module".

Note that as a rule, any production use of DITA will likely require local versions of the DITA-provided shell DTDs, if only to do configuration of the domains you need, so unless you are using DITA very informally, you should already have local copies of all the DITA-provided shell DTDs. Just saying.

For this tutorial we want to create a specialization of "props=" called "phase-of-moon" that takes as its value one or more moon phase names (e.g., "full", "new", "waning", "waxing", etc.). We will call our domain "moonPhaseProp". (Domains must have unique names within the scope of the shell DTDs or schemas that use them.)

For organizing the files, I like to create a separate directory to put my local shell DTDs and specializations in. For this tutorial assume we're putting everything in the directory dtd/myspecs within the normal DITA Open Toolkit distribution structure. (It can go anywhere as long as you configure the entity resolution catalogs appropriately, but for initial development and testing I find it convenient to use relative paths to the various declaration components as that eliminates a variable from the configuration (resolution via catalogs) that can lead to confusing errors. Once you've established that the declarations are correct you should replace all relative paths with absolute URLs or (if you must) PUBLIC IDs that are resolved via catalogs. For my development work I use the OxygenXML editor, which makes it easy to set up catalog configurations for testing resolution via catalogs (and generally testing the correctness of all the parts). Similar tools like XML Spy are probably comparable (but I don't use them).

Step 1 is to create the attribute domain declaration:

1.a. Create a file named moonPhasePropsDomain.ent

1.b. In that file, create these two declarations:


<!ENTITY % moon-phase-props-d-attribute 
   "phase-of-moon 
     CDATA 
     #IMPLIED
   " 
>

<!ENTITY moon-phase-props-d-att
  "a(props  phase-of-moon)"
>

The first declaration declares the "phase-of-moon=" attribute and puts it in a parameter entity so we can add it to the DITA-defined %selection-atts parameter entity via the %props-attribute-extensions configuration parameter entity.

The second declaration is the domain declaration string for the attribute domain. It will be added to the value of the "domains=" attribute declared for each topic-type element type.

You should of course add an appropriate descriptive header to the file as well as a little documentation for the attribute itself.

This is all that is required for the attribute domain module.

Step 2 is to integrate the domain into your local copy of each shell DTD. The pattern is the same for each shell. For this tutorial I'm using a copy of the topic.dtd shell.

2.a. Find the comment that reads "DOMAIN ATTRIBUTE DECLARATIONS". Following that comment, add this declaration:

<!ENTITY % moon-phase-props-d-dec     
  SYSTEM "moonPhasePropsDomain.ent"                                                
>
%moon-phase-props-d-dec;

This pulls in the attribute domain module.

2.b. Find the comment that reads "DOMAIN ATTRIBUTE EXTENSIONS". Following that comment you should see a declaration for the %props-attribute-extensions parameter entity. It will probably be declared as an empty string.

Modify the entity replacement text to include a reference to the %moon-phase-props-d-attribute parameter entity:

<!ENTITY % props-attribute-extensions  
  "%moon-phase-props-d-attribute;"
>

This adds the "phase-of-moon=" attribute to the %selection-atts parameter entity which is then included in the %univ-atts parameter entity, making this new attribute available on most elements (some elements, such as title, are not selection candidates).

2.c. Find the comment that reads "DOMAINS ATTRIBUTE OVERRIDE". Following that you should see the declaration of the text entity included-domains and it should include references to a number of "x-d-att" text entities.

To this entity add a reference to the moon-phase-props-d-att text entity:

<!ENTITY included-domains 
  "&hi-d-att;
   &ut-d-att;
   &moon-phase-props-d-att;
   "                
>

This formally declares your props= attribute specialization so that DITA 1.1 processors will know that "phase-of-moon=" is in fact a conditional attribute and that they should filter on it as appropriate.

That's all there is to it. Now just repeat Step 2 for each shell DTD you use and you're done.

Step 3 is to test your declarations to make sure they work. This is simply a matter of creating an XML document that uses your local shell DTD as its DTD and verifying that the "phase-of-moon=" attribute is now available on all elements that allow the selection attributes.

Tagging the Old Posts

2007-03-07T10:21:00.000-06:00

Now that Blogger lets you tag your posts with descriptive tags I'm going through and tagging all my old posts to help with retrieval (which is pretty bad right now--not sure how to address that without creating some sort of hand-crafted index over the posts).

If you're subscribed to this blog as a feed this may cause you to get re-fed all the old posts.

If this does happen, I apologize for any stuffing of feed reader inboxes this causes.

Nothing To Say?

2007-03-07T10:06:00.000-06:00

I haven't posted in a long while, what with the holidays and work and vacation and being sick and ...

Since the first of the year I've been very busy with client work, most of it DITA-related (creating sophisticated specializations, doing data analysis of documents that don't have an obvious mapping to DITA, etc.). Very interesting stuff--I've learned a lot about DITA and the Open Toolkit and XSD schemas but nothing that translates directly into pithy blog posts (although I do plan to write a tutorial on creating DITA specializations, which turns out to be remarkably easy once you get the pattern down).

In the meantime, I haven't really been doing much with any interesting technology nor have I seen much interesting coming down the pike (although Mike Kay's recent posting about assertions in XML Schema is pretty interesting--that could be very powerful if the Working Group can get it right). [And let me say that all the WS-* and identity standards stuff just bores me so totally to tears that I can't stand it--I'm sure it's important stuff but I just don't see how at the end of the day it's really going to matter much to our day to day and if it does I'm certainly not going to be anything other than a naive user of it....]

So I thought I should post something just to remind people that I'm still out here.

Some of the topics that are on my list to talk about, but that will require a good bit of time to discuss clearly and cogently, include:

- Why Norm just doesn't get what's wrong with DocBook and right with DITA, namely specialization

- So much more the DITA Open Toolkit could do with relationship tables

- Using DITA maps to model time-specific versions and similar configurations

- Reforming DITA's linking semantics and addressing infrastructure (a road map for DITA 2.0)

So here's hoping I have a little more time to write about these things in the future...