The Cafes

An Open Letter to My Public Library

2012-01-19T17:20:51Z

Dear Librarians,

I’d like to thank you for the work you’ve done putting the library catalog online. The ability to reserve books online (and then renew them online when I don’t finish them on time) has been invaluable. It has dramatically increased my use of the library. Now when I need a book I routinely check the library first rather than ordering it from Amazon. It’s cheaper, the book gets to me faster; and when I’m done, the book no longer takes up space in my apartment. Excellent! Kudos all around.

And now you have eBooks so I don’t even have to go to the library to pick up my reservations! Regrettably the selection of eBooks is somewhat thinner and more oversubscribed; and yes, I know this is partially the publishers’ fault. Still, for the books that are available, it’s wonderful knowing that even the thickest physics text or mathematical tome isn’t going to weigh more than a small eReader or tablet. It makes reviewing calculus on the subway so much more practical.

I’d like to make a friendly suggestion for ramping this up a notch, making the library even more useful, expanding the collection, and increasing monetary donations to the library at the same time. Your circulating collection is large, probably one of the largest in the country, and certainly the largest one I’ve ever had the pleasure to use. Probably 90% of the time the book I’m looking for is available at one of your branches, and you helpfully bring it from wherever it is to my local library where I can pick it up off the reserve shelf. But there’s still that 10% of the time when you happen not to have the book I’m looking for. (And for eBooks that’s more like 90% of the time.) Sometimes that’s because it’s a relatively obscure technical book; but sometimes it just looks like a fluke. For instance, it’s the second book in a trilogy for which you have the first and the third, but somehow missed the middle (or it went missing). Or it’s a novel by an author, most of whose works you already have. It’s something that clearly fits in your collection but just doesn’t happen to be there yet. So off I surf to Amazon where I buy a book I only really want to read once, and that then is going to sit on my shelf untouched for years.

Here’s my idea: I’d rather buy the book for the library and than buy it for myself.

What does this entail? Simple: an online form on your web site where I can request a book that’s not currently in your collection, and where I can simultaneously make a donation with my credit card earmarked for the purchase of that book. You’d receive the information, verify that the book is reasonably suitable for the collection, charge my credit card, and then make the purchase. When the book arrives, you’d add the usual bar codes and magnetic strips and insert it into the catalog. Then you’d put it on the reserve shelf for me, and send me the usual e-mail letting me know it’s arrived. I’d have the customary 10 days to go pick it up and check it out. Of course I’d have to return it in the usual time frame just like any other library book (maybe renewing it once or twice if nobody else had placed a hold on it). At this point you’d put it on the shelves for everybody to browse and read. For eBooks this could be even faster. There’s no technical reason I couldn’t buy the book for the library and be reading it in the time it takes to download.

I think this is a win-win for all concerned. The library gets more books (and more donations); and patrons get to read books without perpetually storing them in their apartments. (I’m sure you know how big a concern that is for New Yorkers.) If you asked for the retail cover price of the book in advance, you’d probably even make a little profit on the deal to cover overhead and other activities. Since I’d get a tax deduction, I could afford to pay a little more than the typical Amazon discount price. And as an added bonus, some of your patrons (me included) have employers who match charitable donations 1:1, so a $40 book purchase would garner you an extra $40 in unrestricted funds to be used for staff salary, computers, building maintenance, or other unsexy but necessary expenses.

What do you think? Sounds like a plan?

A Prediction: P == NP and within 5 years

2011-09-29T11:36:37Z

So after reading this poll of mathematicians and computer scientists who have given the process way more thought than I have, I now suspect that indeed P == NP (though perhaps not usefully so). Why do I think this?

Quite simply, it seems to me that there are a lot of possible attacks on this problem, and that no one understands them all. The mathematicians claiming that P != NP (and especially that it won’t be solved soon) are basically claiming that the math they know won’t lead to a solution, whereas the mathematicians claiming that the problem will be solved are saying that they expect the math they know will lead to a solution. Knowing what won’t work is important, but for making predictions knowing what will is more important.

Of course, it could be that the problem will be solved and the solution will be that P != NP, as most folks seem to expect. Or it could be shown that the problem is undecidable (though that would likely take longer).

Perhaps I’m just an optimist, but I’m betting on a constructive solution to the problem. That is, I expect to see an algorithm soon that “solves” a known NP problem in polynomial time. That polynomial time may still be greater than the age of the universe, but it won’t be NP.

Cafe au lait/con Leche Are Down

2011-03-29T13:56:19Z

Yes, I know Cafe au Lait and Cafe con Leche are down (not that this matters a great deal to anyone right now.) This is the result of a planned maintenance that requires me to do some DNS changes on my end. Unfortunately, I’m a little hobbled at the moment since I’m traveling in Europe without my personal laptop and I’m staying a hotel with pathetic wireless. I’ll try to get it fixed ASAP, but as soon as possible could be as late as next Monday.

Update: I forgot that XOM downloads are also hosted on Cafe con Leche, and these matter a little more than year old news. So I spent a few minutes this morning to dig the password for my DNS server out of some web-based backups (Go Crashplan!) and updated the sites with the new IP addresses. All should be resolved as soon as the changes propagate over the next three hours or so.

Update 2: All is well. Both sites are up.

My New Year’s Resolution

2011-09-29T11:34:59Z

In 2011 my New Year’s resolution is to do more things the easy way, even if it takes longer the first time. I am going to stop using brute force to solve problems. In particular:

I am finally going to memorize how one redirects both stderr and stdout to the same stream. (2>&1 |)
I am going to learn the sed? trick my advisor showed me 20 years ago for repeating a command from the shell history while substituting one word for another, instead of just using the arrow key to backup to and erase the string. (^string1^string2^ or !!:s/string1/string2/ or for global substitution, not just the first occurrence !!:gs/string1/string2/)
I am going to increase my regex fu and use regular expressions consistently instead of just editing 20 lines of copy and paste code. (This would be easier if every editor didn’t have subtly different syntax.)
I am going to use Python to automate repetitive tasks.

XML 2.0

2010-12-05T12:13:04Z

First for the record, I’m speaking only for myself, not my employer, the W3C, Apple, Google, Microsoft, WWWAC, the DNRC, the NFL, etc.

XML 1.1 failed. Why? It broke compatibility with XML 1.0 while not offering anyone any features they needed or wanted. It was not synchronous with tools, parsers, or other specs like XML Schemas. This may not have been crippling had anyone actually wanted XML 1.1, but no one did. There was simply no reason for anyone to upgrade.

By contrast XML did succeed in replacing SGML because:

It was compatible. It was a subset of SGML, not a superset or an incompatible intersection (aside from a couple of very minor technical points no one cared about in practice)
It offered new features people actually wanted.
It was simpler than what it replaced, not more complex.
It put more information into the documents themselves. Documents were more self-contained. You no longer needed to parse a DTD before parsing a document.

To do better we have to fix these flaws. That is, XML 2.0 should be like XML 1.0 was to SGML, not like XML 1.1 was to XML 1.0. That is, it should be:

Compatible with XML 1.0 without upgrading tools.
Add new features lots of folks want (but without breaking backwards compatibility).
Simpler and more efficient.
Put more information into the documents themselves. You no longer need to parse a schema to find the types of elements.

These goals feel contradictory, but I plan to show they’re not; and map out a path forward.

The technical requirement is to maintain compatibility. Existing parsers/tools still work. On a few edge cases, an XML 2.0 parser may report slightly different content than an XML 1.0 parser. E.g. more or less white space. However all XML 2.0 documents are well-formed XML 1.0 documents. It will be possible to write an XML 2.0 parser that is faster, simpler, and easier to use than an XML 1.0 parser. However XML 1.0 parsers will still be able to parse XML 2.0 documents. XML 2.0 parsers will not, however, be able to parse all XML 1.0 documents, just as XML parsers can’t parse arbitrary SGML.

The attitudinal requirement is that XML 2.0 be useful. It has to solve real problems for real users. And it has to solve all the problems XML 1.0 solves too.
Tim Bray’s Skunkworks meets goals #1, #3, and #4 but it didn’t offer #2, so there was never a strong desire to implement it. Languages don’t get replaced simply because there’s something new that does the same thing a little more cleanly or a little better. They get replaced when there’s something new that does the old things better and does new things too.

What we take out

Internal DTD subsets

The internal DTD subset is responsible for much of the complexity and security issues with XML. Get rid of it. XML 2.0 documents cannot contain internal DTD subsets.

Validity

Validity is defined separately. The DOCTYPE declaration, if present, can point to schemas in any language with any validation rules. Perhaps we can use the public identifier to name the type of the schema and the system identifier to point to it. However validation is optional and outside the scope of the spec.

Namespace well-formedness is required

Build in namespaces. Require namespace well-formedness. All namespaces must be absolute URIs.

Neurotic and psychotic documents

All namespace prefixes must be declared on the root element. No prefix may have two different URIs in two different parts of the document. This may require rewriting of namespace prefixes when combining documents, e.g. with XInclude. This is uncommon, but if we have to do it we’ll do it.

Default namespaces may still be declared on any element.

CDATA sections

Eliminate CData sections. They’re unnecessary syntax sugar and just lead to confusion among users and extra work for parser implementers. Ideally I’d also like to eliminate the special treatment of the three character sequence ]]>. However that might break existing parsers.

C1 controls

The only reason C1 controls show up in an XML document is because someone has mislabeled a Windows text file. By forbidding these characters we will catch this problem much earlier.

There are likely some other Unicode compatibility characters we should forbid as well.

DOM and the Infoset

Abandon DOM. Abandon the Infoset. They’re confusing and not what users want or need. Folks can still use them–XML 2.0 is a subset of XML 1.0, after all–but they have no normative standing.

Encourage a variety of different APIs and data models appropriate to their respective domains and languages. However be very clear that the actual text of the document is the normative form. The data model is a representation of the text. The text is not a serialization of the data model.

White space preservation

White space is significant inside an element if and only if xml:space=”preserve”. Otherwise all consecutive white space is collapsed to a single space.

Alternatively, provide a means of identifying elements in which white space should be preserved in the prolog, e.g. through a processing instruction.

There are counter-examples this this–e.g. the HTML pre element–but I think this is what most people want most of the time so it makes sense to make it the default and make white space preservation the one that requires special casing. Some thought is needed to figure out the algorithm though, especially for white space like this

<foo> bar </foo>

and this

<foo>bar/foo>

and this

The <em>quick</em> <strong>brown</strong> fox jumped over the lazy dog.

Exactly which white space is retained, and to which element is it assigned?

Whatever way we go here, use the same rules for all attribute values. Attribute value normalization was a mistake in XML 1.0 anyway. Drop it.

Most character encodings

Use XML 1.0 name rules but base on Unicode properties for all non-ASCII characters. Provide a means of identifying the Unicode version in use. Default to Unicode 2.0, unless the document declares otherwise. This is the change I’m least interested in, because it may break compatibility, and has no known actual use cases. That is, no one has ever been able to present a document that any actual user wants to produce that cannot be encoded with XML 1.0 name rules.

Require UTF-8 or UTF-16 exclusively. In fact, maybe just require UTF-8. No other encodings are permitted. Use the encoding declaration to identify the Unicode version and recognize XML 2.0? e.g.

<?xml version=”1.0″ encoding=”Unicode_5″?>

Fallback to an XML 1.0 processor if this doesn’t work.

It does feel a little ugly to specify version="1.0" for what I’m calling XML 2.0. However, as long as the document adheres to the XML 1.0 grammar and constraints, this is completely legal. Maybe we shouldn’t even call this a new version of XML but give it a new name. Perhaps YML (because Y comes after X)?

standalone declaration

This means nothing in practice. No one relies on it. Get rid of it.

What we add

More entities

Predefine the HTML 4 character entity set. Otherwise eliminate all general entity references. We can make this work with a required system ID that points to a DTD containing the definitions. Of course XML 2.0 parsers will not actually load this DTD, only XML 1.0 parsers will need to load it.

Links

Build in xml:base and xml:id.

Build in some form of simple links sufficient for use of HTML. Perhaps just xml:link or xml:href, nothing fancier. This contains a URL, and is normally considered semantically equivalent to an HTML <a href=''>. I.e. it’s a blue underlined thing you click on.

Possibly we should even ditch the namespace prefixes. E.g base, id, and link/href attributes will simply be defined with these semantics.

Data Structures and Types

The biggest lack of XML 1.0 is a standard means of encoding basic data structures and types used in programming: lists, sets, maps, structs, ints floats, etc. This is why JSON is so popular. It’s not that these things can’t be encoded in XML 1.0, just that there are so many ways they can be encoded, and libraries provide no support for decoding them.

To support this use case, we will allow an optional xml:type or perhaps just type attribute on all elements. The value is a name from a type library such as XSD primitive types. Predefine a basic type library with the minimal types: integer, decimal, string, boolean, date, time. For example,

<sku type="string">H647345</sku>
<date type="date">2010-10-12</date>
<quantity type="integer">12</quantity>
<price type="decimal" units="dollars">3.45</quantity>
<price type="decimal" units="percent">7.25</quantity>

The default simple type is string. I.e. we can instead write

H647345

We do not want the full set of XSD types. In particular, we do not want float, double, short, int, and long. Integers have arbitrary size, and real numbers are expressed in base 10. Parsers may express these with more or less precision as they choose.

We also want list and map types and maybe set:

<crew type="list">
  <name>Fred</name>
  <name>Jane</name>
  <name>Bob</name>
</crew>

<dimensions type="map">
  <width type="decimal" units="cm">34.3</width>
  <length type="decimal" units="cm">120.0</length>
  <height type="decimal" units="cm">3.10</height>
</dimensions>

Here I’ve made the keys simply the element names. Possibly with maps we want to allow or require that the keys also be attributes or even elements, which would enable a broader range of key types.

We probably want to define some sort of simple type defintiion that can be used by parsers rather than explicitly specifying the type of each element. E.g.

<dimensions type="map" valuetype="decimal" keytype="string">
  <dimension key="width" units="cm">34.3</dimension>
  <dimension key="length" units="cm">120.0</dimension>
  <dimension key="height" units="cm">3.10</dimension>
</dimensions>

Parsers and APIs are encouraged to make this content available to clients in a more cooked form appropriate to the programming language rather than as raw strings and nodes. However these types are all advisory, not compulsory. Further note that these types could be further parsed by a library that sits on top of an XML 1.0 parser. An XML 2.0 parser is not required.

TBD: should XML 2.0 parsers treat violations of the constraints on the semantic types (e.g. <amount xml:type=’int’>Bob</amount>) as a fatal well-formedness error or a non-fatal validity error? If we just use the type attribute it would have to be the latter to avoid breaking compatibility with existing documents.

The details of the type system remain to be worked out. How do we name and define new types? What does the syntax of a type definition look like? Do we need collections other than list and map? Exactly which primitive types do we predefine? Will the world really let us get away with integers and decimals or will they scream for int and float? Much work here remains to be done. But the basic idea is sound. We don’t need to reinvent the same type annotations for every vocabulary. Just as XML 1.0 improved on SGML by eliminating the freedom to use different syntax to delimit elements and attributes, XML 2.0 will improve on XML 1.0 by eliminating the freedom to use different syntax to denote types. We will not limit which types one can express. We will simply specify a standard form for denoting type information.

What we can’t do

There are a few minor changes I haven’t been able to figure out how to make while maintaining backward compatibility. These include:

Allowing — to appear in comments
Not making ]]> special
version=”2.0″

If you can figure out how to make these compatible, please do let me know. I’m almost willing to compromise on these minor points of backwards compatibility to simplify the parsing process, but I’m not sure that’s wise.

version="2.0" is the trickiest one. XML 1.0 parsers are not required to error out on this, but in practice many do. Perhaps we should drop the XML declaration completely? I.e. any document with an XML declaration is ipso facto not an XML 2.0 document. Instead all XML 2.0 documents will be identified with a specific DOCTYPE:

<!DOCTYPE XML2 PUBLIC "application/xml+xsd http://example.com/optional/actualschema.xsl" http://www.example.com/xml20.dtd">

As mentioned above, the DTD mentioned by the system identifier is a legal XML 1.0 DTD that defines all the HTML entities. XML 2.0 aware processors will ignore this. The public identifier (which may be empty) contains a MIME media type followed by the URL to the actual schema for the document.

The Way Forward

There’s one other reason XML 1.0 succeeded where XML 1.1 failed: XML 1.0 was designed by a small committee of like-minded folks with a common goal. They didn’t always agree on the route, but they were all driving to the same destination. The W3C pretty much ignored them until they were done. By contrast, XML 1.1 was hobbled by a W3C process that took far longer to accomplish much less. If XML 2.0 is to succeed, it needs to follow XML 1.0, not XML 1.1.

A small group of interested folks should convene outside the W3C and write the spec. One month to agree on goals and requirements; one month to write the first draft. Run it up a flagpole and see who salutes.

Step 2 is to write parsers and APIs for the new draft, and gain some implementation experience. Develop a test suite of sample documents. That will take longer, but is necessary. Work the bugs out of the spec. At the point where the goals seem to be satisfied and the spec is reasonably implemented, present it to the world as a fait accompli. If some organization feels like adopting it as a formal standard, that’s fine, though it’s hardly necessary.

The real goal is to take the lessons of the last 12 years of XML, and apply them to create something even better. Who’s with me?

P.S.

If you want to comment, please be aware that you need to escape < as < and > as >. The comments allow basic HTML but aren’t smart enough to distinguish between plain text and real HTML comments.

Could not load a dependent class com/jcraft/jsch/Logger

2010-07-09T10:53:44Z

Have you ever seen an Ant error message like this?

BUILD FAILED
/Users/elharo/Projects/XOM/build.xml:545: Problem: failed to create task or type scp
Cause: Could not load a dependent class com/jcraft/jsch/Logger
       It is not enough to have Ant's optional JARs
       you need the JAR files that the optional tasks depend upon.
       Ant's optional task dependencies are listed in the manual.
Action: Determine what extra JAR files are needed, and place them in one of:
        -/opt/ant/lib
        -/Users/elharo/.ant/lib
        -a directory added on the command line with the -lib argument

Do not panic, this is a common problem.
The commonest cause is a missing JAR.

This is not a bug; it is a configuration problem

As usual, the ant error message is completely unhelpful, though for once it’s at least technically correct. (Most of the time when ant says, “This is not a bug; it is a configuration problem”, it is in fact a bug and not a configuration problem.) Here’s what’s really happening.

The jsch jar file distributed from http://www.jcraft.com/jsch/ is corrupt. Either they uploaded it wrong or they misconfigured their web server or both. (Update: it looks like a misconfigured server/poorly designed web page. The so-called download link isn’t that at all. You can follow it but not save it.) The jar file with the relevant classes is there, but it’s no good. You can check your local copy by trying to list its contents with jar tvf:

$ jar tvf /usr/share/ant/lib/jsch*jar
java.util.zip.ZipException: error in opening zip file
	at java.util.zip.ZipFile.open(Native Method)
	at java.util.zip.ZipFile.(ZipFile.java:114)
	at java.util.zip.ZipFile.(ZipFile.java:75)
	at sun.tools.jar.Main.list(Main.java:979)
	at sun.tools.jar.Main.run(Main.java:224)
	at sun.tools.jar.Main.main(Main.java:1149)

If you see anything but a list of classes, the problem is that the JSCH jar is no good. To fix it, use this link to sourceforge instead.

Once again I am reminded of the perils of depending on external libraries, especially ones you don’t build or distribute with your own product. In 2010 ssh and scp are mandatory features of any build and deployment system. Secure communications are too important to be left to random third party web sites.

Dn’t Abbrvt

2011-03-15T15:10:56Z

Is req a request or a requisition?

Is res a response, a reservation, a resume, or a result?

Is def a default or a definition?

Is rng a range or a random number generator?

Is v1 version 1 or value 1?

Is e an event, an entity, or an exception?

Is f a file or a float?

Is lst a list or the least value?

Is temp a temporary variable or a temperature reading?

Is rep a representation, a representative, a repetition, or a reputation?

Is tm a time or a trademark? Or even another temporary variable? And if it is a time, is it a timestamp, a time of day, or a duration? (These are three very different things.)

Is admin an administrator, an administrative assistant, or a system administrator?

In context, you can usually figure these things out, but you have to think about them. That’s inefficient. Far better to just spell out what you mean from the get go.

There are a few abbreviations that are so well known and understood that they’re acceptable:

max for maximum
min for minimum
in for InputStream
out for OutputStream
e or ex for an exception in a catch clause (but nowhere else).
num for number, though only when used as prefix as in numTokens, or numHits.

You can use single letter variable names for the occasional quantity that has no meaning other than its type. For example, a string variable can be named s, an int variable i, or a double variable x. However, this should only be used when the program really doesn’t know anything about the nature of the variable other than its type. For example, a method that calculates the cube root of a double may name its argument x; but a method that converts temperature from degrees Fahrenheit to degrees Celsius should name its argument degrees, degreesFahrenheit, or perhaps temperatureFahrenheit.

In addition, there’s nothing at all wrong with using common acronyms that are more recognized than what they stand for: URL, HTML, XML, XSL, etc. However, these are the exceptions, not the rules; and I would still be careful with common abbreviations that mean something else at first glance. EmployeeBO is almost certainly a Business Object, but that wasn’t what you read it as first, was it?

Code should be optimized for reading and comprehension, not for marginally faster typing. You should no more abbreviate names in your code than you do words in your sentences. No1 wnts 2 rd txt wrttn lk ths.

Bruce Eckel is Wrong

2010-04-20T10:59:14Z

Every time the subject of checked versus runtime exceptions comes up, someone cites Bruce Eckel as an argument by authority. This is unfortunate, because, as much as I like and respect Bruce, he is out to sea on this one. Nor is it merely a matter of opinion. In this case, Bruce is factually incorrect. He believes things about checked exceptions that just aren’t true; and I think it’s time to lay his misconceptions to rest once and for all.

Let’s see exactly what Bruce’s mistake is. The following is an extended selection from Thinking in Java, 4th edition, pp. 490-491:

An exception-handling system is a trapdoor that allows your program to abandon execution of the normal sequence of statements. The trapdoors used when an “exceptional condition” occurs, such that normal execution is no longer possible or desirable. Exceptions represent conditions that the current method is unable to handle. The reason exception-handling systems were developed is because the approach of dealing with each possible error condition produced by each function call was too onerous, and programmers simply weren’t doing it. As a result, they were ignoring the errors. It’s worth observing that the issue of programmer convenience in handling errors was a prime motivation for exceptions in the first place.

One of the important guidelines in exception handling is “Don’t catch an exception unless you know what to do with it.” In fact, one of the important goals of exception handling is to move the error-handling code away from the point where the errors occur. This allows you to focus on what you want to accomplish in one section of your code, and how you’re going to deal with problems in a distinct separate section of your code. As a result, your mainline code is not cluttered with error-handling logic, and it’s much easier to understand and maintain. Exception handling also tends to reduce the amount of error-handling code, by allowing one handler to deal with many error sites.

Checked exceptions complicate the scenario a bit, because they force you to add catch clauses in places where you may not be ready to handle an error. This results in the “harmful if swallowed” problem:

try { // ... to do something useful } catch (ObligatoryException e) {} // Gulp!

Do you see the mistake? It’s a common one. Let me repeat it, so it’s really obvious [emphasis mine]:

Checked exceptions complicate the scenario a bit, because they force you to add catch clauses in places where you may not be ready to handle an error.

This is false. You are never forced to add catch clauses where you are not ready to handle an error. This isn’t a matter of opinion. It’s a matter of fact. Checked exceptions do not require catch blocks. They require a catch block OR a throws declaration. Eckel’s entire argument is based on ignoring the possibility of a throws declaration.

While Bruce is absolutely right that you should not catch an exception unless you know what to do with it, this in no way means that you should insert a catch block everywhere a checked exception may be thrown. If you aren’t ready to handle an error at one place, let the exception bubble up. If a checked exception is thrown inside a method where you are not ready to handle it, then the correct response is to add a throws clause to the method indicating that the exception will bubble up from that method. For example,

 public void doSomethingUseful() throws ObligatoryException {
   // ... do something useful that throws an obligatory exception
}

You do not and should not insert a catch block in a method where you cannot do anything reasonable in the catch block. Checked exceptions never meant that every exception had to be caught as soon as it was thrown. It is perfectly acceptable to declare that a method throws a checked exception. Indeed, this is exactly how exceptions are meant to be used. It warns whoever calls your method that they need to be ready for this exceptional condition, and they either need to catch it and handle it themselves; or, they themselves need to declare that they throw it so that they warn their callers.

Yes, if it were true that every checked exception needed to be caught immediately, then checked exceptions would be incredibly inconvenient. However, experienced Java programmers don’t do this. Catching each and every checked exception at the first opportunity is a sure mark of a novice Java developer.

Occasionally, you’ll override a method inherited from a superclass or implement a method declared in interface that does not declare it throws the checked exception that your method throws and thus can’t throw the correct exception. In this case alone, it may be acceptable to wrap the exception in either a checked exception that the original declaration declares or in a runtime exception (if the original declaration does not declare any appropriate checked exceptions). For example,

   @Override
    public void doSomethingUseful() {
        try {
     // ... do something useful that throws an obligatory exception
        }
        catch (ObligatoryException e) {
            throw new ObligatoryRuntimeException (e);
        }

However, you still don’t need to handle the exception before you’re ready for it.

I will note that this situation is a failure of design. When you’re forced to do this, one of two things is broken:

The superclass/interface was not designed properly for extension. Specifically it did not take into account the exceptions overriders/implementers might reasonably want to throw. The method likely should have been declared final and probably shouldn’t be extended at all.
The overriding/implementing method is violating the contract of the method it overrides/implements by doing something it really should not be doing.

A good example of the latter would be letting an IOException wrapped in a RuntimeException escape from the run() method of java.lang.Thread or java.lang.Runnable. These methods do not have sufficient context to handle such an exception, but something further down in the call chain does. The exception should be handled before it bubbles all the way up (though not necessarily in the same method where it’s first thrown).

In a method properly designed for extension, an empty throws clause (or a throws clause that does not match the actual exception) indicates that callers of that method cannot handle and do not expect such an exception. An overriding method that throws a new exception is violating the contract by failing to handle it, the same as it would were it to restrict a precondition or loosen a postcondition. (In essence, this is another form of loosening a postcondition.)

Eckel is not the only one to make the mistake of assuming that checked exceptions must be immediately at handled at the first possible opportunity. For example, he cites Barbara Liskov and Alan Snyder explaining their decision not to include checked exceptions in CLU:

requiring that the text of a handler be attached to the invocation that raises the exception would lead to unreadable programs in which expressions were broken up with handlers

True. Requiring that the text of a handler be attached to the invocation that raises the exception would lead to unreadable programs. Fortunately neither Java nor checked exceptions require any such thing. When handlers are appropriate they can usually be moved to the end of a method, far away from the the invocation that raises the exception. When handlers are inappropriate, you use a throws clause instead. Immediately following every statement that can throw an exception with a catch block is bad form on a par with goto.

Liskov and Snyder continue:

“We felt it was unrealistic to require the programmer to provide handlers in situations where no meaningful action can be taken.”

Also very true, but of course, checked exceptions do not require the programmer to provide handlers in situations where no meaningful action can be taken. When no meaningful action can be taken (out of memory error, stack overflow, class not found, etc.) Java programs throw Errors, not checked exceptions, not even runtime exceptions. Checked exceptions signal environmental problems that programmers cannot prevent or predict, should test for, and most decidedly can handle. To choose the most extreme example, if a production-worthy database system is writing a file and the disk fills up, it should handle the condition gracefully without corrupting the database. A disk full error is neither unforeseeable nor unmanageable. Most checked exceptions aren’t even that tricky to respond to.

What checked exceptions actually do require is that any method that can throw a checked exception warn its callers that the exception may be thrown. That’s all. A checked exception is nothing more and nothing less than part of the return type. Methods may return normally or they may throw exceptions. It makes just as much sense to specify the exceptions that can be thrown by a method as it does to specify that a method returns an int or a String. A method that does not declare the exceptions it throws is incomplete.

A lot of us didn’t really get checked exceptions when Java was released in the mid-nineties. It was a genuinely new idea that I don’t think any programming language before had foreshadowed. Liskov and Snyder wrote the paper quoted here in 1979, and their quotes make sense if you assume they simply didn’t conceive of having different kinds of exceptions in the language. if you only have one kind of exception then it makes sense for it to be a runtime exception rather than a checked exception.

Personally, I didn’t really understand how to use exceptions until the first edition of Effective Java was published. But it’s been 15 years. That’s long enough for the message to get out. In 2010 we know better. Proper error handling requires distinguishing programming errors (runtime exceptions) from environmental problems (checked exceptions). Proper error handling requires correcting programming errors and writing handlers for unpreventable environmental conditions. Proper error handling requires knowing when to catch and knowing when to throw. If you try to work with only one kind of exception, or try to get by with only catch but no throws, then exceptions are going to seem very ugly and inconvenient. But if you use checked and runtime exceptions, and use catch and throws, then your error handling code will be far cleaner, safer, and more maintainable. Error handling without checked exceptions and throws is like arithmetic without * and /. Sure, you can do it, but why would when you when using the features the language offers makes life so much simpler?

Would You Entrust Your Data to These Yokels?

2009-12-31T12:40:19Z

securely to www.delldatasafe.com, but we can't confirm that your connection is secure.

Normally, when you try to connect securely,
sites will present trusted identification to prove that you are
going to the right place. However, this site's identity can't be verified.

What Should I Do?

If you usually connect to
this site without problems, this error could mean that someone is
trying to impersonate the site, and you shouldn't continue.

www.delldatasafe.com uses an invalid security certificate.

The certificate is not trusted because the issuer certificate is unknown.

(Error code: sec_error_unknown_issuer)" title="delldatasafe" width="811" height="602" class="alignnone size-full wp-image-1002887" />

If Dell can’t even manage their public key certificates, how can I trust them to keep my data safe and secure?

Technical Details

www.delldatasafe.com uses an invalid security certificate.
The certificate is not trusted because the issuer certificate is unknown.
(Error code: sec_error_unknown_issuer)

SourceForge for the 21st Century

2009-12-31T12:39:46Z

Lately I’ve been thinking a lot about continuous deployment for reasons I’m not quite yet at liberty to disclose. This has inspired me to improve the XOM release process, to make it more of a one click process, or, to be more accurate, a one ant target process. I can now release a new version simply by typing:

$ ant -Dpassword = secret -Dwebpassword=other_secret release

This not only builds the entire project. It tags the release in CVS, uploads the zip and tar.gz files to IBiblio, and uploads the documentation to my web host. It doesn’t yet file a bug to upload the maven files, but I’m working on that.

During the process of setting this up, I realized that my organization is a little backwards. In particular, I’m pushing all the artifacts from my local system. Instead, I should merely be committing everything to the source code control repository; tagging a release; and then having the further downstream artifacts like the zip and tar.gz files and documentation pulled from source code control onto the Web servers.

There are some commercial products that are organized like this, including ThoughtWorks’s Cruise, but none of the major open source hosting sites such as SourceForge and java.net work like this. Certainly, SourceForge and similar sites have been major contributors to the open source revolution. They have enabled hobbyist developers working in their garages to use tools and techniques of software development that were previously limited to corporations. They have it enabled far-flung developers around the world to collaborate with each other far more effectively than they could do by e-mailing each other tar files. They have removed the burden of system administration from many programmers, thus enabling them to devote more time to writing code. Make no mistake. SourceForge et al. are real force for good in the community.

That said, the state of the art in software development has moved forward significantly since these sites were founded. CVS has mostly been replaced by Subversion. On some projects, Subversion has been been replaced by distributed version control systems such as git and Mercurial. Unit testing and test driven development have moved from extreme practices to standard operating procedure. Continuous integration using products like Hudson and Cruise Control is routine. Nonetheless, most project hosting sites still offer little beyond a source code repository, a bug tracker, and some webspace. Not that that’s not important, but we can do so much more.

It’s time to think about what a modern project hosting site might want to offer and what it might look like.

Continuous Integration

The first step forward, and possibly the hardest, is to add continuous integration capabilities to the existing project hosting repositories. Every time code is checked into SourceForge or Java.net or code.google.com, the project should be built and the tests should be run. Technically, the hard part is understanding every project’s unique build infrastructure. Some projects use ant; some use make; some use maven; and some roll their own. Maven is probably the most constrained of a lot. For the others, it will be necessary to ask project owners which targets to run for which tasks. It’s probably a good idea to auto generate basic ant or maven or make scripts for new projects.

Beyond merely building the code, running the tests has some very serious security implications. Currently project hosting sites do not run third-party code. They store it; they display it; they make it available and bundle it up as tar and zip files; but they don’t even compile it, much less run it. Running arbitrary third-party Java and C code submitted by any random teenager with attention deficit disorder somewhere on the Internet is begging for trouble. 10 years ago I would’ve thought this was insane and impossible. But now, just maybe we can do it.

In fact, there are several services on the Internet today that will run arbitrary third-party code for all comers. Amazon’s EC2 service lets anybody with a credit card run what amounts to a complete rooted Linux box on Amazon’s network. Google’s AppEngine let’s more or less anyone, credit card or no, run Python and Java code inside Google’s cloud. And these are hardly the only such services. Advances in virtualization and security sandboxing have made this possible. That said, it certainly helps to have a real user attached to any code that you run so you know who to blame when it starts spamming the world. However when an application goes rogue whether through malice or incompetence, it is possible to shut it down quickly. It is possible to limit the resources used by anyone test suite, and to limit what else I can see on the same filesystem and the same network.

Codehaus uses Atalassian Bamboo to provide continuous integration, including test running, for their projects. However they’re a relatively small site that’s somewhat picky about the projects they host. They do use a separate server for the continuous integration. I’m not sure what other, if any, security precautions they put in place. Launchpad builds Ubuntu packages, but I’m not sure of they run tests. JavaForge builds Java code and runs the unit tests, apparently on top of Amazon EC2. Assembla will build and run tests, and also uses Amazon EC2. Both of these thereby delegate some of the security issues to Amazon’s virtualized systems.

submit queue

Once we’ve solved the problem of running continuous integration servers on project hosting sites, the next step is to flip them around. The usual process is to commit code to the repository and have the continuous integration server pull the code out of the repository. Then, if the build or tests fail, the continuous integration server goes into red mode and sends out alerts. Wouldn’t it be better if the server never turned red in the first place?

What should happen is that new code gets sent directly to the continuous integration server rather than to the source code repository. The continuous integration server pulls the latest known good build from the repository. then it patches the new code into the build and runs the tests. If the tests pass, the continuous integration server commits the code to the repository. If the tests fail, the code is never committed at all.

So far as I know, no current project hosting sites offer this; and it’s a relatively uncommon feature even among self hosted projects. However, it’s a critical one, especially when accepting contributions from the wide world of programmers, not all of whom have yet learned the importance of test driven development. I suppose such a site could also perform other checks on the source code. For example, it could verify coding conventions or measure the incremental code coverage before and after the check-in. It could automatically reject any patches that did not meet some predetermined measures of quality. That said, automated checks tend to be better used as additional data for humans to evaluate rather than as hard and fast rules. One way this can happen is by offering code metrics to code reviewers. This brings us to the next improvement in the code hosting ecosystem.

Code Reviews

Committing code, even assuming all the tests pass, is still a serious operation. Most open source projects don’t want to allow just anyone to commit code willy-nilly. Usually there’s a core group of committers that reviews all incoming patches and decides whether or not to accept them, to reject them, or to send them back for further work. This is somewhat labor-intensive both on the reviewer and the reviewee.

However, if we move to a submit queue-based system, this can become somewhat more straightforward. The continuous integration server can check every incoming patch regardless of the submitter’s status. If the tests pass, it can send an automatic request for review to a project commiter. If the commiter approves the change, then the continuous integration server can commit it to the source code control repository.

Indeed, it’s probably a good idea to require code reviews for all submitted changes, not just those from new users. After all, it’s not like the project’s owners are immune from introducing bugs. In fact, they probably introduce more than anybody else, if for no other reason than that they commit more code than anyone else. Code reviews are well known for increasing the quality of a code base and avoiding stupid errors, yet they’re one of the lesser used software development practices among open-source programmers. It’s time for that to change. Web-based code review interfaces such as Guido von Rossum’s Rietveld have the potential to really move the community forward here. We should integrate this technology or something equivalent into project hosting sites. code.google.com already offers code review, and a few others like BitBucket do too. The rest should follow.

One-button deployment

The final stage of software development is deployment. Eventually the software has to ship to and be installed by its intended users. Here is one area where open source projects have a significantly easier time than a lot of commercial projects, especially enterprise projects. The deployment process for many open source projects consists of little more than uploading a few jar files and some documentation to the right directories on the right Web servers. This should become a one-button operation.

All of project owners should have to do to release a new version is choose a version number and push a button. The server should pull all the code, documentation, and configuration information out of the source code repository; build everything; and put all the finished artifacts in the right locations. No further manual work should be required. This does require that absolutely everything needed to release goes into the repository; not only code but also HTML files, images, config files, and more. The only things that don’t go into the repository are the artifacts that are built from these components: jar files, zip files, Javadoc, etc.

Maven comes close to this, but it still builds and deploys from a local system rather than from the version control repository. This should be turned around. Ideally, maven deploy should work with nothing more than a pom.xml file. Deploying shouldn’t need to access the local maven repository or the local copy of the source code at all.

Summary

There might even be a startup idea in here somewhere. Open source projects aren’t the only ones that would like to offload some of the routine system administration tasks involved in running source code control repositories, continuous integration servers, bug trackers, and deployment pipelines. More likely, what’s really needed are some tweaks in and additions to the existing project hosting services. Or perhaps we can even take advantage of the advances in virtualization technology to install these services on top of Amazon EC2 and similar platforms.

But one thing is for certain: if open source projects are to keep pace with and surpass closed systems, then their software development practices need to be at least as good and probably better than the state-of-the-art in the overall software development community. In order to do that, it’s time to upgrade our tools.