Technical on NewsGator Online

Pure JavaScript HTML Parser

John Resig — Mon, 05 May 2008 08:50:45 GMT

Recently I was having a little bit of fun and decided to go about writing a pure JavaScript HTML parser. Some might remember my one project, env.js, which ported the native browser JavaScript features to the server-side (powered by Rhino). One thing that was lacking from that project was an HTML parser (it parsed strict XML only).

I've been toying with the ability to port env.js to other platforms (Spidermonkey derivatives and the ECMAScript 4 Reference Implementation) and if I were to do so I would need an HTML parser. Because of this fact it became easiest to just write an HTML parser in pure JavaScript.

I did some digging to see what people had previously built, but the landscape was pretty bleak. The only one that I could find was one made by Erik Arvidsson - a simple SAX-style HTML parser. Considering that this contained only the most basic parsing - and none of the actual, complicated, HTML logic there was still a lot of work left to be done.

(I also contemplated porting the HTML 5 parser, wholesale, but that seemed like a herculean effort.)

However, the result is one that I'm quite pleased with. It won't match the compliance of html5lib, nor the speed of a pure XML parser, but it's able to get the job done with little fuss - while still being highly portable.

htmlparser.js:

4 Libraries in One!

There were four pieces of functionality that I wanted to implement with this library:

A SAX-style API

Handles tag, text, and comments with callbacks. For example, let's say you wanted to implement a simple HTML to XML serialization scheme - you could do so using the following:

var results = "";

HTMLParser("<p id=test>hello <i>world", {
start: function( tag, attrs, unary ) {
results += "<" + tag;

for ( var i = 0; i < attrs.length; i++ )
results += " " + attrs[i].name + '="' + attrs[i].escaped + '"';

results += (unary ? "/" : "") + ">";
},
end: function( tag ) {
results += "</" + tag + ">";
},
chars: function( text ) {
results += text;
},
comment: function( text ) {
results += "";
}
});

results == '<p id="test">hello <i>world</i></p>"

XML Serializer

Now, there's no need to worry about implementing the above, since it's included directly in the library, as well. Just feed in HTML and it spits back an XML string.

var results = HTMLtoXML("<p>Data: <input disabled>")
results == "<p>Data: <input disabled="disabled"/></p>"

DOM Builder

If you're using the HTML parser to inject into an existing DOM document (or within an existing DOM element) then htmlparser.js provides a simple method for handling that:

// The following is appended into the document body
HTMLtoDOM("<p>Hello <b>World", document)

// The follow is appended into the specified element
HTMLtoDOM("<p>Hello <b>World", document.getElementById("test"))

DOM Document Creator

This is a more-advanced version of the DOM builder - it includes logic for handling the overall structure of a web page, returning a new DOM document.

A couple points are enforced by this method:

There will always be a html, head, body, and title element.
There will only be one html, head, body, and title element (if the user specifies more, then will be moved to the appropriate locations and merged).
link and base elements are forced into the head.

You would use the method like so:

var dom = HTMLtoDOM("<p>Data: <input disabled>");
dom.getElementsByTagName("body").length == 1
dom.getElementsByTagName("p").length == 1

While this library doesn't cover the full gamut of possible weirdness that HTML provides, it does handle a lot of the most obvious stuff. All of the following are accounted for:

Unclosed Tags:

HTMLtoXML("<p><b>Hello") == '<p><b>Hello</b></p>'
Empty Elements:

HTMLtoXML("<img src=test.jpg>") == '<img src="test.jpg"/>'
Block vs. Inline Elements:

HTMLtoXML("<b>Hello <p>John") == '<b>Hello </b><p>John</p>'
Self-closing Elements:

HTMLtoXML("<p>Hello<p>World") == '<p>Hello</p><p>World</p>'
Attributes Without Values:

HTMLtoXML("<input disabled>") == '<input disabled="disabled"/>'

Note: It does not take into account where in the document an element should exist. Right now you can put block elements in a head or th inside a p and it'll happily accept them. It's not entirely clear how the logic should work for those, but it's something that I'm open to exploring.

You can test a lot of this out in the live demo.

While I doubt this will cover all weird HTML cases - it should handle most of the obvious ones - at least making HTML parsing in JavaScript feasible.

FancyUpload: Swiff meets Ajax

Dion Almaer — Mon, 28 Apr 2008 13:37:17 GMT

Harald Kirschner has created a new version of FancyUpload "a file-input replacement which features an unobtrusive, multiple-file selection menu and queued upload with an animated progress bar."

A good example is the Queued Photo Uploader which is coded by:

PLAIN TEXT

JAVASCRIPT:

var swiffy = new FancyUpload2($('demo-status'), $('demo-list'), {
'url': $('form-demo').action,
'fieldName': 'photoupload',
'path': '../../source/Swiff.Uploader.swf',
'onLoad': function() {
$('demo-status').removeClass('hide');
$('demo-fallback').destroy();
}
});
/**
* Various interactions
*/
$('demo-browse-all').addEvent('click', function() {
swiffy.browse();
return false;
});
$('demo-browse-images').addEvent('click', function() {
swiffy.browse({'Images (*.jpg, *.jpeg, *.gif, *.png)': '*.jpg; *.jpeg; *.gif; *.png'});
return false;
});
$('demo-clear').addEvent('click', function() {
swiffy.removeFile();
return false;
});
$('demo-upload').addEvent('click', function() {
swiffy.upload();
return false;
});

Aptana Cloud: Develop on your desktop, sync out to the cloud

Dion Almaer — Mon, 28 Apr 2008 19:41:13 GMT

Aptana have announced their cloud platform initiative, Aptana Cloud.

Aptana Cloud plugs right into your IDE to provide instant deployment, smart synchronization, and seamless migration as you scale. Aptana Cloud is ideal for developers who use scripting languages to create Ajax, Facebook, mySpace and all other sorts of Web applications.

The key is that this isn’t a infrastructure play, which they clearly point out:

Aptana Cloud is architected to complement Cloud infrastructure providers like Amazon, Google, Joyent and others. To get started we’ve selected Joyent who serves up some of the largest of all Facebook apps.

This shows that their platform is designed to go meta, allowing you to deploy to various clouds in the future.

I think that the number one meme from Web 2.0 Expo last week was the “cloud”, coming off of the excitement of Google App Engine. With Aptana Cloud we will see sophisticated tools to make us productive in the cloud. I am very excited to see that it won’t be too long until developers will be able to build an application, hit DEPLOY, and be done. This is a huge win.

For developers:

IDE plug-in integrates Cloud development, deployment and management life-cycles right into Aptana Studio in either its standalone or Eclipse based editions
Instant deployment of projects to Cloud
One click sync your project to the Cloud, or provide fine grained sync control too
Integrated service management consoles
Configure desired memory size and disk size
Develop and instantly preview remote files right inside your Studio desktop environment
Subversion source control.

As Ajax developers, the vision of Jaxer in the cloud is very interesting too. The entire application using JavaScript, and deployed up into the cloud, all through the nice IDE.

I was also pleased to read that not only Ruby on Rails, but Python is on the docket. After developing Django applications and playing with Google App Engine, I would love to be able to use Studio for Python code too. Not that Emacs (X or GNU) isn’t great, Steve!

Darryl Taft wrote:

Aptana adds extra value via IDE integration, deployment automation and active monitoring and notification services, Hakman said. “It’s like the ease and simplicity between iTunes on your desktop and its connectivity to services on the Web,” he said.

For developers, the IDE plug-in integrates cloud development, deployment and management lifecycles right into Aptana Studio in either its standalone or Eclipse-based editions, Hakman said. “The ability to deploy stuff to the cloud from Eclipse is part of this as well.”

Other developer features include instant deployment of projects to the cloud; one click can sync your project to the cloud or provide fine-grained sync control; the technology features integrated cloud services management, enables users to provision their cloud right from Aptana Studio, configure desired memory size and disk size, develop and instantly preview remote files right inside the Studio desktop environment, and includes Subversion Source Control.

Can’t wait go get an invite. If you want one too, request an account.

Also, Aptana Studio just passed 1,5 million downloads. Impressive.

Ajax Accessibility

John Resig — Tue, 29 Apr 2008 00:35:35 GMT

A common concern of most Ajax applications has been around their resulting accessibility. While, arguably, it's possible to design some form of a usable web page without the use of JavaScript it should be possible - with the additional scripting information - to provide a better experience to users. It's at this point that the ARIA specification comes into play. A large set of interaction is defined within it which is able to help web applications communicate directly to a screen reader in an effective manner.

To get a feel for what this interaction looks like, take the example of ARIA Live Regions (more info). With this functionality it would be possible to keep a live-updated list of users and allow the screen reader to keep up-to-date.

Observe the following ARIA-marked-up HTML:

<b>Active Users:</b>
<p id="users-desc">A list of the currently-connected users.</p>
<ol aria-live="polite" aria-relevant="additions removals"
aria-describedby="users-desc" id="users">
<li>John</li>
<li>Mary</li>
<li>Ted</li>
<li>Jane</li>
</ol>

A couple settings are used to make this piece of HTML particularly interactive to the screen reader (the actual JavaScript that will update this list is left out - but needless to say nothing, in particular, is needed beyond simple DOM insertion and removal).

aria-live="polite" How polite the live area is (as in, how likely is it to butt in to what the user is currently listening to/interacting with). The default is 'polite' - in that it waits until all forms of user interaction have been completed before describing the updates to the user.
aria-relevant="additions removals" Only notify the user about new node additions and removals. Since we wish to provide the user with a live list of users we can expect them to be both transitioning online and offline - this will give us the appropriate level of updates to make this possible.
aria-describedby="users-desc" A pointer to the element that describes the contents of the live area. If the user
wishes to know more about what the contents of the field represent this element can be read to them.

What's most important about all of this, though, is that ARIA isn't just a pipe dream of functionality: It's implemented, today, in Firefox 2 and even more-so in the upcoming Firefox 3.

The Google Reader team recently took advantage of this and added full ARIA support to their application. It's safe to say that Google Reader is not a trivial application by any stretch, allowing this to demonstrate the feasibility of ARIA within large-scale web application projects.

In the course of their implementation they built a tool, AxsJAX, which injects ARIA usability enhancements into many pages using a bookmarklet, greasemonkey, or Fire Vox (a Firefox screenreader). They started by seeding a number of Google applications with this drop-in accessibility support (along with a few others, including the XKCD web comic).

I continue to be impressed with what can be accomplished with ARIA - and seeing the work that Google has been putting forth by implementing this functionality in their applications has been incredibly enlightening and encouraging.

One-line batch script to delete empty directories

oldnewthing — Thu, 17 Apr 2008 14:00:00 GMT

You don't need a whole 260KB program to do it. This batch file does the trick just fine:

for /f "usebackq" %%d in (`"dir /ad/b/s | sort /R"`) do rd "%%d"

I call it rdempty.cmd.

This is the long-forgotten follow-up to Performing an operation in each subdirectory of a directory tree from batch. We're using the same technique as in that article, but pumping the result through "| sort /R" to reverse the order of the enumeration so we enumerate the directories bottom-up rather than top-down. This is important for deleting empty directories because you have to remove the subdirectories before you remove the parent.

Disclaimer: I doubt anybody actually enjoys working with batch files, but that doesn't mean that tips on using it more effectively aren't valid. If you would rather gouge your eyes out than use the confusing command prompt batch language, then you are more than welcome to use the scripting language of your choice instead. At no point in this article am I saying that this is the only way or the best way to do it. But it's definitely smaller than a 260KB program.

Classy Query

John Resig — Tue, 01 Apr 2008 03:26:18 GMT

Over the past two years, seeing hundreds of thousands of people use jQuery, a major point has become apparent: jQuery simply isn't able to scale to handle the development needs of most power users. It lacks the clarity and power functionality that most developers need in order build applications in a collaborative environment.

For this reason I've put a lot of work into a new library (which sits on top of jQuery), called:

Classy Query

» View Source to Classy Query

If one thing has become apparent to me it's that users enjoy working with the typical Class-style of object creation and inheritance. Because of this I've constructed the entirety of Classy Query in an classical manner (you can inherit and override any piece of functionality that you desire).

Before we go too far, let's take a look at the type of clearly-defined code that you can now write with Classy Query:

You can even sub-class functionality, creating your own pieces of encapsulation. All of the class and inheritance functionality is taken care of by the Simple JavaScript Inheritance code released last week.

jQuery.DOM.boldWrapInner = jQuery.DOM.wrapInner.extend({
attach: function(elem){
this._super(elem, "<b></b>");
}
});

jQuery.querySelectorAll("div").forEach(function(elem){
jQuery.DOM.boldWrapInner(elem);
});

Additionally you can treat pieces of functionality as if they were attachable behaviors, like so:

jQuery.querySelectorAll("div").attach(new jQuery.DOM.boldWrapInner());

There's a number of core features encompassed in Classy Query that really help to transform the traditional jQuery experience into something much more usable.

Class creation

The full Simple JavaScript Inheritance API is included in Classy Query, allowing you to extend and create JavaScript 'classes', like so:

var Person = jQuery.Class.create({
init: function(name){
this.name = name;
}
});

var ClassyDude = Person.extend({
sipWine: function(){
return "Has a beautiful bouquet.";
}
});

Inheritance

It's possible to inherit from any existing piece of Classy Query functionality, supporting a true 'programming in the large' mindset. You can inherit from, or completely override, existing methods with ease (especially since they're all classes).

jQuery.Effects.collapse = jQuery.Effects.buildAnimation.extend({
attach: function(elem){
this._super(elem, {height: "hide", width: "hide"});
}
});

Behaviors

Behaviors tend to encapsulate a portion of functionality which is applied against an element (such as making a table sortable or making an item draggable). Classy Query supports a technique to make this particular implementation trivial.

jQuery.querySelectorAll("div")
.attach(new jQuery.DOM.addClassName("current"));

No more 'this'

The 'this' keyword is confusing and frequently misused. To counter this all access to elements (within functions) is done by the first argument, rather than through 'this'.

jQuery.querySelectorAll("div").forEach(function(elem){
jQuery.DOM.addClassName(elem, "current");
});

Re-Structuring

The hierarchy of jQuery has been completely re-organized. Rather than having a single, flat, interface through which to access all methods functionality has been broken down into individual groupings of methods. These groupings tend to, also, coordinate with the jQuery Documentation for convenience.

Additionally, nearly all the method names have been re-named in order to provide a greater level of clarity to developers. Frequently it was found that the concise method names of jQuery provided too much confusion to developers just getting started with the library. The end result is a library that is easier to read and understand for developers.

The full list of categories and changed methods can be found in the following table:

jQuery	Classy Query
jQuery	jQuery.querySelectorAll
each	forEach
prepend	jQuery.DOM.prepend
append	jQuery.DOM.append
before	jQuery.DOM.insertBefore
after	jQuery.DOM.insertAfter
wrap	jQuery.DOM.wrap
wrapInner	jQuery.DOM.wrapInner
wrapAll	jQuery.DOM.wrapAll
clone	jQuery.DOM.clone
empty	jQuery.DOM.empty
remove	jQuery.DOM.remove
replaceWith	jQuery.DOM.replaceWith
removeAttr	jQuery.DOM.removeAttribute
addClass	jQuery.DOM.addClassName
hasClass	jQuery.DOM.hasClassName
removeClass	jQuery.DOM.removeClassName
offset	jQuery.DOM.getOffset
text	jQuery.DOM.getText
text	jQuery.DOM.setText
html	jQuery.DOM.getHTML
html	jQuery.DOM.setHTML
attr	jQuery.DOM.getAttribute
attr	jQuery.DOM.setAttribute
val	jQuery.DOM.getValue
val	jQuery.DOM.setValue
height	jQuery.DOM.getHeight
height	jQuery.DOM.setHeight
width	jQuery.DOM.getWidth
width	jQuery.DOM.setWidth
css	jQuery.DOM.getCSS
css	jQuery.DOM.setCSS
children	jQuery.Traverse.getChildElements
find	jQuery.Traverse.getDescendantElements
next	jQuery.Traverse.getNextSiblingElements
nextAll	jQuery.Traverse.getAllNextSiblingElements
parent	jQuery.Traverse.getParentElements
parents	jQuery.Traverse.getAncestorElements
prev	jQuery.Traverse.getPreviousSiblingElements
prevAll	jQuery.Traverse.getAllPreviousSiblingElements
siblings	jQuery.Traverse.getSiblingElements
filter	jQuery.Traverse.filterSelector
bind	jQuery.Events.addEventListener
unbind	jQuery.Events.removeEventListener
trigger	jQuery.Events.triggerEvent
hover	jQuery.Events.hover
toggle	jQuery.Events.toggle
show	jQuery.Effects.show
hide	jQuery.Effects.hide
toggle	jQuery.Effects.toggle
animate	jQuery.Effects.buildAnimation
queue	jQuery.Effects.queue
dequeue	jQuery.Effects.dequeue
$.ajax	jQuery.Ajax.request
load	jQuery.Ajax.loadAndInsert
ajaxSetup	jQuery.Ajax.setup
serialize	jQuery.Ajax.getSerializedString
serializeArray	jQuery.Ajax.getSerializedArray

querySelector and querySelectorAll

Finally, rather than having a generic, catch-all, method as jQuery selector functionality is deferred to the new querySelector and querySelectorAll methods. These methods only support pure CSS 1-3 selectors (nothing extra) in an attempt to avoid any unpleasant situations when a browser's native querySelectorAll is implemented.

jQuery.querySelectorAll("div").forEach(function(elem){
jQuery.Effect.hide( elem );
});

jQuery.Effect.hide( jQuery.querySelector("#item") )

» View Source to Classy Query

I hope this particular library will be of use to the general jQuery-using population. I hope we can start to have an open dialog, moving forward, getting everyone to adopt a more standard approach to JavaScript development. There's no reason why we shouldn't be using this tried-and-true approach to application development. It's worked in an untold number of existing applications, it's bound to work here as well. Enjoy.

JavaScript running to the server

Dion Almaer — Tue, 10 Jul 2007 14:24:33 GMT

JavaScript continues to get more talk on the server (yes, years after Netscape Enterprise Server!).

John Resig spent a weekend hacking in the zone, to bring the browser to the server.

This resulting environment allows you to do interesting things such as:

Screen scraping

PLAIN TEXT

JAVASCRIPT:

load("env.js");
window.location = "http://alistapart.com/";
window.onload = function(){
load("dist/jquery.js");
print("Newest A List Apart Posts:");
$("h4.title").each(function(){
print(" - " + this.textContent);
});
};

Web Framework

PLAIN TEXT

JAVASCRIPT:

window.onload = function(){
print("Content-type: text/html\n");
if ( location.href == "/" )
show_home();
print( document.innerHTML );
};
function show_home(){
document.load("index.html");
document.getElementById("time").innerHTML = (new Date()).toString();
}

Tom Insam and Ash Berlin also created mod_js, which is a very simple JS embed. Others pointed out mod_wxjs, haxe, and the like.

The beat goes on.

A JavaScript Module Pattern

Dion Almaer — Wed, 13 Jun 2007 11:01:07 GMT

Eric Miraglia, of Yahoo!, has documented his explanation of what Douglas Crockford calls the JavaScript Module pattern.

Eric discusses the steps:

Create a namespace object: If you’re using YUI, you can use the YAHOO.namespace() method
Assign the return value of an anonymous function to your namespace object
Add “private” methods and variables in the anonymous function prior to the return statement
Do something useful with the pattern

PLAIN TEXT

JAVASCRIPT:

YAHOO.myProject.myModule = function () {
//"private" variables:
var myPrivateVar = "I can be accessed only from within YAHOO.myProject.myModule."
//"private" method:
var myPrivateMethod = function () {
YAHOO.log("I can be accessed only from within YAHOO.myProject.myModule");
}
return {
myPublicProperty: "I'm accessible as YAHOO.myProject.myModule.myPublicProperty."
myPublicMethod: function () {
YAHOO.log("I'm accessible as YAHOO.myProject.myModule.myPublicMethod.");
//Within myProject, I can access "private" vars and methods:
YAHOO.log(myPrivateVar);
YAHOO.log(myPrivateMethod());
//The native scope of myPublicMethod is myProject; we can
//access public members using "this":
YAHOO.log(this.myPublicProperty);
}
};
}(); // the parens here cause the anonymous function to execute and return

Note that this technique is very generic, and you can do the same thing with other frameworks, or without any frameworks!

Javascript LRU Cache

Dion Almaer — Wed, 30 May 2007 12:49:11 GMT

Monsur Hossain has created a simple Javascript LRU Cache:

It is loosely based on ASP.NET's Cache, and includes many caching options such as absolute expiration, sliding expiration, cache priority, and a callback function. It can be used to cache data locally in the user's browser, saving a server roundtrip in AJAX heavy applications.

Usage

PLAIN TEXT

JAVASCRIPT:

var cache = new Cache();
cache.setItem("A", "1", {expirationAbsolute: null,
expirationSliding: 60,
priority: CachePriority.High,
callback: function(k, v) { alert('removed ' + k); }
});
cache.getItem("A");
cache.clear();

Implementing a syntax-higlighting JavaScript editor in JavaScript

Dion Almaer — Fri, 25 May 2007 17:40:30 GMT

Marijn Haverbeke has not only been able to create a syntax-highlighting JavaScript editor in JavaScript, but he has written up the story of the brutal odyssey to the dark side of the DOM tree.

The story is a great romp through the never-never-land of JavaScript, DOM, and browser "quirks". It is well worth a read to find out about fun with selection, designMode, parsing, and much much more.

A huge kudos to Marijn.

(via Dean)

Dtrace Firefox to get inside the black box

Dion Almaer — Tue, 15 May 2007 13:26:25 GMT

John Rice gave a presentation at JavaOne on using Dtrace to instrument Firefox.

The JavaScript VM has long been a black box. We long for monitoring tools that give us access to the runtime as we are used to on the JVM or CLR. We may get there with Silverlight in the future.

Adding Dtrace to Mozilla is far from new, and although it isn't an easy tweak, enables you to look deep into the innards of the JavaScript engine.

___ OVERLAP TIMES: ___ ______ ELAPSED _____
FILE FUNCTION COUNT AVG(us) SUM(us)
clock.html getSeconds 12 8 104
clock.html padZero 36 7 275
clock.html getMinutes 12 45 548
clock.html getElementById 12 53 640
clock.html getHours 12 59 714
clock.html setTimeout 12 67 807
clock.html startTime 12 1219 14637

jscsscomp: JavaScript and CSS files compressor

Dion Almaer — Thu, 26 Apr 2007 12:58:54 GMT

We love to play with the plumbing don't we. jscsscomp is the latest compressor that uses Nicolas Martin PHP version of the Dean Edwards JavaScript Compressor.

With a swish of mod_rewrite:

RewriteEngine on

RewriteCond %{REQUEST_FILENAME} -f
RewriteRule ^(.*\.)(js|css)$ jscsscomp/jscsscomp.php?q=$1$2 [L,NC]

you can get your JavaScript like this:

PLAIN TEXT

HTML:

<script src = "/jscsscomp/yahoo.js, dom.js, event.js, effects/dragdrop.js, slider.js"></script>

Digital Web Magazine - Seven JavaScript Techniques You Should Be Using Today

gleddy — Tue, 24 Apr 2007 11:30:00 GMT

JSLT: A JavaScript alternative to XSLT

Dion Almaer — Wed, 11 Apr 2007 11:34:48 GMT

Rik Arends has created JSLT, a pure JavaScript replacement for XSLT.

JSLT is a browser based templating language like XSLT, but instead of
using XML to encode the template logic, it uses normal Javascript with a
few extensions. You can transform XML with it or just template with
javascript variables. The JSLT processor parses the template using
a recursive tokenizing parser and generates javascript code for fast
dynamic (re-)execution. The general structure is:
template[js]template{inline-xpath}[js]template. All code is available
under the LGPL license.

You can visit a real-time example page that recompiles on each keystroke.

If you almost liked XSLT, and like JavaScript, maybe JSLT is for you.

ZParse: Self-definable Javascript Templating

Dion Almaer — Mon, 09 Apr 2007 10:40:46 GMT

ZParse is a multi-purpose, extensible, self-defineable Javascript template parsing system created by Rizqi Ahmad.

What does this mean? ZParse lets you define your templating rules. It acts like a preprocessor that parse strings that contain defined delimiters and tags and parse it to a javascript function. It is similar to TrimPath JavaScript Template, but differs in that instead of defining the syntax inside the script it allows you to create and define the syntax yourself as you need.

PLAIN TEXT

HTML:

<% var name = "Solid Snake", address = "somewhere"; %>
<div id="card">
<:if name && address:>
<div>name: ${name} </div>
<div>address: ${address} </div>
<:/if:>
<:elseif name:>
<div>name: ${name} </div>
<:/elseif:>
<:else:>
no card loaded.
<:/else:>
</div>

If you wish you had yacc or bison in your next project, give it a whirl!

Operating System Configuration Setting for SQL Server 2005

Thu, 04 Jan 2007 16:23:53 GMT

One operating system level configuration setting that most DBAs and SysAdmins overlook is the Server Optimization tab under File and Printer Sharing for Microsoft Networks Properties (shown below).

SQL Server 2005 needs high network throughput with minimal memory devoted to file caching so that more memory is available to SQL Server 2005. The default setting when you install Windows Server 2003 Standard or above is "Maximize data throughput for file sharing". For both SQL Server 2005 and for IIS 6.0 (a database server and web server, respectively), you should change this to "Maximize data throughput for network applications" (like you see below).

Technorati Tag: SQL Server

It's Never Been Built Before

Fri, 17 Nov 2006 07:59:59 GMT

In Microsoft Project and the Gantt Waterfall, many commenters wondered why software projects can't be treated like any other construction or engineering project:

I am not sure why it is so difficult to estimate software development? Is it a mystery, magic, is there a man behind the curtain that every project depends on?
I mean it's simple, come on! It's like a designer and general contractor coming over and estimating how long it will take to remodel your kitchen.

But software projects truly aren't like other engineering projects. I don't say this out of a sense of entitlement, or out of some misguided attempt to obtain special treatment for software developers. I say it because the only kind of software we ever build is unproven, experimental software. Sam Guckenheimer explains:

To overcome the gap, you must recognize that software engineering is not like other engineering. When you build a bridge, road, or house, for example, you can safely study hundreds of very similar examples. Indeed, most of the time, economics dictate that you build the current one almost exactly like the last to take the risk out of the project.
With software, if someone has built a system just like you need, or close to what you need, then chances are you can license it commercially (or even find it as freeware). No sane business is going to spend money on building software that it can buy more economically. With thousands of software products available for commercial license, it is almost always cheaper to buy. Because the decision to build software must be based on sound return on investment and risk analysis, the software projects that get built will almost invariably be those that are not available commercially.
This business context has a profound effect on the nature of software projects. It means that software projects that are easy and low risk, because theyve been done before, dont get funded. The only new software development projects undertaken are those that havent been done before or those whose predecessors are not publicly available. This business reality, more than any other factor, is what makes software development so hard and risky, which makes attention to process so important.

One kitchen remodelling project is much like another, and the next airplane you build will be nearly identical to the last five airplanes you built. Sure, there are some variables, some tweaks to the process over time, but it's a glorified factory assembly line. In software development, if you're repeating the same project over and over, you won't have a job for very long. At least not on this continent. If all you need is a stock airplane, you buy one. We're paid to build high-risk, experimental airplanes.

The appropriately named happy-go-lucky left this comment on the same post which explains the distinction quite succinctly:

The problem originates in the fact that "Software Project" shares the word "project" with "engineering project" or "construction project."
"Ah-ha!" our hapless software managers cry, as they roll-out the money for a copy of MS Project. And the software developers cry all the way 'til the end of the "project."
Software invention is just that. It has more in common with Mr. Edison's Menlo Park laboratory than the local hi-rise construction site.
Of course, there are far fewer variables on a work site than in software invention. Gravity remains the same, elements don't change and concrete has known properties. Things aren't that stable when operating systems, hardware, software and business processes come together in "the perfect speculation."

In software engineering projects, you aren't subject to God's rules. You get to play God: define, build, and control an entire universe in a box. It's a process of invention and evolution, not an assembly line. Can software development be managed? Sure. But it's a mistake to try managing it the same way you'd manage a kitchen remodel.

The Last Responsible Moment

Wed, 18 Oct 2006 07:59:59 GMT

In Lean Software Development: An Agile Toolkit, Mary and Tom Poppendieck describe a counter-intuitive technique for making better decisions:

Concurrent software development means starting development when only partial requirements are known and developing in short iterations that provide the feedback that causes the system to emerge. Concurrent development makes it possible to delay commitment until the last responsible moment, that is, the moment at which failing to make a decision eliminates an important alternative. If commitments are delayed beyond the last responsible moment, then decisions are made by default, which is generally not a good approach to making decisions.

Paradoxically, it's possible to make better decisions by not deciding. I'm a world class procrastinator, so what's to stop me from reading this as carte blanche? Why do today what I can put off until tomorrow?

Making decisions at the Last Responsible Moment isn't procrastination; it's inspired laziness. It's a solid, fundamental risk avoidance strategy. Decisions made too early in a project are hugely risky. Early decisions often result in work that has to be thrown away. Even worse, those early decisions can have crippling and unavoidable consequences for the entire future of the project.

Early in a project, you should make as few binding decisions as you can get away with. This doesn't mean you stop working, of course-- you adapt to the highly variable nature of software development. Often, having the guts to say "I don't know" is your best decision. Immediately followed by "..but we're working on it."

Jeremy Miller participated in a TDD panel with Mary Poppendieck last year, and he logically connects the dots between the Last Responsible Moment and YAGNI:

The key is to make decisions as late as you can responsibly wait because that is the point at which you have the most information on which to base the decision. In software design it means you forgo creating generalized solutions or class structures until you know that theyre justified or necessary.

I think there's a natural human tendency to build or buy things in anticipation of future needs, however unlikely. Isn't that the Boy Scout motto-- Be Prepared?

As a former Scout myself, I think we should resist our natural tendency to prepare too far in advance. My workshop is chock full of unused tools I thought I might need. Why do I have this air compressor? When was the last time I used my wet/dry vac? Have I ever used that metric socket set? It's a complete waste of money and garage space. Plus all the time I spent agonizing over the selection of these tools I don't use. I've adopted the Last Responsible Moment approach for my workshop. I force myself to only buy tools that I've used before, or tools that I have a very specific need for on a project I'm about to start.

Be prepared. But for tomorrow, not next year. Deciding too late is dangerous, but deciding too early in the rapidly changing world of software development is arguably even more dangerous. Let the principle of Last Responsible Moment be your guide.

The True Cost of .NET Exceptions -- Solution

ricom — Mon, 25 Sep 2006 17:58:41 GMT

Well, once again my elite readers have made a solution posting by me nearly redundant. There were many good viewpoints in response to my article including some especially excellent comments that were spot on.

Now there are several things worth saying, but I think perhaps the most important one, in my opinion, was hit on directly by Ian Griffiths and James Curran.

Ian: "Presumably throwing an exception causes the CPU to fetch code and data it would otherwise not have executed..."

James: "I think what Jon's article is overlooking is that his timing is done in isolation..."

And these are crucial observations that I talked about in a previous article (see the section called "Fear the Interference" at the end). In this particular case we're doing nothing but throwing exceptions. So naturally all the code associated with the throw path is in the cache, all the data for resolving the locations of the catch blocks is in the cache, etc. etc. Basically what we have here is much closer to a measurement of the minimum an exception could possibly cost than a typical cost. Fair enough, minimum cost is useful too, but it is sort of an understatement by definition.

I've mentioned some of the additional costs but let me make a list that's somewhat more complete.

In a normal situation you can expect additional cache misses due to the thrown exception accessing resident data that is not normally in the cache
In a normal situation you can expect additional page faults due to the thrown exception accessing non-resident code and data, not normally in your workingset

noteably throwing the exception will require the CLR to find the location of the finally and catch blocks based on the current IP and the return IP of every frame until the exception is handled plus the filter block (if any, VB can have one)
additional construction cost and name resolution in order to create the frames for diagnostic purposes, including reading of metadata etc.
both of the above items typically access "cold" code and data hence hard page faults are probable if you have memory pressure at all

we try to put code and data that is used infrequently far from data that is used frequently to improve locality, this works against you if you force the cold to be hot.
the cost of the hard page faults, if any, will dwarf everything else

Typical catch situations are significantly deeper than the test case therefore the above effects would tend to be magnified (increasing the likelihood of page faults)
Finally blocks have to be run anyway so their cost isn't really "exceptional" however whatever code is in the filter blocks (if present) and catch block is subject to the same issues as the above

Some people thought the ring transition cost was important, in my opinion it isn't especially and it is at least somewhat accounted for in the test as written (except to the same extent that everything else is cheaper because the whole test fits comfortably in L2). I'm not sure there even is a ring transition for a thrown exception in fact, although some kernel32 code runs, I think everything that needs to be done can be done in user mode. But in any case that cost would be included.

Generally, microbenchmarks are designed to hide certain costs and magnify others. This is not a bad thing, the trick is to make sure you know what you're getting.

Is the metric the one of interest to you?
Is the workload representative?
Is the environment representative?

Just those three quick checks will get you far.

Hello PLINQ

joe@bluebytesoftware.com — Wed, 13 Sep 2006 11:48:33 GMT

LINQ coaxes developers into writing declarative queries that specify what is to be computed instead of how to compute the results. This is in contrast to the lion's share of imperative programs written today, which are huge rat nests of for-loops, switch statements, and function calls. The result of this new direction? Computationally intensive filters, projections, reductions, sorts, and joins can be evaluated in parallel... transparently... with little-to-no extra input from the developer. The more data the better.

If you buy the hypothesis--still unproven--that developers will write large swaths of code using LINQ, then by inference, they will now also be writing large swaths of implicitly data parallel code. This, my friends, is very good for taking advantage of multi-core processors.

If you want to get a little glimpse of what I've been spending my time working on, check out these (brief) stories about Parallel LINQ (aka PLINQ), a parallel query execution engine for LINQ:

Microsoft's PLinq to Speed Program Execution (eWeek)
MS eyes multicore technology (InfoWorld)

We've spent many, many months now cranking out a fully functional prototype. The numbers were impressive enough to catch the eye of some key people around the company. And the rest is history... (well, not quite yet...)

I'll no doubt be disclosing more about this in the coming weeks.

(Note: I am in no way committing to any sort of product or release timeframe. This technology is quite early in the lifecycle, and, while unlikely, might never actually make the light of day... Label this puppy as "research" for now.)