My GitHub repo has the library in the context of a Roxy framework so that I could write some unit tests. Those tests not only give me confidence that the code works, they also show you how to use the code. Here’s an example template I used in a unit test:

<curio:item>
  <curio:type src="1"/>
  <curio:name src="2"/>
  <curio:location>
    <curio:label src="3"/>
  </curio:location>
  <curio:acquired-date>
    <curio:start src="4" ns="http://davidcassel.net/csv" action="reformat-date"/>
    <curio:end src="5" ns="http://davidcassel.net/csv" action="reformat-date"/>
  </curio:acquired-date>
  <curio:description src="6"/>
  <curio:notes src="7"/>
  <curio:owner src="8"/>
</curio:item>

As you can see, a lot of the elements have a “src” attribute. The number corresponds to a field position in the CSV line. If the corresponding value is empty, the element gets skipped. The “ns” and “action” attributes specify a function that you want to run on an element after it has values put in. (At some point, I’ll try to add an “at” attribute so you can specify the library module; right now it’s assumed to be in csv-lib.xqy.) Although my examples have the action attribute on leaf nodes, it can be higher up the chain.

Here’s an example CSV line:

pin,Pilatus,"Pilatus mountain",7/1/1984,7/31/1984,,"a word, and another",dcassel

The library will handle commas embedded in quoted strings. And for completeness, here’s the generated XML:

<curio:item xmlns:curio="http://davidcassel.net/curio">
  <curio:type>pin</curio:type>
  <curio:name>Pilatus</curio:name>
  <curio:location>
    <curio:label>Pilatus mountain</curio:label>
  </curio:location>
  <curio:acquired-date>
    <curio:start>1984-07-01</curio:start>
    <curio:end>1984-07-31</curio:end>
  </curio:acquired-date>
  <curio:notes>a word, and another</curio:notes>
  <curio:owner>dcassel</curio:owner>
</curio:item>

After you’ve downloaded the repo, here are the steps for you to be able to run the unit tests:

1. Check whether the ports I’m using conflict with any app servers you already have set up. You’ll find them in deploy/build.properties (app-port, test-port, xcc-port)
2. ./ml local bootstrap  <– use ml.bat if you’re on Windows
3. ./ml local restart  <– if necessary — output from step 1 will tell you if you need to restart
4. ./ml local deploy modules

Assuming all went well, you should now be able to point your browser to http://localhost:8022/test/ and run the csv-lib unit tests.

My local instance of MarkLogic is configured for 16 task server threads, which I can see by going to Configure -> Groups -> Default -> Task Server in the Admin UI. By clicking on the Status tab here, I’ll be able to watch the tasks get worked off.

Normal Priority

To play with the queue, I’ll start by setting up a simple module to simulate a long running task. I’ll call this write-log.xqy:

xquery version "1.0-ml";

declare variable $priority external;
declare variable $id external;

xdmp:sleep(5000),
xdmp:log(fn:concat("task priority: ", $priority, "; id=", $id))

Sleep for 5 seconds, then write out the parameters. Very simple. Now I’ll fire up Query Console and spawn a bunch of these tasks:

for $i in (1 to 500)
let $priority := "normal"
return
  xdmp:spawn(
    "/write-log.xqy",
    (xs:QName("priority"), $priority,
     xs:QName("id"), $i),
    <options xmlns="xdmp:eval"><priority>{$priority}</priority></options>
  )

I spawn 500 tasks with “normal” priority. After running this, I can refresh the Task Server Status page and see a bunch of tasks in the queue, getting worked off by the 16 threads. Watching the log file, I see 16 of the log messages bunched together as the threads wrap up around the same time; then 5 seconds later, another group of 16.

Higher Priority

The new option in MarkLogic 5 is to choose between “normal” and “higher” priority. Let’s see what happens when they mix. We’ll use the same write-log.xqy as before, but we’ll change what we do in Query Console:

for $i in (1 to 500)
let $priority := if ($i <= 250) then "normal" else "higher"
return
  xdmp:spawn(
    "/write-log.xqy",
    (xs:QName("priority"), $priority,
     xs:QName("id"), $i),
    <options xmlns="xdmp:eval"><priority>{$priority}</priority></options>
  )

We launch 250 normal priority tasks followed by 250 higher priority tasks. Higher priority tasks get a separate queue and their own batch of threads. That means that both sets of tasks can proceed in parallel. We see this when we look at the Task Server Status page and see 32 running threads, and in the log file where we see 32 of the log messages bunched together. Tasks 1-16 and tasks 251-266 complete around the same time; about five seconds later, we see tasks 17-32 and tasks 267-282.

Using Priorities

From the way this works, an application can have most tasks running as normal priority, but when higher priority tasks come along, they’ll get to run without interrupting the normal ones. If you make heavy use of the higher priority tasks, be conscious of the extra threads that will be run on your behalf.

xdmp:document-insert(
  "/test.xml",
  <doc>
    <count/>
  </doc>
)

This triggers an error:

[1.0-ml] XDMP-RANGEINDEX: xdmp:eval(“xdmp:document-insert(&#10; &quot;/test.xml&quot;,&#10; <doc>&#…”, (), <options xmlns=”xdmp:eval”><database>13528717381355350321</database><root>/Users/dcassel/gi…</options>) — Range index error: int fn:doc(“/test.xml”)/doc/count: XDMP-CAST: (err:FORG0001) Invalid cast: xs:untypedAtomic(“”) cast as xs:int

So what’s going on? When you set up a range index on an element, MarkLogic will add any new values to the index. The value of the <count/> element above is “” (empty string), which has no valid interpretation as a number. You’ll encounter the same problem with any non-string element or attribute range index (I see it a lot with dates that don’t match the required format).

The Solution

There are basically two choices if you don’t have an actual value to put in an element. The first is to assume a sensible default. For an element called count, that’s probably zero. The other approach is to skip the element altogether. While that can be simple for the insert, it does make an update a little more complex, as you need to check whether you will insert a element or replace an existing one:

declare function local:update-count($doc, $new-ct)
{
  if (fn:exists($doc/count)) then
    xdmp:node-replace($doc/count, <count>{$new-ct}</count>)
  else
    xdmp:node-insert-child($doc, <count>{$new-ct}</count>)
};

Besides complexity, the other factor to consider is whether a missing value is likely to be provided later. If the value might never be provided (for instance, you have a lot of possible elements, but only a few usually get filled), then you’re better off skipping them — the empty elements don’t server a purpose. If it is likely that an element will get a value, then populating it with a reasonable default keeps your code a little simpler.

Before we go any further, let’s be clear that this is neither documented nor supported in the current version — and if it becomes official, the interface with the feature may very well change. If you are interested in this feature, please contact Stephen Buxton in Product Management and let him know.

With all that out of the way, let’s take a look.

ImageMagick is “a software suite to create, edit, compose or convert bitmap images”, as stated on its web site. If you have ImageMagick installed on your server when MarkLogic 5 starts up, you’ll see something like this line slide by in your ErrorLog.txt:

2012-01-09 17:10:41.557 Info: ImageMagick 6.7.4-4 2012-01-09 Q16 http://www.imagemagick.org

If you don’t have it installed, you’ll still see some reference to ImageMagick, but it will be a message indicating that it was not able to load ImageMagick.

Regular readers will have seen that I’m now running a site with the MarkLogic Express license: CurioVault.com. The site lets users upload information about collectibles they have, including when and where they got the item, an interesting story connected to it, and a picture of it. CurioVault is more about the stories than the pictures. It’s not Flickr — I don’t want the site to get bogged down with a bunch of multi-megabyte images. So, I set up the code so that I can specify a maximum size for an image in bytes, and automatically scale down any uploaded images that exceed that size. I also set up a maximum image dimension (in pixels) to use when scaling. Here’s the code that does it:

declare function item:scale-image($img as binary()) as binary()
{
  if (xdmp:binary-size($img) > $MAX-IMG-SIZE) then
    let $wand := magick:read-image(magick:wand(), $img)
    let $height := magick:get-image-height($wand)
    let $width := magick:get-image-width($wand)
    let $wand :=
      magick:scale-image(
        $wand,
        fn:min((fn:ceiling($width * $MAX-IMG-DIM div $height), $MAX-IMG-DIM)),
        fn:min((fn:ceiling($height * $MAX-IMG-DIM div $width), $MAX-IMG-DIM)))
    return magick:write-image($wand)
  else
    $img
};

The trickery with the 2nd and 3rd parameters to magick:scale-image() is to make sure the aspect ratio remains the same, while setting a maximum for either dimension. But simple as that, I pass in binary image that a user uploaded and when it comes back, I know that it won’t be huge. (People who do a lot of image manipulation may have suggestions for a better approach; such comments are welcome, but don’t let that distract you from celebrating this feature!) Previously, to accomplish this would have required calls out to MLJAM or a service.

This integration is so new that even the internal documentation is mostly references to ImageMagick’s documentation at this point. If you want to play around a bit, you can map the code above to the corresponding functions on ImageMagick’s documentation and probably figure out how to make some other kinds of calls.

Be sure to let Stephen know what you think.

Next up for CurioVault: letting users crop images at upload time.

I’ve collected lapel pins since 1984. In that time, I’ve gotten more than 150 and I’ve started to forget when I got some of them, where I got them, and the stories that went with them. Hence CurioVault.com. Give it a look!

The site runs on a computer I bought for $150 on Amazon Marketplace — a used box with Intel Core 2 Duo, 2GB of RAM (upgradable to 4GB) and a 500 GB hard drive. Not a very hefty machine, but good enough to get a small site online.

I’ve seen many examples of MarkLogic being a great solution for huge data sets. But the productivity advantages apply just as well when dealing with a small site. Great tools help on big jobs and little. Throw in the hope that a small site will one day grow up to be a big one, and I’m excited to be building a site for myself on MarkLogic.

Yes, we’re talking about Model-View-Controller. I’ve had a number of conversations with people about whether MVC makes sense in an XQuery world, and I wanted to share my reasons for thinking it does.

The conversation typically centers on the Model part of MVC. The argument against MVC in XQuery points out that a big part of the Model’s job is to get rid of the impedance mismatch between the way data is stored (perhaps rows and columns in a relational database) and the way it is used (perhaps Java objects). In XQuery, data is both stored and used in XML, so there is no mismatch, ergo a Model is overkill. That’s a valid point as far as it goes — after all, that is one of the benefits of working with XQuery in general and MarkLogic in particular.

When we started with our framework, we took a VC approach — we used Views to separate out the presentation logic, and we used Controllers to handle request inputs. However, we still used library modules to do the work of taking those request inputs and turning them into some data to present. We wanted to isolate that logic so that we could write unit tests. Before long, we recognized that we were, in fact, building Models.

So what role does a model play in XQuery? A simpler one. Once you leave the impedance mismatch problem behind, the benefit you are left with is encapsulation. Many of our applications need functions to register, log in, and log out users, along with updating passwords and identifying the current user. Each of these is a function that we store together in a library module, which we write unit tests for and can easily pull out of one project and put into another. That’s our user model. Likewise, many applications have some central concepts to them that lend themselves to building a model, even though there is no corresponding table, as we would have in the relational world.

Yes, this is a thinner version of how people think of a Model in the relational world. But there are benefits to having divisions between the parts of your application, and MVC is an approach that is familiar and (when used properly) has been useful.

You may have heard about the recent release of MarkLogic 5. We at MarkLogic are excited about lots of great improvements in this release, which have been written about in other places. I’m pretty stoked about the Express License. Express is similar to the old Community License in that it’s available for anyone to use, but the big change is that you’re now allowed to go into production. There are some limits, as you might guess: 40 GB of data and 2 cores.

I found myself wondering how little one could spend and put something out there.

Hosting Options

There are a number of options when it comes to putting your software creation out there. There are basic web hosting (like this site), Virtual Private Hosts, Dedicated Hosts, colocation, using your own data center, and then there’s Amazon’s EC2. Ramping a VPS or above up to a good amount of memory pushes the other options above the cost of an EC2 instance. Even getting FIOS Small Business Internet would start at $70/month. Let’s take a look at what we could do with EC2.

Amazon EC2 Standard Small Instance

The Express License is limited to 2 CPUs, so we’ll be looking at the small instance. If we go with Linux, the cost per hour is $0.085 per hour. There are 24 * 365 = 8760 hours in a year, or 730 hours per month. Amazon will charge you for each hour in which your instance is busy working. Let’s take the worst case and say that we’re running at 100% (after all, we’re hoping people will show and use your site, right?). In that case, Amazon will charge you $0.085/hour * 730 hours = $62.05 per month. Not too shabby.

Amazon EC2 Reserved Small Instance

Amazon also offers reserved instances, where you pay a fixed fee for 1 or 3 years, getting you a reduced hourly rate. If you decide to go with one year of a Reserved Small Instance, you’ll pay a one-time fee of $227.50, plus just $0.03 per hour. Averaging that out, we get ($227/year + $0.03/hour * 8760 hours/year) / 12 months = $40.86 per month.

Okay, that’s slightly above the title $40/month, but “MarkLogic on $40.86 per month” just doesn’t have the same ring to it, does it?

Oh, if you go with the 3-year term, it looks like it’s ($350/3 years + $0.03/hour * 8760 hours/year * 3 years) / 36 months = $31.62 per month, but three years seems like a long commitment for this level of service. Hopefully, you’ll be raking in the revenue and upgrading before that time is up.

Garage Stage

Lots of startups begin with big dreams and small bank accounts. I know, I’ve been there. But I think it’s fair to say that someone who’s looking to start a business can handle $40 per month in exchange for the productivity and performance they’ll get from MarkLogic. (If not, I humbly suggest you may be working on a hobby rather than a business.) If I’d had MarkLogic when I started Trovz….

Keeping It Real

Okay, let’s face it — there are some caveats on this approach. You’ve got the data limit of 40 GB. That should hold you for a while. There’s a limit of 2 CPUs, which will keep you from upgrading to a bigger EC2 instance. Truth be told, you won’t get the same performance this way that you would on a dedicated server with more CPUs and more RAM. But that costs more. All things considered, this looks like a real opportunity, especially considering that the cost estimates above assume that the server is busy 24×7 — in practice, I don’t know what utilization percentage is likely (that would vary project to project), but I would guess it’s going to be less than 100%. I have a project that I may launch this way at some point.

Here’s the bottom line: Express License + Amazon EC2 offers a cheap way to make use of some mighty powerful software. It may be enough to get you started. And when you outgrow what Express gives you, MarkLogic will be there to help you grow.

Final caveat: I haven’t set up an instance myself, just looked at Amazon’s documentation. Take the time to check out their documentation yourself before you launch. Also, I haven’t factored in the cost of bandwidth or EBS Volumes, although those costs appear minimal to me. Do your own due diligence, but I think you’ll like what you find. 

Modules Database

1. Deploying code is a transactional update — there’s no need to worry that the server will be reading your source files while you’re in the middle of updating them.
2. Related to #1, if you’re working in a cluster and an application server is present on multiple instances, deploying to a modules database that spans those instances will let the application server on all instances update to the new code at the same time.
3. When you’re migrating an application from one server to another, you can use XQSync to move the source code as well as the data.
4. Using a modules database is actually required for CPF. In a database where you set up content processing, check the Configure tab for the domain — the evaluation context has to be a database.

File System

1. When you make a change in your code, there’s no deploy step. Just hit save and then refresh your browser (or however you are accessing results). Very simple.
2. You’ve got your code in a version control system like SVN or GIT (right?), and those expect to live on the file system. I guess you could set up WebDAV and do an “svn checkout …” right into a modules database, but that feels a little weird to me. Your code probably lives in a file system directory somewhere, and you interact with the version control system from there.
3. When deploying to an integration or production server, I’ve sometimes pointed the application server to a soft link, then update the soft link on the file system after setting up a directory with the new code. I think that addresses point #1 under Module Databases, but not #2 unless you do it on a volume that is visible to all instances.

What Do I Do?

Common practice among my colleagues seems to be use the file system while developing on your local box, then deploy to a modules database for integration or production servers. I’ve actually been using a modules database even for local work lately, because I’ve got an ant script that makes it simple and fast. Even with that, every now and then I forget to deploy.

Where does your code live? Any reasons I haven’t covered?

Direct

This is the simpler case, so I’ll take it first. Direct means that your XQuery code has the XML you want to build:

declare namespace blog = "http://davidcassel.net/blog";

<blog:example simple="true">
  <blog:pointless/>
</blog:example>

XML is a natural data structure for XQuery and this is a very simple way to construct it.

Computed

A computed element is less direct, but it has its benefits. Let’s take a look:

declare namespace blog = "http://davidcassel.net/blog";

element { xs:QName("blog:example") } {
  attribute simple { "true" },
  element { xs:QName("blog:pointless") } { }
}

Benefits

The direct method is more concise, so why would we use the computed? There are two reasons why I typically end up using them.

A Conditional Attribute

Sometimes you want to include an attribute only under certain circumstances. The trick is, this doesn’t work:

<blog:example { if ($condition) then simple="true" else () }/>

That’s not valid syntax — the attribute either has to be there or not, though its value can be the result of an XQuery expression. One way we can conditionally include an attribute is to build the element different in the if and else branches of the condition:

if ($condition) then
  <blog:example simple="true"/>
else
  <blog:example/>

As you can imagine, that can be a lousy way to go if the element has some complexity to it, for instance if it has a lot of attributes. This is one place where computed construction makes your life easier:

declare namespace blog = "http://davidcassel.net/blog";

element { xs:QName("blog:example") } {
  if ($condition) then attribute simple { "true" } else (),
  element { xs:QName("blog:pointless") } { }
}

Computing the Name

Another case where the computed version comes in handy is when the name of the element itself will be the result of a computation. Let’s suppose you have a map that you want to turn into a set of elements.

<root>{
  for $key in map:keys($map)
  return element { $key } { map:get($map, $key) }
}</root>

There’s really no other way to do this case, but it’s simple to do this way.

Any other cases you can think of where computed elements are the easier way to go?

Anyway, as we set up our XQuery standards, we chose to rely on Douglas Crockford’s JSLint to set the standards for our JavaScript code. I found the quote below in a README file as part of JSLint and I thought it was worth sharing.

The place to express yourself in programming is in the quality of your ideas,and the efficiency of execution. The role of style is the same as in literature. A great writer doesn’t express himself by putting the spaces before his commas instead of after, or by putting extra spaces inside his parentheses. A great writer will slavishly conform to some rules of style, and that in no way constrains his power to express himself creatively. See for example William Strunk’s The Elements of Style [http://www.crockford.com/wrrrld/style.html].

This applies to programming as well. Conforming to a consistent style improves readability, and frees you to express yourself in ways that matter. JSLint here plays the part of a stern but benevolent editor, helping you to get the style right so that you can focus your creative energy where it is most needed.

As someone who writes both code and prose, I could relate well to both aspects of what he was saying. I keep my copy of Strunk pretty close at handy.