The Max Dew Point

Revisiting MarkLogic Aggregate User Defined Functions

2014-05-25T06:53:00.003-06:00

So I figure it has been long enough since my last blog post and it is about time that I post something. It this blog post I'm going to revisit User Defined Functions in MarkLogic. To be completely honest, my first try at a UDF proved that I could write one, but the end result was rather lame. In fact, if you go back and look at you'll see that the native code didn't do anything that couldn't be out performed, or at least matched, by XQuery code.

So for my second attempt I thought I'd do something that would provide more value. That is when I thought that a good use would be to bucket values in a map. Using my first UDF as a starting point I decided to write a UDF that would bucket values based off of a regular expression. The key in the map will be the matching portion of the regular expression, while the values will be the strings that share the same string matching the regular expression. For example, if you passed the regular expression "^.", it will group the values in a map by the first character.

Before going any farther, if you aren't familiar with UDFs in MarkLogic and how they work I encourage you to read Dave Cassel's blog post on the matter. He provides a clear view of what is going on behind the scenes.

Since this works with regular expressions as well, I had a good start already in from my first attempt. Just as before, I read the first argument passed and compile a regular expression with it.

The first new step to do this was to determine which C++ structure I'd use to collect the information. Obviously a vector wouldn't work for this scenario. I was first drawn to std::map, but quickly discovered it only stored one value per key. That is when I was drawn to std::multimap. So the map function of my UDF basically adds a std::pair the multimap with the first being the regex matching portion of the string and second being the full string. Nothing too much more complex than my original go at a UDF.

(*(match),*(new String(cp_str,cur.collation()))));
} else if (capture_overflow) {
size_t str_length = strlen(cur.get());
char cp_str[str_length];
strcpy(cp_str,cur.get());
buckets.insert(std::pair(*(new String(overflow_bucket.c_str(),cur.collation())),*(new String(cp_str,cur.collation()))));
}
}
}
}
]]>

The reduce function actually does the opposite of reduce to result set by inserting each pair from the one multimap into the current multimap.

void Bucketeer::
reduce(const AggregateUDF* _o, Reporter& reporter)
{
/* Merge matches found */
const Bucketeer *o = (const Bucketeer*)_o;
typename std::multimap o_buckets = o->buckets;
for (typename std::multimap::iterator it = o_buckets.begin(); it != o_buckets.end(); it++) {
buckets.insert(typename std::pair (it->first, it->second));
}
}
]]>

The encode and decode functions are where it actually got a bit more tricky. From the MarkLogic documentation, the functions "encode/decode the implementation-specific state on your objects" and "you can encode data members in any order, but you must be consistent between encode and decode." I might have been able to work some magic with the encode(*void, size_t)/decode(*void, size_t) signatures to just pass the whole multimap in one go, but didn't feel comfortable enough going that route (If any C++ masters have pointers (pun intended) on the matter, feel free to share). Instead I used a pattern that is a flattened representation of the multimap. First the key count is encoded. Then each key is encoded followed by the count of values of that key and its values. This way, as the information is decoded in the decode function, the decode is provided the proper information to decode in the proper order due to the counts and each key/value pair is inserted into the multimap.

void Bucketeer::
encode(Encoder& e, Reporter& reporter)
{
std::map keys_w_count;
int key_count = 0;
// determine key count and collect unique keys
for (
typename std::multimap::iterator it = buckets.begin();
it != buckets.end();
// set the iterator to the upperbound of the current key
it = buckets.upper_bound(it->first)
) {
// store key with count of entries for that key
keys_w_count.insert(std::pair (it->first,buckets.count(it->first)));
// increment key count
key_count++;
}
// encode key count
e.encode(key_count);
for (std::map::iterator it = keys_w_count.begin(); it != keys_w_count.end(); ++it) {
// encode key value
e.encode((it->first));
// encode count of unique values associated with key
e.encode(it->second);
std::pair ::iterator, typename std::multimap::iterator> ret;
// get iterator for key range
ret = buckets.equal_range(it->first);
// encode each value for key
for (typename std::multimap::iterator it2 = ret.first; it2 != ret.second; ++it2) {
e.encode(it2->second);
}
}
}

template
void Bucketeer::
decode(Decoder& d, Reporter& reporter)
{
int key_count;
// decode key count
d.decode(key_count);
for (int i = 0; i < key_count;i++) { String key; int key_size; // decode key d.decode(key); // decode count of key values d.decode(key_size); for (int j = 0; j < key_size; j++) { T value; // decode value for key d.decode(value); // insert key/value pair into our multimap buckets.insert(typename std::pair (key, value));
}
}
}
]]>

The finish function then starts a MarkLogic map object and then iterates over the unique keys, first writing the key and then writing each of the values of that key.

void Bucketeer::
finish(OutputSequence& os, Reporter& reporter)
{
// start the MarkLogic map
os.startMap();
// interate over the unique keys
// note the use of buckets.upper_bound() to ensure we get a key only once
for (typename std::multimap::iterator it = buckets.begin(); it != buckets.end(); it = buckets.upper_bound(it->first)) {
std::pair ::iterator, typename std::multimap::iterator> ret;
// get the range for the key values
ret = buckets.equal_range(it->first);
// write the map key
os.writeMapKey(it->first);
// iterate over the values for current key
for (typename std::multimap::iterator it2 = ret.first; it2 != ret.second; ++it2) {
// write the value
os.writeValue(it2->second);
}
}
os.endMap();
}]]>

What was really cool was after doing this I thought I would like to do a similar thing with dateTime. For example group by year or year and month, etc. Thanks to C++ templates I was able to write an abstract class that implements the reduce, encode, decode, and finish functions. I then extend Bucketeer and most of the work from the regex implementation carries over. Really the most difficult part was extracting the year, month, etc. from the dateTime. The only reason that was so difficult was because it took me a while to determine the value of marklogic::DateTime wasn't a Unix timestamp, but actually FILETIME. Once I figured that out I was able to find this handy information.

So this UDF allows you to bucket by dateTime attributes like year, month, etc. with "dateTime" or regular expression with "regex". You can see the final code here: https://github.com/ryanjdew/ml-bucketeer.

XML Memory Operations 1.0.x Release

2013-10-31T08:20:00.000-06:00

Version 1.0.3 of the XML memory update library is now available on GitHub from the master branch at https://github.com/ryanjdew/XQuery-XML-Memory-Operations. It currently requires MarkLogic 6. With this release there are a lot of stability enhancements as well as adjustments to some perhaps excessive strong typing (I do plan in the future to patch the v0.4.x branch, which is compatible with ML 4+, with some of the more stabilizing enhancements).

Previously, it was required that when using the mem:copy to begin a transaction and then queueing up operations to be done on nodes for that transaction required there be 1 or more nodes to change. The library now allows zero or more nodes and gracefully handles that situation. Further more, previously it would break if you ever just did a copy on invalid XML. This is also handled more gracefully now and will provide you the expected results (although, I still feel that if XML is namespaced and you have a schema, you should probably follow the schema).

With this 1.0 release, the library was also re-released under the Apache 2.0 License. This was done since the Apache 2.0 License seems to be more friendly with patents, etc.

Finally, in preparation for MarkLogic 7, I added a second library that you could import instead of the regular one that will allow you to perform operations in a more 'functionally pure' way (read no side-effects). With a change like that, this means that this library could fairly easily be ported to other XQuery implementations that implement Michael Kay's proposal of maps as outlined here which will be implemented in MarkLogic 7. While I haven't seen any significant performance differences between the two approaches, it is further validation that most problems can be solved in a functional way. Pretty cool.

Below are examples contrasting the two approaches.

xquery version "1.0-ml";
import module namespace mem = "http://maxdewpoint.blogspot.com/memory-operations" at "/memory-operations.xqy";
import module namespace mem-fun = "http://maxdewpoint.blogspot.com/memory-operations/functional" at "/memory-operations-functional.xqy";

declare variable $test-xml :=

This is a title

This is a paragraph.

;

(: stateful way :)
let $new-xml :=
let $transaction-id := mem:copy($test-xml)
let $_ := (
mem:replace($transaction-id,$test-xml/head/title,element title {"This is so awesome!"}),
mem:insert-child($transaction-id,$test-xml/body/div/p,attribute data-info {"This is also awesome!"})
)
return mem:execute($transaction-id)

(: functional way :)

let $functional-new-xml :=
mem-fun:execute(
mem-fun:insert-child(
mem-fun:replace(
mem-fun:copy($test-xml),
$test-xml/head/title,
element title {"This is so awesome!"}
),
$test-xml/body/div/p,
attribute data-info {"This is also awesome!"}
)
)

return ($new-xml, $functional-new-xml)

It is my hope that this library will gain wider adoption between the changes enhancing usability, stabilization, and patent friendly license. Let me know if you have any comments or suggestions.

The Expiring Constitution: How Technology is Up-ending a Government

2013-08-07T18:45:00.000-06:00

The title to this article may be troubling to some. Some may even dismiss it as unpatriotic and anti-American. That would be unfortunate as this is in no way meant to discount the achievement of the Founding Fathers in bringing about the U.S. Constitution. Even if the Founding Fathers could have foreseen the technological advancements we have achieved since their time, accounting for those advancements in the Constitution would have further threatened what already appeared to be a fragile situation.

So you’re probably thinking “What do you mean ‘expiring Constitution?’” Technology has completely changed the government landscape. For example, I will bring up Federalist Papers No. 10 and No. 51 which are pillars to the founding of the U.S. Constitution. In these papers James Madison makes the argument for a large republic because it brings about enough diversity that there won’t be a solid majority of citizens to infringe on the rights of a minority citizens. Essentially, Madison relies on the inefficiencies of his time period in communicating and sharing opinions.

How do we see some of the consequences of the technology on this outdated principal Madison relied so heavily on? Looking back in history, prior to the current era of the Republican and Democratic parties, we see that political parties changed out about every 30-40 years (Interestingly enough, in the bigger picture of things, not too far off of the 20 year cycle in which Jefferson said a revolution is needed.).

So what changed that has caused the Republicans and Democrats to hold on for so long? Well, within a decade prior to the era of Democrats and Republicans, the first national newspaper was founded. By the time Republicans and Democrats should have been finding themselves shown the door, communication became efficient enough to solidify over-arching opinions over a large population. Madison’s fear of a majority faction that would go against the interests of minority groups is now being realized.

Now let us fast forward to today and take a look at the 4th amendment and reasonable expectation of privacy. If it isn’t obvious already, my example is going to address the NSA programs (Not the allegations of use of mal-ware and similar tactics. Those are totally different stories.). Many people act like the NSA programs of monitoring activities on the internet is like having the government invade their home. Any technologist familiar with how the internet works will have to admit it is more like being in a high school locker room and noticing someone gawking. Creepy and perhaps disturbing, but you aren’t exactly doing much to prevent the view.

Another issue is the technology as a medium for transferring information. At some point there is a ‘middle-man’, whether it is your ISP or your phone provider. The information provided to them to connect you with where you want to go is theirs to keep and can be shared however they really see fit. There is excellent article on this topic that I’m sure I’m giving no justice to.

So in this case, what is reasonable the expectation of privacy? It depends on how tech-savvy you are and how much you trust your ‘middle-man’. Rightly so, people are concerned that the actions of the NSA may harm the openness of the Internet. Oddly enough it is the openness of the Internet that has allowed the NSA to do what it does and it is attempts to prevent the NSA from having access to our information that will kill the openness of the Internet.

So what do we do to solve the issue? We need people familiar with technology, such as Douglas Rushkoff, to make reasonable laws surrounding our digital privacy. We need technological innovations, such as peer-to-peer networking, that provide ample privacy, but prevents those innovations from becoming nesting grounds for child abuse and other horrendous acts. We need to meticulously comb over our laws as to what we deem criminal and do some soul searching as to whether those laws make sense. Lastly, but perhaps most important, we need to change our elections so that we can have a government that the American people can trust again.

I'm sure there are many opinions on this and you're more than welcome to comment. Please be civil, though. I am moderating any comments that are made.

Coding by Superstition: XQuery Sanitization

2013-05-25T21:28:00.000-06:00

There is a great talk given at The Markup Conference 2011 that covers the potential dangers of XQuery injection. The emphasis of that talk was to use sanitization as a method for preventing XQuery injection. In the example given, REST APIs were used to execute queries and the REST APIs were created in such a way that the end client could pass the exact query to be executed.¹

While there is no argument that such an approach can be effective, I would argue with the question of "Why would such an exploitable API even exist?" The power of dynamically generated XML can be leveraged, while maintaining a hardened system that isn't open to XQuery injection. I will show how this can be done with two examples of dynamically built order by clauses. The first could be potentially be exploited with XQuery injection, while the second has been hardened to prevent such an attack.

xquery version "1.0-ml";

declare function local:prolog($element as element()) {
if ($element instance of element(value))
then
fn:string-join(
local:prolog-ns($element/*),
" "
)
else ()
};

declare function local:prolog-ns($element as element()) {
let $qname := fn:resolve-QName(fn:string($element),$element)
let $prefix := fn:prefix-from-QName($qname)
where $prefix
return
fn:concat("declare namespace ", $prefix, " = '", fn:namespace-uri-from-QName($qname), "';")
};

declare function local:order-by($element as element()) {
typeswitch($element)
case element(score) return "cts:score($r) descending"
default return
fn:concat(
"$r/descendant-or-self::",
fn:string($element/element),"[1]",
if ($element[attribute])
then fn:concat("/@",$element/attribute/fn:string())
else (),
" ",
if ($element/@direction eq "descending")
then "descending"
else "ascending"
)
};

let $options := <options>
<sort>
<score/>
<value xmlns:test="http://maxdewpoint.blogspot.com/test-fn-data">
<element>test:id</element>
</value>
<value xmlns:html="http://www.w3.org/1999/xhtml">
<element>html:div</element>
<attribute>id</attribute>
</value>
</sort>
</options>
return
xdmp:eval(
fn:concat(
"xquery version '1.0-ml';
",
fn:string-join(
local:prolog($options/sort/*),
" "
)
,"

for $r in cts:search(fn:collection(), cts:and-query(()))[1 to 50]
", if ($options/sort[*])
then fn:concat("order by ", fn:string-join(local:order-by($options/sort/*),", "), " ")
else (),
"return $r"
)
)

You will notice that in the code above values are taken directly from values in the options XML that we pass in order to generate the order by clause in our XQuery. Those are potential areas that could be subject to XQuery injection.

Below we have code that doesn't use any of the values directly in generating our XQuery, but it makes use of the XML options to determine how to create external variables which then are used in our order by clause. This method of XQuery hardening prevents XQuery injection and affords you the ease of not having to track which inputs need sanitized. Further, you don't have the worry about the trade off of sanitization and potentially affecting the quality of the application's inputs.

xquery version "1.0-ml";

declare function local:prolog($element as element(), $position as xs:integer) {
if ($element instance of element(value))
then
fn:string-join(
local:prolog-var($element/*, $position),
" "
)
else ()
};

declare function local:prolog-var($element as element(), $position as xs:integer) {
fn:concat(
"declare variable ",
local:var-name($element, $position),
" external;")
};

declare function local:var-name($element as element(), $position as xs:integer) {
fn:concat(
"$",
if ($element instance of element(element))
then "element"
else "attribute",
"-qn-",
fn:string($position))
};

declare function local:order-by($element as element(), $position as xs:integer) {
typeswitch($element)
case element(score) return "cts:score($r) descending"
default return
fn:concat(
"$r/descendant-or-self::*[fn:node-name(.) eq ",
local:var-name($element/element, $position),
"][1]",
if ($element[attribute])
then fn:concat("/@*[fn:node-name(.) eq ",$element/attribute/local:var-name(., $position), ']')
else (),
" ",
if ($element/@direction eq "descending")
then "descending"
else "ascending"
)
};

let $options := <options>
<sort>
<score/>
<value xmlns:test="http://maxdewpoint.blogspot.com/test-fn-data">
<element>test:id</element>
</value>
<value xmlns:html="http://www.w3.org/1999/xhtml">
<element>html:div</element>
<attribute>id</attribute>
</value>
</sort>
</options>
return
xdmp:eval(
fn:concat(
"xquery version '1.0-ml';
",
fn:string-join(
for $e at $pos in $options/sort/*
return local:prolog($e,$pos),
" "
)
,"

for $r in cts:search(fn:collection(), cts:and-query(()))[1 to 50]
", if ($options/sort[*])
then
fn:concat("order by ",
fn:string-join(
for $e at $pos in $options/sort/*
return local:order-by($e, $pos),", "), " ")
else (),
"return $r"
),
(
for $e at $pos in $options/sort/*
return
if ($e instance of element(value))
then
for $e in $e/*
return (xs:QName(fn:replace(local:var-name($e,$pos), "^\$","")),fn:resolve-QName(fn:string($e),$e))
else ()
)
)

[1] van der Vlist, Eric. “XQuery Injection: Easy to exploit, easy to prevent....” Presented at Balisage: The Markup Conference 2011, Montréal, Canada, August 2 - 5, 2011. In Proceedings of Balisage: The Markup Conference 2011. Balisage Series on Markup Technologies, vol. 7 (2011). doi:10.4242/BalisageVol7.Vlist02.

Coding by Superstition: Encapsulation

2013-05-02T06:55:00.000-06:00

Encapsulation is great tool in software engineering. It makes life easier when a programmer doesn't have to rework out in his mind the details of an operation each time each they revisit a problem. It is far better to just pass values to the black box and get back what you need. This is precisely why MarkLogic provided private modifiers and XQuery 3.0 provided the private annotation (%private).

Unfortunately, some encapsulation practices from the object-oriented world can sneak into the functional world of XQuery. I'm talking about the dreaded getters and setters. There are few times, if any, that an XQuery program should have setters (In actuality, XQuery without extensions cannot have setters.). Setters can really cause issues by introducing side effects that could cause you to lose out on potential optimizations that could be made (http://en.wikipedia.org/wiki/Functional_programming#Pure_functions). In fact, if I were part MarkLogic Technical Support, I would push MarkLogic to not even mention xdmp:set in any of their documentation. Since ideally we wouldn't have any setters, why would we prefix functions with 'get'?

Going even further, for simple constant (per transaction) values, why wouldn't we just use global variables since they are immutable? This can make code more clear as to what is going on. Otherwise meaning can easily be lost as a variable is passed around through functions if special care isn't taken.

Coding by Superstition: XQuery Booleans

2013-02-23T07:29:00.000-07:00

The next few posts I make are going to be part of a series I am calling, "Coding by Superstition." This series, which I will probably revisit from time to time, is to address programming practices that I have encountered that seem to indicate a lack of understanding. This first post is about use of booleans in XQuery.

I've come across some interesting code dealing with booleans in XQuery. Here are a few examples:

let $items as xs:string* := ('test', 'values', 'to', 'iterate', 'over')
$bool1 as xs:boolean := if ($items) then fn:true() else fn:false(),
$bool2 as xs:boolean? := if (fn:subsequence($items,6)) then fn:true() else (),
$bool3 as xs:boolean* := for $i in $items return $i eq "some-value",
$bool4 as xs:boolean* := for $i in $items return $i eq "iterate"

return (
if ($bool1)
then "bool1: true"
else (),
if ($bool2)
then "bool2: true"
else (),
if ($bool3)
then "bool3: true"
else (),
if ($bool4)
then "bool4: true"
else ()
)

I think these interesting examples come out due to a combination of two things; 1.) not understanding the somewhat complex rules behind effective boolean values and 2.) not realizing the effects of function mapping in MarkLogic's XQuery version "1.0-ml". Below are summaries of the effective boolean rules that can be found here.

Empty sequences are treated as false
If the first item in a sequence is a node it is treated as true
If it is a single boolean value it is treated as that boolean value
If it is a single string, then if it is zero length it is treated as false; otherwise it is treated as true
If it is a single numeric value, then if the value is NaN or zero it is treated as false; otherwise it is treated as true
All other scenarios throw an error

Now throw in function mapping from MarkLogic, and you almost never encounter the last scenario. It that case, if any of the items in a sequence's effective boolean value results in true then the value used is true. That is why, outside of predicates, I feel it is best to avoid effective boolean values and be explicit in what your condition is. Below are some better boolean examples.

let $items as xs:string* := ('test', 'values', 'to', 'iterate', 'over')
$bool1 as xs:boolean := fn:exists($items[. ne '']),
$bool2 as xs:boolean := fn:exists(fn:subsequence($items,6)),
$bool3 as xs:boolean := some $i in $items satisfies $i eq "some-value",
$bool4 as xs:boolean := every $i in $items satisfies $i instance of xs:string

return (
if ($bool1)
then "bool1: true"
else (),
if ($bool2)
then "bool2: true"
else (),
if ($bool3)
then "bool3: true"
else (),
if ($bool4)
then "bool4: true"
else ()
)

A New Way to Transform XML

2013-02-16T19:01:00.001-07:00

In the MarkLogic Application Developer's guide there is a example of a recursive typeswitch transform. This is very efficient to use, but might not be as flexible as you would like. For example, you might want a central place to make transformation calls, but allow the user calling have more power over the transformation.

One solution to this is the Translator design pattern found here. By passing function references, the user can have more control.

Below is an approach that I came up with that is kind of a hybrid between the 2 previous design patterns. It uses functions associated with xs:QNames. The QNames are used to find elements to transform with the associated function. Below I converted the example given in the Application Developer's Guide to use my new method. Note that you will want to take into consideration performance when looking at this new method. The performance difference will also change depending on just how much of the XML is being changed with the transform.

xquery version "1.0-ml";
import module namespace mem = "http://maxdewpoint.blogspot.com/memory-operations" at "/memory-operations.xqy";
(: Functions for transforming xml :)
declare function local:bar($node as node()) as node()*
{
{$node/node()}
};
declare function local:baz($node as node()) as node()*
{
{$node/node()}
};
declare function local:buzz($node as node()) as node()*
{
{$node/node()}
};
declare function local:foo($node as node()) as node()*
{
{$node/node()}
};

(: This functions returns QNames of elements to change followed by the function that makes the change :)
declare function local:instructions() as item()*
{
xs:QName('bar'),local:bar#1,xs:QName('baz'),local:baz#1,xs:QName('buzz'),local:buzz#1,xs:QName('foo'),local:foo#1
};

declare function local:process($node as node()) as node()*
{
local:process($node,local:instructions#0)
};
(: process node :)
declare function local:process($node as node(),$instruction-funs as function() as item()*+) as node()*
{
let $transaction-id := mem:copy($node),
$instructions := $instruction-funs ! .()
return (
fn:map-pairs(
function($qname,$fun){
let $nodes-to-mod := $node/descendant-or-self::*[fn:node-name(.) eq $qname]
where fn:exists($nodes-to-mod)
return mem:transform($transaction-id,$nodes-to-mod, $fun)
},
$instructions[. instance of xs:QName],
$instructions[. instance of function(*)]
),
mem:execute($transaction-id)
)
};

let $x :=
foo
bar
baz
buzz

foo

return
local:process($x)

Revisiting The Sliding Window in MarkLogic 6

2013-02-15T07:06:00.000-07:00

Like my previous post this code revisits my implementation of a sliding window. For more information on sliding windows go here.

xquery version "1.0-ml";

declare function local:sliding-window(
$sequence as item()*,
$only-start as xs:boolean,
$start-condition as function(*),
$only-end as xs:boolean,
$end-condition as function(*),
$return as function(*)
) {
let $windows as map:map := map:map(),
$sequence-size as xs:unsignedInt := fn:count($sequence)
for $item at $pos in $sequence
let $start as xs:boolean :=
((fn:not($only-start) and 1 eq $pos) or
(switch (fn:function-arity($start-condition))
case 2 return $start-condition($item,$pos)
default return $start-condition($item))
),
$_ as empty-sequence() :=
if ($start)
then
let $window-map := map:map()
return (
map:put($window-map,'start',$item),
map:put($window-map,'start-pos',$pos),
map:put($windows,local:_next-key($windows),$window-map)
) else ()
for $window-key in map:keys($windows)
let $window-map as map:map := map:get($windows,$window-key),
$end as xs:boolean :=
((fn:not($only-end) and $sequence-size eq $pos) or
(switch (fn:function-arity($end-condition))
case 4 return $end-condition($item,$pos,map:get($window-map,'start'),map:get($window-map,'start-pos'))
case 3 return $end-condition($item,$pos,map:get($window-map,'start'))
case 2 return $end-condition($item,$pos)
default return $end-condition($item))
),
$_ as empty-sequence() := map:put($window-map,'window',(map:get($window-map,'window'),$item))
where $end
return
(
switch (fn:function-arity($return))
case 5 return $return(map:get($window-map,'window'),map:get($window-map,'start'),map:get($window-map,'start-pos'),$item, $pos)
case 4 return $return(map:get($window-map,'window'),map:get($window-map,'start'),map:get($window-map,'start-pos'),$item)
case 3 return $return(map:get($window-map,'window'),map:get($window-map,'start'),map:get($window-map,'start-pos'))
case 2 return $return(map:get($window-map,'window'),map:get($window-map,'start'))
default return $return(map:get($window-map,'window')),
map:delete($windows,$window-key)
)

};

declare function local:_next-key($windows as map:map) as xs:string {
fn:string((fn:max(map:keys($windows) ! xs:unsignedInt(.)), 0)[1] + 1)
};

local:sliding-window(
(: sequence :)
(2, 4, 6, 8, 10, 12, 14),
(: only start? :)
fn:false(),
(: lambda for start condition :)
function($start) {fn:true()},
(: only end? :)
fn:true(),
(: lambda for end condition :)
function($end, $end-pos, $start, $start-pos) { $end-pos - $start-pos eq 2 },
(: lambda for returning window :)
function($window as item()*, $first as item(), $first-pos as xs:unsignedInt, $last as item()) {<window>{ $first,$last }</window>}
)

Revisiting The Tumbling Window in ML 6

2013-02-15T06:56:00.001-07:00

I was thinking about my previous post I made on tumbling windows in XQuery 1.0. Now with MarkLogic 6 the task has become much more simple and reusable. In the example below I use inline functions for the start/end conditions and for the returned output. For more information on tumbling windows go here.

xquery version "1.0-ml";

declare function local:tumbling-window(
$sequence as item()*,
$only-start as xs:boolean,
$start-condition as function(*),
$only-end as xs:boolean,
$end-condition as function(*),
$return as function(*)
) {
let $map as map:map := map:map(),
$sequence-size as xs:unsignedInt := fn:count($sequence)
for $item at $pos in $sequence
let $start as xs:boolean :=
((fn:not($only-start) and 1 eq $pos) or
(switch (fn:function-arity($start-condition))
case 2 return $start-condition($item,$pos)
default return $start-condition($item))
),
$started as xs:boolean := map:count($map) gt 0,
$_ as empty-sequence() := if ($start and fn:not($started)) then (map:put($map,'start',$item),map:put($map,'start-pos',$pos)) else (),
$end as xs:boolean :=
((fn:not($only-end) and $sequence-size eq $pos) or
(switch (fn:function-arity($end-condition))
case 4 return $end-condition($item,$pos,map:get($map,'start'),map:get($map,'start-pos'))
case 3 return $end-condition($item,$pos,map:get($map,'start'))
case 2 return $end-condition($item,$pos)
default return $end-condition($item))
),
$window-started as xs:boolean := $start or $started,
$_ as empty-sequence() := if ($window-started) then map:put($map,'window',(map:get($map,'window'),$item)) else ()
where $end and $window-started
return
(
switch (fn:function-arity($return))
case 5 return $return(map:get($map,'window'),map:get($map,'start'),map:get($map,'start-pos'),$item, $pos)
case 4 return $return(map:get($map,'window'),map:get($map,'start'),map:get($map,'start-pos'),$item)
case 3 return $return(map:get($map,'window'),map:get($map,'start'),map:get($map,'start-pos'))
case 2 return $return(map:get($map,'window'),map:get($map,'start'))
default return $return(map:get($map,'window')),
map:clear($map)
)

};

local:tumbling-window(
(: sequence :)
(2, 4, 6, 8, 10, 12, 14),
(: only start? :)
fn:false(),
(: lambda for start condition :)
function($start) {fn:true()},
(: only end? :)
fn:true(),
(: lambda for end condition :)
function($end, $end-pos, $start, $start-pos) { $end-pos - $start-pos eq 2 },
(: lambda for returning window :)
function($window as item()*, $first as item(), $first-pos as xs:unsignedInt, $last as item()) {{ $first,$last }}
)

Extreme Unob: Prototype to Production HTML Applications

2013-02-02T14:36:00.000-07:00

I kind of poke fun with the title of this post, but there is a great idea Ryan Semerau calls "Non-obtrusive HTML Replacing (non-ob)". If you haven't haven't read his article on it, I encourage you to do so here. This post is about building on those ideas.

While Ryan's solution successfully removes some duplication of work in the scenario he describes, there remains the somewhat painstaking task of punching out holes for the dynamic data. My proposal to solve this issue is to add a third mixture to the solution that would bind the content from the database to the content in the HTML. Now I could go off and create my own binding, but why not make use of standards?

If the prototype HTML contains microdata and we also add microdata to the content in the database, then we have the binding that we are looking for. This is great because we already have a standard AND it encourages semantic (read SEO friendly) web pages. Now that won't solve all side cases, but those that understand project management will tell you that the sooner you get to the last troublesome 10% of project work the better. Using this standard, you could potentially drop the HTML in your database and immediately begin work on that last 10% of what is required to build a new page in your application.

Below I provide some grossly over simplified example code. The code is meant to convey an idea more than anything, seeing as I haven't even tested it and it makes calls to imaginary functions. (Just to point out, the functions in the mem namespace aren't imaginary and can be found here.)

let $html as element(html) := html:html-for-page($page-uri),
$content-for-page := content:page-content($page-uri),
$transaction-id := mem:copy($html),
$binding-lambda := function ($html, $content) {
for $prop in $html/descendant-or-self::*[@itemprop]
let $prop-name := fn:string($prop/@itemprop),
$prop-value := $content/descendant-or-self::*[@itemprop eq $prop-name]/node()
return
if ($prop-name eq 'url')
then mem:replace-value($transaction-id, ($prop/(@href|@src|@data))[1], $prop-value)
else mem:replace($transaction-id, $prop, element {fn:node-name($prop)} {$prop/@*,$prop-value})
},
$_populate-content :=
for $distinct-type in fn:distinct-values($content-for-page/@itemtype)
return
fn:map-pairs(
$binding-lambda,
$html//*[@itemtype eq $distinct-type],
$content-for-page[@itemtype eq $distinct-type]
)
return mem:execute($transaction-id)

Making xdmp:from-json Results Work on ML5 and ML6

2013-01-08T17:20:00.000-07:00

You might have the situation where you're caught in-between MarkLogic versions. One environment has MarkLogic 5 and the other has MarkLogic 6. One of the notable changes in ML6 is how JSON is handled. Particularly, if you are using the xdmp:from-json function, the output you get is different.

This is great for a better representation of JSON, but can be a pain for you to have to go back and change old code to match up exactly. Further, if your code needs to be backwards compatible, there is work to be done on your part.

Don't worry though, because I've done the work for you. By passing the output of xdmp:from-json to the following function you will get results matching the pattern of ML 5 whether or not you have moved to ML 6.

xquery version "1.0-ml";

declare namespace json = 'http://marklogic.com/xdmp/json';

declare function local:normalize-from-json($json as item()) {
if (fn:type-available('json:object'))
then local:_normalize-from-json($json)
else $json
};

declare function local:_normalize-from-json($json as item()) {
typeswitch(document { $json })
case document-node(element(json:object)) return
let $map := map:map(),
$_ := for $map-key in map:keys($json)
return map:put($map, $map-key, local:_normalize-from-json(map:get($json,$map-key)))
return $map
case document-node(element(json:array)) return
local:normalize-from-json(xdmp:apply(xdmp:function(xs:QName('json:array-values')),$json))
default return
$json
};

(: Without xdmp:apply alternative :)
(:
declare function local:_normalize-from-json($json as item()) {
typeswitch(document { $json })
case document-node(element(json:object)) return
let $map := map:map(),
$_ := for $map-key in map:keys($json)
return map:put($map, $map-key, local:_normalize-from-json(map:get($json,$map-key)))
return $map
case document-node(element(map:map)) return
let $map := map:map(),
$_ := for $map-key in map:keys($json)
return map:put($map, $map-key, local:_normalize-from-json(map:get($json,$map-key)))
return $map
case $doc as document-node(element(json:array)) return
local:normalize-from-json(
for $n in $doc/json:array/*
return
if (fn:exists($n/@xsi:type))
then fn:data($n)
else if (fn:exists($n/json:object))
then map:map($n/*)
else $n/*)
default return
$json
};
:)

let $from-json := xdmp:from-json('[{"some-key": [45683, 231321]}, "this is a string", 123]'),
$from-json-old-format := local:normalize-from-json($from-json)
return $from-json-old-format

While this helps for the in-between phase and is convenient, know that there are definitely advantages to be had with the new output from xdmp:from-json in ML 6. So be sure to make a note to revisit any code you are using this patch for.

Cleaning Up Empty Directories

2013-01-06T21:57:00.000-07:00

If you are like me, you don't like clutter. Particularly I don't like a lot of empty directories in my MarkLogic Database. You could avoid the problem altogether by not having MarkLogic create the directories, but there is a level of convenience in being able to go directly to a file through WebDAV to edit it.

So I came up with the following query to find the empty directories in your database. It uses cts:uris with the return type of map:map. Then I take the difference of the property uris and the document uris to get the list of directories only. Then I filter with an xdmp:estimate on the contents of the directory. If the xdmp:estimate is 0 then the directory is empty.

xquery version "1.0-ml";

let $directories-map := (cts:uris('/',('properties','map'),cts:directory-query('/','infinity'))
-
cts:uris('/',('document','map'),cts:directory-query('/','infinity')))
return map:keys($directories-map)[xdmp:estimate(cts:search(fn:collection(),cts:directory-query(.,'infinity'))) eq 0]

When 100% Isn't Your Best

2012-12-22T12:19:00.001-07:00

Lately I've been looking at making code more compliant with XQuery standards that have been setup in a tool called Sonar that checks for compliance in our code. After closely looking through the code it has become obvious why there are varying levels of severity for each of the rules. Some rules are almost 100% of the time bad to break, if not 100% of the time (this is where different opinions come into play). Other rules, the tool should be simply engaging the developer to rethink the solution they came up with, offering tips of how the developer MIGHT better their code.

If a complex code base has a 100% compliance with the set of rules, I would begin to wonder if the tool for verifying compliance was being properly used. With 100% compliance, I would suspect either the set of rules don't offer much in the way of engaging the developer or the tool is driving development instead of supplementing development.

If we really just wanted to see the 100% on our Sonar dashboard, why don't we just write a tool that will make the offending code compliant for us? The obvious answer is that the impact on performance or outcome could not be completely understood by our tool.

For example, it is generally bad practice to use the double forward slash (//) to find items in an XML document. Hopefully you know your XML structure well enough that you can provide the direct path (Sometimes you honestly don't know.). Even outside the instances where you don't know your structure enough to know the direct path to what your finding, the tool couldn't know that you have made the element you are looking for a root fragment which means you actually get increased performance using the // method within a cts:search.

Also there is a matter of not having functions in predicates. This is generally a bad practice when retrieving content from the database, but it is often better after that point than the alternative of writing a FLWOR with only a simple where clause to filter out the sequence. It is one thing if the developer changes it to a FLWOR instead of a simple XPath for readability, but another if they are doing to be "compliant."

Tools are just that, tools. Due to their limited capability, their measure of 100% doesn't always align with what is truly 100%. That being said, it always good to get in that 94+% range. An A is an A, right?

Server Level Memorization in MarkLogic

2012-12-18T12:21:00.000-07:00

There are some operations that are performed that are more expensive than what you would normally like to incur. If you have an operation like that needs to be repeated over and over again, that is where memorization comes in handy.

Memorization is technique that can be use in purely functional languages or on code that is written to be purely functional. Code is purely functional when the output is always given set inputs.

Like most things, memorization can come at a price. The decreased run time comes with an increase of memory consumption. If you use the technique too liberally it can end up hurting more than helping. That is why you need to be careful about not over-doing it.

A good rule to follow when using memorization (beyond what the language already does for you at a lower level) is only use it when the operation is expensive and the number of unique input values are limited. If you try this technique and all your input value combinations are unique you gain nothing and actually degrade performance by performing more checks and using more memory than you need to.

The following is an XQuery module to aide in memorization. It makes use of dynamic function calls, server fields to store results, and xdmp:describe and xdmp:quote to analyze inputs to ensure that inputs have not changed. If the inputs haven't changed for the provided label string and the operation has already been executed it returns the previously determined results. Otherwise it actually executes the function with its arguments and returns the results while storing it for use next time.

xquery version "1.0-ml";

module namespace memorize = "http://maxdewpoint.blogspot.com/memorize";

declare function memorize:memorize(
$cache-label as xs:string,
$fun as function(*)
) {
memorize:memorize($cache-label, $fun, ())
};

declare function memorize:memorize(
$cache-label as xs:string,
$fun as function(*),
$arg1 as item()*
) {
memorize:memorize($cache-label, $fun, $arg1, ())
};

declare function memorize:memorize(
$cache-label as xs:string,
$fun as function(*),
$arg1 as item()*,
$arg2 as item()*
) {
memorize:memorize($cache-label, $fun, $arg1, $arg2, ())
};

declare function memorize:memorize(
$cache-label as xs:string,
$fun as function(*),
$arg1 as item()*,
$arg2 as item()*,
$arg3 as item()*
) {
memorize:memorize($cache-label, $fun, $arg1, $arg2, $arg3, ())
};

declare function memorize:memorize(
$cache-label as xs:string,
$fun as function(*),
$arg1 as item()*,
$arg2 as item()*,
$arg3 as item()*,
$arg4 as item()*
) {
memorize:memorize($cache-label, $fun, $arg1, $arg2, $arg3, $arg4, ())
};

declare function memorize:memorize(
$cache-label as xs:string,
$fun as function(*),
$arg1 as item()*,
$arg2 as item()*,
$arg3 as item()*,
$arg4 as item()*,
$arg5 as item()*
) {
memorize:memorize($cache-label, $fun, $arg1, $arg2, $arg3, $arg4, $arg5, ())
};

declare function memorize:memorize(
$cache-label as xs:string,
$fun as function(*),
$arg1 as item()*,
$arg2 as item()*,
$arg3 as item()*,
$arg4 as item()*,
$arg5 as item()*,
$arg6 as item()*
) {
memorize:memorize($cache-label, $fun, $arg1, $arg2, $arg3, $arg4, $arg5, $arg6, ())
};

declare function memorize:memorize(
$cache-label as xs:string,
$fun as function(*),
$arg1 as item()*,
$arg2 as item()*,
$arg3 as item()*,
$arg4 as item()*,
$arg5 as item()*,
$arg6 as item()*,
$arg7 as item()*
) {
memorize:memorize($cache-label, $fun, $arg1, $arg2, $arg3, $arg4, $arg5, $arg6, $arg7, ())
};

declare function memorize:memorize(
$cache-label as xs:string,
$fun as function(*),
$arg1 as item()*,
$arg2 as item()*,
$arg3 as item()*,
$arg4 as item()*,
$arg5 as item()*,
$arg6 as item()*,
$arg7 as item()*,
$arg8 as item()*
) {
let $cache-hash := memorize:_cache-hash($fun, $arg1, $arg2, $arg3, $arg4, $arg5, $arg6, $arg7, $arg8),
$stored-value := xdmp:get-server-field(fn:concat('{http://maxdewpoint.blogspot.com/memorize}',$cache-label))
return
if ($stored-value instance of map:map)
then
if (map:get(
$stored-value,
'cache-hash'
)
eq
$cache-hash
)
then map:get(
$stored-value,
'value'
)
else
memorize:_memorize($cache-label, $cache-hash, $fun, $arg1, $arg2, $arg3, $arg4, $arg5, $arg6, $arg7, $arg8)
else
memorize:_memorize($cache-label, $cache-hash, $fun, $arg1, $arg2, $arg3, $arg4, $arg5, $arg6, $arg7, $arg8)
};

declare function memorize:_memorize(
$cache-label as xs:string,
$cache-hash as xs:string,
$fun as function(*),
$arg1 as item()*,
$arg2 as item()*,
$arg3 as item()*,
$arg4 as item()*,
$arg5 as item()*,
$arg6 as item()*,
$arg7 as item()*,
$arg8 as item()*
) {
map:get(
(: set server field with map:map containing input hash and output value :)
xdmp:set-server-field(
fn:concat('{http://maxdewpoint.blogspot.com/memorize}',$cache-label),
map:map()[
map:put(.,'cache-hash',$cache-hash),
map:put(.,
'value',
switch(fn:function-arity($fun))
case 0 return $fun()
case 1 return $fun($arg1)
case 2 return $fun($arg1,$arg2)
case 3 return $fun($arg1,$arg2,$arg3)
case 4 return $fun($arg1,$arg2,$arg3,$arg4)
case 5 return $fun($arg1,$arg2,$arg3,$arg4,$arg5)
case 6 return $fun($arg1,$arg2,$arg3,$arg4,$arg5,$arg6)
case 7 return $fun($arg1,$arg2,$arg3,$arg4,$arg5,$arg6,$arg7)
case 8 return $fun($arg1,$arg2,$arg3,$arg4,$arg5,$arg6,$arg7,$arg8)
default return 'too many arguments'
),
fn:true()
]
),
'value'
)
};

declare function memorize:_cache-hash(
$fun as function(*),
$arg1 as item()*,
$arg2 as item()*,
$arg3 as item()*,
$arg4 as item()*,
$arg5 as item()*,
$arg6 as item()*,
$arg7 as item()*,
$arg8 as item()*
) {
(: Using sha1 to hash and the base64 option for a shorter string. :)
xdmp:hmac-sha1(
'memorize',
fn:string-join(
($fun,'|',
$arg1,'|',
$arg2,'|',
$arg3,'|',
$arg4,'|',
$arg5,'|',
$arg6,'|',
$arg7,'|',
$arg8) ! (
typeswitch(.)
(: if it is a function ensure we get the long hand namespaced name :)
case function(*) return
fn:concat(xdmp:key-from-QName(fn:function-name(.)),'#',fn:string(fn:function-arity(.)))
(: want to use xdmp:quote if it is XML contained in the database since the value could change, but the description would remain the same. :)
case node() return
if (fn:exists(fn:base-uri(.)))
then xdmp:quote(.)
else xdmp:describe(., (),())),
(: by default use xdmp:describe since with no sequence limit and no character limit :)
default return
xdmp:describe(., (),())),
''
),
'base64'
)
};

Copy XML with Validate Lax

2012-12-08T10:34:00.000-07:00

Sometimes you might like to efficiently copy a chuck of XML and have it completely disassociated from its parent node. Fortunately, that is exactly what the validate lax expression does according to the XQuery standards (It also does validation on any schema in the context of the XQuery, but your XML should either match the schema or not have a schema for the document in context anyway, right?).

However in MarkLogic with XQuery version '1.0-ml' the standard isn't followed in this case. If you run the following code in QConsole, you will see that '1.0-ml' will merely return the same node back to you if it is determined to be valid. Compare that to running the same code with version '1.0'.

xquery version "1.0-ml";

let $child-element := (element div { element a {}})/a,
$lax-validated-child := validate lax { $child-element }
return $lax-validated-child is $child-element
;
xquery version "1.0";

let $child-element := (element div { element a {}})/a,
$lax-validated-child := validate lax { $child-element }
return $lax-validated-child is $child-element

Now you might think it is not worth it to lose all the other MarkLogic goodness. Never fear, you can get most of the MarkLogic goodness back using the following:

xquery version "1.0";
(: or xquery version '3.0' if want those functions in ML 6:)

declare namespace cts = "http://marklogic.com/cts";
declare namespace xdmp = "http://marklogic.com/xdmp";
declare namespace map = "http://marklogic.com/xdmp/map";

declare option xdmp:mapping "true";

(: Note that you now need the fn:collection() or fn:doc() :)
cts:search(fn:collection()/article, cts:and-query(()))[1]

Update: You can also use 'declare option xdmp:copy-on-validate "true";' in version 1.0-ml.

If the Software World Ran Like U.S. Politics

2012-11-06T07:35:00.000-07:00

In celebration of “The political calls and ads are finally coming to an end” day, I thought I would share a little analogy that I have about if the software world were run like the U.S. Political system. (I know, I live in Utah, not exactly a swing state. In my defense, though, the Mia Love vs. Jim Matheson contest has gotten pretty heated and tiresome. Also, we get plenty of Romney vs. Obama through those “series of tubes” called the Internet.)

Imagine a world where software engineers and architects were elected to their position. Beyond that, the general population felt that, rather than just being responsible for electing those engineers and architects who qualified, they needed to be involved with the details of how those they elected went about their work. Campaigns would be put together based on the camp in which people fell.

The current Google administration would blame the current problems on the legacy Microsoft APIs of the former Microsoft administration. Meanwhile, Microsoft would constantly try to remind the people of the 'good old days' of Windows 3.1 and how they, like how Lincoln freed the blacks with the abolition of slavery, had freed the bean counters with Excel Spreadsheets. Apple would be attacked for catering to the top fill-in-the-blank percent of users, while Google would be attacked for the redistribution of mobile swagger with their cheap, but effective Android mobile devices.

Journalists would write of the “evils” of functional languages and praise those of the object oriented world. Op eds would be written to inform the population that a vote for MarkLogic is a vote for Microsoft. Watch dog groups would be formed to scrutinize should a for loop be used when it is “obvious” a while loop was the loop of choice. Every four years, should the mood of the nation change, the current architects would be replaced just as they were starting to replace the previously installed systems.

Vendors would form lobbyist groups and promise the world in order to ensure that money was funneled their way. These vendors would also vie for exemptions from the technical mandates handed down from on high. All these dealings would occur behind closed doors out the view of the public eye.

It would be a nightmare to develop and architect software according to the knowledge of the general population. When filling a software engineer position, the best companies hire those that are qualified and can be trusted in that position. I don't pretend to know all the solutions to all the political issues, just as I would be put out by 'non-technical people' pretending to know all the answers to all the technical issues. So why can't we elect people to office, simply knowing they are qualified and can be trusted?

Higher Order Functions in MarkLogic 6

2012-10-06T11:31:00.000-06:00

Lately I've been playing around with higher order functions in MarkLogic 6. After a while I came up with some code for executing XQuery based off of XML. The first example that I give shows how I can use XML to configure how a function is called. This example shows XML determining how xdmp:tidy is called.

xquery version "1.0-ml";

declare function local:html-handler($html as xs:string) {
local:html-handler($html, fn:false())
};

declare function local:html-handler($html as xs:string, $show-tidy-warnings as xs:boolean) {
local:html-handler($html, $show-tidy-warnings, ())
};

declare function local:html-handler($html as xs:string, $show-tidy-warnings as xs:boolean, $options as xs:string*) {
xdmp:tidy($html,
<options xmlns="xdmp:tidy">{
$options ! (element {fn:QName("xdmp:tidy",fn:substring-before(.,'='))} {fn:substring-after(.,'=')} )
}</options>)[$show-tidy-warnings or fn:position() eq 2]
};

let $html := "<html><head><title>My page</title></head><body><div></div></body></html>"
let $execution-descriptor :=
<execution-descriptor xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<function ns="http://www.w3.org/2005/xquery-local-functions" arity="3">html-handler</function>
<arg-sets>
<arg-set>
<item xsi:type="xs:boolean">true</item>
</arg-set>
<arg-set>
<item xsi:type="xs:string">indent=yes</item>
<item xsi:type="xs:string">indent-spaces=2</item>
<item xsi:type="xs:string">markup=yes</item>
</arg-set>
</arg-sets>
</execution-descriptor>

let $funct := fn:function-lookup(fn:QName($execution-descriptor/function/@ns,$execution-descriptor/function),$execution-descriptor/function/@arity)

return
switch(fn:function-arity($funct))
case 1 return $funct($html)
case 2 return $funct($html,$execution-descriptor/arg-sets/arg-set[1]/item/fn:data(.))
case 3 return $funct($html,$execution-descriptor/arg-sets/arg-set[1]/item/fn:data(.),$execution-descriptor/arg-sets/arg-set[2]/item/fn:data(.))
default return ()

In the previous case there might not be any obvious power to it. You could easily just have XML that could change the options you pass to xdmp:tidy already. So the question is, "What is the big deal?" Hopefully the following example expresses the power a little more.

xquery version "1.0-ml";

import module namespace mem = "http://maxdewpoint.blogspot.com/memory-operations" at "/MaxDewPoint/memory-operations.xqy";

declare namespace xs="http://www.w3.org/2001/XMLSchema";
declare namespace xsi="http://www.w3.org/2001/XMLSchema-instance";

declare function local:innermost($nodes as node()*) as node()* {
$nodes except $nodes/ancestor::node()
};

declare function local:descriptor-executor($descriptor as element(execution-descriptor)) {
let $child-descriptors := $descriptor//execution-descriptor
return
if (fn:exists($child-descriptors))
then
local:descriptor-executor(mem:transform(
local:innermost($child-descriptors),
function ($node as node()) as node()* {
local:wrap-result(local:descriptor-execute($node))
}
))
else
local:descriptor-execute($descriptor)
};

declare function local:descriptor-execute($descriptor as element(execution-descriptor)) {
let $funct := fn:function-lookup(fn:QName($descriptor/function/@ns,$descriptor/function),$descriptor/function/@arity)
return
switch(fn:function-arity($funct))
case 0 return
$funct()
case 1 return
$funct(local:process-arg-set($descriptor/arg-sets/arg-set[1]))
case 2 return
$funct(local:process-arg-set($descriptor/arg-sets/arg-set[1]),local:process-arg-set($descriptor/arg-sets/arg-set[2]))
case 3 return
$funct(local:process-arg-set($descriptor/arg-sets/arg-set[1]),local:process-arg-set($descriptor/arg-sets/arg-set[2]),local:process-arg-set($descriptor/arg-sets/arg-set[3]))
default return ()
};

declare function local:process-arg-set($arg-set as element(arg-set)) {
$arg-set/* !
(
typeswitch(.)
case element(item) return fn:data(.)
case element(node) return xdmp:value(fn:string(.))
case element(function-ref) return fn:function-lookup(fn:QName(./@ns,.),./@arity)
default return ()
)
};

declare function local:wrap-result($result as item()*) {
$result !
(
typeswitch(.)
case node() return element node {xdmp:describe(.,(),())}
case function(*) return element function-ref { attribute arity {fn:function-arity(.)}, fn:function-name(.) ! (attribute ns {fn:namespace-uri-from-QName(.)}, fn:local-name-from-QName(.))}
default return element item { attribute xsi:type {local:determine-type(.)},fn:data(.)}
)
};

declare function local:determine-type($item as item()) {
typeswitch($item)
case xs:dateTime return "xs:dateTime"
case xs:date return "xs:date"
case xs:duration return "xs:duration"
case xs:float return "xs:float"
case xs:double return "xs:double"
case xs:decimal return "xs:decimal"
case xs:integer return "xs:integer"
case xs:boolean return "xs:boolean"
case xs:QName return "xs:QName"
case xs:integer return "xs:integer"
default return "xs:untypedAtomic"
};

local:descriptor-executor(
<function ns="http://www.w3.org/2005/xpath-functions" arity="2">string-join</function>
<arg-sets>
<arg-set>
<execution-descriptor>
<function ns="http://www.w3.org/2005/xpath-functions" arity="3">map-pairs</function>
<arg-sets>
<arg-set>
<function-ref ns="http://www.w3.org/2005/xpath-functions" arity="2">concat</function-ref>
</arg-set>
<arg-set>
<item xsi:type="xs:string">indent=</item>
<item xsi:type="xs:string">indent-spaces=</item>
<item xsi:type="xs:string">markup=</item>
</arg-set>
<arg-set>
<item xsi:type="xs:string">yes</item>
<item xsi:type="xs:string">2</item>
<item xsi:type="xs:string">yes</item>
</arg-set>
</arg-sets>
</execution-descriptor>
</arg-set>
<arg-set>
<item xsi:type="xs:string">;</item>
</arg-set>
</arg-sets>
</execution-descriptor>)

In the last example, the output is completely determined by the XML. You could play around with this code in the QConsole, changing the execution-descriptor to whatever it is you want to execute. Try running the following in the QConsole. Pretty neat...

local:descriptor-executor(<execution-descriptor xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<function ns="http://maxdewpoint.blogspot.com/memory-operations" arity="2">insert-child</function>
<arg-sets>
<arg-set>{local:wrap-result(fn:collection()[1]/*)}</arg-set>
<arg-set>{local:wrap-result(attribute new-attribute{"added"})}</arg-set>
</arg-sets>
</execution-descriptor>)

My First MarkLogic Aggregate User-Defined Function

2012-09-27T07:55:00.000-06:00

I wrote my first Aggregate User-Defined Function in MarkLogic. There were a couple stumbling blocks that I encountered and had to brush up on my C/C++, but I think it was pretty straight forward, thanks to MarkLogic's documentation and examples.

The native function that I created compares regular expressions against a lexicon of string values and returns the matches. The results were very encouraging, which I will cover later. First I'll address my stumbling blocks.

My first stumbling block, was a fairly easy and simple one. I didn't pay enough attention to the namespace I used to install my plugin and then proceeded to try to call my plugin without the namespace I indicated. Following the example provided by MarkLogic, I used 'native' as the namespace and then proceeded to call it without the namespace like "regex-audf" instead of the correct way "native/regex-audf".

My second issue encountered was not paying attention to the Object Lifetime. What my plugin does is a bit different from the examples provided by MarkLogic in that I'm accumulating strings that match a regular expression, instead of a running tally of some sort.

Being over eager in trying to clean up after myself, I originally tried to delete the strings I accumulated in the close function of my AUDF. I quickly realized that wouldn't work and subsequently came across the following in the documentation: "The clone and close methods of your aggregate UDF may be called many times per job."

You can take a look at my code on GitHub. If you notice things obviously wrong, be kind, I've been out of C++ for a while so I could have easily missed something(s). :)

Here are some performance numbers using my plugin:

cts:values fn:matches
PT0.005568S
regex audf
PT0.001605S
cts:values reverse fn:matches
PT0.00506S
reverse regex audf
PT0.026693S

You will notice that right now my 'reverse regex audf' which identifies regular expressions in a lexicon that match a passed parameter, is actually much slower than using fn:matches. Perhaps I will need to create my own Custom Allocator to get around that issue. So much to do, so little time...

Memory Updates By Passing Function References

2012-09-21T20:26:00.001-06:00

I was so excited about the new ability to pass function references I had to add use it in my XQuery-XML-Memory-Operations library that I created. The version is now at 0.5.0, which only works with MarkLogic 6. If you still need it to work with older versions of MarkLogic you can use the v0.4.2 tag.

You will also notice that mem:copy($node) now returns a string which is your transaction id. The transaction will need to be passed as the first parameter to any operations you want to perform and then to the mem:execute($transaction-id) function when ready. It actually looks pretty slick when used with the map operator (!).

xquery version "1.0-ml";
import module namespace mem = "http://maxdewpoint.blogspot.com/memory-operations" at "/memory-operations.xqy";

declare variable $test-xml := <html>
<head>
<title>This is a title</title>
</head>

<body>
<div id="div1">
<p class="p1">This is a paragraph.</p>
<p class="p2">This is a paragraph.</p>
<p class="p3">This is a paragraph.</p>
<p class="p4">This is a paragraph.</p>
<p class="p5">This is a paragraph.</p>
</div>
<div id="div2">
<p class="p1">This is a paragraph.</p>
<p class="p2">This is a paragraph.</p>
<p class="p3">This is a paragraph.</p>
<p class="p4">This is a paragraph.</p>
<p class="p5">This is a paragraph.</p>
</div>
</body>
</html>;

mem:copy($test-xml) !
(
mem:transform(.,$test-xml/head/title,function($node as node()) as node()* {element new-title {"This is so awesome!"}}),
mem:execute(.)
)

More MarkLogic 6 Goodness

2012-09-19T06:27:00.001-06:00

String Concatenation

You can now do string concatenation with double pipes (||).

Map Operator

The map operator (!) is added. This allows for simple iteration over atomic types much like the / in XPath with nodes.

Default Values For External Variables

Now you can provide a default value for external variables.

Clark Notation For Calling Functions

This allows you to provide a fully qualified namespace instead of having to declare a namespace. For example, Q{http://marklogic.com/xdmp}functions() instead of xdmp:functions()

Fun With Functions

Speaking of xdmp:functions() that is a pretty cool function that will return all functions that are in scope.

Below I provide an example that makes use of these new found features.

xquery version "3.0";

(: Default vaules for external variables :)
declare variable $item external := ();

(: Specify the namespace with Q followed by clark notation :)
Q{http://marklogic.com/xdmp}functions() (: xdmp:functions returns all functions in scope :) !
(: Support for the map operator :)
element function {
element name {fn:function-name(.)},
element arity {fn:function-arity(.)},
element describe {Q{http://marklogic.com/xdmp}describe(.)}
}

MarkLogic 6.0 Released

2012-09-17T07:41:00.000-06:00

Highlights include the following:

Now can specify the XQuery version as "3.0" (Note that not all of XQuery 3.0 is supported yet and seems to be the same as "1.0-ml" at this point.)
Many new XPath functions added from the XPath 3.0 specs.
Now have the switch case expression available for comparing atomic types
Inline functions/Dynamic function calls (This is huge! Especially when coupled the higher order XPath functions added.)
Try/Catch statements that align with the XQuery 3.0 spec.
Aggregate User Defined Functions (Can't wait to start digging into that!)

This was a really great release for MarkLogic. I'm excited to get playing around with these new features. I must say MarkLogic caught me off guard. I was hoping to have more XQuery 3.0 coolness ready before they made a release like this.

A few things to look forward to in the future:

Some XQuery 3.0 FLWOR love

Generate unique IDs for the collision paranoid

2012-08-30T07:37:00.001-06:00

There were posts recently to the MarkLogic Mailing List about avoiding collisions of ids in your database. The focus of the posts were about checking for items with the same id in the database. My argument is that rather than use something collision apt as xdmp:random, generate something more unique.

So I came up with the following using xdmp:request, fn:current-dateTime, and fn:generate-id concatenated and passed to xdmp:sha1. This provides uniqueness based off of, a request, a moment in time, and a node in memory.

xdmp:sha1(
fn:concat(
(: Unique per request :)
xdmp:request(),
(: Unique per moment in time :)
fn:current-dateTime(),
(: Unique per node in memory :)
fn:generate-id(text {()})
)
)

If you feel that it is still necessary, you can use fn:doc-available to determine whether or not a document at that location (if that is how your database works) exists already or not. I think you will find a collision very unlikely however, with 16^40 different possible ids.

Static checking a library module

2012-08-28T07:06:00.000-06:00

I was very excited to come across the static check option for xdmp:eval. I immediately envisioned a firstline of defense against bad code reaching your code base. I was equally disappointed to see the following error when I tried it on library module: XDMP-EVALLIBMOD. I understand why you wouldn't be able to actually evaluate a library module, but there should be no reason you couldn't static check it.

I didn't let that stop me. It is really simple to change a library module to a main module for the purpose of static checking. Here are the differences between a main module and a library module:

1. The main module doesn't have a module declaration. http://www.w3.org/TR/xquery-30/#id-module-declaration
2. The functions of a main module must be in the local namespace.
3. The main module has a query body beyond the prolog.

With a little fancy footwork we can make use of fn:concat and fn:replace to convert a library module to a main module for the sake of static checking.

xdmp:eval(
(: if module declaration exists change it to a main module:)
if (fn:matches($module,'(^|[ ])\s*module\s+namespace\s+.+?=.*?;'))
then
(: retrieve namespace name for module :)
let $namespace := fn:replace($module,'.*(^|[ ])\s*module\s+namespace\s+([^\s=]+)\s*=.*','$2','s'),
(: check module to see if a default function namespace declaration already exists :)
$has-default-namespace := fn:matches($module, '(declare\s+default\s+function\s+namespace\s+["''])([^"''])*(["''])')
return
(: add empty sequence for query body :)
fn:concat(
(: change function calles to local namespace :)
fn:replace(
(: change function declarations to local namespace :)
fn:replace(
(: change module namespace declaration to just a namespace declaration :)
fn:replace(
$module,
'(^|[ ])\s*module\s+(namespace\s+.+?=.*?;)',
(: if a default function namespace declaration doesn't already exist set it to the local namespace :)
fn:concat('$1 declare $2', if (fn:not($has-default-namespace)) then ' declare default function namespace "http://www.w3.org/2005/xquery-local-functions";' else ())
),
'(^|[ ])\s*declare[\s ]+function[\s ]+([^\s ][^:\(]+:)?([^:][^\(]+\()',
'$1declare function local:$3'
),
fn:concat('(^|\s)(',$namespace,':)'),
if ($has-default-namespace) then '$1local:' else ''
),
' ()'
)
else $module,
(),
<options xmlns="xdmp:eval"><static-check>true</static-check></options>)

The many ways to detect an empty sequence

2012-08-15T07:54:00.001-06:00

I was recently thinking about the different ways that an empty sequence could be detected in XQuery. There were 7 ways that came to my mind. I thought I would compare the different ways performance wise. Unfortunately/fortunately there was no drastic differences. Profiling on the various functions I wrote, the elapsed times shifted around enough between the functions enough that there was no clear benefit of using one over the other.

Anyway, here is the code I ran in QConsole if you want to play around yourself (Best to have results returned as text in its current form.). Each of these functions just iterate over a set and returns exactly what was passed in. If anything, it is nice to know you have options and there are many ways things can be accomplished in XQuery.

If you come up with other ways please feel free to share.

xquery version "1.0-ml";

declare namespace prof = "http://marklogic.com/xdmp/profile";

declare variable $set as item()+ := (1 to 25);

declare function local:empty($items as item()*) {
if(empty($items))
then ()
else ($items[1],local:empty(subsequence($items,2)))
};

declare function local:exists($items as item()*) {
if(exists($items))
then ($items[1],local:exists(subsequence($items,2)))
else ()
};

declare function local:typeswitch-empty($items as item()*) {
typeswitch($items)
case empty-sequence() return ()
default return ($items[1],local:typeswitch-empty(subsequence($items,2)))
};

declare function local:typeswitch-exists($items as item()*) {
typeswitch($items)
case item()+ return ($items[1],local:typeswitch-exists(subsequence($items,2)))
default return ()
};

declare function local:instance-of-empty($items as item()*) {
if ($items instance of empty-sequence())
then ()
else ($items[1],local:instance-of-empty(subsequence($items,2)))
};

declare function local:instance-of-exists($items as item()*) {
if ($items instance of item()+)
then ($items[1],local:instance-of-exists(subsequence($items,2)))
else ()
};

declare function local:inferred-boolean($items as item()*) {
if ($items)
then ($items[1],local:inferred-boolean(subsequence($items,2)))
else ()
};
prof:enable(xdmp:request()),
"local:empty",
prof:value('for $i in (1 to 1000) return local:empty($set)')[1]/prof:metadata/prof:overall-elapsed/string(),
"local:exists",
prof:value('for $i in (1 to 1000) return local:exists($set)')[1]/prof:metadata/prof:overall-elapsed/string(),
"local:typeswitch-empty",
prof:value('for $i in (1 to 1000) return local:typeswitch-empty($set)')[1]/prof:metadata/prof:overall-elapsed/string(),
"local:typeswitch-exists",
prof:value('for $i in (1 to 1000) return local:typeswitch-exists($set)')[1]/prof:metadata/prof:overall-elapsed/string(),
"local:instance-of-empty",
prof:value('for $i in (1 to 1000) return local:instance-of-empty($set)')[1]/prof:metadata/prof:overall-elapsed/string(),
"local:instance-of-exists",
prof:value('for $i in (1 to 1000) return local:instance-of-exists($set)')[1]/prof:metadata/prof:overall-elapsed/string(),
"local:inferred-boolean",
prof:value('for $i in (1 to 1000) return local:inferred-boolean($set)')[1]/prof:metadata/prof:overall-elapsed/string()

Performance Enhanced i18n Library for MarkLogic

2012-08-08T21:00:00.000-06:00

When creating an application, it is probably a good idea to think ahead to internationalization. This can be achieved with key/value pairs stored your MarkLogic database. MarkLogic is fast, but if the number of strings you use grows to be substantial enough, the cost of retrieving these strings from the database for each request can become expensive. This why I created a library for dealing with the i18n strings in a efficient way.

The first step that I take for performance is to store the values in map:map xml in the database. This xml can be loaded into a map:map object which will give you great performance when looking for values based off of a key because it works much like a hash table look up (I imagine that is what is happening behind the scenes.). This can be substantially faster than just using a predicate to filter based off of an attribute value.

map:map(bundle($locale, $bundle,$query)/map:map)

The second step is to retain that map loaded into memory for any other calls to obtain values from your i18n bundle. This way the first call to the bundle takes the brunt of the execution time, while lookups that follow are ready to go in memory.

map:map(bundle($locale, $bundle,$query)/map:map)[map:put($current-bundles,$base-key,.),fn:true()]

The final step is to make use of server fields. Server fields are values that are shared between all requests to a server in your MarkLogic cluster. There is no reason that you should need to load your bundle with every request when there are no changes made to your i18n bundle. So I pull bundles from the database and store them into a server field only when changes have been made to the bundle. This increases performance by not only having the items already in memory and decreases memory usage by having one map:map object per bundle shared between requests instead of having a map:map object per bundle per request. In my implementation I use the uri lexicon and the MarkLogic last-modified property.

(: This loads a bundle into memory. :)
declare function load-bundle($locale as xs:string, $bundle as xs:string, $query as cts:query?) as map:map {
(: Calculate the key for the bundle which changes with every update to the bundle :)
let $base-key as xs:string := fn:concat('/resources','/',
(:As of MarkLogic 4.2 the result of cts:register is always the same, provided the same query. So in this case we can use it to generate a unique id :)
if (fn:exists($query))
then fn:string(cts:register($query))
else 'default',
'/',$locale,'/',$bundle)
return
if (fn:exists(map:get($current-bundles,$base-key)))
then map:get($current-bundles,$base-key)
else

let $timestamp-key as xs:string := fn:concat($base-key,'/timestamp'),
$timestamp as xs:string := fn:string(xdmp:get-server-field($timestamp-key))
(: Try to retrieve bundle from memory. :)
return
(: If it exists in memory in a server field return it. :)
if (fn:exists($timestamp) and fn:exists(cts:uris((),('limit=1'),
cts:and-query((
cts:element-query(
xs:QName('i18n'),
cts:and-query((
cts:element-attribute-value-query(xs:QName('i18n'),xs:QName('locale'), $locale, 'exact'),
cts:element-attribute-value-query(xs:QName('i18n'),xs:QName('id'), $bundle, 'exact')
))
),
$query,
cts:properties-query(
cts:element-value-query(xs:QName('prop:last-modified'),$timestamp,'exact')
)
))
)))
then xdmp:get-server-field($base-key)
(: Otherwise load it. :)
else (
xdmp:set-server-field($base-key,map:map(bundle($locale, $bundle,$query)[xdmp:set-server-field($timestamp-key,property::prop:last-modified)]/map:map))[map:put($current-bundles,$base-key,.),fn:true()]
)
};

Checkout the library on GitHub.