BaseX 6.6: XMLPrague Release

Thu, 24 Mar 2011 00:27:11 +0100

Just in time for the XMLPrague conference, we are proud to announce the release of BaseX 6.6. It offers many new features, such as:

full support for Higher Order Functions, introduced with XQuery 3.0 (dynamic function invocation, inlined functions, etc.),
native support of the EXPath ZIP and HTTP specifications and other new XQuery modules,
a new COPY Command to clone existing databases,
revised CSV and text importers for different encoding and XML formats,
additional XQuery 3.0 features (decimal-format and context item declarations, format functions, etc.), and
numerous speedups, reduced memory consumption and stability fixes for XQuery and XQuery Update operations.

Feel free to download our new release, or check out our Project Page at GitHub for the latest sources.

Christian, BaseX Team

BaseX 6.5 Review

Thu, 27 Jan 2011 15:04:00 +0100

A very active BaseX user took the time to write a review about BaseX 6.5 and related changes. Thanks to Jan Vlčinský!

Background Information:

Topics :

Installation
Storage: Text Compression, Backups, Import
XQuery 3.0: Group By, Try/Catch, Switch, Serialization
Built-In Functions: EXPath, HTTP, DB, Fulltext
GUI: Views, General Features and Hints
API: Language Bindings
File Associations

Review

“ Goal of this article is to find out what the new version 6.5 of BaseX brings and to make sure that we can safely use it our project. In addition, I want to learn some basic GUI skills.

The new website of the project looks very nice. Being completely redesigned it offers much more information in a clearer way. Also worth to mention is the short video tutorial that shows first steps with the GUI mode of BaseX. Although the old documentation was already usable, the new Wiki seems much more matured.

Now What’s New?

Installation

In contrast to BaseX 6.4, the new version 6.5 comes as a bundled download, containing BaseX GUI and BaseX Server, as well as the available APIs, REST server, sample documents and starting scripts for popular platforms (exe, dmg, zip). Using windows, an installer guides through the installation process. A simple JAR file is also included and serves as entry point for the individual scripts.

Storage

Text Compression reduces the size of text nodes up to 50%. Although I haven’t thoroughly tested it, my experience shows that disk space requirements have already been very reasonable in previous versions. In my case, a database consumes approximately the same amount of disk space as the imported XML files. For some cases the disk requirements double, depending on indexes and overall document characteristics.

Enhanced Collection Support means that documents can be stored in folders and subfolders. It was already working well when I tested it with version 6.4. For example, the following query returns all documents folder1 contains:

collection(“hierarchicCollection/folder1”)

Backups Using the GUI this feature can be found under Database/Manage. It’s also possible to call the Backup Command. A zip archive that contains the database is created without prompting and stored at the location of the database folder. Restoring works pretty much the same way. The most recent database backup (timestamp) is restored - again without any prompting. At first this seemed surprising, but I like the simplicity and straight-forward manner of the approach. In the end it seems very well usable.

Import Of CSV, HTML And Text enables the user to create new databases from CSV and Text data out of the box. Select the required input format in the ‘Parsing’ tab of the ‘Create Database’ dialog. The ‘SET PARSER’ command can be used to import data like this from a command line. At the moment tab, comma, and semicolon work as field separators. There’s also no need for column headers. If a CSV file contains headers, they are stored in the first record. Plain text is imported in a way that each individual line is stored as a text node child of a ‘line’ element. All in all the import function is very useful as it creates an internal XML representation and enables the user to explore and search the data using XQuery or one of the interactive visualizations.

Below are a few examples for imported CSV and Text data.

date
   num


   2009-04-22
   3


  Dear all,

  We are excited to announce the
   relaunch of our homepage, which now
  includes commercial offerings such as
   professional support, individual
  software solutions, and training courses:

   http://basex.org/

XQuery Related

XQuery 3.0 (former XQuery 1.1) is still a working draft and not yet fully implemented in BaseX. For a list of currently supported functionality visit the documentation.

Group By the ‘Group By’ clause extends the FLWOR expression and has been a long-awaited feature among real-life users. I tested its functionality without stumbling over any problems. I would recommend to read the specification of ‘Group By’ first, as there are some specialties that seem odd at first.

Try/Catch For example, the try/catch block comes in handy in functions and for type related operations.

let $txtval := “123.15”
let $intval := try {
xs:integer($txtval)
} catch * {0}
return $intval

Switch A switch construct replaces bulky if-else chains in an elegant way. For example:

let $animal := “Fish”
return
switch ($animal)
case “Cow” return “Moo”
case “Cat” return “Meow”
case “Duck” return “Quack”
default return “What’s that odd noise?”

Serialization Serializing XML to text works very well. In the past I faced problems that lines were starting with an extra space. This issue seems to be resolved. In the following example I create a CSV file using semicolons as field delimiter:

(: Using the XML created by importing CSV file:)
declare option output:method “text”;
for $line in //record
let $date := $line/field[@col = 0]
let $num := $line/field[@col = 1]
return text {concat($date, “;”, $num, codepoints-to-string(10))}

New And Modified Functions XQuery 3.0 offers some really handy functions. Visit the BaseX wiki and the specification for a complete overview. Below are some functions that I find very useful:

get first, get all but first
# fn:head()
# fn:tail()
element id generation
# fn:generate-id()
access environment
# fn:environment-variable()
# fn:available-environment-variables()
quick “dir” for documents
# fn:uri-collection()
formating values into readable text
# fn:format-integer()
# fn:format-number()
# fn:format-dateTime()
# fn:format-date()
# fn:format-time()

New Built-In Functions

File The EXPath module has been implemented which offers functions to perform file system related operations. Included are functions to read, write and access the file system. A list of the complete functionality including a short explanation can be found in the wiki. All functionality is subject to change as the specification is still a W3C Candidate Module.

HTTP Another novelty is the HTTP client module that contains an XQuery function to send HTTP requests and handle responses. As this function is quite young I’m yet to be fully convinced by this feature.

DB There’s a new module that allows to perform general database functions. An overview is listed here. I still miss a way to delete single nodes one by one. E.g. a document is stored multiple times if I insert it repeatedly. But the only way to delete this document is to delete it together with all documents that have the same name.

Fulltext Although I don’t use the full text functionality myself, there are some new functions worth to mention. The W3C Full Text recommendation is extended by functions to mark results, extract relevant full text sections and explicitly access indexes and score values. An additional feature is the built-in Lucene and Snowball Stemming for XQuery Full Text. Again, consult the documentation fur details.

GUI

The GUI has already been excellent in older versions and got even better. Each view is shortly presented, although all of them have already been part of previous versions.

XQuery Editor For me this is the best environment to write XQuery code I have found so far. I also own XMLSpy, but BaseX is by far better. While typing it tells me the exact location of any error in my query.
Text View The text view resides on place two of my most used visualizations. Database contents and XQuery results are displayed here. When I click on the home button and a database is opened, the view shows the original input XML data. There’s also a quick search option. I would like the Folder View to display query results. I am aware that it sounds easy but is a complicated task at the same time.
Map View I call the Map View ‘a tree with rectangular branches’. It looks impressive but I don’t use it. Although it might be useful for closer inspection of the data, as the number of displayed nodes is reduced.
Tree View This is a real tree visualization with the root being placed at the top.
Folder View I like this one a lot. People who like Microsoft Internet Explorer XML representation with folding option will also find this very useful.
Table View Being new to me it looks very promising but I still have to learn to use it to my advantage. I would like to control the table columns in some way.
Plot View This is a great tool if you want to gain some quick insight into your data. It requires some thinking at first, but then it is a very efficient tool for visualization.
Explorer View The Explorer View allows for some simple filtering without the use of XQuery.

General features and hints

Context Menu There lies great value in using your right mouse button. Call ‘Copy Path’ from the context menu on any node to get an XPath location path for the selected node - which is really handy for writing queries. This feature is available in most views.

Navigate The Document Tree Navigate into subtrees by clicking on nodes. Navigate back up via the ‘Back’ button in the menu bar, using backspace or the context menu. To reset all views press the ‘Home’ button.

I still miss an option to select and delete a particular document from a collection. So far I have to use XQuery and enter the document name manually. Being already very useful, some more instructions would greatly help to release the full potential of the GUI.

Conclusion In conclusion, the GUI is the best place to explore heaps of XML data (and more: CSV, Text, …). It features short responsive times and is well designed. There are really a lot of options to evaluate XML data and its atoms as well. Yet the documentation is kind of short, so there are always new gimmicks to be found.

API

Although I personally use the Python binding the API seems to have evolved since the last version. Binding queries to variables now allows parametrical queries. The wiki shows some examples with PHP. Most APIs offer about the same functionality, I guess, which includes creation of a session, creation of queries, binding a variable to a query, execute a query and fetch results in an iterative manner, if required.

I haven’t yet tested the Python binding with the actual release. But I expect it to be very usable like with older releases. Note, that Python uses one global socket which might lead to problems with concurrent sessions.

Installation

EXE-Installer I provide a quick overview on the installation process using Windows and the new EXE Installer.

File Associations Be aware that the installer associates all XML and .xq files with BaseX by default. I played around with the associations and it seems very promising and handy. If I open an XML file BaseX creates a database with the name of the input file. In case such a database already exists, it is silently overwritten. My recommendation would be to accept file associations as it is a very comfortable way to explore XML files. It’s also much faster than using the Internet Explorer.

To protect other databases I just use some naming convention - two underscores for databases that I want to keep safe are sufficient, as common XML files do not use underscores. This way, nothing is overwritten accidentally. If I want an imported XML document to become my long term database, I simply rename it via Database-Manage-Rename.

When it comes to .xq files (XQuery) BaseX is simply the best editor I have found so far. These files are opened in a new BaseX window where one can edit and save them subsequently. If you want to test the query, just open the corresponding database. By default, BaseX is only associated with .xq files - I would recommend to add .xquery files manually.

Summary

The release of BaseX 6.5 and all related changes including the website and commercial offerings show, that the authors take their work very seriously.

One finds great support in the mailing list! It’s a bit slower than google search (although not much), but provides real customized answers to your problems.

I think BaseX features a great GUI to explore XML and develop XQuery, plus a server, that is capable of handling lots of data, yet consuming a reasonable amount of ressources. In addition, the API allows to use BaseX through many different programming languages. ”

BaseX 6.5, Homepage Relaunch

Thu, 27 Jan 2011 10:50:33 +0100

Dear all,

We are excited to announce the relaunch of our homepage, which now includes commercial offerings such as professional support, individual software solutions, and training courses:

http://basex.org

Next, we have released Version 6.5 of BaseX, which offers the following new features:

= Storage
- Text compression, shrinking text nodes up to 50%
- Enhanced collection support
- Backup functionality
- Built-in importers: HTML, Text, CSV

= XQuery
- XQuery 3.0 features, including GROUP BY and SWITCH statements
- New file, HTTP, database and full-text functions
- Built-in Lucene and Snowball Stemming for XQuery Full Text

= GUI
- New TreeView visualization
- Improved database management

= APIs
- JAX-RX now supports hierarchical paths and external variables
- Language Bindings extended with query iterators

You can be sure that the project itself will remain Open Source.

We are looking forward to your responses,
Christian

BaseX – The XML Database

BaseX 6.6: XMLPrague Release

BaseX 6.5 Review

Review

Now What’s New?

Installation

Storage

XQuery Related

New Built-In Functions

GUI

General features and hints

API

Installation

Summary

BaseX 6.5, Homepage Relaunch