Tuesday, January 8, 2013

Automatic remapping of Windows-1252 characters to Unicode

Nowadays, almost everything uses Unicode, which supports a large (very large!) number of characters. Unicode assign a code (number) to each character, and in most cases this code is represented with:

  • 2 bytes when in memory (e.g. in Java and Windows since NT);
  • A variable number of bytes when sent over the wire (typically UTF-8 encoded).

Before Unicode, a single byte per character was used for Western languages, with the ISO Latin-1 encoding. This encoding was fine for most Western characters but didn't contain some useful characters, such as curved quotes, the euro sign, the trademark sign, plus many others. In their infinite wisdom, Microsoft decided to use some reserved codes of Latin-1 for those "useful" characters, creating the Windows 1252 encoding.

Unicode is based on Latin-1, and not Windows-1252, which means that the code for all those "useful characters" is higher than 255 in Unicode. The problem is that documents encoded with Windows-1252 are often incorrectly opened as Latin-1. The mistake is easy to do, as it works "in most cases". The error is so common, that the HTML5 spec says that a browser should parse document advertised as using Latin-1 as Windows-1252 (not trusting the advertised encoding!).

But should you incorrectly take that Windows-1252 encoded file as a Latin-1 encoded file, and pass along its content to a system where it is saved as Unicode, you might end up with control characters; if the text is sent back to the browser, still as Unicode, those control characters will show as squares, instead of the curved quotes, euro sign, or trademark signs you originally intended.

Luckily, there is a way to safely and automatically fix incorrectly encoded documents. This is done by changing the code for characters that exist in Windows-1252, but not Latin-1, to their valid Unicode code, using a simple conversion table. And Orbeon automatically does this for you. And of course, configuration properties allow you to disable this automatic conversion, or even to setup your own custom conversion.

Wednesday, December 12, 2012

Orbeon Forms 4.0 M19

Today we released Orbeon Forms 4.0 M19 (Milestone 19). Like 4.0 M18 and the previous milestones this is not a final release.
In this release, the following issues were addressed:
  • Form Builder
    • Copy/paste of control loses translation (#625)
    • Enable XPath type annotations by default in Builder (#630)
    • When creating a new form, dialog shouldn't show a 'cancel' link (#590)
    • Form settings: textarea is not tall enough (#622)
    • Setting help to blank doesn't remove help (#577)
    • Custom XML: focus taken out of XML editor on click (#642)
    • Custom XML: dialog is too narrow, not high enough (#643)
    • Should not be able to set the focus on a readonly CodeMirror (#645)
    • Regression: Action editor: input/output values can't be set (#647)
    • 403 error when testing form containing an XPath error (#627)
    • Wizard view by default for orbeon/controls
    • Make fields required in Action Editor
  • Form Runner
    • Required asterisk doesn't show in dialogs (#639)
    • fr:number is now documented (#546)
    • Add fr-grid-col-<x> classes on cols and fr-grid-<id> on table (a4bcef1766)
    • More files converted to LESS
  • XForms engine
More information is available in the in-progress release notes for 4.0.

You can download the builds using these links:
Don't forget to grab a trial license for the PE version.

Please send feedback:

Monday, December 10, 2012

Creating REST services with Google Apps Script

There are times when you'd like to create a service, want the service to be available publicly, and want to do this quickly, maybe for testing or for a demo you're putting together. In those cases, Google Apps Script might just be the solution you're looking for.

As an example, let's create a service that tells us if a number it receives is even. The number will be passed as a request parameter, e.g. ...?number=42, and will provide an XML response, e.g. <result>true</result>. Then, we will call this service from a form created with Form Builder, to show, next to an input field, whether the typed number is even or odd. Let's start by creating and deploying the service:

  1. To create a new script, load Google Drive, click on Create, and under More, choose Script. Click on Untitled project, and name it IsEven.

  2. Edit your script, or in this case copy-pate the following code in the editor:

    function doGet(request) {
        var result = '' + (request.parameter.number % 2 == 0) + '';
        return ContentService.createTextOutput(result)

  3. To deploy your service, make sure it is first saved, then go to File | Manage Versions, click on Save New Version, and click OK to close the dialog. Go to Publish | Deploy as web app…, in Who has access to the app choose Anyone even anonymous, click Deploy. Copy and save somewhere the URL given to you in the following dialog: this is the URL of your script.

  4. Test the service by pasting the URL in a new tab of your browser and adding ?parameter=42. The service should respond <result>true</result>.

Now let's call the service from a form we create with Form Builder:

  1. In Form Builder, create a new form, create an input field, type a label and name it number, create and output field, type a label and name it even. Your form might look like:

  2. Define an HTTP service, by clicking on Add under HTTP Service in the sidebar. Name the service is-even, in Resource URL paste the URL from step 3 above. Under Serialization, choose HTML form. In Request Body enter <number/>.

  3. Define a action, by clicking on Add under Actions in the sidebar. Name it check-even, under React To choose Value Change, under Control choose number, in Service to call to choose is-even. In Set Service Request Values click the plus icon choose number and type /number, in Destination Control choose even and type /result.

  4. Test the form by clicking on the Test button. Type 42 and hit enter: the output next to it should show true. Type 43 and hit enter: similarly the output should show false.

Congratulations, you just created a service and a form calling that service, and all this without leaving your web browser.

Tuesday, December 4, 2012

Orbeon Forms 4.0 M18

Today we released Orbeon Forms 4.0 M18 (Milestone 18). Like 4.0 M17 and the previous milestones this is not a final release.
In this release, the following issues were addressed:
  • Form Builder
    • Form Builder should use logo specified in oxf.fr.default-logo.uri (#616)
    • Permissions dialog layout scrambled (#570)
    • Hover over button doesn't show proper button editor (#553)
  • Form Runner
    • FB summary page doesn't show app/form name for new forms with MySQL persistence layer (#611)
    • eXist persistence not to throw an error when the database is empty (#614)
    • Form Runner home page: message for no forms, include forms without permission (#94)
    • Don't highlight title on hover (#615)
    • Lots of CSS and cosmetic improvements
    • fr:number focus in/out loses formatting (#586)
    • Wizard: better navigation buttons
  • XForms engine
    • Server sets index of todo instead of list when re-adding first list (#591)
    • Non-visible dialog content should be non-relevant (#15)
    • Tooltip for a trigger hint in a dialog shows below the dialog (#620)
    • Add deferred flag to xf:rebuild/xf:revalidate actions
    • Move xforms.css to LESS
    • Refactoring of control bindings support
More information is available in the in-progress release notes for 4.0.

You can download the builds using these links:
Don't forget to grab a trial license for the PE version.

Please send feedback:

Form Runner Wizard View

One of the cool new features of Orbeon Forms 4.0 is the wizard view.

By default with Form Runner all the form sections appear in the same page, on top of each other. If your form is large that means that you have to scroll to fill out the entire form.

With the wizard view, top-level sections instead appear in a navigation area to the left, and only a single top-level section is shown at any given time.

You can navigate between sections by clicking on a section title, or you can use the navigation arrows.

Errors on your form appear at the bottom as usual, and sections that contain errors are highlighted in red. If you click on an error you are taken directly to the section and control containing the error.

The wizard view is optional - you can use the regular view instead, and you can enable this view per form, per app, or globally with a property:
We hope you enjoy this new feature!

Wednesday, November 28, 2012

Simplify your queries with Oracle's dense_rank

Data persistence for forms created with Form Builder is done through a REST API. You can implement that API, or use one of the provided implementations. Orbeon's implementation for relational databases, for now specifically MySQL and Oracle, always uses inserts instead of doing updates or deletes1. For this, tables have a timestamp column, and when data is "updated", a new row is added with the current timestamp. So to retrieve data, we need to get the row matching a condition that has the latest timestamp.

Let's transpose this problem to an equivalent one on the emp table, to make it easier for you to run the queries below in your own database2. On the emp table, a similar problem is to find for each job, the employee with the most recent hire date3. This can be done with nested queries:
select e1.* 
  from emp e1,
       (  select job, max(hiredate) hiredate 
            from emp 
        group by job ) e2
 where e1.job = e2.job
       and e1.hiredate = e2.hiredate ;
A bit complicated isn't it? This is where Oracle's dense_rankcomes in. This analytic function allows you to rank data. The following adds a rank column, and for each job, the employee with the latest hire date will get a rank of 1, the employee hire just before that a rank of 2, and so on.
select e.*,
    dense_rank() over (partition by job order by hiredate desc) rank
from emp e;
From here, we just need to keep only the employee with rank = 1 to get the expected result:
select * from
    (select e.*,
            dense_rank() over
            (partition by job order by hiredate desc) rank
       from emp e)
 where rank = 1 ;

1 This allows auditing and allows an admin to retrieve older version of the data if necessary.
2 In Oracle, this table is normally found in the scott schema, and you don't have it, or would like to import it in another database, you can download the SQL to create the table and add data.
3 For the purpose of this discussion, we'll assume there is only one employee for each job with the most recent hire date.

Orbeon Forms 4.0 M17

Today we released Orbeon Forms 4.0 M17 (Milestone 17). Like 4.0 M16 and the previous milestones this is not a final release.

Due to the Thanksgiving break, M17 is a rather small update, but it happens to contain two important new bits:
  • a new landing page for the Form Runner and Orbeon Forms  examples (#165)
  • a new "wizard" mode which allows showing forms one section at a time
In addition, the following issues were addressed:
  • Form Builder
    • No default button set in some dialogs (#571)
  • Form Runner
    • fr:alert-dialog with static label crashes (#607)
    • fr:error-summary: improved recalculate and use xxforms-visited/unvisited (ab3ff7f890)
  • XForms engine
    • xxforms:value() function (1021d5eade)
    • New `focusable` mode for XBL components (#526)
    • TinyMCE with required content doesn't reliably show error (#301)
    • Change of visited state requires refresh (84a13e93c8)
    • New xxforms-visited/unvisited events (f6c02b4377)
    • Add deferred flag to xf:recalculate action (3339a5c2c2)
More information is available in the in-progress release notes for 4.0.

You can download the builds using these links:
Don't forget to grab a trial license for the PE version.

Please send feedback: