Digital Sonata's Blog

Workaround for conversion of Unicode Vietnamese to ANSI

2012-04-27T18:58:00.000+10:00

A few months ago, we had an interesting project which involved Vietnamese. As a part of the project, we have hit a minor snag.

It looks like the first support for Vietnamese appeared in 1990s, which means the ANSI codepage was built in a hurry already when Unicode was either in the works or already out. The tonal nature of the language demands complex combinations of diacritics to be used. Not just regular grave accents, umlauts and such. Simply put, there were not enough slots for the new characters, so the creators of the Vietnamese codepage stuffed these new characters wherever, in whatever way possible.

Today very few people use ANSI, however, it is still needed for several reasons: legacy being one (people still work with mainframes, you know), and compliance another. Of course, there is a tried and true function WideStringToMultiByte which works like a Swiss chronometre. That is, for most languages - except Vietnamese. There are posts by Microsoft folks stating that "Vietnamese is a complex language on Windows" (duh!), but not really telling how to fix it. I asked around, no one replied, as expected (I love how Stackoverflow people react when they can't answer the question :-) ).

After scrutinising the result I saw what's the problem. It seems that the decomposing routine during the conversion is unable to handle some combinations of Unicode characters. Manually decomposing some characters to their equivalents worked for me.

I use an esoteric language called Clarion to design our tools and some components, so my original code is in Clarion. A few days ago, Mark Jacobs from Critical Research contacted me requesting help with the same issue, and kindly converted my Clarion source code to C++ more familiar to the rest of the world. Thanks, Mark!

Get Clarion source code here and Mark's C++ here.

LinguaSys - USA Today

2010-12-10T16:54:00.005+11:00

An article about an Android app built by our partners in LinguaSys using Carabao as a backend (among others), is published in USA Today and other news sources.

ALTA 2010

2010-12-06T14:03:00.005+11:00

Digital Sonata was invited to present at the Australian Language Technology Association 2010 workshop on Thursday, December 9, 2010. Overview and directions are here.

Vadim Berman will be speaking some time between 3:30pm and 5:30pm.

Digital Sonata Signs Long Term Exclusive Agreement with LinguaSys™ For Use Of Carabao For Machine Translation

2010-06-07T10:24:00.003+10:00

Digital Sonata signed a long term deal with LinguaSys™ for the exclusive use of Carabao in machine translation (MT) solutions on March 24, 2010.

Carabao is a hybrid language translation system using both statistical and rules-based methodologies. Vadim Berman, CEO of Digital Sonata in Australia, and Chief Technology Officer and a co-founder of LinguaSys, is the author of Carabao. Berman has a wealth of dedicated experience in the field of MT and text analysis.

Brian Garr, CEO of LinguaSys said, “We are very excited that this incredible technology from Digital Sonata will help us create the next generation of language translation solutions.”

LinguaSys is a new next generation machine translation company. LinguaSys’ Carabao language middleware uses language processing methodologies offering excellent comprehension in the least amount of time at low cost. LinguaSys enables enterprises to translate volumes of information, including text chat, e-mail, web pages and documents, quickly, accurately and automatically. LinguaSys provides the creation of new MT languages, customized lexical services, ease of use, compatibility with existing natural language software, security behind the firewall, availability, integration and lower memory requirements.

Carabao Language Kit 1.7.0.0 released

2010-05-03T12:58:00.003+10:00

The version 1.7.0.0 is now available for download.

Fixed:

Handling of control priority greater than 2, when some of the members have no feasible agreement graph. The result was, that some parts of the sequence worked, and some didn't.
Truncation of very long sentences

Added:

A utility to validate and correct rule unit values
A generic support for formatted processing, e.g. HTML, XML, SGML including embedded formatting elements in the text flow
GUI to test formatted processing in Carabao Test Console
Automatic conversion of double-byte space characters into standard single-byte

Improved:

Regular expressions for segmentation into character classes for double-byte languages
Perl-compatible regular expressions have been introduced for unknown heuristics
Frequency-based backtracking added to the tokenization algorithm
Unicode clipboard support in Carabao desktop suites is now bidirectional: when leaving the application and when coming back to the application

Publication in Multilingual

2010-04-19T20:59:00.003+10:00

My article on evaluation of emerging language technologies was published in Multingual:

http://multilingual.texterity.com/multilingual/20100405?pg=59#pg64

English - Swedish OLIF dictionary released

2010-01-27T09:40:00.005+11:00

Engish - Swedish OLIF dictionary added to the list of OLIF lexicons distributed by Digital Sonata. The dictionary is available for download from http://www.digitalsonata.com/download.aspx?type=linguisticData.

Bilingual OLIF dictionaries released

2010-01-10T13:16:00.008+11:00

Digital Sonata released a set of low-cost royalty-free bilingual dictionaries in OLIF format, optimized for use in NLP and content management applications. Translation, part of the speech, and a thesaurus article is included. The dictionaries are available at http://www.digitalsonata.com/download.aspx?type=linguisticData. Currently the following dictionaries are available:

English -> Finnish

English -> French

English -> German

English -> Japanese

English -> Korean

English -> Russian

English -> Spanish

Carabao Language Kit 1.6.2.1 released

2010-01-05T19:38:00.004+11:00

The version 1.6.2.1 is now available for download.

Fixed:

Transliteration to empty string
Partial transliteration

Added:

Change log which allows distributed collaboration

Improved:

Processing speed
Entry matching accuracy

Running IIS7 on Windows Vista Home

2009-07-22T13:21:00.003+10:00

Many people are stumbled when trying to use their Windows Vista Home machines for web development involving IIS deployment. Usually the tricky part is the error message that ASPX pages are not handled. As the management console is not installed on some Vista editions, the workaround is to edit the configuration file directly.

The configuration file is C:\Windows\System32\inetsrv\config\applicationHost.config. Simply overwrite the <handlers> section with the following:

<handlers accessPolicy="Script, Read">
<add name="TraceHandler-Integrated" path="trace.axd" verb="GET,HEAD,POST,DEBUG" type="System.Web.Handlers.TraceHandler" preCondition="integratedMode" />
<add name="WebAdminHandler-Integrated" path="WebAdmin.axd" verb="GET,DEBUG" type="System.Web.Handlers.WebAdminHandler" preCondition="integratedMode" />
<add name="AssemblyResourceLoader-Integrated" path="WebResource.axd" verb="GET,DEBUG" type="System.Web.Handlers.AssemblyResourceLoader" preCondition="integratedMode" />
<add name="PageHandlerFactory-Integrated" path="*.aspx" verb="GET,HEAD,POST,DEBUG" type="System.Web.UI.PageHandlerFactory" preCondition="integratedMode" />
<add name="SimpleHandlerFactory-Integrated" path="*.ashx" verb="GET,HEAD,POST,DEBUG" type="System.Web.UI.SimpleHandlerFactory" preCondition="integratedMode" />
<add name="WebServiceHandlerFactory-Integrated" path="*.asmx" verb="GET,HEAD,POST,DEBUG" type="System.Web.Services.Protocols.WebServiceHandlerFactory, System.Web.Services, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" preCondition="integratedMode" />
<add name="HttpRemotingHandlerFactory-rem-Integrated" path="*.rem" verb="GET,HEAD,POST,DEBUG" type="System.Runtime.Remoting.Channels.Http.HttpRemotingHandlerFactory, System.Runtime.Remoting, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" preCondition="integratedMode" />
<add name="HttpRemotingHandlerFactory-soap-Integrated" path="*.soap" verb="GET,HEAD,POST,DEBUG" type="System.Runtime.Remoting.Channels.Http.HttpRemotingHandlerFactory, System.Runtime.Remoting, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" preCondition="integratedMode" />
<add name="AXD-ISAPI-2.0" path="*.axd" verb="GET,HEAD,POST,DEBUG" modules="IsapiModule" scriptProcessor="%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll" preCondition="classicMode,runtimeVersionv2.0,bitness32" responseBufferLimit="0" />
<add name="PageHandlerFactory-ISAPI-2.0" path="*.aspx" verb="GET,HEAD,POST,DEBUG" modules="IsapiModule" scriptProcessor="%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll" preCondition="classicMode,runtimeVersionv2.0,bitness32" responseBufferLimit="0" />
<add name="SimpleHandlerFactory-ISAPI-2.0" path="*.ashx" verb="GET,HEAD,POST,DEBUG" modules="IsapiModule" scriptProcessor="%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll" preCondition="classicMode,runtimeVersionv2.0,bitness32" responseBufferLimit="0" />
<add name="WebServiceHandlerFactory-ISAPI-2.0" path="*.asmx" verb="GET,HEAD,POST,DEBUG" modules="IsapiModule" scriptProcessor="%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll" preCondition="classicMode,runtimeVersionv2.0,bitness32" responseBufferLimit="0" />
<add name="HttpRemotingHandlerFactory-rem-ISAPI-2.0" path="*.rem" verb="GET,HEAD,POST,DEBUG" modules="IsapiModule" scriptProcessor="%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll" preCondition="classicMode,runtimeVersionv2.0,bitness32" responseBufferLimit="0" />
<add name="HttpRemotingHandlerFactory-soap-ISAPI-2.0" path="*.soap" verb="GET,HEAD,POST,DEBUG" modules="IsapiModule" scriptProcessor="%windir%\Microsoft.NET\Framework\v2.0.50727\aspnet_isapi.dll" preCondition="classicMode,runtimeVersionv2.0,bitness32" responseBufferLimit="0" />
<add name="ASPClassic" path="*.asp" verb="GET,HEAD,POST" modules="IsapiModule" scriptProcessor="%windir%\system32\inetsrv\asp.dll" resourceType="File" />
<add name="SecurityCertificate" path="*.cer" verb="GET,HEAD,POST" modules="IsapiModule" scriptProcessor="%windir%\system32\inetsrv\asp.dll" resourceType="File" />
<add name="SSINC-stm" path="*.stm" verb="GET,POST" modules="ServerSideIncludeModule" resourceType="File" />
<add name="SSINC-shtm" path="*.shtm" verb="GET,POST" modules="ServerSideIncludeModule" resourceType="File" />
<add name="SSINC-shtml" path="*.shtml" verb="GET,POST" modules="ServerSideIncludeModule" resourceType="File" />
<add name="ISAPI-dll" path="*.dll" verb="*" modules="IsapiModule" resourceType="File" requireAccess="Execute" allowPathInfo="true" />
<add name="CGI-exe" path="*.exe" verb="*" modules="CgiModule" resourceType="File" requireAccess="Execute" allowPathInfo="true" />

</handlers>

Done.

BlackBerry connection tip

2009-04-28T15:25:00.010+10:00

We developed a small client application for BlackBerry phones that connects to MiaMia servers. A lot of things can be said about BlackBerry adherence to standards, but I don't want to use cuss words here. The main problem is that these phones were initially devised to serve as "corporate walkie-talkies", and only recently their design was stretched to accommodate the needs of the general public. In a nutshell, this is a hardware equivalent of Lotus Notes - a combination of lack of respect for standards and a kungfu grip on the corporate customers.

As a result, once in a while developers encounter what can be described as a "chastity belt" - when it comes to communications with external world. Look for blackberry connection problems in Google or read this frustrated rant that sums it all. And if, as a BlackBerry user, you encountered problems with MSN Messenger, IRC, or anything else - same reason.

By default, when possible and a proxy-based connection (MDS, BES, BIS) exists, BlackBerry seems to re-route all requests there. It is not at all certain that your tiny app will be allowed to go further. Chances are, it will be stopped, and possibly hang. However, there are two possibilities that should work in most cases:

direct TCP/IP via GPRS network
wireless connection

To cut the long story short, the easiest way to provide maximum coverage is to create a tiny function that will be used like this:

 protected String getDeviceSpecificUrlParameters()
{
      try {
 if (net.rim.device.api.system.WLANInfo.getWLANState() == net.rim.device.api.system.WLANInfo.WLAN_STATE_CONNECTED)
              {
  return ";deviceside=true;interface=wifi;ConnectionSetup=delayed;retrynocontext=true";
              }
          } catch (Exception e) {} // doesn't matter, just go elsewhere
 return ";deviceside=true";
}

then call it whenever you make a connection, e.g.

 ...
Connector.open(url + getDeviceSpecificUrlParameters());
...

If your application is designed to work across different platforms (like ours), you can re-implement that getDeviceSpecificUrlParameters() for whatever platform you need.

A word of warning: if it's a brand new phone, and the access to GPRS is not configured, you may need to explain your users how to set up their APN parameters. The carrier-specific parameters can be found here, here, and here.

Making peace with Vista

2009-04-14T17:33:00.009+10:00

This is not directly related to the subject of the blog, but is worth sharing, since so many people are having problems with this.

My development machine runs Windows Vista. I'm not a fan of it (probably, like the majority), but I have to eat the same food my users do. Many people experience the same phenomenon, where Vista suddenly freezes without any provocation. It gets progressively worse with time. In my case, a few days ago it came to a point where the freezing occurred every half an hour.

I tried different things: switching off visual styles, removing the useless Windows Defender, etc. The performance improved, of course, but the freeze attacks still remained.

Finally, I found this tip. (In Vista Home Basic, it is a bit different: you need to navigate to Control Panel -> System -> Performance -> Adjust indexing options.) It all makes sense: usually in databases functional indices are paramount, and when the system builds them, everything else is secondary. The most important analogy: if an index is corrupt, all kinds of bad things are likely to happen. Except that why would we need them in a desktop OS? It turns out that the reason is the ultra-annoying search, which is a part of the Start menu. I suppose the reason for this nuisance was to show that it's no worse OS X. Well, this is dumb. The old search was fine, and those who need better capabilities, generally use something else. In Vista, as a result, a background process is indexing nearly every file in the system. Why not make the indexing a scheduled task, and fall back to un-indexed search when necessary?

After I reduced the population of files to index to the contents of Start menu only, suddenly Vista became the most stable, high-performance system I have ever seen - I kid you not! On the other hand, if you do need and like that new bloated search, or need the thumbnails, simply go and rebuild the index. Also at this opportunity review what extensions you want to index.

Oh, and of course, I assume no responsibility if you do something silly to your machine, so don't just push everything without thinking.

Guide to sequences uploaded

2009-03-04T16:59:00.003+11:00

We uploaded a short guide to building and debugging the sequences. It is available at our whitepaper download page.

Carabao Language Kit 1.5.0.1 released

2009-02-20T18:11:00.003+11:00

The version 1.5.0.1 is now available for download. Lots of changes and enhancements thanks to ongoing development of Chinese (not in the default database though).

Fixed:

Regression: "phantom capitalization" of re-used words
Regression: sequence style forcing / avoiding
Repositioning errors in sentences with attached tokens
Sequence processing in languages not using white spaces
Regression: single member sequence processing

Added:

Lattice-based processing for speech recognition and OCR application usage
Optional and unmapped members in sequences
Members in sequences which are validated but not mapped
Possibility to get a crosslingual representation (components only: DeepAnalyzer and Translation Server)
Possibility to load content from a disambiguated crosslingual representation
GUI in Translation Console to enable lattice-based processing
GUI in Translation Console to enable loading crosslingual representation
GUI in Translation Console to hint the system about the expected domains in the text
Analysis mode in Translation Console, when the source and target languages are the same and no styles are enforced / avoided
Capability of using the white space as a delimiter in languages that don't have white spaces
Smart quotes and other delimiters

Improved:

Dictionary GUI - presents thesaurus from another language, if missing in the current language
Sequence builder GUI - color coding of members which are not mapped, or contain conditions producing empty sets

Carabao Language Kit 1.2.3.0 released

2008-12-08T09:00:00.003+11:00

The version 1.2.3.0 is now available for download.

Fixed:

Handling of single quotes as syntax delimiters in English

Added:

A segmentation mode more effectively handling languages that don't use white spaces (e.g. Chinese, Japanese, Korean, Thai). In this mode, different character classes are broken into tokens (e.g. Chinese, and then immediately English). The remaining unidentified part is run through unknown heuristic identifier.

Automatic conversion for Unicode clipboard data into the currently active encoding in tokens table

Better warning when attempting to overwrite the current token

A utility to rebuild semantic links cache

Improved:

In some systems, the table of tokens with every update was adding a new set of system icons (minimize, restore, maximize) to the MDI frame window. The maximize option now causes the window to be set roughly to the full client area, but not in maximize mode

Free source code section

2008-09-08T16:28:00.002+10:00

We added a small source code section on our Download page, where we will post freebies for developers.

Carabao Language Kit 1.2.0.0 released

2008-09-08T08:16:00.003+10:00

The version 1.2.0.0 is now available for download.

Fixed:

Unknown patterns were translated as hypernyms

Regression: certain category-based sequences were omitted on second execution because of a malfunctioning guess scan caching mechanism

In analytical mode (Carabao DeepAnalyzer), there was a mismatch between word index number and an idiom member index, in sentences with attached tokens such as 'em, 'm

When copying a token with 1 rule units or less, the text is always reset to the original

Added:

Capability to match numbers as patterns

When a translation is not found, the engine tries to fall back to a matching hypernym instead

New methods to Carabao DeepAnalyzer that enable accessing the members of the detected idioms

New methods to Carabao CDA that enable accessing the unknown heuristics table

New sequences

Russian morphological exceptions

Improved:

If an "unknown pattern" is forced to match a known word, it will not create a new guess if a guess with a same hypernym already exists. For example, if you force to check, whether a known word can be a city, a new record will not be created, if there is already a guess with a known city
Automatic input language switching in locator fields
Locator fields are pre-filled with the list of all existing languages in the database, eliminating the need to jump to the next language

The surreal world of VoIP

2008-06-05T09:48:00.002+10:00

We're currently working on a massive project involving telephony and mobile technologies, and I had to look for VoIP vendors to cater for relatively simple needs of my client. I have to say that while the needs are simple, the traffic is extremely high, so in monetary terms, it could be a nice deal for the VoIP vendors.

But... is VoIP a strange industry or what. I don't know who runs all these companies - but:

if you don't know what IVR, DID, PSTN, or all their other "secret handshakes" mean - they won't even talk to you. Forget about the forums, they are even less helpful than Usenet.
the responses come usually after weeks, and they are of the type "I just transferred your inquiry to our sales representative". Obviously, unless you kick and scream, the sales rep won't get back
you have to re-tell the story over and over again

Ah yes, but the internet is full of dotcomish optimism about mashups and other kewl stuff. Awesome, dude.

Of course, there are also people who do need paying customers, and those who are able to concentrate, and - surprise surprise - they were the ones who got the job eventually.

Published in MultiLingual

2008-05-01T10:02:00.005+10:00

My article about real-world applications of machine translation has been published in MultiLingual Computing, the leading industry magazine for globalization, international software development and language technology.

This part is for subscribers only though.

Localization can be a matter of life and death

2008-04-30T15:28:00.004+10:00

This story (A Cellphone's Missing Dot Kills Two People, Puts Three More in Jail) reads like it was taken from a Tarantino or Coen brothers' movie.

We are published at ELRA

2008-04-23T09:06:00.005+10:00

After a few months of evaluations, agreements, and inspections, our linguistic data is published at European Linguistic Resources Association's website. The Russian - English OLIF dictionary is sold at quite a price, while the freebie Swahili, Czech and Cebuano dictionaries are distributed for free (although ELRA takes postage and media charges).

It is important to mention that all this data can be created from (usually free) ASCII dictionaries on the net using Carabao Linguist Edition.

Clarification: OLIF is Open Lexicon Interchange Format backed by SAP, especially created for NLP oriented lexica. The official website is www.olif.net.

Server transition

2008-04-01T16:45:00.003+11:00

We just moved to a new server. Much better performance, but there might be some minor technical glitches in the next few days. Thank you for your patience.

Hype, hype, hype

2008-03-14T01:01:00.003+11:00

Got this article from TAUS today.

Man, is this annoying or what. With claims like these, the MT industry is going to move from Web 2.0 to Dot Com Bubble Burst 2.0.

I do hold high respect for Asia Online's team. Phillip Koehn co-authored some papers with Franz Och, the guy behind Language Weaver and Google Translate. Dion Wiggins has some super-duper credentials as an IT businessman, although with no experience at all in MT or NLP (and this industry is very different).

From the technological point of view, Koehn did the right thing (IMHO) going in the hybrid direction instead of looking for the philosopher's stone of pure ideal engine that builds itself and with some kind of divination deduces stuff that hand-crafted rules can't. It is also refreshing to see that they decided to use a high quality corpus.

But - c'mon, people!

The article contains odd bits such as:

...adding a syntax component to a SMT system "seriously degrades throughput performance from 5,000 words a minute to only around 300" on a machine with 4 high speed CPUs

Did they mean the SMT system? Because how can you measure speed of all SMT systems?

If this is the SMT system (i.e. they already built the kernel), this probably means they are going to use Moses. Which calls for more questions. Koehn created Pharaoh and Moses; both are open-source; both have been around for a while; yet I never heard of a commercial or even semi-commercial application that uses them. And I know at least one huge translation agency that launched a project to create their own MTs based on Moses.

There is also EuroMatrix, an all-you-can-spend research project, where Koehn also took part, and which, just like all the other euro-science-charity projects, produced nothing (yeah, OK, there is a Czech English lexicon, which is a huge deal, right?).

The website of Asia Online looks nice. Obviously, lots of Wikipedia articles, SMT for dummies complete with scientifically-looking formulas, media buzz. Not a word about the actual engine, no screenshots, no demos.

One thing that particularly captured my attention was the Careers page. As of now, they are looking for:

country office managers in every Asian country, including Thailand

"content procurement" managers that read content and make sure it is well-translated (good luck with that) before feeding it to the lexicon builders

the coolest thing: programmers in Thailand: C++ - probably to fix bugs in Moses, and C# to hack the front-end. Now this last part is kind of OK, except that one of the requirements is "Have database skills in MS SQL Server, Oracle Database and etc". Data and stuff. In other words, on this stage they did not decide yet what backend they are going to use. Which means, there are no design specs.

So they have no people responsible for the content. They have no design specs. They have no solid plan. They have no people who worked and produced anything of this class.

They did not even research their markets properly. If they even looked at Wikipedia articles about the Philippines, or talked to one or two Filipinos, they'd learn that English, not Tagalog, is the lingua franca of the Philippines when it comes to written language. 90% of the major newspapers, all the official correspondence, all the commercial documents in the Philippines are in English.

All they have is the prototype of a system that was never shown to work in a real-world environment, and a bunch of British guys who rented an office in "beautiful downtown Bangkok" and announced that they will conquer the world in a year.

And, of course, the usual reservations about statistical MT apply.

Carabao Language Kit 1.1.0.1 released

2008-03-11T01:10:00.001+11:00

The version 1.1.0.1 is now available for download - mostly to fix the regressions reported in 1.1.0.0.

Fixed:

Crash when using sequence extraction option (regression from 1.1.0.0)

Added:

Capability to import sequences by data entry directly from the Sequence Sheet

Capability to manually set sequence descriptions
Some sequences for multi-word entity extraction
More morphological exceptions for Russian

Improved:

Processing speed and memory consumption - further boost

Token Sheet (words & sequences) GUI

What may happen in the next 100 years - predictions from 1900

2008-03-03T00:37:00.001+11:00

A scan of an interesting article from December 1900's Ladies' Home Journal: What May Happen in the Next 100 Years

Some of the predictions are astonishingly accurate. Some are quire funny.
There's a prediction about language:

There will be No C, X or Q in our every-day alphabet. They will be abandoned because unnecessary. Spelling by sound will have been adopted, first by the newspapers. English will be a language of condensed words expressing condensed ideas, and will be more extensively spoken than any other. Russian will rank second.

Of course, it was before two world wars, which introduced corrections into the statistics.

Well done with the "condensed words" and "condensed ideas", and the propagation of English (definitely not a given in 1900).