Indentation and syntax highlighting of Basic code examples

Basic code examples were hard to read in LibreOffice help, because code blocks were not indented. Also, people got used to syntax highlighting in editors.
By LibreOffice 4.0 I made the code examples indented with some scripts and a bit of manual work. I also tweaked help-to-wiki conversion script, so online WikiHelp of LibreOffice 4.0 will feature syntax highlighting of Basic code examples.
For the local help content, my idea was to re-use SyntaxHighlighter class from svtools in help compiler, so we could add the colors build-time, without modifying the source and create work for translators. Dávid Vastag, a university student, who worked with Novell Hungary as a trainee, helped me to implement this feature in help compiler. The following screenshots show how an example Basic code is rendered in different versions of LibreOffice help.

LibreOffice 3.6

LibreOffice 4.0

LibreOffice 4.1

Size reduction of local help

When I studied help XML files, I noticed that many of the tags and attributes are not needed run-time. Some of them are even completely obsolete, and are not needed at all. However a mass clean-up in the source would not be desirable, it can cause extra work for translators for example. My idea was to apply a stylesheet to each file in the help compilation phase. I created compact.xsl, which removes comments, whitespace, and unnecessary tags and attributes. The size of en-US Windows MSI help pack decreased from 7.83 MB to 5.14 MB – 34% less!

LibreLogo poster (original PDF and OpenDocument versions)

The simple programmable vector graphics of LibreOffice 4 is quite useful in OpenDocument template development. The templates.libreoffice.org site has got a new section called LibreLogo, and some interesting new Logo related templates:

Blank sheet single line and double line music papers (see ODT and PDF1, PDF2) and their generator program.

Striped border drawing program. The program asks for the border and margin width, and the list of the colors (eg. stripes of a national flag or the rainbow). For example, a new template based on this Logo template is a “red-white-green” bordered ODF template for announcements of Hungarian local authorities for elections.

Simple crossword generator for teachers and (language) learners. The program will ask (1) the letter size of the crossword in pt, (2) the solution (the vertical word) and (3) the horizontal words. Word order, plus or missing words don’t matter, because the program asks new words, if needed.

Crop mark drawing program for ODF and PDF documents with detailed instructions about postprocessing of LibreOffice generated PDFs, eg. combining crop marks with existing PDFs, PDF cropbox setting, RGB/CMYK color conversion.

Chessboard generator. Typesetting chessboard manually or by notation in LibreOffice 4 using standard Unicode chess figures. (More about this chess drawing program and the next one: Logo for desktop publishing)

Chessboard generator 2. Typesetting chessboard by notation in LibreOffice 4, using a traditional chess font face, Chess Merida.

Drawing and programming are fun, and it’s often useful to combine them. The aim of LibreLogo, the new experimental LibreOffice module + Writer toolbar to give an easy, Logo-like, Python based programming environment to the excellent vector graphics of LibreOffice. Here (and in my next posts) I will show examples of LibreLogo usage in school/office environment.

Fine graphics with printing quality. PDF and SVG exports of the OpenDocument format support printing and DTP programs: click on the first picture for the SVG export (Warning! It may halt Firefox 15).

Precise drawing. Turtle moving commands of LibreLogo supports printing measurements, and combining with manual handling of drawing objects in LibreOffice, it can save expensive decorative paper, as for these 3D Christmas stars (made for the local School Christmas fair). PDF, PDF2 (with spirals), ODT (with LibreLogo source).

Fancy, unique documents. This Christmas background created by random snowflakes: Unicode character “❄” (U+2744) with random sizes, positions and angles. The result of the LibreLogo program has been modified (overlapping of snowflakes, title) manually. PDF, ODT (with LibreLogo source).

More information: my LibreOffice conference presentation and a blog post also about LibreLogo from Cor Nouws.

One cannot imagine simpler file format, than a tab separated text. When I translated OpenOffice.org into Hungarian, I built a tool set around it, and translated it happily. But not every translator was a programmer, or capable of using scripts. Translators demanded PO file format, which is a quasi-standard file format for localization in the open source world. Translate Toolkit and Pootle became part of the localization process, en-US SDF file was converted to POT (PO Template) files, and translated PO files were converted back to SDF manually.

When LibreOffice project started, I wanted to amend the process for two reasons.

1. Huge SDF files blew the git repository. Git does not work well with large, frequently updated text files.
2. Manual conversion from PO to SDF was a tedious job.

Therefore I implemented to store PO files in git, and convert them to SDF automatically during the build. In LibreOffice 3.4 we used po2oo from Translate Toolkit for the back-conversion. In LibreOffice 3.5 po2oo was replaced by po2lo written by Miklos Vajna, which was 30x as fast as po2oo.

But why convert translations from one format to another, why not use PO files directly? This is what was developed in feature/killsdf branch in the past few months. Most of the programming work was done by a young developer, Tamas Zolnai, who was a trainee at Novell Hungary in the summer, and now he is a fellow of FSF.hu Foundation (FSF.hu sponsors his work on LibreOffice). Now localization tools extract translatable content from the source code directly to PO files, and they read PO files directly during the build. SDF files are not generated any more. At the same time localization tools were refactored. Perl scripts have gone. All tools are in C++ now. Further cleanup and optimizations are on the way on master branch. I fixed the last remaining issues after the merge in Munich Hackfest 2012.

Not much have happened since then, we added new languages, new dictionaries, new features, and cleanup continued, but the size of installer remained approximately the same: 198MB for LibreOffice 3.6.0.

However, in LibreOffice 3.7 there will be some improvement in this field. A couple of days ago Istvan Turi – who spent 6 weeks with LibreOffice hacking as a trainee at Novell Hungary – committed a few patches and I also committed one, which removed all language dependent agenda/fax/letter/report and spreadsheet style templates. Approximately 2000 files were removed from git. Even more were removed from installer, because these templates were not localized into all supported languages, so en-US templates were copied for them. Size of installer has been decreased to 185MB, 6.57% less than before.

The good news is that we did not loose functionality here. Localization will be done runtime. Translators can translate strings from these templates in .po files. I bet that many language teams did not know about these templates, now they have the chance to localize them properly.

x-comment and KeyID in Pootle

This is how it will look like in Pootle, when we will start to translate LibreOffice 3.6. Next to the KeyID the x-comment will appear, if there is one. Now it is up to translators and developers to write good comments for the problematic parts of the UI. Give context, explain things, when someting is non-trivial. Please send patches, make new translators’ job easier.

Example from svx/source/dialog/sdstring.src:

String RID_SVXSTR_GRDT10
{
    Text [ en-US ] = "Diagonal 1l";
    Text [ x-comment ] = "l means left";
};

Currently there are no comments in xml configuration files, but if there were, an entry would look like this:

<prop oor:name="VerbUIName">
    <value xml:lang="x-comment">This is the Open command.</value>
    <value xml:lang="en-US">~Open</value>
</prop>

One of the main concepts of “sentence checking” is to break the bad habit of grammar checkers, the annoyingly frequent false alarms. Reporting all potential (but usually not) problems isn’t able to replace proofreading, but very frustrating: “the most useless feature ever added to Microsoft Word [...] With this feature, an infinite number of monkeys will analyze your writing and present you with useless grammar complaints while not alerting you to actual grammatical errors because computers don’t understand grammar. Sure, it sounds great on a box—or a promotional Web site—but anyone who knows, knows that grammar checking is a sham. Just say no.” (Jason Snell), “Computer grammar checking really is terrible. [...] The things they are good at, like spotting the occasional the the typing error, are very easy there are very few of them. For the most part, accepting the advice of a computer grammar checker on your prose will make it much worse, sometimes hilariously incoherent. If you want an amusing way to whiling away a rainy afternoon, take a piece of literary prose you consider sublimely masterful and run the Microsoft Word™ grammar checker on it, accepting all the suggested changes.” (Geoffrey K. Pullum, cited by Wikipedia), “My take is that we should encourage students to spell check and revise accordingly, but skip the grammar check and proofread instead.” (Mark Pennington). One of the most important improvements of the last few Microsoft Office versions was lowering the “sensitivity” of the grammar checker, responding to customer feedback (source).

In the next few examples I will show how can we make more precise grammar checking:

First rule: we don’t need to catch all mistakes. In fact, we cannot catch all mistakes. We have got only “monkeys”, and often debatable and controversial concepts about grammatical correctness.

Example 1. Capitalization

The following Lightproof rule searches the sentence beginning lower-case letters, and suggests upper-case ones:

^([a-z]) -> =\1.upper() # Missing capitalization?

Unfortunately, there will be many false alarms, especially after abbreviations, because Lightproof has a very simple default sentence boundary detection based on only the paragraph beginning, and sentence ending punctuation (full stop, exclamation mark and question mark). We can check the previous word for abbreviations by a fast Python regex object called abbrev:

^([a-z]) <- not abbrev.search(word(-1)) ->
    =\1.upper() # Missing capitalization?
[code]
# pattern matching for common English abbreviations
abbrev = re.compile("(?i)\b([a-z]|acct|approx|\
appt|apr|apt|assoc|asst|aug|ave|avg|co(nt|rp)?|\
ct|dec|defn|dept|dr|eg|equip|esp|est|etc|excl|\
ext|feb|fri|ft|govt?|hrs?|ib(id)?|ie|in(c|t)?|\
jan|jr|jul|lit|ln|mar|max|mi(n|sc)?|mon|Mrs?|\
mun|natl?|neg?|no(rm|s|v)?|nw|obj|oct|org|orig|\
pl|pos|prev|proj|psi|qty|rd|rec|rel|reqd?|resp|\
rev|sat|sci|se(p|pt)?|spec(if)?|sq|sr|st|subj|\
sun|sw|temp|thurs|tot|tues|univ|var|vs)\.")

The sentence segmentation is better, but not enough for a non-intrusive grammar checker (see also the similar decision of Raphael Mudge, author of the more sophisticated and resource-intensive AtD grammar checker here). We will limit the default checking only for paragraph capitalization, more precisely, checking for the first sentence of a paragraph. The difference is important: lower-case list items seem paragraphs for grammar checker clients of LibreOffice, so we check only the paragraphs with more sentences:

^([a-z]) <- re.search("^[a-z].*[.?!] [A-Z]", TEXT)
    and not abbrev.search(TEXT) -> = \1.upper()
    # Missing capitalization?

(The string variable TEXT contains the full text of the paragraph.)

Second rule: we have to limit false alarms to zero or near zero.

Example 2. Article a/an

The choice of "a" or "an" is determined by phonetic rules and in some cases, also spelling conventions, see Wikipedia. We have to care about the non-standard pronunciation and writing, too, like the form “an HTML” where the letter H pronounced as [ˈheɪtʃ] (see Wikipedia and BBC), and the rare, but similarly allowed spelling conventions: “an hotel”, “an historic” etc. A relevant Lightproof rule to test the “a”:

# pattern "vow" matches words beginning with vowels:
vow [aeiouAEIOU]\w*

a {vow} <- {vow} <> {vow}.upper() and not
    ({vow} in aA or {vow}.lower() in aA)
    and spell({vow}) -> an {vow} # Bad article?
[code]
aA = set(["eucalypti", "eucalyptus", "Eucharist", "Eucharistic",
"euchre", "euchred", "euchring", "Euclid", "euclidean", "Eudora",
"eugene", "Eugenia", "eugenic", "eugenically", "eugenicist",
...])

One of the most interesting elements of the condition is the final spelling dictionary lookup
by the function spell(): this step limits the grammar checking for the known words. Missing words from the exception list (here: “aA”) and from the spelling dictionary won’t result false alarms, for example for the expression “a uremic” or the rare “a usuress”.

Third rule: Don’t hurt people unnecessarily.

Example 3. Spacing

Instead of annoying multiple spaces, like

"  +" -> " " # Remove repeating spaces or use tabulators/styles for formatting.

it’s better to check only double (maybe triple) spaces between words. There is living typewriter tradition in the digital age among the users of word processors, too, see sentence spacing with double spaces. So the proposed grammar checker has got three options for spacing: a default option for word spacing, and two optional for sentence spacing and the bad positioning with multiple spaces.

Example 4. Grammar checking based on detailed language data

Recent English module contains only the following example for morphological analysis:

([Tt])his {abc} <- option("grammar") and morph({abc}, "Ns") ->
    \1hese {abc}\n\1his, {abc} # Did you mean:

There are several things here to limit false alarms. This is an optional rule (see the option("grammar") condition). The morph() function searches unambiguously plural nouns (“Ns”) using Hunspell and language data of Hunspell dictionaries, and the rule checks only lower-case words after "This" or "this", so the likely bad expression “this mice” will be detected, but not in the “Why is this Mice of Men a challenged book?”. Extending the English dictionary of LibreOffice with part-of-speech and morphological data will help to add more sophisticated grammar checking rules, for example with conversion of the huge rule set of LanguageTool development.

Other concept of sentence checking for LibreOffice is to provide optional tools for proofreading, pre-press formatting and desktop publishing. Not so precise grammar checking is one of these tools, eg. the option “capitalization” of the suggested grammar checker of LibreOffice extends the checking of the capitalization for all sentences, not only for the first ones in the paragraphs.

Feature lists of the recent English module

Default checks

• Punctuation (unnecessary spaces before punctuation, missing spaces after punctuation, hyphen instead of n-dash, real double quotation marks and multiplication sign)
• A/an article (with the described improvements)
• Word spacing (not sentence spacing)
• Paragraph capitalization (unwitting paragraph breaks)
• Simple word duplication (and and, or or, for for, the the)
• Longer explanations, using relevant Wikipedia articles (see first screenshot)

Optional features

• More grammar checks (a few examples using Hunspell morphological analyzer and the English dictionary)

• More punctuation checks (n-dash/m-dash, unpaired quotation marks and parentheses, typographical apostrophe and ellipsis)…
• Measurement conversion (lb/kg, mph/km/h, °F/°C, ft/yd/in/m/cm/mm, mile/km, gal/pint/l)
• Thousand separation (common or ISO)
• Sentence capitalization
• Other word duplication
• Sentence spacing (double spaces)
• Formatting with spaces (more than 3 spaces)
• Settings (a little hidden yet): in Tools » Extension manager… choose “English spelling etc. dictionaries” extension and click on its Options button.

Features of Lightproof rule editor

• LibreOffice extension, downloading, description
• Special grammar checking component + LibreOffice toolbar for rule compiling and debugging
• Run-time compilation and update of its grammar checking component
• Debugging of user code (rule conditions and Python user functions)

  

  ]]> http://libreoffice.hu/2011/12/08/grammar-checking-in-libreoffice/feed/ 2