eMpTy Pages

A Short Guide to Measuring and Comparing Machine Translation Engines

2012-01-25T15:01:00.000-08:00

This is an article from the Asia Online November 2011 Newsletter that provides useful advice for meaningful comparisons of MT engines and is authored by Dion Wiggins, CEO of Asia Online. So the next time somebody promises you a BLEU of 60, be skeptical, and make sure you get the proper context and assurances that it was properly done.

“What is your BLEU score?” This is the single most irrelevant question relating to translation quality, yet one of the most frequently asked. BLEU scores and other translation quality metrics greatly depend on many factors that must be understood in order for a score to be meaningful. A BLEU score of 20 in some cases can be better than a BLEU score of 50 or vice versa. Without understanding how a test set was measured and other details such as language pair and domain complexity, a BLEU score without context is not much more than a meaningless number. (For a primer on BLEU look here.)

BLEU scores and other translation quality metrics will vary based upon:

The test set being measured: Different test sets will give very different scores. A test set that is out of domain will usually score lower than a test set that is in the domain of the translation engine being tested. The quality of the segments in the test set should be gold standard (i.e. validated as correct by humans). Lower quality test set data will give a less meaningful score.
How many human reference translations were used: If there is more than one human reference translation, the resulting BLEU score will be higher as there are more opportunities for the machine translation to match part of the reference.
The complexity of the language pair: Spanish is a simpler language in terms of grammar and structure than Finnish or Chinese relative to English. Typically if the source or target language is relatively more complex, the BLEU score will be lower.
The complexity of the domain: A patent has far more complex text and structure than a children’s story book. Very different metric scores will be calculated based on the complexity of the domain. It is not practical to compare two different test sets and conclude that one translation engine is better than the other.
The capitalization of the segments being measured: When comparing metrics, the most common form of measurement is Case Insensitive. However when publishing, Case Sensitive is also important and may also be measured.
The measurement software: There are many measurement tools for translation quality. Each may vary slightly with respect to how a score is calculated, or the settings for the measure tools may not be set the same. The same measurement software should be used for all measurements. Asia Online provides Language Studio™ Pro free of charge and this software measures the scores, for a given test set, for a variety of quality metrics.

It is clear from the above list of variables that a BLEU score number by itself has no real meaning.

How BLEU scores and other translation metrics are measured

With BLEU scores, a higher score indicates higher quality. A BLEU score is not a linear metric. A 2 BLEU point increase from 20 to 22 will be considerably more noticeable than the same increase from 50 to 52. F-Measure and METEOR also work in this manner where a higher score is also better. For Translation Error Rate (TER), a lower score is a better score. Language Studio™ Pro supports all of these metrics and can be downloaded for free.

Basic Test Set Criteria Checklist

The criteria specified by this checklist are absolute. Not complying with any of the checklist items will result in a score that is unreliable and less meaningful.

Test Set Data should be very high quality: If the test set data are of low quality, then the measurement delivered will not be reliable.
Test set should be in domain: The test set should represent the type of information that you are going to translate. The domain, writing style and vocabulary should be representative of what you intend to translate. Testing on out-of-domain text will not result in a useful metric.
Test Set Data must not be included in the training Data: If you are creating an SMT engine, then you must make sure that the data you are testing with or very similar data are not in the data that the engine was trained with. If the test data are in the training data the scores will be artificially high and will not represent the level of quality that will be output when other "blind" data are translated.
Test Set Data should be data that can be translated: Test set segments should have a minimal amount of dates, times, numbers and names. While a valid part a segment, they are not parts of the segment that are translated; they are usually transformed or mapped. The focus for a test set should be on words that are to be translated.
Test Set Data should have segments that are at between 8 and 15 words in length: Short segments will artificially raise the quality scores as most metrics do not take into account segment length. Short segments are more likely to get a perfect match of the entire phrase, which is not a translation and is more like 100% match with a translation memory. The longer the segment, the more opportunity there is for variations on what is being translated. This will result in artificially lower scores, even if the translation is good. A small number of segments shorter than 8 words or longer than 15 words are acceptable, but these should be limited.
Test set should be at least 1,000 segments: While it is possible to get a metric from shorter test sets, a reasonable statistic representation of the metric can only be created when there are sufficient segments to build statistics from. When there are only a low number of segments, small anomalies in one or two segments can raise or reduce the test set score artificially.Be skeptical of scores from test sets that only contain a few hundred sentences.

Comparing Translation Engines - Initial Assessment Checklist

Language Studio™ can be used for calculating BLEU, TER, F-Measure and METEOR scores.

All conditions of the Basic Test Set Criteria must be met: If any condition is not met, then the results of the test could be flawed and not meaningful or reliable.

Test set must be consistent:

Test sets should be “blind”: If the MT engine has seen the test set before or included the test set data in the training data, then the quality of the output will be artificially high and not represent the true quality of the system.
Tests must be carried out transparently: Where possible, submit the data yourself to the MT engine and get it back immediately. Do not rely on a third party to submit the data. If there are no tools or APIs for test set submission, the test set should be returned within 10 minutes of being submitted to the vendor via email. This removes any possibility of the MT vendor tampering with the output or fine tuning the engine based on the output.
Word Segmentation and Tokenization must be consistent: If Word Segmentation is required (i.e. for languages such as Chinese, Japanese and Thai) then the same word segmentation tool should be used on the reference translations and all the machine translation outputs. The same tokenization should also be used. Language Studio™ Pro provides a simple means to ensure all tokenization is consistent with its embedded tokenization technology.

Ability to Improve is More Important than Initial Translation Engine Quality

The initial scores of a machine translation engine, while indicative of initial quality, should be viewed as a starting point for rapid improvement which is measured by the test set and BLEU scores. Depending on the volume and quality of data provided to the SMT vendor for training, the quality may be lower or higher. Most often, more important than the initial quality is how quickly the translation engine quality improves.

Frequently a new translation engine will have gaps in vocabulary and grammatical coverage. Other machine translation vendors’ engines do not improve at all or merely improve very little unless huge volumes of data are added to the initial training data. Most vendors recommend retraining once you have gathered a volume of additional data that is at least 20% of the size of the initial training data that the engine was trained on. Even when this volume of data is added, only a small improvement is achieved. As a result, very few translation engines evolve in quality much further than their initial quality.

In stark contrast, Language Studio™ translation engines are created with millions of sentences of data that Asia Online has prepared in addition to the data that the customer provides. The translation engines improve rapidly with a very small amount of feedback. It is not uncommon to get a 1-2 BLEU score improvement with as little as a few thousand post-edited sentences. Language Studio has a unique 4 step approach that leverages the benefits of Clean Data SMT and manufactures additional learning data by directly analyzing the edits made to the machine translated output.

Consequently, only a small amount of post-edited feedback can improve Language Studio™ translation engine quality quite considerably, and it can do so at speeds much faster and with far less effort than with other machine translation vendors. Asia Online provides complimentary Incremental Improvement Trainings to encourage rapid translation engine quality improvement with every full customization and also offers additional complimentary Incremental Improvement Trainings when word packages are purchased, greatly reducing Total Cost of Ownership (TCO).

An investment in quality at the development stages of a translation engine impacts and reduces the cost of post editing directly, while increasing post editing productivity. While the development of some rules, normalization, glossary and non-translatable term work will assist in the rate of improvement, the fastest and most efficient way to improve Language Studio™ engines is to post edit the translations and feed them back into Language Studio™ for processing. The edits will be analyzed and new training data will be generated, directly addressing the primary cause of most errors. In other words, just post editing as part of a normal project will result in an immediate improvement. Little or no other extra effort is needed. By leveraging the standard post editing process, the effort and cost of improvement as well as the volume of data required in order to improve is greatly reduced.

Depending on the initial training data provided by the client, a small number of Incremental Improvement Trainings are usually sufficient for most Language Studio™ translation engines to improve to a quality level approaching near-human quality.

Other machine translation vendors are now also claiming to build systems based on Clean Data SMT. Closer investigation reveals that their definition of “cleaning” is not the same as Asia Online. Removing formatting tags is not cleaning data. Language Studio™ analyzes translation memories and other training data and ensures that only the highest quality in domain data from trusted sources is included in the creation of your custom engine. The result is that improvements are rapid. Even with just a few thousand segments edited, the improvements are notable. When combined with Language Studio™ hybrid rules and an SMT approach to machine translation the quality of the translation output can increase by as much as 10, 20 or even 30 BLEU points between versions.

Comparing Translation Engines – Translation Quality Improvement Assessment

Comparing Versions: When comparing improvements between versions of a translation engine from a single vendor, it is possible to work with just one test set, but the vendor must ensure that the test set remains “blind” and that the scores are not biased towards the test set. Only then can a meaningful representation of quality improvement be achieved.
Comparing Machine Translation Vendors: When comparing translation engine output from different vendors, a second “blind” test set is often needed to measure improvement. While you can use the first test set, it is often difficult to ensure that the vendor did not adapt its system to better suit and be biased towards the test set and in doing so delivering an artificially high score. It is also possible for the test set data to be added to engines training data which will also bias the score.

As a general rule, if you cannot be 100% certain that the vendor has not included the first test set data or adapted the engine to suit the test set, then a second “blind” test set is required. When a second test set is used, a measurement should be taken from the original translation engine and compared to the improved translation engine to give a meaningful result that can be trusted and relied upon.

Bringing It All Together

The table below shows a real world example of a version 1 translation engine from Asia Online and an improved version after feedback. Additional rules were added to the translation to meet specific client requirements, which resulted in considerable improvement in translation quality. This is part of Asia Online’s standard customization process. Language Studio™ puts a very high level of control in the customer’s hands where rules, runtime glossaries, non-translatable terms and other customization features ensure the quality of the output is as close to human quality and requires the least amount of editing possible.

BLEU Score Comparisons Case Sensitive	Asia Online
BLEU Score Comparisons Case Sensitive	V1 SMT	V2 SMT	V2 SMT + Rules	Google	Bing	Systran
Reference 1	36.05	45.96	56.59	30.58	29.64	21.01
Reference 2	35.80	39.31	48.85	32.05	29.94	22.56
Reference 3	38.65	52.31	65.03	35.51	33.17	24.68
Combined References	50.45	66.52	80.48	44.58	41.65	30.26
Case Insensitive
Reference 1	41.30	52.65	59.25	32.18	31.49	22.49
Reference 2	41.01	45.32	51.24	33.67	31.64	23.88
Reference 3	43.99	58.97	67.49	37.15	35.01	25.92
Combined References	56.83	74.35	82.89	46.26	43.68	31.68

*Language Pair: English into French. Domain: Information Technology.

It can be seen clearly from the scores above that when all three human reference translations are combined the BLEU score is significantly higher and that the BLEU scores vary considerably between each of the human reference translations. The impact of the improvement and the application of client specific rules can also be seen, raising the case sensitive BLEU score from 50.45 to 80.48 (an increase of 30.03 in just one improvement iteration). One interesting side effect of having multiple human references is that it is often possible to judge the quality of the human reference also. In the example above, the machine translation output is much closer to human reference 3, indicating a higher quality reference. The client later confirmed that the editor who prepared the reference was a senior editor and more skilled than the other 2 editors who prepared human reference 1 and 2.

A BLEU score, as with other translation metrics, is just a meaningless number unless it is established in a controlled environment. Asking “What is your BLEU score?” could result in any one of the above scores being given. When controls are applied, translation metrics can be used both to measure improvements in a translation engine and compare translation engines from different vendors. However, while automated metrics are useful, the ultimate measurement is still a human assessment. Language Studio™ Pro also provides tools to assist in delivering balanced, repeatable and meaningful metrics for human quality assessment.

Review: Most Popular Blog Posts from 2011

2011-12-29T16:22:00.001-08:00

“Blogs are about sharing with authenticity. A good blog can help you really connect deeply with your audience in a meaningful way because the content is not only relevant but insightful and personal. I think most enterprises miss that point. When you do it right, your customers will walk away not only having learned something new but will also feel much more connected to your brand.” David Armano EVP, Global Innovation & Integration at Edelman Digital

Don’t say anything online that you wouldn’t want plastered on a billboard with your face on it. -- Erin Bury

One of the things that I enjoy about blogging is the feedback that one gets, and the continuing and evolving discussion that sometimes comes forth from these posts. I find it helps to clarify my thinking on what really matters, and the critical feedback one gets, on assumptions that may previously go unquestioned is very useful in just evolving my own thinking on these issues. The feedback and the rankings helps me, and others too, I think, to understand what strikes a chord in the reader community, and can also sometimes help to guide further evolutionary thinking on the subjects at hand. This is is a ranking of the most popular (Unique Visitors and Page Views) posts of the year based on the data provided by Google Analytics.

Analysis of the Shutdown Announcements of the Google Translate API and the subsequent posts on what this may mean for the translation industry were by far the most popular posts of the year. The original post authored by Dion Wiggins was also referenced by the Atlantic and other mainstream media and still continues to be an influential view on the announcement today, probably much more so than any other publicly offered opinion in the professional translation industry.
The Continuing Saga & Evolution of Machine Translation was coverage of the IMTT 7th Conference in Cordoba triggered active debates and discussions MT, automation and translator compensation in several forums and clearly struck a chord for many.
The Future of Translation Memory (TM) is a posting that continues to receive high new visit rates long after it was originally published.
The Building Momentum for Post-Edited Machine Translation (PEMT) a number of case studies on the increasing use of post-edited MT to meet business timeliness and production cost requirements.
Has Google Translate Reached the Limits of its Ongoing Improvement? More evidence that more data Is not always better especially for MT, but even for Search, and the many reasons to consider the data quality, yet again.
The Growing Interest & Concern About the Future of Professional Translation About reactions to the changes underway in translation
Standards: the Importance of Measurement A guest post by Valeria Cannavina on how standards can drive quality improvements
The Moses Madness and Dead Flowers A post that questions some of the assumptions made by “instant Moses” advocates and challenges the long-term value of these experiments. Strong opinions voiced in the comments.
Translation Crowdsourcing An exploration of the driving forces underlying successful translation crowdsourcing efforts.
An Exploration of Post-Editing MT – Part I Discussion on the nature and compensation of post-editing MT work.

Please Repeat: Influence is NOT Popularity! -- Brian Solis

While reader traffic is one way to measure the impact of articles, there are also other ways that capture the relative influence of individual posts. PostRank is one such measure that I think monitors how others reference the posts, and monitors where and when content generates meaningful interactions across the web. They provide a truer picture of the relative influence and impact of individual blog posts, and thus I include the latest PostRank snapshot here. (You can link to the posts through the table on the right of this blog text). This table shows that some articles that may not have had high direct readership may actually be much more useful to readers and it is interesting to see how different the two lists are though it is clear that the analysis of the Google Translate API shutdown/pay-wall was a major hit no matter how you look at it.

It is also interesting to note that some older posts continue to strike a chord with readers and remain active in terms of visibility because the themes are longer lived and also perhaps because they ring true. The original post on standards and some of the posts discussing disintermediation were also posts that generate continuing interest and continue to show up in both the Google Analytics and PostRank ratings.

I have noticed that we are getting more clarity on post-editing MT work in many different ways including new models for more equitable compensation. I am hoping to highlight best practices in this area in the coming year as I believe it will be critical to ongoing adoption and success with MT technology. I also think there will be much more to share on best practices of post-editing MT and I expect that we may find that it is not quite the dreaded beast it has often been portrayed to be.

Social Media is not just a set of new channels for marketing messages. It’s an opportunity for organizations to align with the marketplace and start delivering on behalf of customers -- Valeria Maltoni, conversationagent.com

I would also like to invite some of you to contribute to the discussion in this blog (guest posts) and assure you that I believe in open discourse and think it is useful for many different viewpoints to be aired to get closer to the “truth”. So please don’t hesitate to send me contributions that you think might be interesting to the audience that has been following this blog. I thank you for your support and I hope that the content here will continue to earn your interest and comments to extend the discussion beyond my thoughts on key issues.

For those who are not aware, there are some very interesting videos from presentations at TAUS that I reported on in the 4th ranked posting above on PEMT momentum.

Videos of presentations and panels at the recent TAUS User Conference in Santa Clara are now available on YouTube for everyone. The links below will take you to playlists on specific themes:

I wish you all a wonderful holiday season and look forward to sharing observations in the coming year, a year that many say will be a turning point across many dimensions.

The Moses Madness and Dead Flowers

2011-12-02T01:48:00.001-08:00

Machine translation technology has an unfortunate history of overpromising and under delivering. At least 50 years of doing this and sometimes it seems that the torture will never stop. MT enthusiasts continue to make promises that often greatly exceed the realistic possibilities. Recently, in various conversations, I have seen that the level of unwarranted exuberance around the possibilities with the Moses Open Source SMT technology is rising to peak levels. This is especially true in the LSP community. While most technologies go through a single hype cycle, MT seems destined to go through several of these cycles with each new approach and the latest of these is what I call Moses Madness. It has become fashionable of late to build instant DIY MT engines, using tools that help you with the mechanics of running the software that is “Moses”. While some of these tools greatly simplify the mechanical process of running the Moses software, they do not give you any insight into what is really going on inside the magic box or any clues to what you are doing at all. Moses is a wonderful technology and it enables all kinds of experimentation that furthers the art and science of data-driven MT, but it does require some knowledge and understanding for real success. It is possible to get a quick and dirty MT engine together using some of these tools, but for long-term strategic translation production leverage, I am not so sure. Thus it is my sense that we are at the peak of the hype cycle for DIY Moses.

I would like to present a somewhat contrarian viewpoint to much of what you will hear at TAUS - “Let a thousand MT systems bloom”, and other online forums on getting started with instant MT approaches. IMO Moses and especially instant Moses is clearly not the final answer. While Moses is a starting point for real development, it should not be mistaken as the final destination. I think there are a number of reasons that you should pause before you jump in, and at least build up some knowledge before taking the dive. I have attempted to enumerate some of these reasons, but I am sure some will disagree. Anyway, I hope an open discussion will be valuable in reaching a more sustainable and accurate view of the reality and so here goes, even though perhaps I am rushing in where angels fear to tread. And of course my opinion on this matter is not impartial, given my involvement with Asia Online.

The Sheer Complexity
As you can see from the official description, Moses is an open source project that makes its home in the academic research community. This link describes some of the conferences where people with some expertise and understanding of what Moses actually does convene and share information. Take a look at the program committee of these conferences to get a sense of what the focus might be. Now take a look at the “ step-by-step guide”, which students in NLP are expected to be able to handle. It is what you would have to do to build an MT system if did not have the DIY kit. Most of the instant/simplified Moses engine services in the market focus on simplifying this and only this aspect of developing an MT engine.

Clearly it would be good to have some knowledge of what is going on in the magic box BEFORE you begin, and perhaps it would even be really nice to have some limited team expertise with computational linguistics to make your exploration more useful. Remember that hiding complexity is not quite the same as removing complexity, and it would be smart to not underestimate this complexity BEFORE you begin. Anybody who has ventured into this has probably realized already, that while some of the complexity has been hidden, there is still much that is ugly and complicated to deal with in Moses world, and often it feels like the blind leading the blind.

I have noticed that many in professional translation industry have trouble even with basics like MT system BLEU scoring, and even some alleged MT experts barely know how to measure BLEU accurately and fairly. Thus I am skeptical that LSPs will be able to jump into this with any real level of competence in the short term. A level of competence that assures or at least raises the probability of business success i.e. enhances long-term translation productivity. Though it is possible that a hardy few will learn over the next 2-5 years, it is also clear that NLP and computational linguistics is not for everyone. The level and extent of knowledge required is simply too specialized and vast. As Richard Feynman said:”I think it’s much more interesting to live not knowing, than to have answers which might be wrong.” (Though he was talking about beauty, curiosity and mostly about doubt).

Alon Lavie, AMTA President, CMU NLP professor and President of Safaba (which develops hosted MT solutions that are largely built on top of Moses) says:
“ I am of course a strong supporter, and am extremely enthusiastic about Moses and what it has accomplished in both academic research and in the commercial space. I also think there is indeed a lot of value in the various DIY offerings (commercial and Achim's M4L efforts). But these efforts primarily target and solve the *engineering complexity* of deploying Moses. While this undoubtedly is a critical bottleneck, I think there is a potential pitfall here that users that are not MT experts (the vast majority) would come to believe that that's all it takes to build a state-of-the-art MT system. The technology is actually complex and is getting more complex and involved to master. Users may be disappointed with what they get from DIY Moses, and more detrimentally, become convinced that that's the best they can accomplish, when in fact letting expert MT developers do the work can result in far better performance results. I think this is an important message to communicate to potential users, but I'm not sure how best to communicate this message.”

Thus, I will join Alon in trying to convey the message that Moses is a starting point in your exploration of MT and not the final answer, and that experience, expertise and knowledge matter. Perhaps, a way to understand the complexity issue better, is to use some analogies.

The sewing machine/tailor analogy: Moses can be perhaps be viewed as a very basic sewing machine. You still need to understand how to cut cloth, stitching technique, fabric and lining selection, measurement, pocket technique (?), final fit modifications and so on to make clothes. Tailors do it better and expert tailors that only focus on men's suits do it even better than you or I would with the same sewing equipment. The closest to a ready made suit would be the free MT engines, except in this analogy they are only available in one size. Expertise really does matter folks if you want to customize-to-fit.

The DIY car analogy: In this analogy, Moses is the car engine and perhaps a very basic chassis, one that would be dangerous on a highway or bumpy roads. The DIY task is to build a car that can actually be used as transportation. This will require some understanding of auto systems design, matching key components to each other, tires, braking systems, body design and so on. Finally you also need to learn to drive and you would want the car to turn right when you want to. Again, expert mechanics are more likely to be successful even though there are some great DIY kits out there for NASCAR enthusiasts.

The Learning Curve
Even if you do have a team with some NLP expertise, remember that working with any complex technology involves a process of learning and usually an apprenticeship to get to a point of real skill. The people who build SMT engines at Microsoft, Google, Asia Online and other MT research teams have built thousands of MT engines during their MT careers. The skills developed and lessons learned during this experience are not easily replicated and embedded into open source code. Failure is often the best teacher and most of these teams have failed often enough to understand the many pitfalls along an SMT engine development path. To expect that any “instant” Moses solution is going to capture and encapsulate all of this is naïve and and somewhat arrogant. This is the kind of skill where expertise builds slowly, and comes after much experimentation across many different kinds of data and use case scenarios. Just as professional tailors and expert mechanics are likely to produce better results, MT experts who work across many different use scenarios are likely to produce much better results than a do-it-yourself enthusiast might. These results translate into long-term savings that should far exceed an initially higher price.

The objective of MT deployment for most LSP users is to increase translation productivity. (Very few have reached the next phase where they are translating new content that would never be translated were it not for MT). Thus getting the best possible systems that produce the highest possible MT output quality really matters to achieve this core objective of achieving measurable translation productivity. To put this in simpler terms, the difference between instant Moses systems and expert MT systems could be as much as 4,000 words/day versus 10,000+ words a day. Expert MT engine developers like Asia Online have multi-dimensional approaches, NLP skills, and many specialized tools in place to extract the maximum amount of information out of the data they have available. The use of these tools is guided by two team members with deep expertise on the inner workings of Moses and SMT in general. The learning process driving the development of these comprehensive tools takes years, and they enable Asia Online custom systems to produce superior translation output to the free online MT engines consistently. One team member has literally written the book on SMT and created Moses and thus one could presume is quite likely to have the expertise to develop better MT systems than most.

I have already heard from several translators who when asked to post-edit “instant Moses” output they know is inferior, simply run the same source material through Google/Bing and edit that instead, to improve their own personal productivity and save themselves some anguish. So if your Moses engine is not as good as these public engines you will find that translators will simply bypass them whenever they can. And they may not actually tell you that they are doing this. Post-editors will generally choose the best MT output they can get access to, so beware if your engine does not compare well. And buyers, insist on seeing how these instant MT engines compare to the public free engines on a meaningful and comprehensive test set, not just a 100 or so sentences.

However, I am also aware that some Moses initiatives have produced great results e.g. Autodesk,(for you doubters on the value of PEMT, here is clear evidence from a customer viewpoint) and here I would caution against any extrapolation of these results and expectation to achieve this for any and every Moses attempt. The team that produced these systems were more technically capable and knowledgeable than most, and I am also aware that that their training data was better suited for SMT than most of the TM you will find in the TDA or on the web. And even here, I would argue that MT experts would probably produce better results with the same data especially with the Asian languages where other support tools and processes become much more imperative.

As others have stated before me, the global population of people who actually understand how these data-driven systems work is really quite tiny, miniscule in fact. If you are building Moses systems you should be comparing yourself to the public free engines, as you may find that all your effort was much ado about nothing. One would hope that you will produce systems that compare favorably to these “free” options. And if your competition includes the lads and lassies at Microsoft and Google, one would hope that you know more about how to do this than pushing the instant Make-my-engine button. The financial cost of ignorance is substantially higher than most are able to define in terms of lost opportunity costs, and learning costs (a.k.a. mistakes) should be factored into a real TCO (Total Cost of Ownership).

The bottom line: Success with SMT requires very specialized skills that include, some NLP background, massive data handling skills, knowledge of parallel computing processing, linguistic data management tools, corpus analysis and linguistic structural analysis capabilities for optimal results not to mention a culture that nurtures collaboration with translators.

The Data, the Data, The Data
Moses is a data-driven technology and thus is highly dependent on the data that is used. Data volume is required to get good output from the systems and thus users have to gather the data from public sources and it is important to normalize and prepare the data for optimal performance. Most LSPs will not have the data or skills needed to gather the data in an optimal way. I have seen two major SMT engineering initiatives up close, one where training data was scraped off the web by spider programs, and another where data was not allowed to go into training data if it had not passed several human linguistic quality assessment checks. The differing impact of these approaches is quite striking. The dirty data approach requires substantially larger amounts of new data to see any ongoing improvement, while the clean data approach can produce compelling improvement results with much less new data.

This ability to respond to small amounts of corrective feedback is a critical condition for ongoing improvement, and for continued improvements in productivity e.g. raising PEMT throughput up to 15,000+ words/day in the shortest time possible. I have already stated that I was surprised how little attention is paid to data quality in instant Moses approaches presented at TAUS. And while data volume matters, for high quality domain-focused systems, the data you exclude may be more important than what you include. We are in a phase of the web's development where ‘ Big Data” is solving many semantic and linguistic problems, but we have also seen that data is not always the solution to better MT systems.

The upfront data analysis and data preparation, the development of “good” tuning and test sets are critical to the the short and long-term quality and evolution of an MT engine. This is something that takes experience and experimentation to understand and be skillful at. Experts can add huge value at this formative stage. Remember that this is a technology where “Garbage In Garbage Out” (GIGO) will be particularly true. Many who understand how bad TM can get don’t need any further elaboration on this, even though some people in the SMT community remain unconvinced that clean data does matter.

Many of the people who have jumped into instant Moses, do not realize that to get your initial MT engine to improve, will require very large amounts of new data with a standard Moses approach. The rule of thumb I have heard used frequently is that you need 20-25% of the initial training data volume to see meaningful improvements. Thus, if you used 10 million words to build your system, you will need 2-3 million new words to see the system noticeably improve. So most of these instant systems are as good as they are ever going to get when the first engine is produced. In contrast, Asia Online systems can improve dramatically with as little as a few thousand sentences (a single project) and are architected and designed from the outset to improve continuously over time with focused and targeted corrective feedback.

Given the difficulty of getting large amounts of new data, users need systems that can respond to small amounts of corrective feedback and yet show noticeable improvements. One of the major deficiencies of historical MT systems has been the lack of user control, the inability of users to make any meaningful impact on the quality of raw output produced on an ongoing basis.This ability to CONTINUALLY steer the MT engine with financially feasible amounts (i.e. relatively small) of corrective feedback is a key to getting the best long-term productivity results and ROI. I think as users get more informed on how to work with this technology, they will zero in on this ability of some expert MT systems. IMO, it is the single most important criterion when evaluating competitive MT systems:

What do I have to do to improve the raw system output quality once an initial engine is in place?
And, how much effort/data is required to get meaningful and measurable improvements?
Measurable = Rising average throughput of post-editors (By hundreds or thousands of more words a day, and often a multiple of what is possible with instant MT).

The issue of data cleaning is also not well understood. While it is helpful to remove tags and formatting information, it is also important to validate the linguistics and the quality of translations in addition to this to avoid GIGO results. Users should take care to keep data in the cleanest possible state (format wise and linguistically) as it can provide real long-term business production leverage on a scale greater than most TM data can. What most successful users will find is that 90%+ of the time spent in developing the highest quality engines is spent in corpus and data analysis, data preparation and organization, error detection and correction. The Moses step is a tiny component of the whole process of developing superior MT engines.

Control & Data Security
One of the reasons why it may make sense to use Moses sometimes is to keep your data and training and translation activity REALLY REALLY private (e.g. translations of interrogation transcripts where persuasion involving water might be used). The need for security and privacy especially makes sense for national security applications, but I find it hard to understand the resistance some global companies have, to working in the cloud when a lot of this MT and PEMT content ends up on the web anyway. For most companies cloud computing simply makes sense and spares the user from the substantial IT burden of maintaining the hardware infrastructure needed to play at the highest professional level. (Asia Online actually makes it’s full training and translation environment available for on-premise installation for large enterprise customers like LexisNexis who process hundreds of millions of words a day and have suitable computing and human resource expertise to handle this).

I have heard of several LSPs who have spent $10K–$20K on servers that will probably only do Moses training once a year. If you do not have the data to drive an improvement in your Moses engine, what is the point of having these kinds of servers? There is no point in trying to re-train an engine when you don’t have enough new data to make any noticeable impact. This is a technology that just makes much more sense in the cloud, for scalability, extensibility, security and effective control. Cloud solutions are often more secure than on-premise installations at LSPs because cloud service providers can afford the IT staff that has deep expertise on computer security, data protection and data availability management. (BTW I have also seen what happens when hacks try and manage 200 servers = not pretty). Like many other things in today’s world, IT (Information Technology) has become so specialized and complex that it makes more sense to outsource much of it, and work in the cloud rather than try and do it on your own with a meager and barely trained staff. Compare your IT staff capabilities to any cloud service provider. Even Microsoft Office is finally making the transition to the cloud. Some analysts are even saying that the shift to the cloud will challenge the dominance of older stalwarts like HP, Microsoft, Intel, SAP, RIM, Oracle, Cisco, Dell and that a third of these companies may not be around in in 2020. Remember DEC and Wang? In a world where tablets, smartphones and mobile platforms will increasingly drive global commerce, the desktop/server perspective of traditional IT is already fading, and makes less sense with each passing day. It is ironic to see LSPs jumping on the “On-Premise Server” train just as it about to reach the end of the line.

Cloud based MT can also be setup to be always improving (assuming you have more than basic Moses MT) as new data is added regularly and feedback gathered from users as Google and Bing do. Setting up this kind of infrastructure is a significant undertaking and most Moses users will never get to that point, but this is how the best MT systems will continue to evolve. What some may find is, that their domain focused MT system may be better than the public engines in January, but by June this may no longer be true. You should realize that you are dealing with a moving target and most public engines will continue to improve. All the expert MT developers are constantly updating and enhancing their technology, most have already moved beyond the phrase-based SMT that Moses is today, and are incorporating linguistics in various forms. This can only be done because they understand what they are doing. Some of these enhancements may make it back to Moses years later but the productivity edge will remain with experts in the foreseeable future and I expect in 2012 we will see several case studies where expert MT systems outperform instant Moses systems by significant margins. So my advice; Be wary of any kind of instant MT solution that is not free.

I started the eMpTy Pages blog in early 2010, and one of my earliest posts was on the importance of clean data for SMT. It was blasphemy at the time to question the value of sheer data volume for SMT, but in the period since then, many have validated that working with consolidated TM from multiple sources, trusted though they may be, is a tricky affair and data quality does matter. Pooling data can work sometimes but will also fail often without cleaning and standardization.

The origin of the phrase “Let a thousand flowers bloom” is attributed to a misquote of Mao Zedong. The results for Chinese intellectuals who took Mao seriously were quite unfortunate. Fortunately we live in better times (I think?) and this phrase is not likely to have such dire consequences today. However, while a thousand MT systems may bloom (or at least be seeded), I predict that many will fade and die quickly. This is not necessarily bad, as hopefully institutional, community and industry learning will take place, and some practitioners may actually discover that they now have a much better appreciation for corpus linguistics and some of the skills that drive the creation of better MT systems. The experimental evidence from many failed experiments with Moses will also provide useful information for MT experts and further enhance the state of the art and science of MT. The learning curve for this technology is long and arduous and it may take a while for the dust to settle from the current hype, but I fully expect that by December 21st, 2012 it will be clear that expertise, experience and knowledge does matter with something as complex as Moses. Dead flowers are also used to fertilize gardens and help other plants thrive, and as long as we have the long view, we will continue to move onward and upward. I will restate my prediction, that the best MT systems will still come from close collaboration between MT experts with linguists, translators, LSPs and insight drawn from experience and failure.

And you can send me dead flowers every morning
Send me dead flowers by the mail
Send me dead flowers to my wedding
And I won't forget to put roses on your grave

A celebration for dead flowers

Wanted: A Fair and Simple Compensation Scheme for MT Post-Editing

2011-11-29T17:37:00.001-08:00

As the subject of fair and equitable compensation to post-editors of MT is important to the ongoing momentum of MT, I would like to introduce some people who have examined this issue, and have made an attempt (however imperfect) to developing a solution. The initial response to many such initiatives often seems to be criticism of how the approach fails. I am hoping that the dialogue on these ideas can rise above this, to more constructive and pragmatic advice or feedback to help the continuing evolution of this approach to reach more widely accepted levels of accuracy. The MemSource approach is something that measures the effort after the work is done. Used together with other initiatives that attempt to provide some measure of the post-editing task a priori, I think it could have great value in developing new compensation models that make sense to all the stakeholders in the professional translation world. It is important to develop new ways to measure MT quality and post-editing difficulty as this will become increasingly more common in the professional translation world.

This is a guest post by David Canek, CEO of MemSource Technologies. I have not edited David’s article other than selecting some phrases that I felt were worth highlighting for a reader who skims the page.

======================================

Throughout 2011 MemSource, a provider of a cloud-based translation environment and CAT tool, has run a number of workshops, exploring the impact of machine translation on the traditional translation workflow. We had lots of debates with translation buyers, LSPs, as well as translators on machine translation post-editing and specifically on how it should be compensated. We have shared our findings at the 2011 Localization World in Barcelona and we thought it may be interesting to also share them here, on the eMpTy Pages blog.

Translation Buyers and MT

While the majority of translation buyers still need to discover machine translation, there are many organizations whose progress with MT goes beyond the pilot phase. The innovators, among them many software companies, have successfully used machine translation to make the traditional translation process more efficient. One headache still remains: A simple and fair compensation scheme for machine translation post-editing. Today, typically a flat reduction of the “normal” translation rate is negotiated with the vendor, disregarding the actual effort of the translator spent on post-editing a specific document, let alone a specific segment. This can be rather imprecise, even unfair as MT quality can vary significantly from document to document, and of course segment to segment.

Translators and MT

There is a myth that all translators dislike machine translation post-editing. In fact many translators have started MT post-editing as their standard translation workflow long before anyone requested them to do so. They themselves chose to use MT because it helped them increase their productivity. Then, some years later, they were approached by their LSP/client regarding MT. Perhaps it went like this?

Dear translator,

We have introduced this great new technology, it is called machine translation. It will help you speed up your translation and – by the way we will cut your rates by 30%.

All the best...

Of course, none of the translators could be happy at the face of this news. The innovative translators - already using MT to speed up their translations - would not be happy because nothing would change for them except that their rates would get cut. The less innovative also had no reason to be happy – they had to adapt to a new translation method and their rates got cut – without any guarantee that the new translation workflow would actually speed up their translation process.

LSPs and MT

Language service providers, generally speaking, are not too fast to adopt machine translation. This may come as a surprise, as LSPs should be most interested in slashing their costs with intelligent use of MT. However, LSPs, it seems, face specific obstacles, which make MT adoption not a simple task. In contrast to translation buyers, LSPs have to cope with limited resources, yet on the other hand have to tackle multiple language pairs and subject domains, spanning across all of their clients. Training a custom MT engine in this context is a bit challenging. The available online MT services, such as Google Translate or Microsoft Translator, are perceived by many LSPs as inadequate, mainly because of “confidentiality” concerns. The – growing – minority of LSPs that have started using custom MT engines report mixed results but are generally quite optimistic about the output.

Getting the right MT technology in place is important but not enough. LSPs need to make sure that there is ROI on the new technology. That means they need to modify their translation workflow to include machine translation and most of all have to make sure the new workflow makes translating faster, i.e. cheaper. This means that they will have to renegotiate rates with their translators. All of this is far from trivial and if not done carefully, it can cause more trouble than good.

Fair Compensation for MT Post-editing

MT is an innovative technology that will eventually (though not equally across all language pairs and domains) make human translation faster, i.e. cheaper. It is important that all stakeholders benefit from this increased efficiency: Translation buyers, LSPs and translators.

Above all, compensation for MT post-editing should be fair. There can be different ways. Some translation buyers run regular productivity tests and, based on the results, apply a flat discount on translations supported by MT (I believe Autodesk has a fairly sophisticated approach to this). At MemSource we have tried to come up with a different, perhaps complementary, approach, which is based on the editing distance between the MT output and the post-edited translation. Indeed, quite simple. We call this the Post-editing Analysis. In fact this approach is an extension of the traditional “TRADOS discount scheme”, which long ago became a standard method for analyzing translation memory matches and the related discounts in the translation industry.

Post-editing Analysis: How It Works

When a translation for a segment can be retrieved from translation memory (a 100% match), the translation rate for that segment is reduced – typically to just 10% of the normal rate. A similar approach can be applied to MT post-editing. If the MT output for a segment is approved by the post-editor as correct, then we can say we have a 100% match and the post-editing rate should be very moderate for that segment. If, on the other hand, the post-editing effort is heavy and the machine translated output needs to be completely rewritten for a segment, a full translation rate should be paid. In the post-editing analysis, there is, of course an entire scale ranging from 0% to 100% when calculating the similarity (editing distance) between the MT output and its post-edited version. The rates can be adjusted accordingly.

The advantages of the post-editing analysis:

· Simple

· Transparent

· Measurable at segment-level

· Extension of the established TM discount scheme

There are also some disadvantages. Namely, the analysis can be run only after the post-editing has been carried out, which means that any discounts can be determined only after the translation job is completed. Another objection could be that the editing distance is a simplification of the actual effort of the post-editor. Indeed, this could be valid and a more complex approach could be applied. However, our goal was to come up with a simple and efficient approach, which could be easily implemented into today’s CAT workbenches and translation environments.

Interested to Know More and Experiment?

More details on the MemSource Post-editing analysis, incl. a sample post-editing analysis can be found on our wiki. If you are interested to share your experiences with MT post-editing initiatives and/or find out more about our efforts in this space, sign up for a webinar, etc. write to labs@memsource.com

David Canek is the founder and CEO of MemSource Technologies, a software company providing cloud translation technology. David, a graduate from Translation and Comparative Studies, received his education at Charles University, Prague, Humboldt University in Berlin and the University of Vienna. His professional experience includes business development and product management roles in the software and translation industries. David is keen on pursuing innovative trends in the translation industry, such as machine translation post-editing or cloud-based translation technologies and has presented on these topics at leading industry conferences, such as Localization World, Tekom, ATA and others.

The Building Momentum for Post-Edited Machine Translation (PEMT)

2011-10-28T12:08:00.001-07:00

This is an (opinionated) summary of interesting findings from a flurry of conferences that I attended earlier this month. The conferences were the TAUS User Conference, Localization World and tekom. Even though it is tiring to have so many so close together, it is interesting to see what sticks out a few weeks later. For me TAUS and tekom were clearly worthwhile, and Localization World was not, and I believe that #LWSV is an event that is losing it’s mojo in spite of big attendance numbers.

Some of the big themes that stand out (mostly from TAUS) were:

Detailed case studies that provide clear and specific evidence that customized MT enhances and improves the productivity of traditional (TEP) translation processes
The Instant on-demand Moses MT engine parade
Initial attempts at defining post-editing effort and difficulty from MemoQ and Memosource
A future session on the multilingual web from speakers who actually are involved with big perspective, global web-wide changes and requirements
More MT hyperbole
The bigger context and content production chain for translation that is visible at tekom
Post-editor feedback at tekom
The lack of innovation in most of the content presented at Localization World

The archived twitter stream from TAUS (#tausuc11) is available here, the tekom tag is #tcworld11 and Localization World is #lwsv. Many of the TAUS presentations will be available as web video shortly and I recommend that you check some of them out.

PEMT Case Studies

In the last month I have seen several case studies that document the time and cost savings and overall consistency benefits of good customized MT systems. At TAUS, Caterpillar indicated that their demand for translation was rising rapidly and thus they instituted their famed controlled language (Caterpillar English) based translation production process using MT. The MT process was initially more expensive since 100% of the segments needed to be reviewed but they are now seeing better results on their quality measurements from MT than from human translators on Brazilian Portuguese and Russian according to Don Johnson, Caterpillar. They expect to expand to new kinds of content as these engines mature.

Catherine Dove of PayPal described how the human translation process got bogged down on review and rework cycles (to ensure PayPal brand’s tone and style was intact) and was unable to meet production requirements of 15K words per week with a 3 day turnaround in 25 languages. They found that “machine-aided human translation” delivers better, more consistent terminology in the first pass and thus they were able to focus more on style and fluency. Deadlines are easier to meet and she also commented that MT can handle tags better than humans. They also focus on source cleanup and improvement to leverage the MT efforts and interestingly the MT is also useful in catching errors in the authoring phase. PayPal uses an “edit distance” measurement to determine the amount of rework and have found that the MT process reduces this effort by 20% on 8 of 10 languages they are using MT on. An additional benefit is that there is a new quality improvement process in place that should continue to yield increasing benefits.

A PEMT user case study was also presented by Asia Online and Sajan at the Localization Research Conference in September 2011. The global enterprise customer is a major information technology software developer, hardware/IT OEM manufacturer, and comprehensive IT services provider for mission critical enterprise systems in 100+ countries. This company had a legacy MT system developed internally that had been used in the past by the key customer stakeholders. Sajan and Asia Online customized English to Chinese and English to Spanish engines for this customer. These MT systems have been delivering translated output that even beats the first pass output from their human translators due to the highly technical terminology, especially in Chinese. A summary of the use case is provided below:

27 million words have been processed by this client using MT
Large amounts of quality TM (many millions of words) and glossaries were provided and these engines are expected to continue to improve with additional feedback.
The customized engine was focused on the broad IT domain and was intended to translate new documentation and support content from English into Chinese and Spanish.
A key objective of the project was to eliminate the need for full translation and limit it to MT + Post-editing as a new modified production process.
The custom engine output delivered higher quality than their first pass human translators especially in Chinese
All output was proof read to deliver publication quality.
Using Asia Online Language Studio the customer saved 60% in costs and 77% in time over previous production processes based on their own structured time and cost measurements.
The client also produces an MT product, but the business units prefer to use Asia Online because of considerable quality and cost differences.
Client extremely impressed with result especially when compared to the output of their own engine.
The new pricing model enabled by MT creates a situation where the higher the volume the more beneficial the outcome.

The video presentation below by Sajan begins at 27 minutes (in case you want to skip over the Asia Online part) and even if you only watch the Sajan presentation for 5 minutes you will get a clear sense for the benefit delivered by the PEMT process.

A session on the multilingual web at TAUS by the trio Bruno Fernandez Ruiz, Yahoo! Fellow and Vice President, Bill Dolan, Head of NLP Research, Microsoft, Addison Phillips, Chair, W3C Internationalization Group / Amazon also produced many interesting observations such as:

The impact of “Big Data” and the cloud will affect language perspectives of the future and the tools and processes of the future need to change to handle the new floating content.
Future applications will be built once and go to multiple platforms (PC, Web, Mobile, Tablets)
The number of small nuggets of information that need to be translated instantly will increase dramatically
HTML5 will enable publishers to be much freer in information creation and transformation processes and together with CSS3 and Javascript can handle translation of flowing data across multiple platforms
Semantics have not proven to be necessary to solve a lot of MT problems contrary to what many believed even 5 years ago. Big Data will help us to solve many linguistic problems that involve semantics
Linking text to location and topic to find cultural meaning will become more important to developing a larger translation perspective
Engagement around content happens in communities where there is a definable culture, language and values dimension
While data availability continues to explode for the major languages we are seeing a digital divide for the smaller languages and users will need to engage in translation to make more content in these languages happen
Even small GUI projects of 2,000 words are found to have better results with MT + crowdsourcing than with professional translation
More translation will be of words and small phrases where MT + crowdsourcing can outperform HT
User s need to be involved in improving MT and several choices can be presented to users to determine the “best” ones
The community that cares about solving language translation problems will grow beyond the professional translation industry.

At TAUS, there were several presentations on Moses tools and instant Moses MT engines via a one or two step push button approach. While these tools facilitate the creation of “quick and dirty data” MT engines, I am skeptical of the value of this approach for real production quality engines where the objective is provide long-term translation production productivity. As Austin Powers once said, “This is emPHASIS on the wrong syllABLE" My professional experience is that the key to long-term success (i.e. really good MT systems) is to really clean the data and this means more than removing formatting tags and removing the most obvious crap. This is harder than most think. Real cleaning also involves linguistic and bilingual human supervised alignment analysis. Also, I have seen that it takes perhaps thousands of attempts across many different language pairs to understand what is happening when you throw data into the hopper, and that this learning is critical to fundamental success with MT and developing continuous improvement architectures. I expect that some Moses initiatives will produce decent gist engines, but are unlikely to do much better than Google/Bing for the most part. I disagree with Jaap’s call to the community to produce thousands of MT systems, what we really need to see are a few hundred really good, kick-ass systems, rather than thousands that do not even measure up to the free online engines. And so far, getting a really good MT engine is not possible without real engagement from linguists and translators and more effort than pushing a button. We all need to be wary of instant solutions, with thousands of MT engines produced rapidly but all lacking in quality and "new" super semantic approaches that promise to solve the automated translation problem without human assistance. I predict that the best systems will still come from close collaboration with linguists and translators and insight borne from experience.

I was also excited to see the initiative from MemoQ to establish a measure of translator productivity or post-editing effort expended, by creating an open source measurement of post-edited output, where the assumption is that an untouched segment is a good one. MemoQ will use an open and published edit distance algorithm that could be helpful in establishing better pricing for MT post-editing and they also stressed the high value of terminology in building productivity. While there is already much criticism of the approach, I think this is a great first step to formulating a useful measurement. At tekom I also got a chance to see the scheme that MemSource has developed where post-edited output is mapped back to a fuzzy matching scheme to establish a more equitable post-editing pricing scheme than advocated by some LSPs. I look forward to seeing this idea spread and hope to cover it in more detail in the coming months.

Localization World was a disappointing affair and I was struck by how mundane, unimaginative and irrelevant much of the content of the conference was. While the focus of the keynotes was apparently innovation, I found the @sarahcuda presentation interesting, but not very compelling or convincing at all in terms of insight into innovation. The second day keynote was just plain bad, filled with clichés and obvious truisms e.g. “You have to have a localization plan” or “I like to sort ideas in a funnel”. (Somebody needs to tell Tapling that he is not the CEO anymore even though it might say so on his card). I heard several others complain about the quality of many sessions, and apparently in some sessions audience members were openly upset. The MT sessions were really weak in comparison to TAUS and rather than broadening the discussion they succeeded in mostly making them vague and insubstantial. The most interesting (and innovative) sessions that I was witness to were the Smartling use case studies and a pre-conference session on Social Translation. Both of these sessions focused on how the production model is changing and both were not particularly well attended. I am sure that there were others that were worthwhile (or maybe not), but it appears that this conference will matter less and less in terms of producing compelling and relevant content that provides value in the Web 2.0 world. This event is useful to meet with people but I truly wonder how many will attend for the quality of the content.

The tekom event is a good event to get a sense for how technical business translation fits into the overall content creation chain and also see how synergies could be created within this chain. There were many excellent sessions and it is the kind of event that helps you to broaden your perspective and understand how you fit into a bigger picture and ecosystem. The event has 3300 visitors so it is also a much larger perspective in terms of many different views points. I had a detailed conversation with some translators about post-editing. They were most concerned about the compensation structure and post-editor recruitment practices. They specifically pointed out how unfair the SDL practice of paying post-editors 60% of standard rates was, and asked that more equitable and fair systems be put into place. LSPs and buyers would be wise to heed this feedback if they want to be able to recruit quality people in future. I got a close look at the MemSource approach to making this more fair, and I think that this approach which measures the actual work done at a segment level should be acceptable to many. This approach measures the effort after the fact. However, we still need to do more on making the difficulty of the task before the translators begin more transparent. This begins with an understanding of how good the individual MT system is and how much effort is needed to get to production quality levels. This is an area that I hope to explore further in the coming weeks.

I continue to see more progress on the PEMT front and I now have good data of measurable productivity even on a language pair as tough as English to Hungarian. I expect that a partnership of language and MT experts will be more likely to produce compelling results than many DIY initiatives, but hopefully we learn from all the efforts being made.

The Growing Interest & Concern About the Future of Professional Translation

2011-09-27T14:28:00.001-07:00

I have noticed of late that every conference has a session or two that focuses on the future. Probably because many sense that change is in the air. Some of you may also have noticed that the protest from some quarters has grown more strident or even outright rude, to some of the ideas presented at these future outlook sessions. The most vocal protests seem to be directed at predictions about the increasing use of machine translation, anything about “good enough” quality and the process and production process changes necessary to deal with the increasing translation volume. (There are still some who think that the data deluge is a myth).

Some feel personally threatened by those who speak on these subjects and rush to kill or at least stab the messenger. I think they miss the point that what is happening in translation, is just part of a larger upheaval in the way global enterprises are interacting with customers. The forces causing change in translation are also creating upheaval in marketing, operations and product development departments as many analysts have remarked for some time now. The discussion in the professional translation blogosphere is polarized enough (translators vs. technology advocates) that dialogue is difficult, but hopefully we all continue to speak with increasing clarity, so that the polemic subsides. The truth is that none of us really knows the definite future, but that should not stop us from making educated (or even wild) guesses at where current trends may lead. (I highly recommend you skim The Cluetrain Manifesto to get a sense for the broader forces at play.)

Brian Solis has a new book coming out that describes the overall change quite succinctly. The End of Business As Usual (his new book) explores each layer of the complex consumer revolution that is changing the future of business, media, and culture. As consumers further connect with one another, a vast and efficient information network takes shape and begins to steer experiences, decisions, and markets. It is nothing short of disruptive.

I was watching the Twitter feed from two conferences last week (LRC XVI in Ireland and Translation Forum in Russia) and I thought it would be interesting to summarize and highlight some of the tweets as they pertain to this changing world and perhaps provide more clarity about the trends from a variety of voices and perspectives. The LRC conference had several speakers from large IT companies who talked about their specific experience, as well as technology vendor and LSP presentations. For those who are not aware, CSA research identifies IT as one of the single largest sectors buying professional translation services. The chart below shows the sectors with the largest share of global business. This chart is also probably a good way to understand where these changes are being felt most strongly.

Here are some Twitter highlights from LRC on the increasing volume of translation, changing content, improving MT and changing translation production needs. I would recommend that you check out @therosettafound for the most complete Twitter trail. I have made minor edits to provide context and clarify abbreviations and have attempted to provide some basic organization to the tweet trail to make it somewhat readable.

@CNGL Changing content consumption and creation models require new translation and localisation models – (according to) @RobVandenberg

@TheRosettaFound We are all authors, the enterprise is going social - implications for localisation?

@ArleLommel Quality even worse than Rob Vandenberg says: we have no real idea what it is/how to measure, especially in terms of customer impact

Issue is NOT MT vs. human translation (HT). It's becoming MT AND HT. Creates new opportunities for domain experts.

Dion Wiggins. LSPs not using MT will put themselves out of business? Prediction: yes in four/five years

CNGL says 25% of translators/linguists use MT. I wonder how many use it but say they don't use it due to (negative) perception (with peers)?

Waiting for translation technology equivalent of iPhone: something that transforms what we do in ways we can't yet imagine.

Tweets from Jason Rickhard’s presentation on Collaborative Translation (i.e. Crowdsourcing) and IT go Social.

@TheRosettaFound Jason of Symantec giving the enterprise perspective, added 15-20 languages to small but popular product, built tech to support this. Not just linguistic but also legal, organizational issues to be resolved in collaborative, paid-for product.

Is collaborative translation bad & not-timely? #lrcconf Not so, a lot of translators = involved users of the content/product they translate.

Review process is different in collaborative translation. Done by voting, not by editors

The smaller the language gets, the more motivated volunteer translators are and the better collaborative translation works.

Is volunteering something for people who don't have to worry that their day-to-day basics are covered?

Does collaborative translation and collaboration mean that content owners "give up the illusion of control" over their content?

Enterprises do collaborative translation for languages they would/could not cover otherwise - true, for both profit and non-profits

Collaborative/Community will not replace existing service providers but open up more content for more languages

Language Service Providers could play an important role in community translation by building, supporting, moderating communities

It's not correct to say Community Translation = bad; Professional Translation = good

Microsoft appoints moderators with a passion for the project/language for community localization

>1,200 users translated Symantec products into 35 languages

If >1,200 were needed to translate 2 small-ish products, how can millions of translators translating 1 ZB be 'managed'?

@ArleLommel Symantec research: Community involvement in support often leads to ~25% reduction in support spend

“Super users” are what make communities scalable. Key is to identify/cultivate them early in the process

Jason Rickard: Dell is a good example of using Facebook for support. One of few companies with real metrics and insight in this area.

Jason Rickard: Symantec has really cool/systemic/well-thought ways to support community

@TheRosettaFound 21st generation localisation is about the user, about user-generated content - Ellen Langer: Give up the Illusion of Control

@ArleLommel Illusion of control? You mean we can have even less control that we have now? That's a scary thought!

@TheRosettaFound The most dramatic shifts driven by the web happened because communities took over - Imagine: 100000s of user translators translating billions of words into 100s of languages - control that!

Seems the deep and complex problems of localisation are a minute drop in the ocean of digital content management

@CNGL Discovery, analysis, transformation - Alex O'Connor tells how CNGL is addressing the grand challenges of digital content management

@TheRosettaFound Is the L10N industry due for a wave of destruction or positive transformation?

@ArleLommel Yes, Most of the mainstream technologies for translators are non-ergonomic and still in 20-year-old paradigms

Tweets from Tony Allen, Product Manager Intel Localisation Solutions presentation

@TheRosettaFound 30+ langs >200k pages >40% localised @ Intel's web presence. Intel: important to have user-driven content, interaction with the customer. Integration important, e.g. multilingual support chat. Integration, Interoperability key issues for Intel L10N. To figure out how content flows, without loss of metadata, interoperates with internal/external range of systems, is crucial.

2.5b netizens, >15b connected devices >1 zetabyte of traffic by 2015 and companies will interact with their customers using social media - type setups; new challenges for localization.

#intel What does it mean for localization infrastructures if we have >1 zetabyte of content in 2015? Current methods won't keep up

@ArleLommel #intel says that interoperability standards are required for cloud to meet future demands. L10n must evolve to meet this need too.

@ArleLommel Alison Toon (#hp) puts it this way: “localization (people) are the garbage collectors of the documentation world”

@TheRosettaFound 600GB of data in Oracle's Translation Platform - We need concise well-structured content - then we're going to be able to deliver efficient translation services - How to get it right: analyze content, identify problems and throw it back into the face of writers and developers. I18N and l10n have to get into the core curriculum at Universities says Paul Leahy (of Oracle), since we spend too much time teaching it.

Tweets from Sajan / Asia Online MT presentation

@TheRosettaFound MT cannot perform magic on bad source text - user-generated non-native-speaker content is often 'bad'

MT errors make me laugh... but human errors make me cry - an old quote from previously recycled presentations... Asia Online

Dirty Data SMT - what kind of translations would you expect? If there are no humans involved you are training on dirty data, says Asia Online. Sajan achieved 60% reduction in costs and 77% time savings for specific project - a miracle? Probably no, let’s see.

Millions of words faster, cheaper, better translated by Sajan using Asia Online - is this phenomenal success transferable? How?

XLIFF contributed to the success of Sajan/Asia Online's MT project. Asia Online's process rejected 26% of TM training data.

Tweets from Martin Orsted, Microsoft presentation

@TheRosettaFound Cloud will lead to improved cycle times and scalability: 100+ languages, millions of words

Extraordinary scale: 106 languages for the next version of Office. Need a process that scales up & down in proportion.

Microsoft: We have fewer people than ever and we are doing more and more languages than ever

Martin: "The Language Game - Make it fun to review Office"... here is a challenge :) Great idea to involve community via game

How can a "Game" approach be used for translation? Levels of experience, quality, domains, complexity; rewards?

No more 'stop & go', just let it flow @robvandenburg >>Continuous publishing requires continuous translation. New workflows

Tweets from Derek Coffey, Welocalize presentation Are we the FedEx or the WallMart of words?

@TheRosettaFound TMS of SDL = burning stacks of cash - Reality: we support your XLIFF, but not your implementation

Lack of collaboration, workflow integration, content integration = most important bottle necks. Welocalize, MemoQ, Kilgray and Ontram working on reference implementation - Derek: Make it compelling for translators to work for us

It's all about the translators and they will seek to maximise their earning potential according to Derek.

Tweets from Future Panel

@TheRosettaFound Many translators don't know what XML looks like

Rob: more collaborative, community translation - Rob: Users who consume content will have a large input into the translation BINGO

Tony: users will drive localisation decision, translation live

Derek: future is in cooking xxx? Open up a whole new market - user generated, currently untranslatable content. HUGE market

Derek: need to re-invent our industry, with focus on supply chain

The end of the big projects - how are we going to make money (question from audience)

From service/product to community - the radical change for enterprises, according to Fred

No spark, big bang, revolution - but continuous change, Derek

Big Spark (Dion): English will no longer remain the almost exclusive source language

The Translation Forum Russia twitter trail has a much more translator oriented focus and is also bilingual. Here are some highlights below, again with minor edits to improve readability.

@antonkunin Listened to an information-packed keynote by @Doug_Lawrence at #tfru this morning. As rates keep falling, translators' income keeps rising.

@ilbarbaro Talking about "the art of interpreting and translation" in the last quarter of 2011 is definitely outdated

Language and "quality" are important for translators, speed and competence for (final) clients. Really?

Translators are the weakest link in the translation process

Bert: here and now translation more important than perfect translation

Bert on fan subbing as an unstoppable new trend in translation

Is Bert anticipating my conclusions? Noah's ark was made and run by amateurs, RMS Titanic by professionals

Carlos Incharraulde: terminology is pivotal in translators training < Primarily as a knowledge transfer tool

To renatobeninatto at who said: Translation companies can do without process standards < I don't agree

@renatobeninatto: Start looking at income rather than price/rates

Evaluating translation is like evaluating haircuts - it's better to be good on time than perfect too late

Few translation companies do like airlines: 1st class/ Economy/ Discount rates – Esselink

Traditional translation models only deal w/ tip of iceberg. New models required for other 90%. Esselink

Good enough revolution. Good enough translation for Wikileaks, for example. Bert Esselink

In 2007 English Russian was $0.22 per word, in 2010 it dropped to $0.16 @Doug_Lawrence

There's much talk on innovation but not much action - don't expect SDL and Lionbridge to be innovative

@Doug_Lawrence all languages except German and French decreased in pricing from 2007 to 2010

@AndreyLeites @ilbarbaro problem-solving is the most important feature translator should acquire - Don't teach translators technology, teach them to solve problems - language is a technology, we need to learn how to use it like technology - 85% of translators are still women

@ilbarbaro 3 points on quality: 1. Quality is never absolute, 2. Quality is defined by the customer, 3. Quality can be measured - it is necessary to learn to define quality requirements (precisely)

@Kilgraymemoq announces that they will open Kilgray Russia before the end of the year

This is of course my biased extraction from the stream, but the original Twitter trail will be there for a few more weeks and you can check it out yourself. It is clear to me from seeing the comments above, that at the enterprise level, MT and Community initiatives will continue to gather momentum. Translation volumes will continue to rise and production processes will have to change to adapt to this. Also, I believe, there are translators who are seeking ways to add value in this changing world and I hope that they will provide the example that leads the way in this changing flux.

And for a completely different view of "the future" check this out.

Understanding Where Machine Translation (MT) Makes Sense

2011-09-12T16:22:00.001-07:00

One of the reasons that many find MT threatening I think, is the claim by some MT enthusiasts that it that it will do EXACTLY the same work that was previously done by multiple individuals in the “translate-edit-proof” chain without the humans, of course. To the best of my knowledge this is not possible today, even though one may produce an occasional sentence where this does indeed happen. If you want final output that is indistinguishable from competent human translation, then you are going to have to use the human “edit-proof” chain to make this happen.

Some in the industry have attempted to restate the potential of MT from Fully Automated High Quality Translation - FAHQT (Notice how that sounds suspiciously like f*&ked?) to Fully Automated Useful Translation – FAUT. However, in some highly technical domains it is actually possible to see that carefully customized MT systems can outperform exclusively human-based production, because it is simply not possible to find as many competent technical translators as are required to get the work done.

We have seen that both Google and Bing have gotten dramatically better since they switched from RbMT to statistical data-driven approaches, but the free MT solutions have yet to deliver real compelling quality outside of some Romance languages, and the quality is usually far from competent human translation. They also offer very little in terms of control, even if you are not concerned about the serious data privacy issues that their use brings to the user. It is usually worthwhile for professionals to work with specialists who can help them customize these systems to the specific purpose they are intended for. MT systems evolve and can get better with with small amounts of corrective feedback if they are designed from the outset to do this. Somebody who has built thousands of MT systems, across many language combinations, is likely to offer more value and skill than most can get from using tools like Moses building a handful of systems, or even the limited dictionary building input possible with many RbMT systems. And how much better can customized systems get than the free systems? Depending on the data volume and quality, it can range from small but very meaningful improvements to significantly better overall quality.

So where does MT make most sense? Given that there is a significant effort required to customize an MT system, it usually makes most sense when you have ongoing high volume, dynamically created source data and tolerant users or any combination thereof. It is also important to understand that the higher the quality requirements, the greater the need for human editing and proofing. The graphic below elaborates on this.

While MT is unlikely to replace human beings in any application where quality is really important, there are a growing number of cases that show that MT is suitable for:

Highly repetitive content where productivity gains with MT can exceed what is possible with just using TM alone
Content that would just not get translated otherwise
Content that simply cannot afford human translation
High value content that is changing every hour and every day but has a short shelf life
Knowledge content that facilitates and enhances the global spread of critical knowledge, especially for health and social services
Content that is created to enhance and accelerate communication with global customers who prefer a self-service model
Real-time customer conversations in social networks and customer support scenarios
Content that does not need to be perfect but just approximately understandable

So while there are some who would say that MT can be used anywhere and everywhere, I would suggest that a better fit for professional use is where you have ongoing volume, and dynamic but high value source content that can enhance international initiatives. To my mind, customized MT does not make sense for one-time, small localization projects where the customization efforts cannot be leveraged frequently. Free online MT might still prove of some value in these cases, to boost productivity, but as language service providers learn to better use and steer MT, I expect that we will see that they will provide translators access to “highly customized internal systems” for project work, and the value to the translators will be very similar to the value provided by high quality translation memory. Simply put – it can and will boost productivity even for things like user documentation and software interfaces.

It is worth understanding that while “good” MT systems can enhance translator productivity in traditional localization projects, they can also enable completely new kinds of translation projects that have larger volumes and much more dynamic content. While we can expect that these systems will continue to improve in quality, they are not likely to produce TEP equivalent output. I expect that these new applications will be a major source for work in the professional translation industry but will require production models that differ from traditional TEP production.

However, we are still at a point in time where there is not a lot of clarity on what post-editing, linguistic steering and MT engine refinement really involve. They do in fact involve many of the same things that are of value in standard localization processes , e.g. unknown word resolution, terminological consistency, DNT lists and style adjustments. They also increasingly include new kinds of linguistic steering designed to “train” the MT system to learn from historical error patterns and corrections. Unfortunately many of the prescriptions on post-editing principles available on LinkedIn and translator forums, are either linked to older generation MT systems (RbMT), systems that really cannot improve much beyond a very limited point or are linked to a specific MT system. In the age of data-driven systems new approaches are necessary and we have only just begun to define this. These new hybrid systems also allow translators and linguists to create linguistic and grammar rules around the pure data patterns. Hopefully we will see much better “user-friendly” post-editing environments that bring powerful error detection and correction utilities into much more linguistically logical and pro-active feedback loops. These tools can only emerge if more savvy translators are involved (and properly compensated) and we successfully liberate translators from dealing with the horrors of file and format conversions and other non-linguistic tedium that translation today requires. This shift to a mostly linguistic focus could also be much easier with better data interchange standards. The best examples of these are from Google and Microsoft rather than the translation industry, thus far.

Talking about standards, possibly the most worthwhile of the initiatives focusing on translation data interchange standards is meeting in Warsaw later this month. The XLIFF symposium IMO is the most concrete and most practical standards discussion going on at the moment and includes academics, LSPs, TAUS, tools vendors and large buyers sharing experiences. The future is all about data flowing in and out of translation processes and we all stand to benefit from real, robust standards that work for all constituencies.

The Continuing Saga & Evolution of Machine Translation

2011-09-05T10:12:00.001-07:00

I recently attended the 7th IMTT Conference in Cordoba, Argentina. I especially enjoy the IMTT events because somehow they have found a content formula that works for both translators and LSPs. You get to see the translation supply chain communicate in real-time. The overall culture of their events is also usually very collaborative, and to my mind the place to see the most open and constructive dialogue between translators and agencies. Some may not be aware that Argentina has a particularly strong concentration of skilled humans who understand the mechanics of localization (especially in FIGS BrPt), and many of the agencies, even small ones, are able to work with pretty much every TM and TMS (Translation Management System) system in the market with more than a basic level of competence. Because of historical decisions made by @RenatoBeninatto many years ago and a great university educational system, Argentina has become a place with a comprehensive and sizable professional translation eco-system.

There were a few presentations that I found especially interesting, including a plenary presentation by Suzanne de Santamarina on the use of quality metrics. You can see some of the twitter trail here and here but basically Suzanne described her very active use of J2450 measurements to improve the dialogue on quality with customers and with her translators. While there clearly is effort and expense involved in implementing this as actively as she has, I think it dramatically improves the conversation regarding translation quality between all the participants, as it is very specific and impersonal and clear about what quality means. It is also a means to build what she called “customer delight” which of course also includes a major service component.

Quality in a product or service is not what the supplier puts in. It is what the customer gets out (of the product/service) and is willing to pay for. A product is not quality because it is hard to make and costs a lot of money, as manufacturers typically believe. This is incompetence. Customers pay only for what is of use to them and gives them value. Nothing else constitutes quality…
~ Peter Drucker

Asia Online makes a software tool available to enable customers to calculate J2450 scores for this very reason. It helps to move the discussion from inactionable complaints like “I don’t like the quality” or “the quality is not good”, to practical error identification and resolution action steps like “Is there a way to reduce frequency of the wrong terminology errors in the system?” Just as proper use of BLEU scores requires care and some expertise so does the use of J2450. Suzanne’s company’s regular and highly structured use of J2450 is such that they can really assess the linguistic quality from project to project with a precision that few have. Her approach is refreshing in its clarity and precision and quite a contrast form the meandering inconclusive discussions on “quality” that you see in LinkedIn. Tools like BLEU and J2450 depend on the skill level of the user, and require an investment of time and effort and repeated use to develop real user competence before one understands the informational insights that their use can provide.

I also enjoyed the presentations by master translators like João Roque Dias and Danilo Nogueira on the craft and art of translation, and enjoyed talking to them about MT and the life of the translator in general. (Yes, MT is sometimes useful even for some of them.) There were several skill focused presentations on Language QA tools, CAT and collaborative tools that were also very interesting. I heard great things about Val Ivonica’s presentation (in Brazilian) on translation productivity tools which I was unable to attend as it coincided with mine. It is interesting that Patricia Bown positioned MemoQ as collaboration software that enables the linear TEP model to evolve, enabling faster turnaround and higher volume. There were many Brazilians present (though some said not enough) and they lived up to their reputation for revelry but unfortunately were thwarted in their (our) attempts to find a karaoke place one evening. Nevertheless they shared their linguistically oriented humor with me and I had no difficulty finding a willing interpreter even though I was often the only person who did not speak the language.

I delivered a presentation on the emerging role of MT as a means to deal with the translation challenges created by the content explosion and new kinds of dynamic product/business related content. The feedback I received was mostly positive and constructive even though there were several very skeptical translators in the crowd. There were some in the audience who have already experienced MT that works and even those who had not worked with customized systems admitted that sometimes MT was useful. I was also on a panel on “The Future of the Industry” which got mixed reviews as some translators felt it was not relevant and others felt it was a tired topic that nobody had any real clarity on. Many feel change is coming but are not clear what this really means and unfortunately for many the end-result of these changes is that customers expect more work for less money. This does mean that there is a certain amount of apprehension amongst the attendees as the future is not quite predictable.

A blog entry by Emily Stewart that pondered upon the theme of technology driven change at the conference a few days later, triggered an interesting and on-going discussion in LinkedIn. Her post which was about the advent of technology in a variety of different markets is thoughtful and worth reading. I also think her conclusion (shown below) is good advice for us all.

Instead of denouncing machine translation as the end of the translation world as we know it, it may be time to take a step back and see what happens. The discussion shouldn’t stop, but perhaps it could become less polemic and instead convert into a deeper conversation on and reflection of what may or may not lie ahead.

While initially there is a lot of focus on the perceived threat (there are some who think that I, together with other over zealous MT developers, am responsible for some of this fear and FUD), I am hoping that the dialogue moves beyond this point. Some MT systems have indeed been used to push rates down unfairly, but as we all begin to better understand these early mishaps, this can and must change. As George W Bush misspoke when he tried to say "Fool me once, shame on you; Fool me twice, shame on me." (I hope you click on the Bush video, it is toooo funny). If it becomes clearer to everybody what it actually takes to “finish off” MT output to required target quality levels, this kind of abuse cannot continue. We need better quality assessment so that this gap can be more clearly defined.

All MT systems are not equal and to have a global post-editing pricing policy is guaranteed to create disenchantment. We all need to better understand where to use MT and where to avoid it. MT cannot easily if ever replace humans, on the same projects that were previously done through a careful human TEP process. If the quality expectations are high, it has to be MT and human. MT makes most sense where there is ongoing volume and information volatility. We also need to better understand how to quickly assess the output quality of different MT systems so that post-editors are compensated fairly. The best MT systems are yet to come and they will be better because they are the product of informed linguistic steering in addition to standard data and MT techniques. We have yet to see useful compensation systems develop for these linguists and this will probably be needed before some of the uneasiness dissipates, but the forces driving this expanding need are strong and hopefully we should realize and recognize the value of these key individuals at some point in the future. This is already true at Asia Online so I imagine it can be done elsewhere.

In terms of disintermediation, I think MT will be only part of the whole picture, as we see more people learn to use motivated communities to get work done. Adobe and others have learnt to use “the crowd” to get traditional localization work done using translation platforms like Lingotek and newcomer Smartling (which might also have obtained the biggest startup investment made by a VC in the translation industry.) Much of the coming change will also come from collaboration software platforms like Lingotek, Smartling and others yet to come, that change how translation projects get done in terms of process flow, and that have a different modus operandi from traditional localization tools born in the TEP world. Translators are required to spend too much time working with data in different formats and too little time on the actual act of translation. New collaboration platforms and real data interchange standards will hopefully enable translators to focus mostly on real linguistic problem solving, and not on managing archaic and arcane format interchange issues.

From my vantage point, I see that -
1) Translation is increasingly done outside of the sphere of the localization world and community based translation initiatives around the world are gathering momentum both in the non-profit and corporate world
2) The volume of translation that can help drive international business initiatives forward is increasing at a substantial rate (5X to perhaps as much as 100X) Interestingly, there are still some who think that this content explosion is a myth.
3) Social network conversations matter and are often more important to translate than having user documentation that is "perfect" and “error-free”. The company to customer communications have also become much more interactive, real-time and urgent and go way beyond the scope of most user documentation.

Thus to approach every translation task with the TEP mindset that made great sense in the 1X or 2X volume days is not useful today. New approaches are needed and new models of automation/collaboration are necessary - and are perhaps the only way that all the changing momentum can be handled effectively. MT is simply one part of the equation and is far from being the whole solution. The need to solve this overall translation challenge is linked to the customers business survival so it gains a kind of momentum of inevitability. Businesses need to translate way more content to remain competitive in global marketplaces that move at internet speeds, thus automation and better collaboration are essential and critical to success and even survival.

We have seen in the last 5 years that many of the largest global organizations have launched MT initiatives on their own, because their LSP vendors were/are stuck in the TEP mindset, and did not realize that their customers had to learn to do dramatically more translation with not very much more money. This is perhaps a clue that in certain volume and time constraint scenarios, MT is necessary. We have seen that global enterprises need to solve this problem with or without vendors who historically managed the bulk of their localization translation. My sense is that this trend is likely to build momentum if LSPs do not learn to offer real MT competence. Real MT competence comes from building custom systems and seeing what works and what does not. Global enterprises will increasingly take this task upon themselves if they cannot find LSPs who can help them solve this problem e.g. TAUS is mostly a buyer driven organization with the key focus of sharing TM and facilitating large scale MT initiatives. The greatest successes presented at TAUS are all in-house initiatives with little LSP involvement. Surely this is because there is a real need, and we see that competitors are willing to share linguistic data and resources to handle this problem. I suspect that the buyer’s motivation is less about saving cents per word on translation costs, and much more about keeping and building customer loyalty and satisfaction in a world with growing global online commerce and information access needs.

My guess is that some of the anxiety on the coming change comes not so much from raw technology like MT, but perhaps it's real origin is the growing awareness that some of the work they are involved with grows less valuable to the customer’s real mission: which is to build and develop international markets. Perhaps the anxiety is really rooted in the fact that they sense that they are not involved in high value work. The real threat is not MT per se, but it is the growing awareness amongst international marketing executives (in global enterprises) that they need to focus on what their customers really care about - more and more often this is something other than getting a really great user manual out. Have you noticed that many leading edge companies like Apple, Sony have dramatically reduced their investment in user manuals? The iPhone simply does not have one (in the box but they do on the web). I am not suggesting that manuals are going away, but it is already clear that their relative value is diminishing. The content that drives global customer adoption and loyalty is changing and thus the relative value of traditional localization (software and documentation) work also changes.

I expect that new translation production models to build success in international markets will involve MT (and other translation automation), crowdsourcing as well as professional oversight and management. It is very likely that old production models like TEP will be increasingly less important, or just one of several approaches to translation projects as new collaboration models gain momentum.I think that the most successful approaches to solving these "new" translation problems will involve a close and constructive collaboration between traditional localization professionals, linguists, MT developers, end-customers and probably others in global enterprise organizations who have never worked in "localization" but are more directly concerned about the quality of the relationship with the final customer across the world. At the end of the day our value as an industry is determined by how useful our input is to the process of building international markets and the requirements for success are changing as we speak.

The conversations at IMTT and the ensuing discussions suggest that while progress is being made in the understanding of translation technology, there is still a long way to go. I hope that at future IMTT conferences we see more discussion of approaches to translation projects where TEP may not make sense and automation and collaboration approaches can help solve different kinds of problems that also further international business initiatives. I expect that IMTT will be a leader in changing the current polemic and also expand the conversation to new stakeholders. This conversation is likely to require much more direct content with product management, international sales and support teams and the final end customer. Hopefully some of us in the industry get to lead or participate in the driving this change through these new conversations.

While change can be difficult it can also be a time of opportunity and a time when leadership changes. Very few try to understand the forces of change better. People often go through a sequential emotional cycle before they learn to cope, and eventually even thrive when facing disruptive change. Those who get stuck at fear and despair, often end up as victims.

This little video shows that effective and heartfelt communication across cultures need not be heavily planned, ponderous or calculated. Sometimes simple and real is enough to create the change and build a connection to your customers.

Where the Hell is Matt? (2008) from Matthew Harding on Vimeo.

Translation Crowdsourcing

2011-08-01T12:59:00.000-07:00

The phenomena of a crowd (or community) stepping forward and doing real translation work, often for no direct financial compensation is something that troubles many in the professional translation world. Mostly because they see this activity, as work being taken away from legitimate professionals or they see it as a ploy to reduce prices. While in some cases their fears may actually be justified, in the most successful uses of this approach I think it is clear that this is not true.

As I have said before, the growing momentum in the volume of content demands new production models. This momentum which exists both in the corporate world and also the general world out there simply cannot be addressed by ONLY using traditional professional translation production models. New needs require new approaches. For those who insist that the data deluge is a fiction, the rest of this post is probably irrelevant.

Another key driving force behind new crowdsourcing initiatives is the need to engage and interact with users in new markets. In new markets having active conversations with locals is key to building brand awareness and really learning about local needs and behavior. This is frequently a more important driver than cost containment as many in the industry think.

In some cases were it not for crowd-based localization efforts, it would simply not be done as it is not economically feasible to undertake the same efforts (and expenses) as are made for FIGS/CJK for “lesser” languages. Thus crowdsourcing is emerging sometimes as a means to get “lesser” languages done.

If we look at some of the most successful examples of crowd-sourced translation in practice, we can see that they have many if not all of the following elements in common.

A Crowd/Community That Is Invested:

· TED Open Translation Project – Volunteer translators are often inspired by the content and wish to share it with their friends and countrymen. June Cohen has said that the volunteer translators in general do better quality work than the many of the paid professionals, who initially did a few translations to seed the project because of their passion for the subject and often their subject matter expertise. This effort has now enabled over 20,000 translations into 80+ languages of really challenging material. Many professionals also volunteer because they believe in the high general value of the content.

· Facebook – Users who wish to build and expand the friend community in their particular language group. This effort has enabled Facebook to grow rapidly in international markets and accomplish very rapid coverage across 60+ languages. Had they used traditional means to do this it may have taken them years to get to the same point. Critics also often miss the point that engaging real users in the translation task also encourages rapid growth of the user base as “user translators” engage friends into their network.

· Microsoft - MVPs (top accredited reseller partners) who wish to make technical support knowledge about Microsoft products more easily and widely available in their markets. Their efforts are rewarded by lower support costs and also an increase in product sales as more and more users look for self-service knowledge base information. Microsoft has been a trail blazer in making large amounts of knowledge base content available via MT, they are now adding crowd based editing to raise the quality of the translated information. Thus the most used and vital information tends to get the most attention and benefits all users.

· Asia Online – Student users provide corrective feedback to continue to improve the translation quality of the Wikipedia and other knowledge content that is initially done by highly customized MT engines and paid translators. The students themselves will be the primary beneficiaries of this content, and their efforts will enable them to access high quality educational information. The volume of this information will likely increase a thousand fold.

· Yeeyan: has 150,000 registered users, who collectively translate 50 to 100 news articles every day from English to Chinese. Since its inception in 2006, the site has grown into a key gateway for Chinese speakers who want to follow international news. It has been so successful that it has attracted the attention of major news sources like The Guardian and ReadWriteWeb. Yeeyan is focused on addressing the problem of ghettoization of information by language through a community collaboration, where members both identify interesting content and also help to translate this content.

· Adobe: This is a much more carefully managed effort designed to engage influential users, partners and customers to help provide relevant information for the broad Adobe User community in China.

T Twitter: The translation center asks Twitter users -- all volunteers -- to help translate Twitter's interface into various languages. Once the basic support pages are translated, a select group of the "most active" translators are invited to work with Twitter to "maintain localized versions of the service." Twitter boasts that its translation center has 200,000 translators, and that the localization process for Dutch and Indonesian took just one month from the first call for involvement to its announcement. The availability of its interface in multiple foreign languages is bound to increase its popularity and effectiveness not only for online marketing but also for social and political activism.

Software Infrastructure That Facilitates Contribution & Participation:

In all of the cases above the companies involved crowdsourced translation initiatives need to invest in software that enables tasks to be parceled out, evolve as tasks change, enable efficient administration, maintain quality, gather feedback, and build self-sustaining eco-systems. The tools developed by dotSUB, Lingotek, Yeeyan and Asia Online are all unique collaboration and translation workflow management tools that enable these kinds of initiatives, They make little or no use of industry standard tools like Trados and TMS because of the highly proprietary, rigidity and archaic nature of these tools. These new-generation tools are much more open and are designed to evolve with technical and process advances on the internet today. It is quite possible that these community efforts could produce tools that supercede many of the tools in use today as these new tools focus on collaboration and sharing assets to enhance the efficiency of the collaborative translation process.

The Importance of Engagement and Higher Purpose:

It is interesting to note that translation is not the primary business of any of the companies listed in the examples above. In every case the goal and intent is to make more information available faster. Even for many of the corporations that are exploring crowdsourcing, the rationale is often more about customer engagement than cost savings. It is also important to note that none of these initiatives could even be attempted without the use of automation and large-scale community support and they are enabling initiatives that would not be possible otherwise. This is also true for Facebook who still had to use professionals to translate legalese that their community was not interested in translating. The role of communities is likely to increase in future as more of the world comes online.

As we move forward we will see much more video and other rich content come online and already it is clear that the old approaches will not enable us to make this new content multilingual in an effective time frame. Crowdsourcing and automated translation will be necessary tools for an organization that seeks to communicate across the globe. As Clay Shirky has pointed out, the ‘cognitive surplus’ of the online population is a force that can be harnessed under the right circumstances and for the right purposes. It is likely that the professional translation world is going to see significant disruption in the coming years, as innovators figure out how to build sustainable models around community engagement, technology and organizational mission. However, as we have already seen, there is much that the crowd has no interest in doing and we should expect that this is not likely to change.

Crowdsourcing is here to stay and is a new mode of production that enables high–volume projects to be undertaken, engage with users and partners more deeply and participate in multilingual social networks where so many branding impressions are being formed. Managing crowdsourcing is also a major opportunity for savvy LSPs who have processes in place to recruit and manage the collaboration of dispersed volunteers and contributors.

eMpTy Pages

A Short Guide to Measuring and Comparing Machine Translation Engines

Review: Most Popular Blog Posts from 2011

The Continuing Saga & Evolution of Machine Translation was coverage of the IMTT 7th Conference in Cordoba triggered active debates and discussions MT, automation and translator compensation in several forums and clearly struck a chord for many.

The Future of Translation Memory (TM) is a posting that continues to receive high new visit rates long after it was originally published.

The Building Momentum for Post-Edited Machine Translation (PEMT) a number of case studies on the increasing use of post-edited MT to meet business timeliness and production cost requirements.

Has Google Translate Reached the Limits of its Ongoing Improvement? More evidence that more data Is not always better especially for MT, but even for Search, and the many reasons to consider the data quality, yet again.

The Growing Interest & Concern About the Future of Professional Translation About reactions to the changes underway in translation

Standards: the Importance of Measurement A guest post by Valeria Cannavina on how standards can drive quality improvements

The Moses Madness and Dead Flowers A post that questions some of the assumptions made by “instant Moses” advocates and challenges the long-term value of these experiments. Strong opinions voiced in the comments.

Translation Crowdsourcing An exploration of the driving forces underlying successful translation crowdsourcing efforts.