Saplo will have a session and talk about predictive text analytics. If you are interested in knowing more please join our session on Tuesday (you can find tickets here). We will be in Berlin between Sunday to Tuesday, if you are interested in meeting us at the conference, or at another place in Berlin, please let us know.

Picture from officialgdc

Saplo is a Red Herring 2011 Global Finalist

Saplo’s text analysis technology have been awarded before but this is something extra since Red Herring is a global competition finding the best and most promising technology companies in the world from thousands of companies. Back in May this year Saplo was awarded the Red Herring Europe Award and thanks to that and the progress during the year Saplo is now been been honored as one of the Global Finalists.

The final award and the Top 100 winners will be announced at a special awards ceremony on December 7. So keep your fingers crossed =)

About the Red Herring Award
The Red Herring editorial team selected the 2011 Finalists as the most innovative companies from a pool of thousands of companies from over 40 nations. These companies, representing past Red Herring Asia, Europe and North America winners.

The Innovative Sweden Exhibition shows the best innovations from Sweden and the first stop was Stanford in the beginning of November. Swedish Institute held a one week program with different theme days such as Future of Mobile and Cracking the Code of Innovation. As one of the 20 selected companies Saplo and our text analytics innovation was a popular visit for hundreds of the top influencers from Silicon Valley that took the chance to visit the event.

Saplo will be a part of the full tour that will continue for several years and visit cities in all continents. Next stop will be Toronto in January 2012 (January 23 to February 12) and then we will visit Washington in March (March 1 – April 30). Later next year the exhibition will visit Brasil,  Rio de Janeiro in May (May 21 – June 24) and São Paulo in July (July 9 – middle of August). The tour will later also visit both Asia and Europe.

For latest information regarding program and not yet released cities please visit Swedish Institute’s website. If you are in one of the cities or interested to have the exhibition in your city please contact Swedish Institute or write us an email.

Hope to see you on the text analytics world tour.

Cache results on client-side

When a result has been processed and returned from Saplo API (e.g. related texts or tags), the result needs to be stored on the client side. Don’t connect and make a request to Saplo API everytime you are about to present a result for a user or on a website. Fetch the result from your local storage server.

Why? Otherwise you will run out of API calls and the response time is much faster.

Store all data

Always store everything you get back from Saplo API. For example when fetching tags for a text always store all tags regardless relevance value and store all information you get from the API about the tag, e.g. relevance and category.

Why? You never know what you want to do with your data in the future. Perhaps you want to search for tags that are related to each other, create trending tag lists, order by relevance or category, show 5 related texts one week and 20 another week.

Store data in its own data structure

Always try to have a separate data structure (persistent structure) for you meta data with relations to your texts. You would also want to have your meta data indexed and searchable.

Why? If the meta data is stored in separate places (e.g. XML-files, or together with each text) it is much harder to query the meta data. Example cases when it’s better to store all data in its own data structure:

• You want to be able to find related and common tags by searching through (count) all tags.
• You want to update all tags from one certain name to another.
• You want to get the trending tags for a time period.
• You want to traverse through your related texts.

Blacklist

If you are using Saplo Tags create a blacklist system that works on article level.

Why? There might be some tags you don’t want to show for a user and these can be different for each text. It’s kind of like an hide/show functionality. Still you might want to be able to index them in a search index.

Handle manual changes

If an editor have changed the results, e.g. added, deleted, edited tags or related texts you have to make sure you handle these cases. For instance re-tagging needs to be forced manually.

Why? If an editor deletes a tag from the result and the text is sent to be tagged again the tag will re-appear which would be really annoying for the editor.

Indicator if text has “auto” result or if it has been approved manually

Use some kind of indicator so you know if a result has been reviewed or not.

Why? Some companies don’t want to push non-reviewed results to the GUI, though they want to have the result searchable and indexed automatically. This indicator can prevent a push to the GUI.

Multiple languages

Saplo API supports English and Swedish. You would need to create two collections (one Swedish and one English) and use one of them as default. If a text is not written in the default language you will not be able to add it to that collection. Than you can add it to the other collection and get results. If it can’t be added to any of the collections it has to be discarded.

Batch job (cron job) and Response Time

Depending on text length and method the response time can various and take longer than expected. Usually a response (tagging, related texts) takes a couple of seconds 2-5 seconds. You would want to keep track of texts that have not yet received results so you can fetch these later.

Send Feedback to Saplo API

Saplo is built using machine learning. This means that we become better the more users and the more feedback the system gets. For example if a tag is wrong we want to know it. Prepare your system so it is easy to send feedback to Saplo API when a user corrects something.

Flexibility for different users

What works for one user/site don’t have to be the correct setup and settings for another. Create a settings file (or admin ui) where the site owner can change properties for their setup. Examples of properties that might be important:

• Be able to change default input parameters to different API methods (e.g. change wait time, change limits, thresholds etc.)
• How tags should be presented; alphabetically, by relevance, by category etc.
• How related texts should be presented; by relevance, by date, how many etc.
• If “auto” results can be pushed to GUI instantly.
• Default collections to use in Saplo API
• API- and Secret keys

With social media becoming more and more an integrated part of people’s lives and awareness, the question is no longer how to obtain information in text, but how to select the relevant parts of it. The information itself is out there readily available.

I have always felt that the core of our prediction method is to extract some entity or number connected to the mood or sentiment in texts and process it according to the needs of a certain issue or costumer.

One could compare the vision (haha) of text analysis to the fitting of a pair of glasses. Each eye needs an individually tailored lens in order to get a perfect impression of the outside world. If the lens is too weak or strong, this impression will be blurred to a smaller or greater extent. But there exists this optimal fitting that permits the viewer to experience a razor sharp picture of the surrounding environment.

The role of a practitioner of our trade is that of the optician’s. In an environment with an abundance of text available, information can easily be blurred by the sheer volumes of sources available. With our technology there is an emerge of tools to obtain relevant information as well as performing a ”read between the lines” analysis. And, as in the case with the glasses, what relevant information constitutes is highly individual and any tool provided by us must therefore contain the possibility to be fine tuned according to the needs of the user.

Enough said about visions. Lately we have been investigating the use of text analysis in risk management. Some exciting results have emerged both in terms of the interpretation of events but more importantly in the predictive power in foreseeing future changes in entities that have text written about them.

Recently, we have been using our sentiment analysis with other techniques in order to predict future credit ratings of companies so that now not only pure economical data is used for such an analysis, but also the information contained in the text in numerous articles about that very company.

The results have been very encouraging, displaying a connection between the average sentiment in articles and actual subsequent credit rating. A perfect example of our aim to link information contained in writing to a real quantity. We hope to see many opportunities of practical importance in the wake of these findings in the near future.

Picture: http://www.seniorliving.org/

In our tiny company, we were then forced to make many hard decision, often after heated debates, that seen in retrospect were followed through, despite hardships. What I find most interesting, is that many decisions we made are only recently starting to to pay off.

Notably long term decisions were;

• to commit to a development process that is based on existing or homegrown machine learning techniques
• to build software that is language independent
• to automate everything
• to use all technologies we could get our hands on, as long as they had elements of prediction technology in them (were scalable in our opinion at the time).
• to build a scalable technology platform that can spawn many types of easy accessible technologies or high level services
• calculations and systems must scale (large amounts of text == more hidden information)
• focus on text analysis (e.g., sound analysis was briefly on the table)

Somewhat later, but nonetheless important, we decided to;

• build a user friendly and scalable API

The task of building a fully language dependent technology platform is a painstakingly long effort, where hundreds of assorted system and coding projects needs to be completed. Thus, it is not entirely easy to defend that we went down that path, considered we primarily were active in the Swedish market the first two and half years. Today, we see growing demand and interest in language independent technology. It is on the other hand easy to defend the long term work on scalable systems, as we now have the capability to provide stable services to larger organizations and can perform large scale analysis of text.

However, the one effort that consistently have kept our head above the water, in a treacherous sea of marvelous competitors, is the constant strive to automate processes and technologies. The automation of system tests, deployment processes and system administration is to a great degree, but not entirely, based on knowledge from the ICT community. Thanks! We are not the first company to build this type of large scale systems with API:s. Still, we have less resources than many competitors and manage to keep good pace with other similar efforts – while being active in more text analysis markets than may seem healthy. Why?

My firm belief is that the decision to commit to predictive technologies, rather than building programs designed to solve a specific problem, is one major reason why we still are competitive. Machine learning (or genetic programming) is by no means a silver bullet. It requires (perhaps less today) good theoretical knowledge of prediction methods (e.g., we get many ideas from the field of psychology) and how to combine them, has a rather high learning curve and requires many non-standard system solutions to work well and scale performance-wise.

The main benefit is that machine learning scales well in terms of how many developers a technology or service require. Take the case of our tagging service, which we built from scratch at Saplo. Excluded below are all API, GUI, support and market efforts required to launch a service/product, which at least triple all numbers:

• (2 p team) firstly, we built a technology that could predict the category of entities in Swedish text
• (2 p team) secondly, we built a technology that could predict entities in Swedish text
• (0.5 p team) thirdly, we shaved of a few aspects that were not fully language independent and ran the same code on English text.
• (1 p team) fourthly, we refined the above techniques to a second generation tagging technology.
• (0.2 p team) to scale to other languages (markets), or support the code base, we now don’t need much basic code work.

A third generation of this service will probably be merged with technology from the other 3 (soon 4) technologies we provide with a small team of dedicated coders.

Traditional rule based coding can also be implemented in smart ways, still in order to scale vertically in terms of languages, text formats, types of predictive tagging tasks, etc the code base will likely contain many special rules and is probably maintained by a larger team. The code will, in my opinion, also be harder to adopt to new complex problems and to combine with other technologies.

A second problem we solved, by using machine learning, is that often we found that we were entirely wrong in our assumptions of what was the best solution for a specific problems. Today, we to a great degree let the technology find the optimal solution to any given problem by rapidly implementing ideas (models) and testing them. The test effort is seldom considered lost time, even if we prove a model has poor standard. For instance, when analyzing sentiment predictions we can see that in some cases a particular technology is significant while in others it is not. It depends entirely on the purpose of the analysis task designated to our API by the end-user (or customer).

A third, perhaps hidden benefit, is that we let the technology predict/decide even minor options for running a service. This greatly reduces the need for complex decisions by the end user (e.g. when API methods are designed).

On an ending note. Yesterday, when watching an documentary about IBM:s Watson I was stricken by similarities between their and our team efforts, despite the apparent size difference between our projects. I’m not indicating we are their equals. What I interpreted as similarities was; The progress of technology based on machine learning can initially be slow – in the long term (in their case the 4:th year) it will accelerate beyond anything rule based programming can provide. In conclusion, I argue that machine learning greatly accelerates the development process in the long term.

Cheers,
Anders Hall,
coder

Fredrik Hörte and Joakim Stenberg has taken Team Saplo all the way to the finals of the prestigious Swedish Business Golf Tournament. The finals will be played in mid October at PGA Catalunya Resort, Barcelona, Spain.

Disclaimer: The image is not from any of the golf courses mentioned in the post, but from a course in France.

The tournament is played as a foursome, where players compete in teams of two, taking alternate shots. Team Saplo entered the tournament August 16 at PGA of Sweden National, together with another 100 teams.
Solid drives and amazing short game got us a third place. Luckily, since Saplo was one of the entrepreneur teams invited by Nya Affärer, that was enough to get us to the semi-finals.

The first semi-finals and the one Saplo was entering took place September 2 at Sand Golf Club together with over 60 other qualified teams from south Sweden. Team Saplo played better golf then ever before and mixed with a good portion of luck we managed amazing 7 strokes under par net after nine holes!

Unfortunately that made us realize that we had a great chance of winning the whole thing and suddenly nerves was a prominent part of the game. For another five holes we managed to keep our great score intact, then it all came crashing down. The last four holes we went from 6 under par net to even. A disaster!

Again, Team Saplo finished third, just missing the finals in Barcelona! We were pretty happy anyway, we had played great for most part of the round and got rewarded with some really great bags from Victorinox. Add to that playing two of the best golf courses in Sweden, PGA of Sweden National and Sand Golf Club, and we were all but sad.

Then September 6 during lunch Fredrik from Team Saplo gets a phone call. Apparently the best entrepreneur team from each of the semi-finals get a spot in the finals at PGA Catalunya in Barcelona!
So, of more then 1300 teams, Team Saplo will be one of eight going to Spain in October! Isn’t that a treat!

Thanks a lot to all of you that make the tournament such a great event! For those of you who understand Swedish, you can read more about the tournament at http://www.foretagsgolfen.com

Picture: Athena’s Pix

1) Filter posts – Read only the posts that are interesting for you
Reading through the Stream there is a big problem finding the most important and interesting information. Facebook has the “Top News” but in Google+ you only get the most recent news.

The picture show an example of how easy the “Filter by Interest” feature could be implemented in the current UI.

I noticed that my stream get noisy with very few users and I need some type of filter to find the most relevant posts. The solution to this problem is to atomically find the most interesting posts and I want to take this one step further than what Facebook have done. I want to analyse the meaning of the content and based on what it is about show only posts that are interesting for me. By using information such as your Sparks, the posts (and links) you shared and the posts you commented on you can use advanced text analytics to predict the relevance of each post based on the users interests.

2) Automatically Recommend Sparks
The first page you see when you visit Sparks can be more interesting. Instead of getting standard Sparks you should get personal recommendations of Sparks that are interesting for you. As before this can be done using what you have written and commented on plus the sparks you already have and in combination with text analysis this will give you some great recommendations.

Get relevant information on a Spark page
Instead of just show the posts on the Spark-page there is an opportunity to show so much more relevant information. For instance;

3) Related Persons to the spark, showing people that often comment or share information in the area. This makes it possible for persons that are interested in an area to find other people to add to circles and to interact with them.

4) Related Sparks, showing other sparks that are related to this one and makes it easy to find more interesting material.

5) Related Posts, find other posts that are about the same subject/event so you can find out about what have happened before this article was written.

The picture shows how related persons, related Sparks and related posts can be implemented in Google+.

Summary
I think there are a few killer features that could be build very easily that would be able to change Google+’s position.
This are my top five
1) Smart filer of Stream
2) Automatic recommendation of Sparks
3) Related Persons
4) Related Sparks
5) Related Posts
Sure there is also an opportunity for Facebook to implement similar features before Google does it. So who will win, Google+ or Facebook?

Note: The features are not science fiction they can all be implemented using existing text analytics technologies.

The supported languages are Swedish and English, with additional languages coming in the future depending on interest.

The module works with Drupal 6.x and PHP 5 or later. If you’re interested in seeing a version for Drupal 7 please contact us. At this moment the module is being reviewed by the Drupal community, but you’re free to download it using Git from the sandbox page.

The Saplo Drupal module supports two of the main features from the Saplo API.

Tagging

Using our module you can – with the push of a button – extract tags from articles.
The benefits of this versus a manual approach are twofold; first of all it’s very convenient to have the tags magically appear with the click of a button. Second of all you add consistency to the tagging, since the Saplo Text Analysis API is based on machine learning and will intelligently analyze and extract the tags the same way every time. It’s like having a dedicated employee just doing your tagging.

Related Articles

The module also provides you with a means to display related content, for example lists of “Related articles” that are often found at the end of news articles.
Similar modules have the capability to do this as well, but are often powered solely on tags. In other words, they make the assumption that articles that share the same tags should be related.
The Saplo API goes beyond that; we use semantic analysis and machine learning to give a higher precision to this relevance.

The caveat is that the module can only give you related content if that content has been previously analyzed. So as you analyze more and more of your articles or text, naturally more related content will be able to be displayed.
We basically establish relationships between different pieces of content, which is measured in a perceptual value. E.g. “this article is 60% similar to that article”. These relationships will be stored in the database powering your Drupal installation; so you’re free to use it in interesting ways if you want to.

Included in the module is also a basic filter, where you can change the relevance thresholds for displayed related content. So if you only want to display articles that are very related, you can easily do so in the administrative interface.

Using the Saplo Drupal Module

Installing the module is easy; you will be able to find the complete guide on the module page. In essence, it goes like this:

1. Download and unpack the module inside your module folder.
2. Enable the module in the Drupal administrative interface.
3. Enter the API Key-pair that you’ve received from signing up at http://saplo.com/signup/free into the modules administrative interface.
4. Enable the service you want (tagging, related articles) for your node types.
5. Insert a new node, and when inside the view click the “Add Saplo Metadata”-button.

The “Add Saplo Metadata”-button is probably the most common way you will be interacting with the module, besides changing settings inside the administrative interface.
Pressing this button sends the text from the node to the Saplo API. Once the Saplo API is done analysing the text it sends back the the extracted metadata that you’ve requested. The data is then saved locally, and displayed when the node is viewed.

Other features

The module implements various useful Drupal hooks. Amongst others we have different permissions that you can enable for users to access the administrative interface and to extract metadata for articles.
We also provide a small template that you can further configure that shows the related articles.