Unified Information Access Blog | Attivio Blog

Communicating Across the Two Worlds of Structured and Unstructured Information, with SQL

2012-05-17T17:56:25Z

One of our colleagues at Attivio has a niece and nephew who are as fluent in Japanese as they are in English. Their mom is Japanese and their dad is American, so they have a completely bilingual household. In one moment they might talk to each other in English, their Mom or Dad might call to them from another room in Japanese, and they will answer in kind, switching between their two languages as easily as switching TV channels.

Unified information access (UIA) technology is a lot like being strongly bilingual, in that UIA also quickly and easily communicates information that spans different worlds — specifically, structured data (databases) and unstructured content (documents/text), whether from internal and external sources.

Just as our colleague's niece and nephew can communicate as easily with anyone when visiting Japan as they can at home, a true UIA platform can also freely communicate with disparate information sources and with other applications; particularly BI tools, self-service dashboards and analytic systems. Doing so requires supporting the widely-used SQL (Structured Query Language) standard, via ODBC/JDBC connectivity.

Background

Much energy and effort has gone into the production of tools and technologies to analyze data. From iPhone and iPad apps to spreadsheets, reporting tools, "self-service" dashboards, various analytic systems, right on up to full-blown ad-hoc drag & drop BI tools, we live in an era where everything is analyzed, and the tools we use for that analysis actually contribute to better decisions. One of the keys to the interoperability of this huge ecosystem is a standards-based approach: the broad use of the Structured Query Language (SQL) is the reason the eco-system exists.

The downside to many of these tools is that they operate only on so-called structured data — until recently, ignoring valuable context contained in unstructured sources. Without an integrated and fully correlated view of the complete picture, organizations will miss out on a much wider world of business insights and understanding; not unlike relying on a really bad language interpreter (poor Bill Murray!):

Unified Information Access

Fortunately, Attivio’s Active Intelligence Enging (AIE) gives you “the best of both worlds.” Because AIE supports querying in SQL via ODBC and JDBC, organizations can use it to explore all information regardless of source or format. By deploying AIE as a back-end unified information source, your users can continue to use BI and other tools they are comfortable with — but now with the added ability to access a far more complete business informational picture, for more informed decisions and deeper understanding that is simply not possible working with structured or unstructured information alone.

One key to making this happen: Active Intelligence SQL (AI-SQL) - a set of full-text function extensions to SQL. AI-SQL functions make it easy for SQL query authors to incorporate AIE's unique features including operations like:

REGEX — find rows based on a pattern in a field.
STARTSWITH — find rows in which a given fields starts with a specified string.
ENDSWITH — find rows in which a given fields ends with a specified string.
NEAR — find rows based on two or more terms being within a specific number of words of each other
ONEAR — find rows based on two or more terms being within a specific number of words of each other in the order they are specified to the function.
FULLTEXTSEARCH — apply a simple query language query as a filter to a specified field.

The fulltextsearch function enables blended search and analytic user interfaces by enabling applications to plug user search box input into a SQL query to allow the user to interact with data - but in a controlled, simple way.

Some examples of using AI-SQL extensions:

select r_regionkey,r_name from all_tables where r_name = regex('e.*e')
select p_partkey,p_container from all_tables where p_container = startswith('brown')
select p_partkey,p_container from all_tables where p_container = endswith('car')
select p_partkey,p_container from all_tables where p_container = near('brown','car','set(distance=1)')
select p_partkey,p_container from all_tables where p_container = onear('brown','car','set(distance=1)')

select company.company, company.ticker, count(news.newsarticleid) 
FROM company INNER JOIN news ON company.ticker = news.ticker
WHERE news.content=fulltextsearch(?UserSearch)
GROUP BY company.company, company.ticker

The Importance of JOIN

It should be noted that it is no easy feat to support a detailed query language like SQL, while also serving as a UIA platform that can ingest and search across countless information sources at massive scale.

Some SQL capabilities, like a single field GROUP BY, are easily accommodated by unstructured search, and are relatively straightforward to handle. For example, given the following query:

SELECT name,count(*) FROM customers GROUP BY name

This can be easily issued as an AIE facet query, asking for facet values and counts on the name field:

query-request> table:customers
facet-request> name

However, other SQL features, like JOIN, are among the most difficult to support; but happily, they are handled by AIE’s patented ability to dynamically JOIN data and content without advance data modeling on an unstructured index. These advanced AIE capabilities allow us to execute all of the queries of the TPC-H benchmark, including TPC-H "Query 3" joining three tables and aggregating the results:

SELECT l_orderkey,  
       SUM(l_extendedprice*(1-l_discount)) as revenue, 
       o_orderdate, 
       o_shippriority
FROM customer,
     orders,
     lineitem
WHERE c_mktsegment = 'BUILDING'
      and c_custkey = o_custkey   
      and l_orderkey = o_orderkey   
      and o_orderdate < '1995-03-15'   
      and l_shipdate >  '1995-03-15'  
GROUP BY l_orderkey,   
         o_orderdate,   
         o_shippriority  
ORDER BY revenue DESC,   
         o_orderdate

Partnering for Success

While AIE supports a wide range of SQL, as with any backend system, some SQL operations will not be available. According to a leading analyst, no relational database actually supports the full ANSI-92 standard.

BI vendors recognize this and provide mechanisms to tune the SQL that a tool tries to issue to a specific backend. To help our customers in ensuring success with a given BI tool, Attivio has implemented a BI vendor certification program, which currently includes Tableau, TIBCO Spotfire and QlikView.

This program certifies tools into two levels:

Gold — The BI tool is certified to work with AIE using standard SQL
Platinum — The BI tool is certified to work with AIE using standard SQL, but also supports the use of AI-SQL extension functions.

Opening up the flexibility of AIE’s universal index to support a powerful industry standard language like SQL via ODBC and JDBC, enabling BI tools and scores of other compatible applications to access and analyze unified information, has proven to be a compelling value proposition for organizations looking for new opportunities to build revenue, cut costs and/or increase competitiveness. Clearly, AIE also speaks the universal language of business success.

Learn more about AIE’s support for SQL via ODBC and JDBC, as well as AIE’s Query-Time JOIN with no data modeling required.

How We Handle Open Source

2012-05-10T11:54:35Z

Attivio, like many other companies, uses open source where appropriate. The Java community in particular has great open source technologies powering some of the hottest technology trends today, from big data (Hadoop and MapReduce) to columnar storage (MonetDB) to IDE frameworks (Eclipse).

There‘s also a trove of infrastructure technologies out there for logging, dependency ingestion, character set normalization, etc. Knowing when to use open source and when to write your own code is where the real value comes in as a developer and more importantly, can save you and the company a great deal of time and resources.

All that being said, there are often bugs and functional gaps in open source code and as a developer you need to have a system in place that allows you to handle these issues. For example, we may find a bug in a particular package and submit a patch but since we don't control the release cycle of the open source projects we can't simply wait around for the fix. We are also very picky about less critical run-time issues such as threads being left around after a process or unit test finishes. Many open source projects assume that they are going to be run as a standalone server that terminates with the JVM and these sorts of assumptions can break or disturb our unit test environments.

We've recently switched to the vendor branching methodology described here against our Subversion repository. This allows us to import external projects into our revision control system.

When we encounter an issue in open source code, we:

Check in a copy of the source code for the class(es) we plan to fix or enhance. It's important to have this copy so we can easily compare and merge new revisions of the upstream project with our own modifications.
Make changes to the class(es) annotating them with //attivio start mod and //attivio end mod to make quick scans for changes that are easier to tease out. We put a reference to the internal developer ticket in the code as well; so later reviewers can back track those changes easily when upgrading upstream code.
Compile and build these changes into a JAR file that contains not only the class files, but also the source files in order to comply with many open source licenses. Generally speaking, there is no intellectual property in this code so we don't have to worry about anyone reviewing our changes.
Deploy this JAR to our internal Maven repository with an Attivio specific revision. We then update our top level POM (Project Object Model) to reference this Attivio version instead of the public Maven repository version.

Lastly, we strive to create a formal ticket, patch and test for contribution back to the open source community. In all fairness, this is our weakest part of the process, but one we are striving to improve. Many of our changes are small in nature and fix either esoteric edge cases or general code cleanliness like the thread example I mentioned above, but most changes are still useful for the wider community.

Enterprise Strategy Group Report - Today's Information Access Requirements Outpace Open Source Search Options

Thinking About Replacing Your Search Engine or Search Appliance?

Software at the Speed of Light

More About Unstructured Information

2012-04-30T12:58:57Z

I recently attended the YES Boston panel on Big Data & Analytics at the Harvard Innovation Center. Overall it was an excellent discussion. At least one panelist indicated that "It feels like 1995 again", referring to that heady period when the Internet emerged and drove the dot-com era forward. Most of the focus was on the superb opportunities that Big Data creates for entrepreneurs. A few panelists also suggested that Big Data would lead to the death of the traditional relational database and data warehouse.

Earlier in the discussion, one panelist characterized Big Data as having "three V's — Volume, Velocity, Variety" etc. This has become the standard way to segment the various use cases that collectively add up to "Big Data", as well as a number of other often cited characteristics like value and complexity. However, another panelist then said that Big Data was mostly about unstructured information. I have written about how all unstructured data is not the same previously. Most "unstructured" in the context of Big Data is "data" — variable length log files, etc. Not truly unstructured CONTENT, which includes articles, web pages, documents, email, etc. It is important to understand the difference, especially from the entrepreneur's perspective. Much of the "volume" in the Big Data that is unstructured DATA — again, log files, mostly — has very low individual value. It is only when analyzed in volume that it becomes interesting and valuable.

One of the more interesting questions at the end of the panel came from Wikibon's Dave Vellante. He asked, more or less, "Why are big data and the new technologies that are emerging to analyze it going to be disruptive to the enterprise data warehouse?"

Here, the panelists' answers seemed uncertain. Several spoke about the challenge of getting centralized IT to produce new information from legacy BI tools. While this is probably impossible to argue with, at least one panelist went too far, saying that data will just be kept in new systems and BI tools will work directly against them. I didn't buy this angle, and in a follow-up conversation with Dave, he agreed with me that the panel mostly missed the mark.

I would have answered Dave's question like this: the key with Big Data is to take the volume of low value items and turn it into high-value analysis. That analysis then needs to be co-mingled with other information that has high item value. This includes email, documents, text in applications, rows in databases, ERP, CRM etc. That isn't disruptive to the eDW in and of itself. Most big data — like behavioral information — won't be interesting to typical corporate decision makers. A few data scientists etc. will analyze click streams and use it to optimize the end user experience. The transactions (hopefully sales) that result from that will go into the eDW. The yield improvements will also be analyzed and tracked over time — again, probably by traditional BI tools.

The best example I can give of a real world case is from Attivio's IT Knowledge Expert solution. ITKE analyzes log files from operating systems and applications to identify events that are interesting and/or problematic. For example, we may drop or summarize informative messages, keep warnings, and correlate errors. This is great because it helps system administrators quickly discover the symptoms of an issue. However, it is the other data — the high value articles, knowledge bases, SharePoint articles created by previous admins, etc., in which the solution to the problem is found. This is why I refer to content as high-value. It explains WHY things happen.

Missing Some Key Points with Big Data

Extreme Information: Completing the Big Data Picture

Attivio IT Knowledge Expert

RESTing Easy

2012-04-05T12:05:15Z

Introduction

Modularity is a hallmark of good application development practice. When attempting to rapidly implement a web-based application, basing the front end on one or more REST APIs is a common choice. A REST API is simple to use and provides for easy client-side bookmarking. Since most of our customers develop some type of custom web UI for AIE-based applications, we've made it simple to build robust, testable REST APIs.

Building a REST API in AIE requires extending PlatformComponent and overriding two methods. Testing the API involves a few lines of code and use of shipped test framework classes (the same we use to test AIE). In the example below, I create a REST API that provides the current total memory and memory in use by the hosting JVM.

A simple REST API

Creating an AIE service that provides a REST API is as simple as extending the PlatformComponent class and overriding a couple of methods.

Override convertCGIRequest

Whenever a service that exposes an HTTP endpoint is accessed with a GET request, the request is converted by AIE into a CgiRequest message. CgiRequest messages contain all of the parameters and headers of the request. The convertCgiRequest method is a hook for the developer to translate the request into a message that the REST service can handle. This can be as simple as translating the request into a StringMessage with the relevant data, but I prefer to use simple inner classes to make requests concrete instead of opaque (this also makes testing easier). Returning null from this method indicates the CGI request is not recognized and the service should return nothing. Alternatively, you can throw an exception that will result in an error getting returned to the caller.

Sample GET request

http://localhost:17001/memtracker?mem=true

Overridden class

/** {@inheritDoc} */
  @Override
  protected PlatformMessage convertCgiRequest(CgiRequest cgi) throws AttivioException {
    String mem = cgi.getCgiParameter("mem");
    if (mem != null) {
      return new MemRequestMessage();  // concrete private message class
    } else {
      return null; // no valid request found... alternatively could throw an exception
    }
  }

The inner message class

public static class MemRequestMessage extends AbstractPlatformMessage {}

Override handleMessage

handleMessage()

/** {@inheritDoc} */
  @Override
  protected PlatformMessage handleMessage(MessageContext context, PlatformMessage msg) throws AttivioException {
    if (msg instanceof MemRequestMessage) {
      return new CgiResponse() {
        @Override
        public void writeResponse(HttpServletResponse resp) throws IOException {
          resp.getWriter().format("%d,%d", Runtime.getRuntime().freeMemory(), Runtime.getRuntime().maxMemory());
        }
      };
    }
  }

Testing

As has been pointed out here, here and here, at Attivio we really believe in testing. We support that philosophy by making it as easy (fast to write/fast to run) to test functionality as possible. To that end, we have developed supporting classes for testing PlatformComponents (including services) without having to start a full AIE system. With the AIE SDK, we ship those supporting classes so that our customers can test easily as well.

Testing the Service

@Test
  public void cgiTest() throws AttivioException, IOException {
    MemoryApiService srv = new MemoryApiService();
    TransformerTestUtils.startTransformer(srv); // mock-up system stuff and start the service
    
    CgiRequest req = new CgiRequest(null);
    req.setCgiParameters("mem=", IOUtils.DEFAULT_ENCODING); // provide the full cgi query string we are testing, use UTF-8 encoding
    CgiResponse respMessage = (CgiResponse) srv.onCall(null, req); // execute the request
    MockHttpServletResponse mockServletResponse = new MockHttpServletResponse();
    respMessage.writeResponse(mockServletResponse);

    String[] results = mockServletResponse.getWrittenResponse().split(",");
    Assert.assertTrue(Long.parseLong(results[0]) < Long.parseLong(results[1]));
    Assert.assertTrue(Long.parseLong(results[0]) > 0);
 }

The MockHttpServletResponse that is available in AIE 3.1 is a very simple class. Below is a partial implementation for your use if needed.

/**
   * A mocked version of a HttpServletResponse
   */
  public class MockHttpServletResponse implements HttpServletResponse {
    private final ByteArrayOutputStream bos = new ByteArrayOutputStream();
    private final PrintWriter writer = new PrintWriter(bos, true);
    /**
     * @return the response written so far
     */
    public String getWrittenResponse() {
      return bos.toString();
    }
    /** {@inheritDoc} */
    @Override
    public PrintWriter getWriter() throws IOException {
      return writer;
    }
   ////  rest of default (unchanged from Eclipse auto-generation) implementation omitted for brevity.

“Something is not Right!” – Don’t Ignore Your Gut When Analyzing Information

2012-03-27T12:32:51Z

Ludwig Bemelmans' classic picture book Madeline has been enjoyed by generations of children — my own daughters included — who are perennially attracted to the story of Madeline, the smallest yet most adventurous of twelve little girls in a Paris boarding school.

The story also has an important lesson on analyzing — and questioning — information.

In Madeline, Miss Clavel, the girls' teacher and caregiver, suddenly awoke one night sensing trouble:

In the middle of the night
Miss Clavel turned on her light
and said, "Something is not right!"

Sure enough, she found Madeline in her bed, in pain from appendicitis. Of course, all turns out well, thanks to Miss Clavel listening to her personal sense that something was not right.

That scene from Madeline somehow came to mind while recently reading Know What You Don't Know: How Great Leaders Prevent Problems Before They Happen (2009) by best-selling author and business professor Michael Roberto. One of the most troubling causes of unseen problems mushrooming into catastrophes, Roberto writes, is an organizational culture that dismisses intuition in favor of hard data:

Some organizations exhibit an intensely analytical culture. They apply quantitative analysis and structured frameworks to solve problems and make decisions. "Data rule the day; without a wealth of statistics and information, one does not persuade others to adopt his or her proposals".

Roberto convincingly drives this point home with a serious real-world medical problem: many hospitals experience high levels of cardiac arrest among admitted patients. According to one study, hospital personnel who did observe some advance warning sign(s) of cardiac arrest alerted a doctor only 25% of the time. Why? Nurses and other staff members might have noticed a change in patient monitoring data, but when observed in isolation, the data did not clearly indicate an urgent problem. Other warning signs are not based on quantifiable data, such as the patient's mental condition and level of fatigue or discomfort.

Often nurses and other staff felt a Miss Clavel-like sense that "Something is not right" with a patient who was indeed approaching cardiac arrest; but, with nominal hard data, if any, to support their concern, they did not feel comfortable alerting a doctor. The consequences of hospital cultures that unwittingly compel caregivers to ignore their intuition are high: once the window of opportunity to avert cardiac arrest closes, a "Code Blue" crisis is at hand — with a survival rate of less than 15%.

Many hospitals nationwide have since implemented a highly successful program to sharply reduce Code Blue incidents. Nurses and staff are actively encouraged to report observed warning signs, as well as concerns not yet supported by observed data, to a new Rapid Response Team. The team will arrive at an affected patient's bedside within minutes and actively diagnose whether further testing or treatment to prevent a cardiac arrest is warranted. Unlike a Code Blue team that "fights the fire" of a full-on heart attack, Roberto writes, a Rapid Response Team "detects the smoke" of a potential heart attack.

Traditional data warehousing and data analytics vendors often present their solutions as a way to make decisions ‘based on objective facts' rather than relying on ‘emotional gut feel.' The problem is, however, the known ‘objective facts' may not provide a complete — or even accurate — picture of what's really going on.

As Roberto's hospital case study illustrates, a gnawing sense that "Something is not right" should be interpreted as an alert that you probably do not have "all the facts," but rather just some facts. That is, you don't know what you don't know. On this key point, Roberto writes:

In highly analytical cultures, my research suggests that employees also may self-censor their concerns...In one case, a manager told me, "I was trained to rely on data [which] pointed in the opposite direction of my [correct] hunch that we had a problem. I relied on the data and ignored that nagging feeling in my gut."

So, listen to your gut, your intuition, as a signal that you need to dig deeper into the matter at hand. Actively seek out further information beyond the hard data available to you. Compare that information with your hard data and "connect the dots" for a far more complete picture, which may well yield surprising new insights.

What I find very exciting is that unified information access (UIA) is playing a vital role in empowering managers and leaders to connect those dots between data and other silos of information to realize those critical new insights.

UIA integrates, joins and presents all related information — structured data and unstructured content to complete the informational picture and significantly expand what organizations "know" to determine with confidence whether "Something is not right."

Analyzing just a single data type or source of information would not only fail to detect those new insights, but — even worse — such a limited, incomplete analysis may well point leaders in the wrong direction.

Stay tuned here on the Attivio blog for more insights on how UIA can help you get that complete informational picture, and, as Roberto writes, how to go beyond being a "problem solver" to becoming a "problem finder".

Apple almost bigger than all retailers combined

2012-03-19T17:18:21Z

As reported by CBS News MoneyWatch recently, Apple's market cap now hovers over $560 billion. The rest of the entire US retail industry is valued just a bit higher, as the chart shows.

(Credit: Bloomberg)

One thing the article makes quite clear is that Apple's platform really is that: products that are platforms, integrating numerous experiences into a unique experience — be that digital (iOS, iTunes) or physical (iPad, iPhone, Macs, Apple Stores). Look at the floor space of the Apple store. You know they all look the same — white and wood, glass stairs, colorfully shirted employees with iPod touches in hand... it is convergence: putting the entire thing together in one place, with consistency.

Convergence is the ultimate competitive advantage.

One way companies can pursue convergence is to start thinking of their information as a strategic asset. Converge it, and you will reap a multitude of benefits. More about convergence...

Convergence: A Winning Business Strategy, Now Ready for Enterprise Information

Convergence: A Winning Business Strategy, Now Ready for Enterprise Information - Part Two

Active Security in AIE

2012-02-13T19:37:57Z

Information retrieval systems have classically been interested in simply retrieving information based on a user's query. Typically these applications were going after data that was either public, such as job listings, or semi-private, such as research papers that were viewable by anyone with an account. Most enterprise data does not fit into either of these categories, though.

Content like last year's performance reviews, employee salary information or M&A research is usually only accessible by a select group(s) of people within an organization. Any system that provides access to this information needs to respect those same permission structures in order to be viable inside a company.

There were two common patterns that legacy search applications used in order to deal with these issues:

Late-bound security

In a late-bound security model, each search result is checked at run time to determine if the user is allowed to view the information. This has the nice effect that any changes are propagated immediately to the search experience, but it also has some significant drawbacks. First off, the back end has to be able to support a high volume of requests for ‘can user X see document Y?' If you have a highly secure system, a single page of 10 results might need to filter through hundreds or thousands of results in order to return the ten specific items the user is allowed to see.

Second, it's impossible to give users accurate summarization information such as facet counts or even total number of search results. Doing either of these would require scanning the full corpus, which might work for a few thousand documents, but would never scale to a few million documents.

At Attivio, we like to call this the easy way out. As long as you can write a for loop and build a web service, rest API or intelligent pidgeon to say yes/no to the "can user X see document Y," you have ‘security'. The code looks like this:

int resultsToShow = 10;
List<Result> resultsToDisplay = new ArrayList<Result>();
for (Result result : mySearchResults) {
  if (securityService.canRead(result, searchUser) {
    resultsToDisplay.add(result);
    if (resultsToDisplay.size() >= resultsToShow) {
      break;
    }
  } 
}
// print out the results

Early-bound security

In early-bound security, user, group and permission information is stored along with the documents in the index. Essentially, you index the list of users and groups (and parent groups) who are allowed to see each document in a field of that document. Then at query time, you append a user's id and their groups (and parent groups) to the query in an AND clause. This has the nice benefit of allowing for fast queries and exact counts for faceting and other aggregations. The downside is that updating security information can be a huge pain. If a user joins a group it's usually not a big deal, but if a group joins another group due to company reorg or a new parent group is created, you end up having to update large portions of the index.

Doing this scheme well is complicated, but it's been the prevailing pattern for implementing security for most search applications for the last 10 years or so.

Using this design a document might look like this:

field	value
url	http://www.acme.com/someSecretProject/verySecretStuff
text	blah blah blah
users	joe, mik
groups	headHonchos, superVillians

If a user named 'Tim' who is a member of the 'engineering' and 'qa' groups logged into the system and ran a query for 'secret' his query would be rewritten as follows:

AND(
  "secret",
  users:tim,
  OR(
    groups:engineering, 
    groups:qa
  )
)

Attivio's Answer — Active Security

The problem with both of the previous methods is that they miss the key design of the source systems, namely that the security information is stored separately from the actual content and is linked by some sort of pointer. In a database that relationship might be modeled as a foreign key from the content table to the ACL table. In a file system, it's a pointer from the file's content to a record in the NTFS/ext3/xyz file system's database for permissioning information.

The active security model maintains this information in a relational structure and then uses our patented JOIN operator to link the two records at query time. We are able to index users and groups as native objects in our index, and traverse up and down the hierarchy in order to determine the full set of permissions for a given document.

All that's required at search time is for the application to pass in a user's id. This scheme also allows us to perform rapid updates of the security information. If one group joins another group, all we need to do is update the one parent group's record.

In this world we have the following objects in our index:

A Document

field	value
id	doc1
url	http://www.acme.com/someSecretProject/verySecretStuff
text	blah blah blah

A User

field	value
userid	user1
username	jsmith
realm	acme

A Group

field	value
groupid	group1
groupname	engineering
members	qa, patty
realm	acme

An ACL

field	value
docid	doc1
users	user57, user88
groups	group1, group77

The user's query simply needs to have the principal set on it:

QueryRequest req = new QueryRequest("secret");
req.setPrincipal(new AttivioPrincipal("acme", "jsmith");

This scheme allows us to independently update each individual record. For example, when a group's members change, you update the group record. When permissions on a document change, you update the ACL. We're able to make these smaller updates much faster than a full document update in an early bound security model.

In addition, this method lets us set up simple connectors to fetch each of the record types from Active Directory, LDAP, or a custom database schema, without having to merge all of that information in with the documents, which also speeds up ingestion time.

Learn more about Active Security and AIE JOIN:

Introducing Active Security

Patent Issued to Attivio for Unique Method of Unifying Multiple Content and Data Sources

Thinking Like a Tester

2012-01-23T16:30:48Z

As a member of what was back then, just a three-person QA team, my heart sank when I read the title of one of our early blog posts stating that quality is job 1 to 3000+. My manager had recently transitioned into another group and his vacancy had yet to be filled. Our CTO, Sid Probstein, had recently closed Attivio's maiden blog post with a promise to expound on Attivio's deliberate approach to quality. So here it is, I thought — had I taken on an impossible task with unrealistic objectives?

Fortunately, though, thinking like a tester once again resuscitated me. The logic of abductive inference compelled me to continue reading Sid's post. While I had to hold my breath until the closing statements convinced me sufficiently that my initial interpretation had been misguided, I was relieved to discover that 3000+ was actually a reference to the number of unit tests covering 81.2% of AIE V1.2. I eyed the 81.2% assertion suspiciously...Ah, things were going to be ok (we are now on AIE V3, and actually have almost 19,000 automated unit tests covering close to 85% of the code base).

Sid's post highlighted the premium Attivio puts on product quality. With a tip of the hat to Lessons Learned in Software Testing (Cem Kaner, James Bach, Bret Pettichord), I'd like to introduce another dynamic and indexable facet of our approach to quality at Attivio: Exploratory Testing.

AIE, Attivio's unified information access platform is incredibly flexible, configurable, and extensible. You can feed multi-language documents into sophisticated workflows via custom clients, command line connectors and in-process configured connectors. You may want your timely insight that matters pushed to an active dashboard, prefer searching/exploring your unified information store via a custom GUI, or perhaps have occasion when only a sick multi-level SQL join will suffice. Of course, it goes without saying that all of this must be secure, stable, scalable, performant, highly available and fault-tolerant.

While we continue to review, expand, and augment the unit tests that are the foundation of our quality strategy, our now fully-staffed QA team has also formally embarked on an exploratory voyage. More than tourists, like C.T. Granville setting sail on a whale watch, Attivio QA is on a quest, a never-ending journey to boldly go where no test or tester has gone before. Everything is fair game: stories, requirements, design, usability and documentation. We delve into all of it, bringing our own experience, curiosity, and skepticism to bear. Most importantly, we take inspiration from our customers who are constantly finding new and interesting ways to deploy our platform, challenging us to look at things from an entirely new perspective.

Recently, a friend sent me a Slate article about two studies that appeared in Cognition, an international journal that publishes theoretical and experimental papers on the study of the mind, which focused on how children learn. The article's author, Alison Gopnik, a professor at UC-Berkeley, who conducted one of the studies, says the studies "provide scientific support for the intuitions many teachers have had all along: Direct instruction really can limit young children's learning. Teaching is a very effective way to get children to learn something specific...But it also makes children less likely to discover unexpected information and to draw unexpected conclusions."

I think this assertion has relevance to people of any age. In the province of software testing, it's reasonable to equate 'direct instruction' with test scripts, whether manual or automated. There is significant value in them (e.g., regression, smoke, and config testing). However, they have the potential to hamstring a tester's most valuable assets: creativity, curiosity, and judgment.

At Attivio, we strive to take full advantage of those assets. The effort produces benefits beyond simply identifying additional defects and regressions. It augments our culture of communication, collaboration, and continuous learning/discovery and enables fresh and deep insight. That this is commensurate with AIE's capabilities is only fitting.

Author Bio

John McEleney is a senior member of the Attivio QA team and has been with Attivio for over three years. Prior to Attivio, John worked on and managed teams at BEA and Plumtree.

What AIE and unified information access mean for developers

2012-01-12T18:55:33Z

There has been a lot of press recently on unified information access and how it enables business users and IT staff to reduce the time it takes to provide information to make better business decisions. The Active Intelligence Engine not only provides value to business users, there are also a number of advantages for developers that make life much easier:

Index now, think later

One of the greatest advantages of AIE is the ability to facet and join on fields without having to do a lot of preprocessing or, more importantly, design work. For example, you might not know ahead of time that your customer database records are linkable to customer comments on your website, but you can easily find out with a single query after both information sources are indexed. A number of POCs and development spikes we have conducted have followed the pattern of indexing everything possible and then trying to infer relationships using queries. Unlike a database where primary and foreign keys must be setup ahead of time, the index does not require this sort of predefined and rigid schema definition. Also, many of these features are so cheap to leave on, that tuning isn't always necessary.

That's not to say that there aren't advantages to doing some tuning of the schema or the ingestion workflows, but it isn't required in order to start seeing the power and ROI of using our system.

Develop locally, deploy globally

Developing for a single node, single JVM system is a straightforward process for most any platform. Some platforms also make it fairly easy to write business logic for a large distributed system. The key advantage we've found is being able to develop and test in a small localized environment, but then deploy to a large distributed system and not be surprised by system behaviors. In addition, it's important to be able to use a standard debugger when developing locally in a fully functional system, but then also to be able to use the same debugger in a distributed environment. AIE topology files provide an abstraction that separates the system functionality from the system deployment. This allows operations teams the ability to scale the system for QA, staging and production environments without having to worry about functional issues with the configuration.

Some other systems like Hadoop force users into somewhat complex development models. AIE's development models strive to support the "I want to do X to Y" in the simplest possible manner. We ship a sample transformer that implements some simple business logic and more importantly, we ship a unit test for the transformer.

Learn one API, let us handle the details

One of the hardest parts of building enterprise wide applications is the need to work with multiple different APIs. In addition, each system has its own idea of what a user is, what it means to have permissions to read a document and more importantly, what a document is to begin with. If you can't define and normalize all of these concepts it's impossible to join, group, categorize and make decisions based on the data. AIE not only provides connectors to these back-end systems, we also handle normalizing each system's data to a standard format that is accessible via our API. A user in Active Directory can have permissions to a document in Documentum and a document in SharePoint. More importantly, the permissions are applied transparently at search time so that developers don't have to worry about doing any sort of post filtering of results.

Attivio's development environment is meant to hide all of the enterprise ugliness from developers and present a single user, group, document, acl, and query concept. If userX can see records in 10 repositories, we handle the details. If you want to join data from your internal SharePoint server to your CRM system based on a support person's contact information, we can do that for you as well.

Author Bio

Since graduating from MIT with a degree in Computer Science, Will Johnson has worked for Altavista and FAST for over 7 years. At Altavista Will developed AV's real time indexing solution used by news aggregators who demanded instantaneous access to news as it arrived. In addition he was one of two engineers responsible for developing the Altavista QIndexer product that was used by the large majority of AV's customers. At FAST, Will developed high speed database connectors as well as developing search UI's and tool sets used across the organization. In addition Will also worked on many of the largest and most complex sales engagements and deployments for customers around the world, specializing in distributed systems for many of the largest internet publishers, directories as well as internal knowledge management systems. Will is a founder, one of the Chief Architects at Attivio and a really nice guy.

The (Real) Semantic Web Requires Machine Learning

2012-01-05T17:57:21Z

We think about the semantic web in two complementary (and equivalent) ways. It can be viewed as:

• A large set of subject-verb-object triples, where the verb is a relation and the subject and object are entities

• As a large graph or network, where the nodes of the graph are entities and the graph's directed edges or arrows are the relations between nodes.

As a reminder, entities are proper names, like people, places, companies, and so on. Relations are meaningful events, outcomes or states, like BORN-IN, WORKS-FOR, MARRIED-TO, and so on. Each entity (like "John O'Neil", "Attivio" or "Newton, MA") has a type (like "PERSON", "COMPANY" or "LOCATION") and each relation is constrained to only accept certain types of entities. For example, WORKS-FOR may require a PERSON as the subject and a COMPANY as the object.

How semantic web information is organized and transmitted is described by a blizzard of technical standards and XML namespaces. Once you escape from that, the basic goals of the semantic web are (1) to allow a lot of useful information about the world to be simply expressed, in a way that (2) allows computers to do useful things with it.

Almost immediately, some problems crop up. As generations of artificial intelligence researchers have learned, it can be really difficult to encode real-world knowledge into predicate logic, which is more-or-less what the semantic web is. The same AI researchers also learned that different people will almost inevitably create knowledge encodings that can't easily be compared, because they use different — sometimes subtly, maddeningly different — basic definitions and concepts. Another difficult problem is to decide when entity names refer to the "same" real-world thing. Even worse, if the entity names are defined in two separate places, when and how should they be merged? For example, do an Internet search for "John O'Neil", and try to decide which of the results refer to how many different people. Believe me, all the results are not for the same person.

As for relations, it's difficult to tell when they really mean the same thing across different knowledge encodings. No matter how careful you are, if you want to use relations to infer new facts, you have few resources to check to see if the combined information is valid.

So, when each web site can define its own entities and relations, independently of any other web site, how do you reconcile entities and relations defined by different people?

One technique is to require (or STRONGLY SUGGEST) the use of a shared ontology. (For our purposes, an ontology is one person's — or one company's — semantic web).

Perhaps, if it were carefully designed, it would be possible to allow anyone to add to it without making it unusable. Wikipedia might serve as an inspiration here. However, this is generally impractical, for a number of reasons:

A lot of smart people have tried to do this in the past, and they've obviously failed.
Wikipedia has grown a community that is good — perhaps too good — at discussing how articles should be written. However, it's not clear that any community could become competent to discuss semantic web issues in detail - and to come into agreement about them.

The major problem is the "open-world" requirement implicit in the semantic web. In a closed world or a limited domain - even if the limited domain isn't small — it's possible to agree on the ontological issues and get to work. Many companies have put a lot of effort into creating their domain ontologies, and some have even found a day-to-day use for them. However, it takes a lot of work, and continuously ongoing work, to maintain a good domain ontology.

Even if companies were willing to open-source their ontologies, their domain is closed — and once you start trying to knit different domain ontologies together, you quickly start seeing the problems discussed above.

By the way, the fact that the semantic web has failed to be widely adopted has, I think, a simple explanation: it's really difficult, much more so than learning HTML, and the practical payoff is not obvious, to put it mildly.

As an aside, Attivio's unified information access architecture allows corporate ontologies to be directly imported, so a user can search through them, or perform SQL queries on them, including joins. Joins, in particular, are a powerful tool for understanding semantic web ontologies, and for using them to improve search and other kinds of business intelligence work. (You can read about our newly awarded join patent here.)

Is there a solution? Can the creation of domain ontologies be automated — or at least made easier? Will something make it possible to combine different domain (and different site) semantic webs — at least with some minimum guarantees about reliability? I think so, and here's why.

At Attivio, we've been working on using statistical machine learning to learn how to extract relations from plain text. We're still working on it — it's a difficult problem — but we're making real progress and I'm pretty sure that we'll discuss the details of our work in future blog posts. For now, though, it's clear to us that there's a real advantage in being able to associate probabilities with the entities and relations that we find in a document, especially when we can accumulate information from millions of documents (or more). If we build a knowledge graph with weights on the entity nodes and relational edges, we start having a way to measure the reliability of different parts of a semantic web. We can also determine, for two separate semantic webs, what entities and relations we know are the same or different, and where we're unsure.

Human ontology builders can't create probabilities like that, since humans are even worse at statistics than they are at semantics. (No blame here — both are really confusing to think about!) However, there's been a lot of research into relation and event extraction, as well as in machine learning using big data (or extreme information, if you prefer). So it's now possible to create tools that substantially help the process of building ontologies.

And, making no promises we'll regret, we hope that we'll be able to talk more about it soon.

Author Bio

John O'Neil has written and designed software for search, natural language processing and machine learning for 10 years. After receiving a Ph.D. in computational linguistics from Harvard University, he has worked for Lingo Motors, where he designed their main commercial product and ended up with his name on a number of their patents, as well as other search engine companies where he worked to increase search relevancy and accuracy. He also worked for over five years at Basis Technology, Inc., where he was the designer and lead developer for the Rosette Linguistics Platform, their language processing and entity extraction suite of products.