For more information, please visit www.imintel.org.

About the Institute for Modern Intelligence (IMI)

The IMI, formed in early 2009,  is a non-profit research institute committed to developing the science, practice, and governance of Modern Intelligence.  IMI is embracing the evolution of the Intelligence Community from its emerging theoretical foundations to the panoply of innovative techniques and the implications for organization and management.  Today, IMI primarily serves the U.S. Army and Intelligence Community.

Contacts
info@imintel.org

Company: Institute for Modern Intelligence (IMI)

Industry: Research and Development

Location: Alexandria, VA, United States of America

Alexandria, VA—The Institute for Modern Intelligence (IMI) announced today that Dr. Suzanne Yoakum-Stover, IMI Executive Director, was named as one of the top 5 information managers of the year by Information Management Magazine http://goo.gl/vRjn.

Dr. Yoakum-Stover was recognized for her work creating a practical, Ultra-Large Scale (ULS) systems solution – a unified dataspace – for data storage, exploration, enrichment, and exploitation that accommodates the diversity of data, semantics, and perspectives without information loss or distortion. Through an innovation that provides a generic interface to all data, she aims to enable all manner of  processing to be put into production coherently within an internet-scale dataspace.  Building upon this foundation her mission is to execute a broad ULS systems research agenda with specific application to intelligence and ultimately inaugurate what she terms “the new discipline of Modern Intelligence” – the science and practice of intelligence at Ultra-Large Scale.

“I was absolutely delighted to be identified  as one of the top information managers of the year.  It is an honor that reflects the community’s recognition and growing acceptance of the IMI’s breakthrough approach for addressing the scale and diversity of data storage, processing, visualization, and management all within a unified, yet completely open dataspace.” – Dr. Yoakum-Stover.

For more information, please visit www.imintel.org.

About the Institute for Modern Intelligence (IMI)

The IMI, formed in early 2009,  is a non-profit research institute committed to developing the science, practice, and governance of Modern Intelligence.  IMI is embracing the evolution of the Intelligence Community from its emerging theoretical foundations to the panoply of innovative techniques and the implications for organization and management.  Today, IMI primarily serves the U.S. Army and Intelligence Community.

Contacts
info@imintel.org

A Data Integration Framework with Full Spectrum Fusion Capabilities

August 2009

ABSTRACT

One of the most challenging problems in intelligence gathering and processing is resolving the issues of syntactic and semantic interoperability of numerous intelligence sources and fusing the intelligence data, information, and knowledge to provide for efficient, accurate and comprehensive analysis. The key features of a successful fusion solution are: 1) The ability to rapidly and seamlessly integrate any source while preserving its original data and semantics; 2) Support for powerful data processing capabilities that can utilize the data and semantics of the integrated sources without limitations. Existing data integration approaches require heavy pre-integration processing (schema harmonization and data normalization) and usually entail loss/distortion of original data and semantics. Processing of the integrated data is defined by, and therefore limited by, the integrating schema – discovering data relationships and fusing data beyond the integrating schema is impossible.

We present a unified data and information integration framework that presents absolutely minimal barriers to incorporating new data and semantics into the integrated system (e.g. no heavy pre-processing or data / data-model conditioning), and embraces the full spectrum of data sources, types, models, and modalities (e.g. text, images, audio, signals).

The approach enables rapid integration of ad-hoc data and data-semantics and results in a multi-layered data store that we call a Unified Data Space (UDS). The UDS supports data fusion to yield information and knowledge while imposing no restrictions on what data must be or how it is to be used, and the diversity of processing by which structural and semantic barriers are overcome.

In this paper we concentrate on the benefits of the approach for data fusion.

Key words: Data Description Framework (DDF), data fusion, data integration, semantic enrichment, structured data.

Introduction

One of the most challenging problems in intelligence gathering and processing is resolving the issues of syntactic and semantic interoperability of numerous intelligence sources and fusing the intelligence data, information, and knowledge to provide for efficient, accurate and comprehensive analysis. The key features of a successful fusion solution are: 1) The ability to rapidly and seamlessly integrate any source while preserving its original data and semantics; 2) Support for powerful data processing capabilities that can utilize the data and semantics of the integrated sources without limitations. Existing data integration approaches require heavy pre-integration processing (schema harmonization and data normalization) and usually entail loss/distortion of original data and semantics. Processing of the integrated data is defined by, and therefore limited by, the integrating schema – discovering data relationships and fusing data beyond the integrating schema is impossible [1 – 3].

We present a unified data and information integration framework that:

• Presents absolutely minimal barriers to incorporating new data and semantics into the integrated system (e.g. no heavy pre-processing or data / data-model conditioning)
  
• Embraces the full spectrum of data sources, types, models, and modalities (e.g. text, images, audio, signals)
  

A detailed description of the architecture and the philosophy of the approach can be found in [4 – 7]. In this paper we concentrate on the benefits of the approach for data fusion:

• The UDS permits “horizontal” data fusion spanning across data from all sources. It can accommodate any number of integration models, without imposing physical or semantic barriers, while facilitating navigation, search, and exploration of the integrated data store. As a result, relationships between data from multiple disparate sources that are difficult or impossible to foresee (and therefore impossible to look for) are revealed.
  
• The multi-layered UDS permits the “vertical” fusion of data and data models, allowing one to backtrack to sources on one hand and connect to knowledge models on the other. It also supports the processing by which data is cultivated to produce information, knowledge, and ultimately understanding. Thus, the UDS represents a viable integrated solution that “matures” with time.
  

  Challenge of Data Integration
  

To be viable within an Ultra-Large Scale (ULS) systems [8] environment consisting of a freely evolving, interdependent collective of human and computational systems, very little of which will ever be under our control, any approach to data and semantic fusion must:

• Present absolutely minimal barriers to incorporating new data and semantics into the integrated system (e.g. no heavy pre-processing or data / data-model conditioning)
  
• Embrace the full spectrum of data sources, types, models, and modalities (e.g. text, images, audio, signals)
  
• Impose no restrictions on what data must be or how it is to be used
  
• Support the diversity of processing by which structural and semantic barriers are overcome to yield information and knowledge
  
• Allow data, information, and knowledge to be re-used according to diverse perspectives
  

To our knowledge, no traditional approach to data integration, physical or virtual, has all of these characteristics [1, 2, 9, 10, 12]. Consequently, traditional approaches fail to provide viable solutions in the “wild”, i.e. for ULS environments that are characterized by decentralization; inherently conflicting, diverse, and unknowable requirements; heterogeneous, changing, and inconsistent elements; normal failures; continuous operation, evolution, and deployment; and immense scale along many dimensions.

To productively address the challenges of ULS systems environments, we developed the Data Integration and Semantic Enrichment Platform (DISEP) and the Data Description Framework (DDF) based on the following principles:

To enable true data integration and unencumbered semantic enrichment,

• Data must be perceived objectively, independent of intended use
  
• Domain data-models must be considered from a higher level of abstraction
  

In our approach, these principles are reflected in two essential decouplings: The decoupling of data from domain data-models, and the decoupling of domain data-models from the model of the integrated data store. As a result, the DISEP is able to serve as a universal platform for data integration and semantic enrichment, and within it, the DDF serves as a universal store for structured data.

Data Description Framework

The DDF supports semantic data integration by establishing a domain-neutral unified store for structured data. To achieve this, we consider structure, vocabulary, semantics, and constraints from a higher level of abstraction from which we then distill a minimal set of elements sufficient to capture any data-model. These are illustrated in Figure 1 and defined as follows:

Sign: A sign
g_i represents a chunk of data, either physically located within a tangible artifact, or contained within an analyst’s mind. Examples of the former include a string of text in a document; an area of pixels within an image; a segment of an audio stream or other signal. As illustrated in Fig. 1, for tangible artifacts, regardless of the type of medium, signs are always associated with a physical extent or quantifiable span within the artifact, which we call a mention. The set of all signs, G = {g_i}, spans across all data sources. In the set, each element is unique: ∀i,j (i ≠ j) g_i ≠ g_j. G is the construct by which the DDF represents data. From the text data shown in Fig. 1, signs G‘ = {‘Suzi’, ‘Tanya’, ‘July 4, 2007′, ‘Bring lunch’, ‘Message1′} contribute to G (i.e. G’
Í G), though many more signs may be identified even from this simple example.

Concept:
A concept
c_i
is an abstract idea, defined either explicitly or implicitly by a source data-model.  For example, the nodes of an ontology, the tag set in an XML Schema Document (XSD), and the attribute / table names in a relational database all represent concepts. In the set of all concepts C = {c_i}, each element is unique: ∀i,j (i ≠ j) c_i ≠ c_j. From the text data shown in Fig. 1, concepts C’ = {‘Message’, ‘Person’, ‘Body_text’} contribute to the full set of concepts C (i.e. C’ Í C).

Predicate: A predicate p_i is an abstract idea used to express a relationship between “things.” Predicates are used in the formation of statements (described below) and may be defined either explicitly or implicitly by a source data-model. For example, the arcs of an ontology, and the attributes of an XML or database schema represent predicates. In the set of all predicates P = {p_i}, each element is unique: ∀i,j (i ≠ j) p_i ≠ p_j. The text example of Fig. 1 contributes predicates P’ = {‘To’, ‘From’, ‘Body’}
to the set of all predicates
P
(i.e. P’ Í P). The only predicate that is “built into” (i.e. defined by) our storage model is the ‘IsInstanceOf‘ predicate, which is used to disambiguate signs to form terms as described below. Concepts and predicates are the constructs by which we link to data-models and, thereby, explicitly expose data-semantics.

Term: A term
t_i is an ordered pair <g_i,c_j> where g_i
∈ G
and c_j
∈ C. Each term represents a disambiguated sign. The process of disambiguation associates a sign with a concept using the ‘IsInstanceOf’
predicate (though not every sign from G is necessarily disambiguated, and not every concept from C is necessarily used for disambiguation). In the set of all terms T = {t_ij}, each element is unique: ∀
i,j,k,l (i ≠ k or j ≠ l) t_ij ≠ t_kl. The text example of Fig. 1 contributes terms T’ = {t₁, t₂, t₃, t₄}
where t₁ = <’Suzi’, person>, t₂ = <’Tanya’, person>, t₃ = <’Bring lunch’, Body_text>, t₄ = <Message1, message>
to the complete set of terms
T (i.e. T’ Í T).

Statement: A statement, s_i is an encodes a binary relationship between a subject and an object mediated by a predicate. A statement is represented by an ordered triple s_ijh = <subject_i, predicate_j, object_h>. Among the set of all statements, each element is unique: ∀
i,j,h,l,m,n (i ≠ l or j ≠ m or h ≠ n) s_ijh ≠ s_lmn. In our model, subject and object may be either a term or statement. The simplest kind of statement is one in which subject and object are terms s0_ijh = <t_i, p_j, t_h>. Statements in which the object is itself another statement represent reifications: s1_klm = <t_k, p_l, s_m>. Finally, a statement in which both subject and object are other statements represents a relationship between statements: s2_xyz = <s_x, p_y, s_z>. The set of all statements S = {s0_ijh} U {s1_klm} U {s2_xyz}. The text example of Fig. 1 shows three statements: S’ = {<t₄, to, t₁>, <t₄, from, t₂>, <t₄, body, t₃>} all with the same subject, which is the term corresponding to the message itself. These statements contribute to the set of all statements, i.e. S’ Í S.

These elementary constructs (sign, concept, predicate, term, and statement) define a data reference model, which we call the Data Description Framework (DDF) [13]. Because it effectively decouples data from data-models, it can encapsulate any sort of data-model. Because it binds knowledge to data, it enables deep data integration and semantic enrichment. By using the DDF as the basis for implementing a stable storage-model, we are able to build a practical data integration platform on commodity database infrastructure.

The reader familiar with the Resource Description Framework (RDF/RDFS) may wonder what is different here. Indeed, RDF and DDF share DNA, so to speak, since both employ a similar level of abstraction and expose semantics. Unlike RDF however, DDF also prescribes the exposure of data as signs which can freely participate in the disambiguations and associations necessary for data integration. In contrast, a datum represented as an RDF literal cannot be explicitly disambiguated or associated. Also, in contrast to DDF signs, which provide a primal level of data integration (to be described below), there is no mechanism in RDF to prevent a single datum from being represented by multiple URIs. This is not a criticism of RDF as these differences reflect the fact that RDF is a meta-model not specifically aimed at data integration. Thus, employing RDF for data integration necessitates building a particular metamodel instance (i.e. a model) in RDF along with rules prescribing the manner of data exposure [3]. In contrast, DDF is a model that makes explicit commitments to support data integration. Because this model represents an abstraction over domain data-models, the DDF can represent data structured by any data-model, and be represented in any metamodel (including RDF).

As illustrated in Figure 2, the DDF forms a layer of data and semantics (Layer 2) in the DISEP that lies between the indigenous source systems (Layer 1) and their data/knowledge models (Layer 3). A more detailed description of the layered architecture is presented in [13]. Layer 1 feeds the layers above, and Layers 2 and 3 interact: Layer 3 provides the semantic context for Layer 2, and Layer 2 participates in the formation of an overarching knowledge model in Layer 3. Together Layers 2 and 3 form what we call the Unified Data Space (UDS).

Demonstration of Fusing Potential

The DDF fusing potential is demonstrated in Fig. 3:

• The UDS integrated (Fig. 4 and Fig. 5) selected data from Freebase [16] (structured data), GovTrack [17] (semi-structured data), and the images and articles from Wikipedia [18, 19] (unstructured data).
• The UDS absorbed the diversity of the data types and modalities: audio, images, unstructured text, and data from the relational database.
• Accommodating data from the structured and semi-structured stores was performed in an automated fashion with little to no up-front preprocessing that is usually needed in traditional integration approaches.
• The unstructured text data and images were represented with the help of DDF constructs via manual or automated entity extraction and face-recognition processes.

Figures 4 and 5 illustrate the process of “DDFying” the data sources from Fig. 3: extracting and disambiguating data, and representing disambiguated data as terms (Fig. 4); associating disambiguated data as statements (Fig. 5).

Horizontal Fusion Across Sources

Utilizing data from multiple sources is complicated by their original disconnect – each data store is built on some data scope and on a specific data-model, and over time, neither the scope of data nor the data-model change much. A traditional integration approach, even a successful one, results in yet another store with its own data and data-models, with the limited ability to evolve – often another candidate for future integration efforts. Significant investment into the integration project may result in the “rise” of a new data store but cannot prevent its “fall.” Fig. 6a) demonstrates a progression of integrated stores – isolated silos of data disconnected from each other and from the data sources.

The Unified Data Space that is built and cultivated on the DDF model allows to incorporate any data source without sacrificing the source’s original data and data-model – DDF unlocks the data source to virtually endless enrichment by different cross-source disambiguations and associations. The DDF UDS is permanently and invariably open to new data sources, and is constantly evolving with its data and data-model expanding. Each participating data store contributes to the expressiveness of the whole Data Space, and in turn benefits from being fused with other data stores by gaining richer semantic and data contents (Fig. 6b).

Using the “fusion” metaphor – a traditional integration approach produces a finished product – an alloy, while the DDF approach produces a melting pot.

The evolving DDF Data Space is representative of the sources and the results of cross-source enrichment. It also takes data utilization and exploitation to a new level and supports navigation, exploration, and querying, without limits.

In the subsequent text, we represent signs, concepts, and predicates using Arial font. Terms are denoted as
[sign, concept] (e.g.
[Adam, Chemist]) and statements are denoted using an intuitive triple representation, e.g. [Adam, Chemist] hasInventoryID [1001,InventoryID].

Fig. 7 demonstrates these capabilities for the following scenario:

• A user (or a process) performed search for the text (i.e. sign) ‘Bush’ and retrieved a number of terms ([George Walker Bush, Name] from Wikipedia, [George W. Bush, President] from Freebase, …) visualized as the nodes.
  

• The user asserts that ([George Walker Bush, Name] from Wikipedia is same as [George W. Bush, President] from Freebase – a new statement [George Walker Bush, Name] sameAs [George W. Bush, President] is created in the UDS and is shown as an arc connecting the two nodes. The figure shows the results of. The dashed ellipses show how the multiple cross-source assertions fuse data from different sources.
• Furthermore, the fused data allows us to break the sources barriers and see the cross-source relationships between data as if they came from the same source. For example, [George Walker Bush, Name] from Wikipedia (ellipse 2) will be associated with [Dick Cheney, Vice_President] from Freebase (ellipse 1).
• The user can semantically enrich the UDS by introducing new DDF constructs, such as signs, terms, and statements. For example, the user asserts that [BUSH George Walker, Face] from one Wikipedia article is associated with [Bush George Herbert Walker, Face] from another Wikipedia article via predicate hasFather (green arc).
• Following the original and asserted relationships between data the user can navigate across the sources, e.g. from ellipse 1 to ellipse 4 in the figure.

Vertical Fusion Across Data and Knowledge

The DDF defines the organization of the Layer 2 of DISEP (Fig. 2) and establishes connections with its other layers – the DISEP provides for a contiguous multi-layer store: from sources in the wild, to the structured data store, and to the knowledge representation.

Data is taken from Layer 1, tied with the semantic elements from Layer 3 to get the structured data store, on which we can perform efficient data processing and enrichment. However, in DDF we may lose the rich data context of the data element from Layer 1, so when we need to perform data analysis in the original data context, we can go back to Layer 1. We also may lose the rich semantic context for the semantic element, so when we need to perform model analysis, we can go to Layer 3. Fig. 8 demonstrates contiguousness of the UDS and vertical enrichments processes.

Fig. 9 illustrates these enrichment processes on our example. After [George Walker Bush, Name] from Wikipedia is fused with [George W. Bush, President] from Freebase, a user (a process) can see in Layer 3 that the concept President is associated with the concept Party and he returns to the Wikipedia source to find out that George Walker Bush is a Republican. A new term [Republican, Party] and a new statement [George Walker Bush, Name] associatedWith [Republican, Party] are created in the Layer 2, enriching the integrated data store that supports exploration, navigation and querying.

The enrichment operations preserve the contiguousness of the UDS: from Layer 1 we can “see” how a particular data element is associated with any abstraction of Layer 3, and from Layer 3 we can “see” how any abstract idea is associated with data. We truly break the barriers not only between the sources, but also between the data and knowledge worlds.

Summary

It is acknowledged that there is no silver bullet for solving the problem of data integration [7], and that all integration approaches face deep challenges associated with scale, performance, query processing, data conditioning / pre-processing, semantic enrichment, viability, and sustainability. The DISEP and DDF serve to address these challenges as follows:

Scalability. The challenge of scale is common to most integrated stores. The “lossless” data representation of the DDF slightly exacerbates this problem because it generally requires several times more storage space than in the original source. Fortunately, distributed database technologies, and cloud computing infrastructure in particular, provide viable means to manage this challenge.

Query Processing and Semantic Exploration. The DDF enables semantic navigation over the DDF Data Space by the action of a series of questions that “surf” across the entire DDF Data Space unimpeded by barriers between source systems. We distinguish two types of navigation and data retrieval on the DDF Data Space:

Exploration. A user navigates the Data Space having limited or no knowledge of the sources and their models. Navigation is data-driven; the result of the initial user’s query is used to generate the next .

Querying is similar to querying of traditional data sources when a user formulates a request assuming knowledge of a data-model. Querying is model-driven; each query is independent from others.

A very important consequence of the unified representation of data and data-semantics in DDF is the unified structure of the queries. This allows querying patterns to be defined and processing optimized. As a result, the ad-hoc querying and exploration of the data store can be performed naturally [15]. Ad-hoc querying of traditional integrated stores has serious performance issues and is constrained by the integration model. Moreover, some queries, such as search over data values, which are natural in the DDF, cannot be implemented in a practical way using other integration solutions.

Semantic enrichment. The scope of semantic enrichment achievable by the DDF is defined by the integration actions that it supports: incorporating new data and data-models, disambiguating data, and building associations between disambiguated data. Because traditional data integration approaches support only some aspects of data and data-model enhancement, their semantic enrichment power is severely limited. In addition, the DDF implementation supports extensive metadata [14] (not discussed in this paper), which may provide additional semantic enrichment by, for example, capturing information quality. Metadata also enable more sophisticated information retrieval processes.

Viability. One of the most important advantages of the DDF over other integration solutions is that it meets the challenge of the viability of an integration solution. The DDF accommodates new data and semantics and allows for virtually endless semantic enhancement through new data disambiguations (i.e. term formation) and new semantics associations (i.e. statements formation).

Sustainability. An integration solution must “…offer some services immediately without any setup time, and improve the services as more investment is made into creating semantic relationships.” [7]. The DDF actually achieves this: Data sources can be integrated in the DDF without heavy preprocessing or data-model harmonization. The Data Space can be explored, and semantic relationships discovered, without a-priori understanding of source data-models. Additional refinement and enrichment of the Data Space serves to increase the effectiveness of Data Space services.

Implementation: General Overview

A DDF data store can be implemented in a variety of ways (e.g. objects, relations, triples). We have a prototype in Oracle 10g and there is an ongoing effort of implementing DDF on the Cloud. The following table gives an overview of the DDF architecture on the Cloud.

Table 1. Architecture of the DDF Integration Solution

Conclusion

The DISEP and DDF support data fusion and unencumbered semantic enrichment by implementing the following key principles:

• Data must be perceived objectively, independent of intended use
• Domain data-models must be considered from a higher level of abstraction

Data fusion approaches, physical and virtual, generally manage to preserve only a portion of the original data and semantics, and present these with yet another single, restrictive data-model. In contrast, the DDF persists and presents the entirety of the source data and semantics by using a higher level abstraction that imposes no particular data-model, yet supports any. The DDF populated with data produces a Unified Data Space that represents the primal integrated data layer of the DISEP. Within this Data Space, the original data and data-models co-exist and may be enriched either through the ingestion and integration of additional data or semantic enhancement.

Over decades of data processing, we have been formalizing our perception of data and then transforming (or even creating) and storing data according to this perception. Unfortunately, there has been very little effort to ensure correctness/durability/objectivity of those perceptions. As a result, we work with numerous models and formats of data, and numerous versions of data buried beneath. The evolution of our perception and understanding of data cannot be reflected in these data stores. New data, which does not conform to the store’s model, also cannot be accommodated. Thus we are trapped in an endless loop of creating and integrating new data stores, each of which deals with only a fraction of the data surrounding us. None of these can be expanded to represent other data, and all are valid for a relatively short time. In DDF, on the other hand, data “lives” alongside data-models, not inside them. This enables loose coupling of data and perceptions (i.e. data-models), and allows multiple perceptions to co-exist in the Data Space.

Without imposing modifications on existing data stores, the DDF can expose their data and semantics for use and re-use, without further increasing data entropy. The DDF Data Space is a live integrated store that evolves with our intentions (i.e. applications) and perceptions (i.e. data-models).

Acknowledgements

The authors thank the following US Army CERDEC I2WD personnel for their continued support: Mr. Anthony Lisuzzo, Director, Mr. Kesny Parent, DCGS-A Branch Chief, and Ms. Virginia Goon IXFL Manager. The authors also thank Mr. Oscar Wood and Mr. David Salmen of Data Tactics Corporation, as well as Mr. Andrew Eick of MissionFoc.us for many productive and stimulating discussions. This work was funded by US Army CERDEC I2WD under contract number W15P7T-06-D-A401/009.

References

[1] Batini, C. et al. A comparative analysis of methodologies for database schema integration. ACM Computing Surveys, (18) 4, 1986.

[2] Bernstein P., and Ho, H. Model Management and Schema Mappings: Theory and Practice. Proceedings of VLDB Conference, 2007.

[3] Booth, D. Why URI Declarations? A comparison of architectural approaches. HP Software, 2008. http://sunsite.informatik.rwth-aachen.de/Publications/CEUR-WS/Vol-422/irsw2008-submission-9.pdf

[4] Franklin, M., Halevy, A., and Maier, D. From Databases to Dataspaces: A New Abstraction for Information Management. ACM SIGMOD Record, 2005.

[5] Halevy, A. et al. Enterprise information integration: successes, challenges and controversies. Proceedings of 24th International Conference on Management of Data, Baltimore, 2005.

[6] Halevy, A. Franklin, M., and Maier, D. Principles of Dataspace Systems. Proceedings of the twenty-fifth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, 2006.

[7] Halevy, A., Rajaraman, A., and Ordille, J.
Data Integration: The Teenage Years. Proceedings of VLDB Conference, 2006.

[8] Northrop, L., et al., Ultra-Large-Scale Systems The Software Challenge of the Future. Pittsburgh: Carnegie Mellon University, 2007.

[9] Omelayenko, B. and Fensel, D. An Analysis of B2B Catalogue Integration Problems. Proceedings of the International Conference on Enterprise Information Systems (ICEIS-2001),
2001.

[10] Parent, C. and Spaccapietra, S. Issues and approaches of database integration. Communications of the ACM, 41(5), 1998.

[11] Sowa, J. Knowledge Representation. Logical, Philosophical, and Computational Foundations. Brooks/Cole, 2000.

[12] Yero, J. Logical vs. Physical Data Integration: A Practical Decision Guide. The DAMA International Symposium & Wilshire Meta-Data Conference. San-Diego, CA, 2008.

[13] Yoakum-Stover, S. and Malyuta, T. Unified Architecture for Integrating Intelligence Data, Proceedings of MIT Information Quality Industry Symposium, MIT, Cambridge, MA, 2008.

[14] Yoakum-Stover, S. and Malyuta, T. Unified Integration Architecture for Intelligence Data, Proceedings of DAMA International Europe Conference, London, UK, 2008.

[15] Yoakum-Stover, S. and Malyuta, T. Unified Data Integration for Situation Management, Proceedings of the 4th IEEE Workshop on Situation Management (SIMA2008) at MILCOM 2008, San Diego CA, 2008.

Data Sources used in demonstration:

[16] Freebase. http://www.freebase.com/

[17] GovTrack.us: Tracking the U.S. Congress. http://www.govtrack.us/

[18] George W. Bush. Wikipedia, The Free Encyclopedia. http://en.wikipedia.org/w/index.php?title = George_W._Bush&oldid=300819162

[19] Bill Clinton. Wikipedia, The Free Encyclopedia. http://en.wikipedia.org/w/index.php?title = Bill_Clinton&oldid=301113520

ACM Journal of Data and Information Quality. Pending decision

Authors

Suzanne Yoakum-Stover, Ph.D.
Institute for Modern Intelligence, Executive Director
Alexandria, VA

Tatiana Malyuta, Ph.D.
Data Tactics Corp., Principal Database Architect

Alexandria, VA
New York City College of Technology, Associate Professor

Abstract

The principal problem spanning the Intelligence Community today is how to integrate the great variety of disparate data into one single coherent repository of knowledge. Current practice whereby all data-models would be merged into a single “Uber-model” simply does not work. We require a solution that remains viable in a freely evolving, interdependent collective of human and computational systems, very little of which will ever be under our control. Our approach is database-centric and proceeds in stages. The first addresses the unified representation of the broad spectrum of artifacts existing within the Intelligence Enterprise today regardless of modality or structure. The second builds upon the foundation provided by the first to address the unified storage of structured data and semantic data integration. In both we embrace the diversity of data-models employed throughout the Intelligence Community. The result is a layered data architecture that can accommodate any kind of data without placing restrictions on vocabulary, structure, semantics, or constraints in a way that addresses today’s intelligence needs while providing a seamless transition path toward a future of Ultra-Large Scale (ULS) systems imbued with semantic technologies.

Introduction

The principal problem spanning the Intelligence Community today is how to integrate the great variety of disparate data stores and streams, both legacy and bleeding-edge, into one coherent repository of knowledge. Pieces of the intelligence puzzle lay scattered in data silos sequestered by the very systems that served to create them. Each of these systems, to include most of today’s Army Programs of Record, was built as an end-to-end solution with its own sensors, processors, and data stores, implemented and operated to achieve a specific intelligence objective. They were never meant to interoperate, share data, or even expose data beyond a narrow mission-focused enclave. The advent of network technologies and protocols, which have effectively eliminated the physical barriers between systems, has done little to bridge the chasm between these data silos. Although we can now transfer data over the wire, disparate and utterly incompatible data-models characterized by straightforward and subtle differences in vocabulary, structure, semantics, and constraints continue to stymie data search, exploration, enrichment, and exploitation efforts. The fundamental problems of data integration remain to be solved.

Current practice in data integration, whereby all data-models would be merged or harmonized, either physically or virtually [Batini 1986, Parent 1998, Halevy 2005, Bernstein 2007] fails to accommodate the demands of our fluid and rapidly growing Intelligence Enterprise. The physical mapping of disparate models into a single canonical data-model [Omelayenko 2001] is simply untenable as the scale and complexity of their subjects quickly overwhelms our tools and methods. Federation approaches share this defect and introduce new ones [Izydor 2007, Yero 2008]. In practice, these approaches provide only the illusion of data integration as they mainly integrate data-models, not the data itself, and in so doing confine all data to a model that is incapable of adapting itself or its contents as our knowledge about the domain evolves.

In all but the most constrained situations, what begins as a perfectly neat solution for a handful of systems quickly becomes intractable with scale, exposing not only the limitations of traditional approaches, but also of our grasp at the foundations of knowledge representation itself. This phenomenon is but one early symptom of our evolution toward Ultra-Large Scale (ULS) systems [Northrop 2006] and as such, invites a completely different approach – one that remains viable in a freely evolving, interdependent collective of data sources / types / modalities / models, analytics, tools, interfaces, mission applications, perspectives, and users, very little of which will ever be under our control. Our objective is to define such a solution.

Conceptual Approach

Our approach to integrating intelligence data in a ULS systems environment is data-centric (as opposed to data-model centric) and proceeds in stages. The first addresses the unified storage of the entire spectrum of intelligence artifacts regardless of modality or representation. The second stage builds upon the foundation provided by the first to address the unified storage of structured data to enable semantic data integration. A third stage (beyond the scope of this paper) addresses unified storage of knowledge models. In all stages we embrace the diversity of domain-specific data-models employed throughout the Intelligence Community by taking a data-model agnostic approach wherein the persistence model makes the least possible commitment to any particular data-model. In the case of “raw” artifacts, this means storing each according to its native representation without the application of structural or semantic transformations. In the case of structured artifacts, it means: a) perceiving data as first-class citizens by de-coupling them from the data-model and b) using an abstract model for the unified representation and persistence of the artifact and integration semantics. A key aspect of our approach is that the character and meaning of the source data and data-model is preserved and made accessible by the data store. The result is a layered Data Architecture and Semantic Integration Framework that can accommodate any kind of data and semantics without placing restrictions on vocabulary, structure, semantics, or constraints, in a way that addresses the needs of the Intelligence Community today while providing a seamless transition path toward a future of ULS systems imbued with semantic technologies.

Scope

The types of intelligence collected by sensors and systems today span the electro-magnetic spectrum to include all manner of signals, audio, video, and images, in addition to so-called human intelligence (e.g. text artifacts such as reports, messages, web pages). Our approach to data integration supports all of these simultaneously regardless of their underlying source data-model, or lack thereof. It does not however, prescribe a solution for data-model harmonization. In particular, our approach imposes no relationship between the data-models to which the artifacts adhere. It does however, allow such relationships, created by external processes of any sort, to be effectively represented and persisted together.

Whereas the business of intelligence is to develop and communicate understanding (which entails the collection, exploitation, and provisioning of intelligence), intelligence business processing includes any automated activity that moves intelligence artifacts with respect to the cognitive hierarchy (see Fig.1a). This includes data collection, semantic enhancement, and fusion from data to information to knowledge, and communication / collaboration to create understanding.

Our Data Architecture and Semantic Integration Framework mirrors both the structure of the cognitive hierarchy and the operations of intelligence business processing. Built atop a collection of indigenous artifacts (see Fig. 1b), Layer 1 of the Framework supports an aspect of collection and rudimentary exploitation of artifact semantics. Layer 2 supports the processing by which data extracted from artifacts is enhanced with semantics to produce information, and the processing by which information is enhanced with richer associations to produce knowledge. Layer 3 supports the management and integration of knowledge models employed by Layers 1 and 2. Finally, Layer 4 supports human computer interfaces through which the analyst “sees” all of this intelligence. This paper focuses on Layers 1 and 2, which together support the provisioning of integrated intelligence at the level of data, information, and knowledge. Layers 3 and 4 will be the subject of a subsequent paper.

Layer 1

The broad and ever-changing spectrum of intelligence artifacts existing within the Intelligence Enterprise today reflects a nearly equally broad and ever-changing spectrum of intelligence collectors, producers, and consumers. The types of artifacts they generate vary tremendously in their modality (e.g. text, images, audio, video, signals), structure (e.g. relational, object, key-value) and representation (e.g. free text, XML, SQL, vector, raster). As this diversity is beyond our control, we term all such artifacts as “indigenous” and the diversity of the external data stores and systems in which they reside as the “wild.”

Layer 1 of our data integration framework addresses the integration of the entire spectrum of artifacts existing in the wild by simply collecting them together in a unified data store. The decision to bring an artifact into the Framework is recorded in Layer 1 by persisting a reference to the artifact along with a minimal set of essential meta-data whose main purpose is: a) to support analysis of artifact content; and b) to provide access to the artifact from the higher layers of the framework (see Fig. 2). The original artifact may also be physically captured in an indigenous data store that sits below Layer 1, however this is not required. In addition to the artifact reference and metadata, the Layer 1 schema also supports associations between artifacts. Any kind of relationship between artifacts can be represented since the set of predicates used to express them is not pre-defined. As described subsequently, predicates are persisted in Layer 3 of the Framework.

Layer 2

Structured Data

Every analyst engaged in intelligence processing either creates or uses structured data. Just as we do not control the sources or format of indigenous artifacts, we also do not control the various methods by which such artifacts might be structured or the data-models employed therein. Thus as the objective of Layer 1 is to represent the diversity of indigenous artifacts regardless of type or format, the objective of Layer 2 is to accommodate the diversity of all structured data regardless of vocabulary, organization, representation, or semantics.

Structured data necessarily adheres to some sort of model, which in general specifies vocabulary, organization, semantics, and constraints. Though not all data-models specify all of these, at minimum, every structured artifact entails a vocabulary reflecting a set of entity types (e.g. person, message) and an organization reflecting their relationships (e.g. message to person). These basic elements are illustrated in the simplified example of Fig. 3. Part (a) of the figure shows a short unstructured text message, and part (b) shows a data-model according to which a message might be structured. Part (c) then shows the original message structured according to the data-model and part (d) shows how that structured message might typically be persisted in a database.

Notice how the database schema is tightly coupled to the data-model that was used to structure the data, and how the raw message is bound to the data-model by the database. In effect, the data-model is imposed on the database, and the data itself is frozen into it such that no additional attributes or relationships are possible (without modifying the database schema). This is a severe shortcoming considering the tremendous variety of ways in which a given artifact might be structured or enhanced with additional attributes and associations. Even for the simple case shown in the figure, we can easily imagine data-models that use different entities (e.g. ‘Individual’ instead of ‘Person’), different relationships (e.g. ‘Sender’ instead of ‘From’), and different organizations (e.g. by including ‘MessageDate’), not to mention the wealth of other information external to the message itself (e.g. about ‘Tanya’) that might be brought to bear.

In a ULS systems environment, it is simply unreasonable to presume that the data-models or the various processes, either automated or manual, that structure data can be controlled or constrained. It is also unreasonable to presume that it is possible to anticipate the totality of their breadth or their application. To the contrary, the urgency and diversity driving our Intelligence Enterprise essentially guarantees that as many different methods for extracting entities, relationships, and events will be brought to bear as our imaginations and increasingly powerful technologies can support. Thus, although we might like to enhance Layer 1 of our Data Architecture and Semantic Integration Framework by exposing all possible extracted elements along with their properties and attributes in order to support efficient information retrieval and broad application, introducing an ever expanding array of fields and tables into a database is as impractical as attempting to accommodate every kind of data and purpose within a single canonical data-model.

The challenge therefore, is to build the next layer of the Framework to accommodate structured data in a way that exposes that structure for use, without imposing the structure on the data store itself. In other words, we must determine a method for storing and managing any kind of structured data, reflecting any data-model, so that it can be shared, efficiently exploited, and extended in unforeseen ways without requiring model-specific storage implementations. In other words, we seek a universal, domain-neutral storage model for structured data.

Data-Model Abstraction

The key to devising a domain-neutral storage model for structured data is to decouple what varies, namely vocabularies and, more generally the data-models, from that which remains constant, namely the source artifact, and ideally the storage structure. To achieve this, we consider structure, vocabulary, semantics, and constraints from a higher level of abstraction from which we then distill a minimal set of elements sufficient to capture any data-model. These are defined as follows:

Sign: A sign, g, is a representation of a chunk of data, either physically located within a tangible artifact, or contained within an analyst’s mind. Examples of the former include a string of text in a document; an object within an image; a segment of audio in an audio stream; a spike in a signal. As illustrated in Fig. 4, regardless of the type of medium, a sign for tangible data is always associated with a physical extent within the artifact and has a quantifiable span, which we call a mention. In contrast, signs that reside in an analyst’s mind become tangible only when she writes down her thoughts. We explicitly include such intangible signs here to support the analyst’s ability to assert information directly into the data store without having to first represent it in a physical artifact. The set of all signs, G = {g_i}, spans across all data sources. In the set, each element is unique: ∀i,j (i
≠ j) à (g_i ≠ g_j). G is the construct by which data are represented. From the text data shown in Fig. 4, signs G’ = {‘Suzi’, ‘Tanya’, ‘July 4, 2007′, ‘Bring lunch’, ‘Message1′} contribute to G (i.e. G’ Í
G), though many more signs may be identified even from this simple example.

Concept:
A concept,
c, is a representation of an abstract idea, defined explicitly or implicitly by a source data-model.  For example, the nodes of an ontology, the tag set in an XML Schema Document (XSD), and the attribute / table names in a relational database all represent concepts. In the set of all concepts C = {c_i}, each element is unique: ∀i,j (i
≠ j) à (c_i ≠ c_j). From the text data shown in Fig. 4, concepts C’ = {‘Message’, ‘Person’, ‘Body_text’} contribute to the full set of concepts C (i.e. C’ Í
C).

Predicate: A predicate,
p, is a representation of an abstract idea used to express a relationship between “things.” Predicates are used in the formation of statements (described below) and may be defined either explicitly or implicitly by a source data-model. For example, the arcs of an ontology, and the attributes of an XML or database schema represent predicates. In the set of all predicates P = {p_i}, each element is unique: ∀i,j (i
≠ j) à (p_i ≠ p_j). The text example of Fig. 4 contributes predicates P’ = {‘To’, ‘From’,
‘Body’} to the set of all predicates P (i.e. P’ Í
P). The only predicate that is “built into” (i.e. defined by) our storage model is the ‘IsInstanceOf’ predicate, which is used to disambiguate signs to form terms as described below. Concepts and predicates are the constructs by which we link to data-models and, thereby, explicitly expose data-semantics.

Term: A term,
t_ij,
is an ordered pair <g_i,c_j> where g_i ∈ G and c_j ∈ C. Each term represents a disambiguated sign. The process of disambiguation associates a sign with a concept using the ‘IsInstanceOf’
predicate (though not every sign from G is necessarily disambiguated, and not every concept from C is necessarily used for disambiguation). In the set of all terms T = {t_ij}, each element is unique: ∀ i,j,k,l
(i ≠ k or
j ≠ l)
à (t_ij ≠ t_kl). The text example of Fig. 4 contributes terms T’ = {t₁, t₂, t₃, t₄} where t₁ = <’Suzi’, person>, t₂ = <’Tanya’, person>, t₃ = <’Bring lunch’, Body_text>, t₄ = <Message1, message> to the complete set of terms T (i.e. T’ Í
T).

Statement: A statement, s, encodes a binary relationship between a subject and an object mediated by a predicate. A statement is represented by an ordered triple s_ijh = <subject_i, predicate_j, object_h>. Among the set of all statements, each element is unique: ∀ i,j,h,l,m,n
(i ≠ l or j ≠ m or h ≠ n)
à (s_ijh ≠ s_lmn). In our model, subject and object may be either a term or statement. The simplest kind of statement is one in which subject and object are terms s0_ijh = <t_i, p_j, t_h>. Statements in which the object is itself another statement represent reifications: s1_klm = <t_k, p_l, s_m>. Finally, a statement in which both subject and object are other statements represents a relationship between statements: s2_xyz = <s_x, p_y, s_z>. The set of all statements S = {s0_ijh} U {s1_klm} U {s2_xyz}. The text example of Fig. 4 shows three statements: S’ = {<t₄, to, t₁>, <t₄, from, t₂>, <t₄, body, t₃>} all with the same subject, which is the term corresponding to the message itself. These statements contribute to the set of all statements, i.e. S’ Í
S.

Note that the above definitions are formulated to be clear and unambiguous with respect to our particular approach and may not match those found in other literature. Throughout the paper, we will denote instances of signs, concepts, predicates, terms, and statements using Arial font within single quotes (e.g. ‘person’).

DDF

Abstracted from the milieu of all possible data-models, these elementary constructs (concept, predicate, sign, term, and statement) provide the fixed-points of a data reference model that will ultimately form the basis of a practical Data Architecture and Semantic Integration Framework which we call the Data Description Framework (DDF). Despite its simplicity, the DDF is an amazingly rich model that can be viewed from at least two different perspectives. From one perspective, the DDF encompasses a synergistic combination of two higher order models lying along different dimensions of abstraction – one that is outward-looking (“extrospective”), one inward-looking (introspective).

The “extrospective” portion of the model is a meta-model formed by (a) C and P, which look outward to domain knowledge (represented in data / knowledge models), and (b) G, which looks outward toward the data. Signs bring data into the DDF as first class entities which may then participate in various, unlimited conceptualizing relationships created by any sort of automated or manual process at any time. Signs provide a fundamental level of data integration (that traditional approaches lack) resulting from having eliminated data-model barriers. Concepts and predicates are to domain knowledge what signs are to data. They are the mechanism by which such knowledge (typically encoded in domain-specific data / knowledge models) is linked into the DDF and exposed by our Data Integration Framework for use and re-use.

The introspective portion of the model is a semantic model formed by T and S which abstract data-model internals to expose structure in a uniform way. Terms bind signs to concepts, exposing the meaning of the data unambiguously with respect to the original source data-model. Statements represent semantic relationships about, within, and between disambiguated data elements.

Together the introspective and “extrospective” models that comprise the DDF enable both horizontal and vertical data integration. The “extrospective” abstraction bridges data and domain knowledge (vertical integration). The instrospective abstraction bridges data structured by various disparate processes (horizontal integration) and binds the two outward looking faces of the “extrospective” model to provide a comprehensive data integration model.

From the second perspective, the DDF may be regarded as a synergistic combination of two interaction patterns – one that decouples, one that binds. DDF achieves decoupling in two ways. First, as a higher order data-model abstraction, DDF effectively decouples data from data-models. Thus, the DDF can encapsulate any sort of data regardless of the source data-model. Second, as a higher order data-structure, DDF effectively decouples structured data from data storage structures. Thus, the DDF can accommodate any data regardless of the source storage structure. As a result, the DDF provides a practical foundation for implementing a stable database that can accommodate any sort of structured data.

The ways in which DDF implements binding are illustrated in Fig. 5. Specifically, sign g binds with concept c to form term t,
and predicate p
binds with term t
to form statement s. The diagram also indicates that a predicate may bind a term and a statement to form reification or a predicate may bind a statement with another statement to form a statement relationship. These bindings allow data to be integrated within and across data-models and continuously enriched into knowledge.

Together these interaction patterns make the DDF a powerful yet practical platform for data integration. Decoupling gives DDF the character of a universal data store and successive bindings progressively move intelligence artifacts (or their constituent elements) upward through the cognitive hierarchy. The result is a universal data integration and semantic enrichment platform that supports data structured by any means, unrestricted associations within and between them, and increasingly rich semantics.

Representational Power

Although the expressiveness of the DDF is sufficient to capture the data and data-semantics of any structured data source, we illustrate this for the relational model since it is the most commonly used. Similar arguments can be made for other model types, such as hierarchical, object-oriented, and graph.

In accordance with common relational formalism [Date 2004], a relation R is defined by the set of attributes A = {A_i} (1 ≤ i ≤ n). The subset of attributes that comprise the primary key are denoted as K={K_l} (1 ≤ l ≤ k), K Í
A. The set of all data values in R is D = {d_ij},
where d_ij is a value on the intersection of attribute A_i and row W_j (1≤ j ≤ m). We can integrate data and its original semantics from R into a DDF data space consisting of G₀, C₀, P₀, T₀, and S₀ according to the following procedure:

• All attributes of R are added to the set of concepts:

  C = C₀ U A
  

• Non-key attributes are added to the set of predicates:

  P = P₀ U (A – K)

• D’ = {d’_i} is the set of unique values of D: ∀i,j (i
  ≠ j) d^‘_i ≠ d^‘_j . The values in D’ that are not already present in G₀ are added to the set of signs:

  G = G₀ U (D’– G₀)

• We build the set of terms T_R = {t_ij} where t_ij=<d_ij, A_i> and 1 ≤ i ≤ n, 1≤ j ≤ m. T’_R is the subset of unique terms of T_R. Terms of T’_R are added to T₀.

  T = T₀ U T’_R

• We build the set of statements S_R = {s_ij} where s_ij = < <d_kj, K>, A_i, <d_ij, A_i> >, d_kj represents the combination of values of the key attributes for the row W_j, A_i
  Í
  A-K, and k+1 ≤ i ≤ n, 1≤ j ≤ m. Statements of S_R are added to S₀:

  S = S₀ U S_R

Representation of R in DDF is lossless (no loss or distortion of data and semantics, even though semantics of R is not explicitly represented in DDF) because we can restore R from DDF:

1. R is contained in statements S, therefore, using processing metadata (described in the following section and shown in Fig. 6), extract from S the statements that originated from R:

   S_R = {s_ij} where s_ij = < <d_kj, K>, C <d_ij, A_i> >
   

2. From S_R restore the structure and rows of R as follows:

The process that was used to build combinations of values of the key attributes can be reversed to get to the relation in its original form:

Therefore, by the integration procedure described above, the data and data-semantics from R are faithfully represented with the DDF. The structure of R itself and its identity integrity are explicitly captured in Layer 3.

This procedure further reveals two powerful and distinguishing features of the DDF:

• The DDF can accommodate data and data-semantics from structured sources without loss or distortion.
• Structured artifacts can be integrated within the DDF in a mechanical fashion without requiring prior knowledge, and or analysis of, their domain-specific data-models.

Integration Power

The DDF exposes four levels of data and semantics (signs, terms, statements, and concepts and predicates), which support four levels of largely independent integration actions or patterns: establishing signs, disambiguation, association, and data-model enhancement:

• Signs are established from mentions asserted by users or data ingest process. Many mentions may relate to the same sign, and it is this “re-use” of signs that provides a primal level data integration.
• Data-model enhancement occurs when processes operating on Layer 3 extend, enhance, or harmonize data-models associated with the incorporated data sources. Since these operations do not affect the original data-models, but serve to establish new overarching data-models, multiple perspectives can co-exist (e.g. both the original and enhanced).
• Disambiguation actions create associations between signs and concepts, either when data is ingested into the DDF, or by a subsequent semantic process. The same sign may be disambiguated in any number of different ways and signs and concepts may be associated regardless of their originating data source. In other words, the sign and the concept to which it is bound may originate from disparate sources.
• Finally, association actions create binary statements between terms and or statements using a predicate. These may originate from semantic relationships expressed within a data source or may be created by a subsequent semantic process. Any term / statement may be associated to any other regardless of their origin.

By virtue of these integration actions, the DDF is able to support an essentially endless process of semantic enrichment and each of these actions may be conducted on any integration level without necessarily affecting the other elements.

Evolutionary Power

The independence of the integration actions imbues the DDF with extraordinary evolutionary power. New sources, modifications of the integrated data sources, and changes of integration models can be accommodated without requiring the data space to be rebuilt. In particular, the growth of a source system, both in terms of data and semantics, can be accommodated by the addition of new DDF elements (signs, terms, statements, concepts, and predicates) associated with that growth. Source system modifications can be handled in several ways depending on how the integrated store is used. Because the dependencies between all of the elements in the Framework are known, we can always define how to proliferate / manage changes and the approaches employed may vary from a source to a source. Finally, new integration models can be introduced by the integration actions described previously and these will simply co-exist with all other models. If an integration model needs to be modified, we can introduce it as a new model or mark the changed elements appropriately and proliferate those changes.

DDF in Relationship to RDF

The reader familiar with the Resource Description Framework (RDF/RDFS) may wonder what is different here. Indeed, RDF and DDF share DNA, so to speak, since both employ a similar level of abstraction and make semantics explicit. Unlike RDF however, DDF also prescribes the exposure of data as signs which can freely participate in the disambiguations and associations necessary for data integration. In other words, data represent themselves directly as signs and participate in the DDF as first-class citizens. Because of the way they are designed, DDF signs provide a primal level of data integration. In contrast, data in RDF are represented either as literals or by URIs. A datum represented as an RDF literal cannot be explicitly disambiguated or associated. Furthermore, because the URI is a first-class citizen, not the datum, there is no mechanism in RDF to prevent a single datum from being represented by multiple URIs and or literals.

This is not a criticism of RDF as these differences reflect the fact that DDF is an abstract model aimed at data integration, whereas, RDF is a meta-model. Thus, employing RDF for data integration necessitates building a model in RDF (i.e. a particular meta-model instance) along with rules prescribing the manner of data exposure. In contrast, DDF is a model that makes explicit commitments to support data integration. Because this model represents an abstraction over domain data-models, the DDF can represent data structured by any data-model, and be represented in any meta-model (including RDF). Fig. 6 illustrates the place of DDF in relation to the models and meta-models.

Implementation

A universal storage model based on DDF can be implemented in a variety of ways (e.g. objects, relations, triples). We have explored and implemented two approaches, the first using relational technology (Oracle, mySQL) and the second using cloud technology (Hadoop / HBase) [Hadoop], [HBase], [Chang 2006]. In both cases we employ the Dimensional Data Modeling (DDM) approach [Kimball 2002] because it nicely captures the business processes associated with moving intelligence artifacts upward through the cognitive hierarchy while accommodating the metadata that intelligence processing requires. In particular, we maintain not only the contextual metadata about the indigenous artifact itself (e.g. the who what when and where of its creation and transmission), but also process metadata regarding the processing by which signs, terms and statements are created. The former are captured in the Layer 1 storage structure as described previously while the latter are accommodated in Layer 2.

As reflected in Fig. 7, the essential intelligence business processes that the DDF captures are semantic disambiguation and association formation. Thus, the DDF storage model consists of two main fact-tables, SemanticFact
and
AssociationFact. The SemanticFact table records metrics relating to the formation and disambiguation of signs, and references dimension tables that record signs, concepts, and process metadata. The signs themselves are represented using two tables, Sign and Mention. The value of a mention is identified by the region of the artifact in which it is localized. The boundary of such a region is recorded in the Mention table. The value of a sign may represent any number of source mentions that are exactly the same or are considered to be the same from the perspective of the process which extracts / identifies them. The Concept dimension records elements from the domain knowledge which includes the source artifacts’ data-models. Each record in the SemanticFact table binds a sign to a concept using ‘isInstanceOf’
semantics.

The AssociationFact table records metrics relating to the formation of associations and references dimension tables that record statements, predicates, and process metadata. Recall that statements come in three types – an assocation between terms (i.e. statement), an association between a term and another statement (i.e. reification), and an association between two statements (statement relation). These are accommodated by the three subclasses of the Statement dimension which are Statement0, Statement1, Statement2 respectively. The Predicate dimension records predicates from the domain knowledge.

The ProcessMetadata package shown in Fig. 7, represents a collection of dimensional tables used to record operational and contextual metadata about the various external processes that create SemanticFact and AssociationFact records. The particular elements and formulation of this metadata would be designed to support the information assurance needs of the Intelligence Community. Typically these would include Date, Time, Creator, and SecurityClassification dimensions.

The DDF does not prescribe or constrain the processing by which the DDF storage model would be populated, and the nature of such processing depends both on the modality and structure (or lack thereof) of the indigenous artifacts. Nevertheless, to illustrate how DDF works, and provide more insight into the relationship between external processes and our Data Integration Framework, the interested reader may find a brief discussion of the processing by which Layers 1 and 2 would be populated in the Appendix.

Relation to Other Approaches

Data quality professionals widely recognize the importance of data integration and the need for efficient data integration approaches to redress a panoply of data quality problems [Lee 2006].

A large body of work exists on data integration approaches [Batini 1986], [Parent 1998], [Halevy 2005],[Bernstein 2007], many of which have contributed to successful Enterprise Information Integration solutions. However, because they all are based on some kind of data-model harmonization (i.e. mapping), they fail to provide practical solution for ULS intelligence data integration. In particular, data-model integration does not address data integration, which intelligence data processing requires. Physical data integration, typical of data warehouse applications, also requires heavy up-front data-model analysis and harmonization as well. This activity is not only resource intensive, it often results in the loss and or distortion of data and its semantics which, in the context of intelligence, may reduce the richness and power of the data. DDF addresses the needs of the Intelligence Community by supporting ad-hoc, lossless data integration without imposing a heavy pre-processing burden.

The Dataspaces approach introduced by Halevy et al. [Franklin 2005], [Halevy 2005], [Halevy 2006-1], [Halevy 2006-2] is similar in philosophy to the DDF in that it supports the co-existence of disparate data sources regardless of their type.

“Dataspaces are not a data integration approach; rather, they are more of a data co-existence
approach. The goal of Dataspaces support is to provide base functionality over all data sources, regardless of how integrated they are” [Franklin 2005].

With this approach, a mediated (general, global) schema serves as an integration model. Individual sources participate in the integration by exposing Local-As-View (LaV) schemas that comply with the mediated schema. In practice, a LaV is implemented as a view on the source, and the mediated schema provides an interface to Dataspaces participants. To support the storage of new associations between data (a shortcoming of virtual integration) it may be necessary to introduce a Local Store for these associations. DDF and Dataspaces represent two different approaches to data management and have different niches: Dataspaces focuses on eliminating the barriers to data access and provides some limited data integration capability, whereas DDF focuses on comprehensive data integration and supports deep semantic enrichment. The DDF can leverage the Dataspaces as a mechanism of data access, and participate within Dataspaces as a semantically rich Integrated Local Store.

A number of commercial products, including the ones that are being used in military applications, e.g. Palantir [Palantir], are based on the object meta-model where data entities are represented by objects with dynamic properties. Although they claim to be able to accommodate data from any structured or unstructured source, in practice they impose a particular, albeit modifiable, data-model on the structured data. This requires heavy pre-processing of the source, as is typical for such solutions, and results in loss and or distortion of source data and data semantics. In addition, customers are dependent on the solution provider as they cannot perform modifications of the integration model.

The Information Model Interoperability Reference Model [Melnik 2000], [Omelayenko 2001], proposed for presenting information on the web, consists of three layers – syntax, object, and semantic. The syntax layer represents serialized data content, similar to our indigenous text artifacts. The semantic layer provides semantics through data-models and languages, and the object layer provides a bridge between the two. Unfortunately however, the IMI does not provide a practical model for the implementation of those layers and their interfaces.

The Data Reference Model (DRM) of the Federal Enterprise Architecture (FEA) aims to provide standards for the description, categorization, and sharing of data [DRF 2005]. Unlike DDF however, it does not resolve the issues of data integration and unfortunately exhibits the typical shortcomings of most physical and virtual data integration approaches.

Finally, the Common Warehouse Model (CWM) [CWM 2001] offers a standardized approach (and tools that support it) for representing and mediating the automated interchange of metadata in warehouse applications that involve multiple data sources and data processing applications. Being focused on metadata integration, as opposed to data integration, the CWM mainly addresses issues relating to Layer 3 of our Data Integration Framework.

Current & Future Work

Today there is a deployed system called the Joint Intelligence Operational Capability in Iraq (JIOC-I) that essentially implements Layer 1 of our Data Integration Framework, though only for text artifacts. Unfortunately, the JIOC-I by itself falls short of a complete integration solution because it does not address structured data in a way that exposes that structure to support further analytical processing and visualization. In other words, it lacks Layer 2. Consequently, there has been much criticism of the JIOC-I, along with various suggestions for “fixing” it (e.g. by extending the schema to accommodate structured data). In contrast, we recognize the JIOC-I as a foundational element (that got it mostly right) and a first step toward a ULS intelligence system that integrates data while embracing data diversity. Indeed, the JIOC-I was the inspiration that led us to develop the layers above, and the DDF in particular.

While several relational database implementations of Layers 1 and 2 of our Data Architecture and Semantic Integration Framework are being developed and tested in various US Army CERDEC I2WD projects, our most ambitious implementation is being made for the Army using cloud computing technology. Our current effort on a 52-node cloud implements all three Layers of the data architecture using the Hadoop Distributed File System (HDFS) and HBase to achieve an Ultra-Large Scale, unified “dataspace” that supports not only a diversity of data, but also a diversity of processing (Hadoop Map / Reduce [Dean 2004] analytics). Although we reserve the performance metrics and details of the DDF implementation within the cloud computing key-value meta-model for a separate paper, it is interesting to note that cloud appears to accommodate the DDF in a particularly efficient manner.

Other key aspects of our Data Integration Framework are described elsewhere. [Yoakum 2008 IQIS] highlights the low barrier to entry for data integration by describing the process for lossless mechanical data ingestion which requires no costly pre-processing or data-model harmonization. Data surfing, drilling, and discovery on the DDF unified data space are described in [Yoakum 2008 DAMA]. [Yoakum 2008 SIMA] addresses the utility of DDF in Situation Management – another activity that requires rapid, ad-hoc data integration. Finally, [Yoakum 2009] describes a formation of a contiguous and therefore navigatable integrated data space that enables “vertical” integration from artifacts to structured data and to knowledge models, and “horizontal” integration of data from various sources.

As they are matured, we anticipate that Layers 3 and 4 of the Framework will provide fertile ground for entirely new work in knowledge interaction and perception. Layer 3 serves as a universal substrate on which to explore, discover, and encode relationships between knowledge models that go well beyond harmonization and integration to include, for example, dissonant perspectives which can not and should not be “harmonized.” Layer 4 provides the lenses through which the human user looks into this cauldron of knowledge, information, and data to explore and make sense of the object of his interest (e.g. a domain, a situation, an entity) according to a chosen perspective. Having all four layers present will close the loop between data and knowledge in both directions so that they may co-evolve to yield more complete and accurate understanding. Atop the immense foundation of integrated data provided by Layers 1 and 2, Layers 3 and 4 will fuel the engines of ULS systems research for a very long way into the future.

Conclusion

The Intelligence Enterprise is inexorably evolving into an Ultra-Large Scale systems world that can not, and will not, be constrained in its processes or products. The data integration problem is but one early symptom of this burgeoning reality. Although this knowledge does not provide a recipe for good solutions, it makes it rather easy to spot bad ones. Unfortunately, current data integration approaches generally represent the latter.

In this paper, we have presented the first two layers of a multi-layer Data Integration and Semantic Enrichment Framework that enables deep semantic data integration in a ULS systems environment. The model on which it is founded, the DDF, supports both horizontal and vertical data integration (i.e. across disparate data-models and from data to knowledge) by embracing the diversity of data / knowledge models and processes by which data is structured. More importantly, the model admits a practical implementation (i.e. “hard running code”) that accommodates artifacts of any modality (e.g. text, audio, images, video, signals) in a single unified data store that enables true data integration and the continuous enrichment of data into knowledge.

References

[Batini 1986] Batini, C. et al. A comparative analysis of methodologies for database schema integration, ACM Computing Surveys, (18) 4, 1986.

[Bernstein 2007] Bernstein P., Ho, H. Model Management and Schema Mappings: Theory and Practice, Proceedings of VLDB Conference, 2007.

[Chang 2006] Chang, F., et. al. Bigtable: A Distributed Storage System for Structured Data. 2006. http://labs.google.com/papers/bigtable.html

[CWM 2001] Object Management Group “Common Warehouse Model (CWM) Specification“, OMG, 2001. http://www.omg.org/docs/ad/01-02-01.pdf

[Date 2004] Date, C. An Introduction to Database Systems, 8^th edition, Addison Wesley, 2004.

[Dean 2004] Dean, J. and Ghemawat, S. MapReduce: Simplified Data Processing on Large Clusters. 2004. http://labs.google.com/papers/mapreduce.html

[DRF 2005] Federal Enterprise Architecture Program “The Data Reference Model“, 2005. http://www.whitehouse.gov/omb/egov/documents/DRM_2_0_Final.pdf

[Franklin 2005] Franklin, M., Halevy, A., and Maier, D. From Databases to Dataspaces: A New Abstraction for Information Management. ACM SIGMOD Record, 2005.

[Hadoop] Apache Hadoop. http://hadoop.apache.org/

[Halevy 2005] Halevy, A. et al. Enterprise information integration: successes, challenges and controversies, Proceedings of 24th International Conference on Management of Data, Baltimore, 2005.

[Halevy 2006 - 1] Halevy, A. Franklin, M., and Maier, D. Principles of Dataspace Systems. Proceedings of the twenty-fifth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, 2006.

[Halevy 2006 - 2] Halevy, A., Rajaraman, A., and Ordille, J.
Data Integration: The Teenage Years. Proceedings of VLDB Conference, 2006.

[HBase] HBase Tutiorial. http://arunma.com/2008/10/26/hbase-tutorial/

[Izydor 2007] Izydor, C. and McCollum, P. BI, Process and Integration Trends. DM Review Magazine, August 2007. http://www.dmreview.com/issues/20070801/1089409-1.html?portal=data_integration

[Kimball 2002] Kimball, R. and Ross, M. The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling, Wiley, 2002.

[Lee 2006] Lee, Y., Pipino, L., Funk, J., Wang, R. Journey to Data Quality, The MIT Press, Cambridge, MA, 2006

[Melnik 2000] Melnik, S. and Decker, S. A layered approach to Information Modeling and Interoperability on the Web. Proc. ECDL’00 Workshop on the Semantic Web, Lisbon, Portugal, Sept 2000. http://infolab.stanford.edu/~melnik/pub/sw00/.

[Northrop 2006] Northrop, L., et al., Ultra-Large-Scale Systems The Software Challenge of the Future, Pittsburgh: Carnegie Mellon University, 2007. http://www.sei.cmu.edu/publications/books/engineering/uls.html

[Omelayenko 2001] Omelayenko, B. and Fensel, D. An Analysis of B2B Catalogue Integration Problems. Proceedings of the International Conference on Enterprise Information Systems (ICEIS-2001), July 7-10, 2001, p. 945-952.

[Palantir] Palantir Technologies. http://www.palantirtech.com/

[Parent 1998] Parent, C. and Spaccapietra, S. Issues and approaches of database integration, Communications of the ACM, 41(5), 1998.

[RDF 2004] RDF Core Working Group “Resource Description Framework (RDF)”, W3C, 2004. http://www.w3.org/RDF/.

[Steinberg 1998] Steinberg, N., Bowman, C. L. and White F. E. Revision to the JDL Data Fusion Model, Joint NATO/IRIS Conference, Quebec City, October 1998.

[Yero 2008] Yero, J. Logical vs. Physical Data Integration: A Practical Decision Guide, The DAMA International Symposium & Wilshire Meta-Data Conference. San-Diego, CA, 2008.

[Yoakum 2008 IQIS] Yoakum-Stover, S. and Malyuta, T. Unified Architecture for Integrating Intelligence Data, Proceedings of MIT Information Quality Industry Symposium, MIT, Cambridge, MA, 2008.  http://blog.systover.net/

[Yoakum 2008 DAMA] Yoakum-Stover, S. and Malyuta, T. Unified Integration Architecture for Intelligence Data, Proceedings of DAMA International Europe Conference, London, UK, 2008.  http://blog.systover.net/

[Yoakum 2008 SIMA] Yoakum-Stover, S. and Malyuta, T. Unified Data Integration for Situation Management, Proceedings of the 4th IEEE Workshop on Situation Management (SIMA 2008) at MILCOM 2008, San Diego CA, 2008.  http://blog.systover.net/

[Yoakum 2009] Yoakum-Stover, S., Malyuta, T., and Antunes, N. A Data Integration Framework with Full Spectrum Fusion Capabilities.
MSS Information Fusion Symposium, Las Vegas, NV, Aug 3-7, 2009. http://blog.systover.net/

Appendix – Processing

Ingestion

Consider first, processes that load indigenous artifacts into Layer 1 either physically or virtually so that they may be unambiguously referenced within Layer 2. Typically these are called ingestion processes. Such processes insert either the entire indigenous artifact, or a reference to its location within the authoritative data source, into Layer 1. In addition, both artifact and process metadata are recorded in the appropriate metadata tables. The former essentially provides a card catalogue for the artifact and the latter provides information assurance.

Unstructured Information

Processes that structure unstructured artifacts generate SemanticFact and AssociationFact records in Layer 2. Each such process necessarily entails a particular data-model. This data-model is persisted in Layer 3. Concepts and predicates from the data-model (or references to them) are also persisted in the Concept
and Predicate dimension tables of Layer 2 along with sufficient metadata to identify and retrieve the data-model source artifact (i.e. schema, ontology, etc..).

Unstructured information processing typically identifies all instances of the concepts within its data-model or type system. For example, a given text extractor may identify all ocurrences of ‘IBM’ and associate them with the concept ‘Company.’ Each such instance is represented as a DDF mention. The position of each mention within the source artifact is recorded in the Mention table (e.g. using beginChar, endChar) and a single record is added to the Sign table using, for example, the actual contents of the span (‘IBM’) as the sign value. Each disambiguation ocurrence (i.e. the association made by the text extractor between a mention and a concept) is recorded in the SemanticFact table along with appropriate process metadata, and a term consisting of <sign, concept> is created in the Term table (if such term does not already exist).

Further semantic processing may identify relationships between elements within the artifact. The elements themselves would have already been recorded as SemanticFacts. For each such relationship, an AssociationFact is recorded along with appropriate process metadata, and a Statement table entry is created.

Unstructured information processing of other than text artifacts is similar, the main differences being that entries in the Mention table will have a different spanCoordinateType, and the method for assigning a sign value will be different. For example, consider object recognition software that extracts faces from within an image of a crowd. For each extracted face, the corresponding rectangular area of the image could be recorded in the Mention table with the help of pixelUpperLeft and pixelLowerRight, and a sign (e.g. ‘Suzi’s faceImage’) would be assigned to all extracted mentions.

Extract-Transform-Load

Consider next, Extract-Transform-Load (ETL) processes that pull data from other structured data sources, typically databases, into Layer 2. The initial phase of the ETL loads the source data-model (e.g. database data dictionary) into Layer 3, and concepts and predicates (or their references) into in the Concept and Predicate
dimension tables of Layer 2. Sufficient metadata necessary to identify and retrieve the data-model source artifact (i.e. schema), are also stored. Subsequent ETL processing, which entails a mapping to the DDF structure, inserts signs, terms, and statements into the SemanticFact and AssociationFact tables along with appropriate process metadata.

Because the ETL process needs only to capture the explicit semantics of the source meta-model (e.g. relational, hierarchical, graph…), one ETL can be developed for a whole class of data stores. For example a discussion of ETL for relational stores may be found in [Yoakum 2008 IQIS].

Interactive

Finally, consider an interactive user interface that enables an analyst to assert semantic and association facts directly into the DDF. The analyst will have the option to use existing concepts, predicates, terms, and statements or to create new ones. In the case of the latter, recorded and asserted mentions will reference the source analyst. Metadata recorded for manual processes with also reference the source analyst

Date: July 14, 2009

Time: 1600 hrs

Location: Suite 313, 510 King Street, Alexandria VA 22314

Highlights of the agenda include a review of the articles of incorporation, bylaws, treasurer’s report, new business, and announcements.

The directors are listed as:

1. Mark Andrew Eick
2. Suzanne Yoakum-Stover
3. Oscar Wood

4th IEEE Workshop on Situation Management (SIMA2008) at MILCOM 2008, San Diego CA, 2008

S. Yoakum-Stover, Ph.D.

Potomac Institute for Policy Studies

US Army CERDEC I2WD Information Exploitation Futures Lab

T. Malyuta, Ph.D.

New York City College of Technology

Computer Systems Technology Department

Abstract

We propose a new solution for data integration and semantic enrichment in support of Situation Management (SIMA). Our solution applies to any modality (e.g. text, images, audio, signals etc.) and embraces the diversity of data sources, types, and models, placing no restrictions on processes, applications, or users. It is database centric and proceeds in stages to address the unified storage of structured data and its semantic enrichment in a way that remains viable in an Ultra-Large Scale systems environment. The result is a layered data integration architecture that can accommodate any kind of data to coherently support the multiplicity of processing required for SIMA.

Challenge of Data Integration in Situation Management

Though generally scoped around a particular set of circumstances, or state of affairs, Situation Management (SIMA) is a mega-process occurring in a heterogeneous and volatile data space resulting from a cacophony of human and automated systems. To understand a situation and engineer the means for managing it, we must organize its data space. In particular, the heavy load of sophisticated processing for the anticipation, recognition, and influence of a situation must be girded with an architecture that enables data sourced from wildly disparate systems, having different modalities, structures, and semantics, to be integrated into one coherent body of situational knowledge.

In most business intelligence applications, data is integrated across information systems to support a choreographed interplay of services comprising an established set of business processes. In contrast, the constituent events in SIMA typically entail information systems that are far more diverse and whose dynamic interplay is less scripted, less repeatable, and therefore less predictable. Since many of these information systems capture data for completely different and unrelated purposes, and were never intended as participants in a coherent process, for SIMA we require a data architecture that enables them to be dynamically re-used or re-purposed. Because every situation is unique and we cannot anticipate all the right “business processes,” we need the capability to quickly fuse data often in high volumes from an ad-hoc set of systems, sometimes with knowledge asserted by analysts, in meaningful ways on the fly.

Traditional approaches to data integration, both physical and virtual [Batini 1986, Parent 1998, Halevy 2005, Bernstein 2007], cannot accommodate the complexity, heterogeneity, and volatility of the SIMA data space. In actual practice, the canonical data-models that underlie such approaches, including federation, are simply too rigid. They cannot adapt their structure to handle new data sources, associations, processes, or applications without heavy manual intervention. Moreover, such approaches generally result in the loss and or distortion of data, semantics, and context, all of which may be useful or even critical in SIMA. Even if initially successful, the IT costs associated with sustaining such systems as well as the human costs resulting from their deficiencies can be devastatingly high.

The scale and complexity of SIMA places it squarely in the domain of Ultra-Large-Scale systems which are characterized by decentralization; inherently conflicting, diverse, and unknowable requirements; heterogeneous, changing and inconsistent elements; normal failures; continuous operation, evolution, and deployment; and immense scale along many dimensions [Northrop 2006]. As such, SIMA demands a supporting data architecture that remains viable in a freely evolving, interdependent collective of systems, people, policies, cultures, and economics, very little of which will ever be under our control. Our objective is to define such a solution.

Data Description Framework

To organize the SIMA data space in a ULS systems environment, we enable semantic data integration by providing for the unified storage of structured data. We embrace the diversity of domain-specific data-models by taking a data-model agnostic approach wherein the integration model makes the least possible commitment to any particular data-model. We achieve this by identifying the universal aspects inherent in all structured data and creating an integration model based on that. A key aspect of our approach is that the character and meaning of the source data-model is preserved and made accessible by the data store. The result is a data architecture that can accommodate any kind of data without placing restrictions on vocabulary, structure, semantics, or constraints, in a way that addresses the needs of the SIMA Community today while providing a seamless transition path toward a future of ULS systems imbued with semantic technologies.

The key to devising a domain-neutral storage model for structured data is to decouple that which varies, namely vocabularies and, more generally the data-models, from that which remains constant, namely the source artifact, and ideally the storage structure. To achieve this, we consider structure, vocabulary, semantics, and constraints from a higher level of abstraction from which we then distill a minimal set of elements sufficient to capture any data-model. These are illustrated in Fig. 1 and defined as follows:

Sign: A sign is a chunk of data, either physically located within a tangible artifact, or contained within an analyst’s mind. Examples of the former include a string of text in a document; an object within an image; a segment of audio in an audio stream; a spike in a signal. As illustrated in Fig. 1, regardless of the type of medium, tangible signs are always associated with a physical extent (i.e. quantifiable span which we call a mention) within the artifact. In contrast, signs that reside in an analyst’s mind become tangible when she writes down her thoughts.

Concept:
A concept is an abstract idea, defined explicitly or implicitly by a source data-model. For example, the nodes of an ontology, the tag set in an XML Schema Document (XSD), and the attribute / table names in a relational database all represent concepts. Concept is an abstraction of such representations, which in the example of Fig. 1 includes Message, Person, and Body_text.

Predicate: A predicate is an abstract idea used to express a relationship between “things.” They are used in the formation of statements (described below) and may be defined either explicitly or implicitly by a source data-model. For example, the arcs of an ontology, and the attributes of an XML or database schema represent predicates. In Fig. 1, To, From, and Body represent predicates.

Term: A term is a disambiguated mention abstracted from the source artifact or asserting analyst. The process of disambiguation associates a mention with a concept, implicitly using the IsInstanceOf
predicate. However, not every such pairing results in a distinct term. All signs that are identical, and that are identified as having the same meaning, are represented by a single term. In the example of Fig. 1, Suzi
IsInstanceOf
Person represents a term.

Statement: A statement encodes a binary relationship between a subject and an object mediated by a predicate. In our design, subject and object may be either a term or statement. The simplest kind of statement is one in which subject and object are terms. Statements in which the object is itself another statement represent reifications. Finally, a statement in which both subject and object are other statements represents a relationship between statements. In Fig. 1, we see three statements, all with the same subject, which is the term corresponding to the message itself.

This organization of these elementary constructs (sign, concept, predicate, term, and statement) defines a data reference model, which we call the Data Description Framework (DDF) [Yoakum 2008 DAMA]. Because it effectively decouples data from data-models and structured data from data-structures, it can encapsulate any sort of data-model and support any data-structure. Because it binds knowledge to data, it enables deep data integration and semantic enrichment. Because it provides a foundation for implementing a stable database, it serves as a practical data integration platform.

In the subsequent text, we represent mentions, concepts, and predicates using Arial font. Terms are denoted as [mention, concept] (e.g. [Adam, Chemist]) and statements are denoted using an intuitive triple representation, e.g. [Adam, Chemist] hasInventoryID [1001,InventoryID].

The Unified Data Space

As illustrated in Fig. 2, the DDF forms a layer of data and semantics (Layer 2) lying between the indigenous source systems (Layer 1) and their knowledge models (Layer 3). Layer 1 feeds the layers above, and Layers 2 and 3 interact: Layer 3 provides semantic context for Layer 2 and Layer 2 participates in the formation of an overarching knowledge model in Layer 3. Together Layers 2 and 3 form what we call the unified DDF data space.

Illustrative Example

To convey a more tangible understanding of the DDF to the user, in this section we present a simplified example that illustrates:

• Loading three disparate data sources into the DDF
• Surveying the resulting integrated data space
• Enhancing the data space with additional semantic associations
• Exploring the enriched data space

Loading the DDF

Loading structured data into a DDF store is a straightforward, mechanical Extract – Transform – Load (ETL) process. This process maps the original data and semantics into the DDF using a pattern that depends primarily on the type of data source because it needs only to capture the structure and semantics of the relational metamodel (not the structure and semantics of a specific instance). For example, our prototype loader works out-of-the-box for most relational databases, extracting data structure and data from the source’s data dictionary and relations as follows:

• Data instances ® signs
• Table attributes ® concepts
• Signs are bound to their respective concepts to form terms
• Predicates are derived from non-key attributes (i.e. concepts) using ‘has’ semantics. For example the predicate derived from the concept Project is hasProject.
• Within a record, terms associated with primary key columns are semantically linked via derived predicates to terms associated with non-primary key columns to form statements. For example, [Adam, ChemistName] hasProject [P1, Project].

Figures 4 and 5 illustrate the result of the mechanical ETL for the three data sources shown in Fig. 3. For the purpose of our illustration, we assume that everything from the sources presented in Fig. 3 is loaded, but this need not be the case. We may freely choose which parts of a data source to load and when to load them. For example, we may choose to load specific views of the source data, or perhaps only the structure of a data source, lazily loading instances only when requested. Finally, the DDF can (and should) capture any desired metadata associated with the source artifacts, the ETL process itself, the quality / strength of semantic and association facts, or any other aspects of the data space elements. For simplicity we do not illustrate this.

Surveying the Unified Data Space Floor

We refer to the integrated data space that results simply from loading data into the DDF as the Unified Data Space Floor. We may explore this space through querying. For example, we may observe the spectrum of semantics of the sign Adam by issuing a query that asks, ‘What is Adam?’ The result set will include all the concepts associated with the sign Adam
across all sources (i.e. ChemistName and Chemist). Note that this simple yet penetrating question cannot be answered by any traditional data integration solution.

Another simple but useful question that traditional data integration solutions cannot answer is: ‘Which data elements (i.e. signs)
in source B also appear in source C?’
The result is: E1001, E2119, and E3327. By looking at the range of concepts associated with this result set, one may glean useful insight for data-model harmonization. For example, we find that E1001 is associated with the concept InventoryID in source B and EquipCode in source C. An analyst might suspect therefore, that that the two concepts are the same, and if confirmed, assert this equivalence at the data-model level. Thus insight obtained by the analysis of data instances may be applied more broadly as knowledge at the data-model level. This is but one example of how Layer 2 can inform Layer 3.

By chaining such queries we can explore semantic associations and traverse unified data space floor. For example, we may ask:

1. Query: What terms are associated with the sign L1?
   

   Result: [E1001, EquipCode], [E3327, EquipCode]
   

   Analyst thinks: ‘This stuff is located in the same lab.’
   

2. Query: What other concepts are associated with signs E1001 and E3327?
   

   Result: InventoryID (from source B)
   

   Analyst thinks: ‘I wonder if EquipCode is the same thing as InventoryID.’
   

3. Query: Which signs of EquipCode match signs of InventoryID?
   

Result: E1001, E2119, E3327

Analyst thinks: ‘The concepts EquipCode and InventoryID probably do mean the same thing.’

1. Query: What other concepts are associated with InventoryID?
   

Result: Chemist

1. Query: Which Chemists are associated with [E1001,InventoryID] and [E3327,InventoryID]?
   

Result: [Adam, Chemist], [Mary, Chemist]

Analyst thinks: ‘Adam and Mary have equipment in the same lab, so they probably know each other.’

These queries illustrate the ability to perform “semantic drilling” into the DDF data space. We can ask series of questions that “surf” across the entire DDF data space unimpeded by barriers between source systems. One need not have specific semantic knowledge of the source systems in order to explore the data space this way and to extract useful insight. In the next section we will illustrate how this insight may be subsequently inserted back into the data space, as additional information and knowledge, to produce further semantic enrichment and fusion.

Enhancing the Data Space

Up to this point, we have discussed the data integration and analytic power of the unified data space floor that results simply by the mechanical loading of data into Layer 2. The breadth of integration, depth of semantic enrichment, and analytic power can all be dramatically improved by building upon this floor, either by an analyst or an automated process. This can be performed at the data instance level (Level 2), the data-model level (Level 3), or the combination of the two. The first regards the assertion of new instances of DDF elements (i.e. signs, terms, concepts, predicates, and statements). The second regards the enhancement and or harmonization of source specific data-models. The third regards the association of concepts and predicates asserted in Level 2 with existing knowledge models in Level 3.

For example, as is illustrated in Fig. 5, we may introduce the predicate isEquivalent and use it to assert the statement that [Ben, ChemistName] isEquivalent [Benjamin, Chemist]. Such statements, created at the data instance level, represent data integration. In addition, we may assert new associations at the data-model level to achieve global data-model integration (e.g. harmonization). This is illustrated in Fig. 6 wherein, concept ChemistName is asserted to be the same as concept Chemist. The result of this assertion is that the meaning of all ChemistName terms becomes sameAs the meaning of all Chemist
terms.

Exploring the Enriched Data Space

As we explore the enriched data space, surfing semantics and drilling associations, we find that previously disjoint regions of the space become reachable via the newly asserted data and associations. For example, having equated the concept ChemistName with Chemist, and InventoryID with EquipCode, an analyst can simply retrieve the projects that are located in a particular lab with basically one query.

Query: Which terms are associated with [L1, lab]?

Result: [E1001, EquipCode], [Adam, Chemist], [P1, Project]

Fig. 6 shows how the asserted associations (dashed) at the data-model level enable additional associations (dotted) to be inferred. This interplay of data and data-model integration is what ultimately allows us to “connect the dots.”

Application to SIMA

To enable the rapid, ad-hoc assimilation of diverse data into situational views useful for SIMA, we must overcome system, structural, and semantic barriers between data sourced from different systems. As illustrated in Fig. 7, traditional data integration approaches attempt to achieve this by imposing a tight commitment to a particular data-model or integration schema (i.e. canonical data-model). Unfortunately, choosing which of the source data element to expose and mapping them to the canonical model inevitably leads to information loss, and or distortion, and the integration schema itself creates yet another semantic barrier.

In contrast, the DDF breaks the barriers between data sources to accommodate all within a single coherent data space. Simply loading data into the DDF in a largely automated fashion produces a fundamental level of data unity – the Unified Data Space Floor. No data-model harmonization need be made and yet non-trivial data integration results. Upon this floor, the DDF supports the construction of deeper integration and semantic enrichment at both the data instance and data-model levels without prescribing or constraining the processing by which such enrichment may be achieved. Any fusion or data integration method can be applied alone or in combination. Moreover, unlike other integration approaches, new data and associations, regardless of their origin, join seamlessly into the unified data space.

The DDF data space also supports the complete spectrum of applications and clients, from generic (i.e. those operating at the level of the DDF structure) to specific (i.e. those that have knowledge of a particular source data-model). Generic clients seamlessly span across the entire data space regardless of data source or associated data-model to perform analysis. Such clients require no modification as new data or semantics are introduced. Specific clients are able to operate with the same semantic depth in the DDF data space as they would on the source system itself since the DDF data space preserves the data and semantics of the source systems. In other words, the expressiveness and search capability native to those systems are retained [Yoakum 2008 JDIQ]. As the data space is increasingly enriched with semantics that bridge data-models, the depth of specific clients is retained while their breadth increasingly widens toward that of a generic client.

Conclusion

Successfully executing the constellation of activities that comprise SIMA, particularly in support of decision-making, requires exploiting information within a dynamic, heterogeneous, and distributed data environment that is largely beyond our control. The challenge therefore, is to dynamically integrate data, information, and knowledge into one coherent intelligence repository to serve as a foundation for SIMA processes and operations. Current practice is insufficient in the face of scale and complexity.

The approach presented in this paper overcomes the shortcomings of traditional data integration approaches using a framework, called the Data Description Framework, which enables the seamless integration of any structured data within and across data sources and models without the loss or distortion of data and semantics. Moreover, the framework supports a practical, stable implementation using any standard database system.

The simple, mechanical loading of source data and semantics into the DDF creates a unified data space floor that exhibits a primary level of data integration unmatched by traditional integration approaches. No up-front, heavy investment in data-model harmonization is required – one simply pours data on the floor. Deeper integration and semantic enrichment may then be pursued with any manual or automated processing operating either at the data instance or data-model levels.

The ultimate analytic power that is enabled by the DDF data space is essentially unlimited and exceeds that of any particular source system or traditional data integration solution at any level. Having the power and flexibility required to organize the transient and complex SIMA data space, it provides the ideal foundation on which to pursue SIMA.

Acknowledgements

The authors would like to thank the following US Army CERDEC I2WD personnel for their continued support: Mr. Anthony Lisuzzo, Director, Mr. Kesny Parent, DCGS-A Branch Chief, Ms. Virginia Goon IXFL Manager, and Mr. Norbert Antunes IXFL Computer Engineer. This work was funded by US Army CERDEC I2WD under contract number W15P7T-06-D-A401/009.

References

[Batini 1986] Batini, C. et al. A comparative analysis of methodologies for database schema integration, ACM Computing Surveys, (18) 4, 1986.

[Bernstein 2007] Bernstein P., Ho, H. Model Management and Schema Mappings: Theory and Practice, Proceedings of VLDB Conference, 2007.

[Halevy 2005] Halevy, A. et al. Enterprise information integration: successes, challenges and controversies, Proceedings of 24th International Conference on Management of Data, Baltimore, 2005.

[Northrop 2006] Northrop, L., et al., Ultra-Large-Scale Systems The Software Challenge of the Future, Pittsburgh: Carnegie Mellon University, 2007. http://www.sei.cmu.edu/publications/books/engineering/uls.html

[Parent 1998] Parent, C. and Spaccapietra, S. Issues and approaches of database integration, Communications of the ACM, 41(5), 1998.

[Yoakum 2008 DAMA] Yoakum-Stover, S. and Malyuta, T. Unified Integration Architecture for Intelligence Data, DAMA International Europe Conference 2008, November 2008, London, UK.

[Yoakum 2008 JDIQ] Yoakum-Stover, S. and Malyuta, T. Unified Architecture for Integrating Intelligence Data, ACM Journal of Data and Information Quality. September 2008. Pending decision.