Can we use ElasticSearch or SOLR as a primary database

2014-08-15T00:31:00.004-07:00

Enterprise applications esp. OLTP apps, do not generally use ElasticSearch or SOLR as a primary database but only as a "Search Index datastore" for Fast-Search/Read-Only operations. The intent of using it as a datastore is probably to demonstrate the fast search architecture and mechanism to do so.

If the web application is a use case like data aggregation, analytics, log analysis etc. using ElasticSearch or SOLR as Datastore makes sense.

Having said this - Yes, in general ElasticSearch or SOLR can be used as a database. But some things to consider before doing so are -

ElasticSearch or SOLR's most common data access pattern is over http. Its not particularly great for batch querying. Furthermore, ElasticSearch or SOLR does NOT stream data. This means we have to be very thoughtful when we design large scale data access patterns with ElasticSearch or SOLR. Pagination and progressive rendering kind of use cases, needs to be designed carefully, so we don't run into performance issues. Though there is good support for such use cases in ElasticSearch or SOLR.
Although SOLR scales horizontally (more machines, more cores, etc..) as well as vertically (more RAM, better machines, etc), its querying capabilities are severely limited compared to those of a mature RDBMS. Though, there are some excellent functions, like the field stats queries, which are quite convenient.
Developers who are used to using relational databases will often have a steep learning curve for developing the right approach to building an application that uses ElasticSearch or SOLR for its large queries or state-full modifications.
Enterprise level APM (Application Performance Monitoring) tools that allow for advanced session management and database monitoring are not useful in such cases.However ElasticSearch has its own monitoring tool -Marvel (developer license is free but need to buy Production License ..Approx $500 per year).
Relational databases are meant to deal with complex data and relationships - and are thus accompanied by state of the art metrics and automated analysis tools. Tools for ElasticSearch or SOLR are still evolving and not yet mature.
Joins: Relational databases support methods for building and optimizing views and queries that join tuples based on simple predicates. In ElasticSearch or SOLR, there aren't any robust methods for joining data across indices. This can be done using application level joins or new concepts in ES like Parent-Child relationships which are not quite sophisticated and might not work for all kinds of join scenarios.
Analytical Functions: Though ElasticSearch or SOLR have in-built analytical functions, they are not as rich as a RDBMS analytical functions library.
Inbuilt Data Security and Authentication Mechanisms : For a RDBMS schema, there are users-roles and previlages. However, a index in ES or Solr does not have any such concept supported yet. Data security is more driven by OS level security features (since its ultimately file system based ).

That said - there are plenty of obvious advantages to SOLR for certain tasks - Schemaless design, loose queries are much easier to run and return meaningful results.
Indexing is done as a matter of default, so most arbitrary queries run pretty effectively (unlike a RDBMS, where you often have to optimize and de-normalize after the fact).

Conclusion: Even though we can use SOLR as an database, we may find that, there is ultimately "no free lunch" - and the cost savings of super-cool lucene text-searches and high-performance, in-memory indexing, are often paid for by less flexibility and adoption of new data access work-flows.

log4j 2 ... new and improved ! Still beta

2013-12-25T21:08:00.002-08:00

Apache Log4j 2 is an upgrade to Log4j that provides significant improvements over its predecessor, Log4j 1.x, and provides many of the improvements available in Logback while fixing some inherent problems in Logback's architecture. Asynchronous Loggers have 10 times higher throughput and orders of magnitude lower latency than Log4j 1.x and Logback. See Asynchronous Logging Performance for details

Samant

Can we use ElasticSearch or SOLR as a primary database

log4j 2 ... new and improved ! Still beta