In fact regionalization via Postleitzahl is handled purely on the client side via a cookie which is evaluated in Javascript. This cookie contains only information about the address of the store and no other personalized information. It is removed in vcl_recv.

Regionalization of the product line and product attributes is handled differently by a separate cookie which contains the region of the user (there are only a few). This cookie is used by Varnish to cache separate versions of the page.

These are quite low spec Hadoop servers. Our main bottleneck is the RAM. So we are in the process of updating the cluster from 8 GB to 16 GB.

May i know the node preference you are using i.e, about the ram and disk space and any other spec.

I am afraid I can not give you a good answer to your questions. Keeping an index of 30 billion documents in Solr is definitely a challange. I guess you will need to use something like Solr Cloud. It might or might not work out. This really depends on your index schema.

Only thing I can answer is, that Solr is able to index 10 million documents per hour. Our highest peak reached 26 million documents in one hour in a single index. Although the query time was really bad during this peak.

I would really recommend building a prototype with a single Solr index and try to find the limits of a single index for your indexing schema and data. From there you can try to scale through multiple index shards.

Regards,
Peter

Great Post ! Thank you.
I am looking to build an index for transactional data (Carts). The source data is in multiple Oracle DBs and, at peak, will get created at upto 10 million rows per hour.

I need the index to a) be kept up to date at near real time, b) be able to query based on 10 attributes and return a Key and pointer to the original RDBMS instance and c) keep enough history data to amount to about 30 billion rows.

It looks like Solr is 1 option but am concerned about manageability and the near real time updating. What are your thoughts on using an RDBMS table for the index ? Anything other option I should consider ?

Thank You,
Peter

If on the other hand the index is too large for a single core and you need a kind of sharding you might be able to reuse your cluster also for Lucene / Solr. But first you need to evaluate the use of your Hadoop Cluster. If the Cluster is also heavily used for Map/Reduce-Jobs, you will not have enough resources for Solr.

Bottom line: If your Hadoop cluster is primarily used for storage and has only a light Map/Reduce load, you can reuse it for running Solr. In all other cases you are better off with a seperate Solr Cluster.

yes, we have evaluated Katta when we started research on this project. At that time we could not get it to run stable. Please keep in mind, that this was at the beginning of 2009. We have filed bug reports for some of the issues we had and they have been fixed, but we decided at that time, that we feel more comfortable with our own solution.

Best regards,
Peter

What is the best way to setup Luence / Solr in a Hadoop env.

1. Should the Luence / Solr have their own servers and storage?
2. Is it best to have Luence / Solr implemented in the same Hadoop cluster — machines in the same rack with the hadoop nodes

Do you have any exemplar architecture diagrams showing a logical or a physical implementation of Luence/Solr with Hadoop

Thanks

cheers
tiru