JERWIN ROY'S BLOG

Cassandra Data Distribution using Partitioners

2018-11-26T13:07:00.002+05:30

Before wading through the partitioners lets have a look how generally data is being distributed in a Cassandra
cluster. It includes a ring like a topology between the nodes. Here the data are being broken into tokens and are circulated
among the nodes through the Cassandra ring. Data is distributed across the cluster by the value of the token by using the
hash technique. Based on the value of the tokens the data will be evenly distributed among the cluster. The main advantage
of this method is the data retrieval process can be quick based on the token range. Now we can have can see what does a
partitioner role in a cluster in detail.

Partitioners

Partitioners allow how row keys should be sorted and how data will be distributed across your nodes in the cluster. The
read/write request to the cluster are evenly distributed when each part of the hash range receives the same number of
tokens on average. Based on the difference in the hash methods partitioners are classified into three types as below:

Murmur3Partitioner
RandomPartitioner
Byte Ordered Partitioner

Murmur3 Partitioner

This is the default partitioning strategy for Cassandra. It provides fast hashing and good performance. It
uses MurmurHash hash values to distribute the data across the clusters.

Random Partitioner

It involves MD5 hash applied to place the keys on the node ring. An MD5 hash provides a natural way of
load balancing keys to nodes. Each data item is mapped to a token by calculating the MD5 hash of its key
The disadvantage of this method is that it causes inefficient range queries when keys specified in the range
might be in another ring.

Byte Ordered Partitioner

This method involves the distribution of the data lexically by key bytes in an ordered manner. It treats
the data as raw bytes, instead of converting them to strings. This is most likely to use when you want a
partitioner that doesn't want to validate the keys as being strings.

How to change a Partitioner?

So based on the need for the application you can choose any one of the above partitioners. In order
to apply it make the below change in the cassandra.yaml file. This file will be generally located in conf
directory.

So this is all about how a data is distributed among the Cassandra cluster and how partitioner helps more
for the sorting of data & retrieving them. Got any queries about partitioners to comment on them.

MongoDB Storage Engines

2018-11-01T13:09:00.001+05:30

The storage engine is a vital component for managing and storing data in the memory as well as disk. MongoDB supports multiple storage engines with unique features for better performance. In this blog, we are going to discuss the various storage engines and their features.

Types of Storage Engine:

Our production workload will be different for each application, some will be writing intensive, some will be read and some required encryption etc. MongoDB provides flexibility to handle such workloads by providing multiple storage engines. Mentioned the storage engines below.

Wired Tiger
MMAPv1
Encrypted
In-memory

Let's see the key features of each storage engines.

Wired Tiger:

Wired Tiger (WT) is the default storage engine from mongo 3.0

WT storage engine uses document-level concurrency control for write operations so multiple clients can modify different documents of a collection at the same time.
It uses only intent locks at the global, database and collection levels when the storage engine detects conflicts between two operations, one will incur a write conflict causing MongoDB to transparently retry that operation
MongoDB utilizes both the WiredTiger internal cache and the filesystem cache. By default the wired tiger cache will use 50% of RAM minus 1 GB or 256 MB.
Efficient use of CPU cores and RAM
Allows for more tuning of storage engine than MMAP
7 to 10X better write performance
80% less storage with compression
Compression minimizes storage use at the expense of additional CPU.

Collection level data in the WiredTiger internal cache is uncompressed and uses a different representation from the on-disk format.

MMAP:

The MMAP Storage engine uses memory mapped files to store its data
A segment of virtual memory which has been assigned a direct byte-for-byte correlation with some portion of a file
It is a traditional storage engine that allow great deal of performance for heavy read applications
Data and indexes are mapped into virtual space
Data access is placed into RAM
When the OS runs out of RAM and an application requests for memory,then it will swap out memory to disk to make space for the newly requested data
The operating system’s virtual memory subsystem manages MongoDB’s memory
Deployments with enough memory to fit the application’s working data set in RAM will achieve the best performance.

Encrypted:

Available in mongodb enterprise only.
The default encryption mode that MongoDB Enterprise uses is the AES256-CBC
All data files are fully encrypted from a file system
Unencrypted state in memory and during transmission
Master keys and database keys are used for encryption
Data is encrypted with the database keys,master key encrypts the database keys
Encryption is not a part in replication keys are not replicated
In replication data is not natively encrypted over the wire
Application Level Encryption provides encryption on a per-field or per-document basis within the application layer

In-memory:

It is available in the enterprise editions starting from version 3.2.6.
Handles ultra high throughput with low latency and high availability

In-memory storage engine is part of general availability

More predictable and low latency on less in-memory infrastructure
Supports high level infrastructure based on zonal sharding
MongodB rich query capability and indexing support

Third-party pluggable storage engines:

MongoDB is providing support for 3rd party storage engines as “modules” that can be independently updated.
When building MongoDB, any storage engine modules will be automatically detected, configured and integrated in the final binaries.
The RocksDB storage engine is the first one to use this new module system for their MongoDB storage integration layer

RocksDB for MongoDB is based on the key-value store optimized for fast storage.
It is developed by facebook and designed to handle write-intensive workloads.

Storage Engine application API:

As mentioned, each application load will be different from other. Choosing the right storage engine will definetly boost the performance. Differentiated storage engine with respect to the workload which helps in choosing the right storage engine.

Comparison chart:

The overall feature comparison for all the storage engines are listed below:

MongoDB Free monitoring on Community version 4.0 explained

2018-11-01T12:51:00.002+05:30

I was just playing around with MongoDB 4.0 community version & noticed a feature called free monitoring. In this article let have a detailed look of free monitoring, how to enable, check status and what are the monitoring metrics that are provided by MongoDB.

Free Monitoring:

Free monitoring is a service that is available in the MongoDB 4.0 Community versions. It provides us with some graphical/statistical data related to currently deployed instances which expire after 24 hours. So let's have a look how can we enable free monitoring & look at what is in it.

Enabling Monitoring:

This command enabled the free monitoring in the instance, one-time enabling is enough it is not necessary to enable each time to check the status.

db.enableFreeMonitoring()

Check Monitoring status:

In order to check the current free monitoring status the above command which let us know the status. When once enabled, the monitored data is uploaded periodically to the cloud & it can be accessed via browsers through the provided URL.

db.getFreeMonitoringStatus()

Graph options:

Let's have a look over the various graph parameters that are included in the free monitoring.

Operation Execution:

This lists out the how many times the operations are being executed in the server. Operations include reads, writes & commands.

Disk Utilization:

Every disk has its own read & write speed. Disk utilization involves the sum of the total speed of data that is being written to and read from the disk. This graph plots out the maximum & average disk used by the drives.

Documents:

This returns the stats of the documents that are returned, inserted, updated & deleted

Memory:

MongoDB uses virtual as well as the resident memory. Resident memory is the memory that is used by RAM. In some cases, the operating system returns the imaginary address when MongoDB requests for memory address which is not real. If the journal is enabled then it returns another address for journal data, both together is the Virtual memory. Journal reference memory alone is known as the mapped memory.

Network IO:

This refers to the total network traffic that is being received and sent by MongoDB in bytes.

Opcounters:

Opcounters are nothing but the total list of the operations count that is being performed by the server. It included the operations such as insert, query, update, delete,getmore, commands.

Replicated Opcounters:

This refers the opcounters that are being replicated to the other secondaries in the replica cluster. The value of replicated opcounters will be present only if the instance is a replica set.

Query Targeting:

This chart depicts the query & objects that are scanned by the process.

Queue:

This parameter lists out the total number of reads,writes that are currently waiting under the queue or waiting for any other lock.

System CPU usage:

It explains about the complete system CPU usage with respect to the some of the specialized factors.

Consistency levels in Apache Cassandra explained

2018-11-01T12:51:00.001+05:30

Cassandra is scalable column-oriented open source NoSQL database. It is the right choice for managing large amounts of structured, semi-structured, and unstructured data across multiple data centers when you need scalability and high availability without compromising performance. In this article, we are going to discuss how the read/write operations are maintained in a cluster and various consistency levels in Cassandra & how can they be applied to our business applications.

According to CAP theorem, it is impossible for a distributed system to simultaneously provide all three guarantees:

Consistency -Every node contains same data at the same time
Availability- At least one node must be available to serve data every time
Partition tolerance -Failure of the system is very rare

"Cassandra is typically classified as an AP system, meaning that availability and partition tolerance are generally considered to be more important than consistency in Cassandra. But Cassandra can be tuned with replication factor and consistency level to also meet C.So Cassandra is eventually consistent."

Replication factor:

Before deep diving into the consistency levels its necessary to understand the term replication factor. It describes how many copies of your data exist. Based on the RF & the consistency levels it is easy to design a very good stable architecture in Cassandra.

The below terms explains how the write/read transactions serve its purpose:

Commit log − The commit log is a crash-recovery mechanism in Cassandra. Every write operation is written to the commit log.

Mem-table − A mem-table is a memory-resident data structure. After commit log, the data will be written to the mem-table. Sometimes, for a single-column family, there will be multiple mem-tables.

SSTable − It is a disk file to which the data is flushed from the mem-table when its contents reach a threshold value

Write path in Cassandra:

When a write is initiated its first captured by the commit logs. Later the data will be captured and stored in the mem-table. Whenever the mem-table is full, data will be written into the SStable data file. All writes are automatically partitioned and replicated throughout the cluster. Cassandra periodically consolidates the SSTables, discarding unnecessary data.

Read path in Cassandra:

For any read operations first, the values are fetched from the mem table and then Cassandra checks the bloom filter(cache) to find the appropriate SSTable that holds the required data.

Consistency Levels :

Consistency levels are used to manage the data consistency versus data availability. Below are the various levels of consistency that can be set to achieve the data consistency in the DB:

ALL- Writes/Reads must be written to the commit log and memtable on all in the cluster.

EACH_QUORUM- Writes/Reads must be written to the commit log and memtable on each quorum of nodes. Quorum is 51% of the nodes in a cluster.

QUORUM- Writes/Reads must be written to the commit log and memtable on a quorum of nodes across all data centers.

LOCAL_QUORUM- Writes/Reads must be written to the commit log and memtable on a quorum of nodes in the same datacenter as the coordinator.

ONE- Writes must/Reads be written to the commit log and memtable of at least one node.

TWO- Writes/Reads must be written to the commit log and memtable of at least two nodes.

THREE- Writes/Reads must be written to the commit log and memtable of at least three nodes.

LOCAL_ONE- Writes/Reads must be sent to and successfully acknowledged by, at least one node in the local datacenter.

ANY- Writes/Reads must be written to at least one node.

How to calculate the DB impact based on these parameters?

Its very easy to calculate the DB impacts for any given RF & read, write Consistency levels. For example, say let us set up a 5 node cluster with 3 RF, Read & Write Consistency level as quorum then the impact would be as below:

1. Your reads are consistent

2. You can survive the loss of 1 node without impacting the application.

3. You can survive the loss of 1 node without data loss.

4. You are really reading from 2 nodes every time.

5. You are really writing to 2 nodes every time.

6. Each node holds 60% of your data.

The same cluster scenario with Read & Write Consistency level as ONE will have the below impact.

1. Your reads are eventually consistent

2. You can survive the loss of 2 nodes without impacting the application.

3. You can survive the loss of no nodes without data loss.

4. You are really reading from 1 node every time.

5. You are really writing to 1 node every time.

6. Each node holds 60% of your data.

Thus the Cassandra cluster architecture can be defined according to our own business need with the optimal use of the resources to yield high performance.

Credits: You can use this Cassandra Parameters for Dummies to find out the impact: https://www.ecyrd.com/cassandracalculator/

Apache solr replication step by step

2018-02-15T12:18:00.002+05:30

Need for Replication :

When there is a large search volume that cannot be handled by a single machine, so you need to distribute searches across multiple read-only copies of the index.
If there is a high volume of indexing which consumes machine resources and reduces search performance on the indexing machine, so you need to separate indexing and searching.
When we want to make a backup of the index

MASTER-SLAVE

Distributes complete copies of a master index to one or more slave servers.
The master server continues to manage updates to the index.
All querying is handled by the slaves.
This enables Solr to scale to provide adequate responsiveness to queries against large search volumes.

Replication Terminology:

Index

A Lucene index is a directory of files. These files make up the searchable and returnable data of a Solr Core.

Distribution

The copying of an index from the master server to all slaves.

Inserts and Deletes

As inserts and deletes occur in the index, the directory remains unchanged. Documents are always inserted into newly created files.
Documents that are deleted are not removed from the files.
They are flagged in the file, deletable and are not removed from the files until the index is optimized.

Master and Slave

A Solr replication master is a single node which receives all updates initially and keeps everything organized.
Solr replication slave nodes receive no updates directly, instead all changes (such as inserts, updates, deletes, etc.) are made against the single master node.
Changes made on the master are distributed to all the slave nodes which service all query requests from the clients.

Repeater

A node that acts as both a master and a slave.

Optimization

A process that compacts the index and merges segments in order to improve query performance.
Optimization should only be run on the master nodes. An optimized index may give query performance gains compared to an index that has become fragmented over a period of time with many updates.
Distributing an optimized index requires a much longer time than the distribution of new segments to an un-optimized index

Snapshot

A directory containing hard links to the data files of an index. Snapshots are distributed from the master nodes when the slaves pull them, "smart copying" any segments the slave node does not have in snapshot directory that contains the hard links to the most recent index data files.

Configuring the Replication RequestHandler on a Master Server:

commit-Triggers replication whenever a commit is performed on the master index.

optimize-Triggers replication whenever the master index is optimized.

startup-Triggers replication whenever the master index starts up.

MASTER SLAVE CONFIGURATION

MASTER

The configuration of the master are configured in the below file.

vim /var/solr/data/fortis/conf/solrconfig.xml

vim /var/solr/data/fortis/core.properties

Make the below changes in the core.properties file

enable.master=true

enable.slave=false

SLAVE

To create a new collection use the below command

/usr/local/solr-6.6.2/bin/solr create -c prod

The configuration of the master are configured in the below file.

vim /var/solr/data/fortis/conf/solrconfig.xml

Make the below changes in the core.properties file

vim /var/solr/data/fortis/core.properties

enable.master=false

enable.slave=true

Now we have setup the master-slave replication in the apache solr.

Apache SOLR introduction & installation

2017-11-27T18:30:00.002+05:30

In this post, we are going to discuss about Apache Solr and how to install them step by step.

Why Solr?

Lucene:

Search storage engine
Solr uses concepts derived from Lucene
Lucene is widely used in many projects among one of them is solr
Used to index & search with high performance.
Solr uses lucene as its backend

Solr:

Search server
Document oriented
Stores data & indexing
Searches including full-text search, stemming,hit-highlighting, faceted-search etc that cannot done by native databases
Vertically and horizontally scalable
Replication for high availability
Sharding for distributed search
Performs in memory,grouping,counting,similar products in single shot
Exposed over HTTP,REST like api.
The DataImportHandler provides a configuration driven way to import data from relational databases or XML files, into Solr in both “full import” and “incremental delta import” mode.

INVERTED INDEX:

Searches document by unique word
Similar to the index back of back

D1- I like Apache services.
D2- They include all kind of database & services support.
D3- I would recommend their services to my clients too.

Working:

Define a schema.
Deploy Solr.
Feed Solr documents for which your users will search.
Expose search functionality in your application.

Solr schema:

No schema
Has index that contain docs
Fields are used to index,search & store

Define a schema

The schema tells Solr about the contents of documents it will be indexing. In the online store example, the schema would define fields for the product name, description, price, manufacturer, and so on.

Defining fields:

Indexing:

Indexing is a technique of adding Document’s content to Solr Index so that we can search them easily. Apache Solr uses Apache Lucene Inverted Index technique to Index it’s documents. That’s why Solr provides very fast searching feature.

Field analyzers: (Analyzer=tokenizer+filters)

Used both during ingestion, when a document is indexed, and at query time.Analyzers may be a single class or series of tokenizer and filter classes.

Alternative words-finish,complete

misspelled-google,gogle

Tokenizers:

Break field data into lexical units, or tokens.

Pre- Stripping html tags

Post-Stemming(replace) tables as collection

Stop word filter (the,is,and)

Filters:(used for indexing)

Examine a stream of tokens and keep them, transform or discard them, or create new ones.
Tokenizers and filters may be combined to form pipelines, or chains, where the output of one is input to the next.
Such a sequence of tokenizers and filters is called an analyzer and the resulting output of an analyzer is used to match query results or build indices.

eg)ram,RAM,Ram

Deploying Solr

Pre requisites:

yum update

yum install java-1.8.0-openjdk.x86_64

java -version

Installation:

wget http://apache.org/dist/lucene/solr/6.6.1/solr-6.6.1.tgz

tar zxvf solr-6.6.1.tgz
cp /opt/solr-6.6.1/bin/install_solr_service.sh .
rm -rf solr-6.6.1
./install_solr_service.sh solr-6.6.1.tgz
ps -ef | grep solr

Feed Solr documents for which your users will search Creating a project(core): A Core is an Index of texts and fields available in all documents. One Solr Instance can contain one or more Solr Cores. /opt/solr-6.6.1/bin/solr create -c jerwin Create new document : We can create a new document to the core with 3 fields and value using the below command on the terminal using curl: curl http://localhost:8983/solr/jerwin/update -d ' [ {"id" : "db1", "company_name" : "Mafiree", "location" : "Nagercoil" } ]'

View:

The inserted data can be viewed with the id specified.

curl http://localhost:8983/solr/jerwin/get?id=db1

Output can be viewed in the browser:

Hope this gives you simple introduction about SOLR,let me know if you have any concerns via comments.

Explaining PMM server by Percona

2017-11-27T18:15:00.003+05:30

Percona Monitoring and Management

PMM Client

It is installed on every database host that you want to monitor. It collects server metrics, general system metrics, and query analytics data for a complete performance overview. Collected data is sent to the PMM Server.

PMM Server

It is the central part of PMM that aggregates collected data and presents it in the form of tables, dashboards, and graphs in a web interface.

PMM Client

PMM Client packages consist of the following:

pmm-admin is a command-line tool for managing PMM Client, for example, adding and removing database instances that you want to monitor.

percona-qan-agent is a service that manages the Query Analytics (QAN) agent as it collects query performance data. It also connects with QAN API in PMM Server and sends over collected data.

node_exporter is a Prometheus exporter that collects general system metrics

mysqld_exporter is a Prometheus exporter that collects MySQL server metrics.

mongodb_exporter is a Prometheus exporter that collects MongoDB server metrics.

proxysql_exporter is a Prometheus exporter that collects ProxySQL performance metrics.

PMM Server

PMM Server consists of the following tools:

Query Analytics (QAN) enables you to analyze MySQL query performance over periods of time. In addition to the client-side QAN agent, it includes the following:

QAN API is the backend for storing and accessing query data collected by percona-qan-agent running on a PMM Client.

QAN Web App is a web application for visualizing collected Query Analytics data.

Metrics Monitor (MM) provides a historical view of metrics that are critical to a MySQL or MongoDB server instance.

The summary table contains top 10 queries ranked by %GTT (percent of grand total time)

Query details-mysql

Performance schema-mysql

Prometheus is a third-party time-series database that connects to exporters running on a PMM Client and aggregates collected metrics.

Consul provides an API that a PMM Client can use to remotely list, add, and remove hosts for Prometheus. It also stores monitoring metadata.

Grafana is a third-party dashboard and graph builder for visualizing data aggregated by Prometheus.

Percona Dashboards is a set of dashboards for Grafana developed by Percona.

Orchestrator is a MySQL replication topology(NODE) management and visualization tool.

All tools can be accessed from the PMM Server web interface (landing page).Special users must be created for topology generation.

PMM server running methods:

Run PMM Server using Docker

Run PMM Server as a virtual appliance(VM ware,VirtualBox)

Run PMM Server using Amazon Machine Image (AMI)

Docker

Docker is a tool designed to make it easier to create, deploy, and run applications by using containers.

It extracts all the needed packages from https://github.com/docker.

Running PMM Server Using Docker

Step 1. Create a PMM Data Container

docker create \
  -v /opt/prometheus/data \
  -v /opt/consul-data \
  -v /var/lib/mysql \
  -v /var/lib/grafana \
  --name pmm-data \
  percona/pmm-server:1.1.5 /bin/true

docker create-instructs the Docker daemon to create a container from an image.

v- initialize data volumes for the container.

name-assigns a custom name for the container

percona/pmm-server:1.1.5 is the name and version tag of the image to derive the container from.

Step 2. Create and Run the PMM Server Container

docker run -d \
  -p 80:80 \
  --volumes-from pmm-data \
  --name pmm-server \
  --restart always \
  percona/pmm-server:1.1.5

run-instructs the docker daemon to run a container from an image.

d-starts the container in detached mode(bg)

p-port

volumes from-mounts volumes from the pmm-data container

restart-defines the container’s restart policy

Verifying PMM Server

Checkout access the PMM web interface using the IP address of the host where the container is running.

Connecting PMM Client to PMM Server

To connect the client to PMM Server, specify the IP address using the pmm-admin config --server command

$ sudo pmm-admin config --server 192.168.100.1
OK, PMM server is alive.

PMM Server | 192.168.100.1
Client Name | ubuntu-amd64

Client Address | 192.168.200.1

Starting Data Collection

After you connect the client to PMM Server, enable data collection from the database instance by adding a monitoring service.

To enable general system metrics, MySQL metrics, and query analytics, run:

sudo pmm-admin add mysql

To enable general system metrics and MongoDB metrics, run:

sudo pmm-admin add mongodb

To see what is being monitored, run:

$ sudo pmm-admin list

Upgrading PMM Server

Stop the server

$ docker stop pmm-server

Remove the old server

$ docker rm pmm-server

Run the new one

$ docker run -d \
  -p 80:80 \
  --volumes-from pmm-data \
  --name pmm-server \
  --restart always \
  percona/pmm-server:1.1.5

Upgrading PMM Client

For Debian or Ubuntu:

$ sudo apt-get update && sudo apt-get install pmm-client

For RedHat or CentOS:

$ yum update pmm-client

Removing PMM Server

Stop and remove the pmm-server container:

$ docker stop pmm-server && docker rm pmm-server

If you also want to discard all collected data, remove the pmm-data container:

$ docker rm pmm-data

Removing PMM Client

If you installed using RPM packages:

$ rpm -e pmm-client

If you installed using YUM:

$ yum remove pmm-client

If you installed using DEB packages:

$ dpkg -r pmm-client

If you installed using APT:

$ apt-get remove pmm-client

Dashboards:

To View current running operations in MongoDB

2015-07-08T17:44:00.002+05:30

Every database includes multiple currentoperations running in the server. In this blog, we shall discuss the process list in MongoDB.

MongoDB is used to display all the documents that contains information on in-progress operations for the database instance which can be done using db.currentop() operation.If you pass a query document to db.currentOp(), the output returns information only for the current operations that match the query.In this post we are going to see the various output that are return by db.currentop().

To view the current active queries in the database:

db.currentOp(
      {
        "active" : true
    }
    )

To view the queries that are running more than ‘x’ seconds in the database:

db.currentOp(
      {
        "active" : true,
        "secs_running" : { "$gt" : 10}
      }
    )

To view the queries that are waiting for a lock and not a read:

db.currentOp().inprog.forEach(
      function(d){
        if(d.waitingForLock && d.lockType != "read")
          printjson(d)
        })

To view all active read queries:

db.currentOp().inprog.forEach(
      function(d){
        if(d.active && d.lockType == "read")
          printjson(d)
        })

To view all active write queries:

db.currentOp().inprog.forEach(
      function(d){
        if(d.waitingForLock && d.lockType != "write")
           printjson(d)
        })

Kill the queries:

Incase if some of the queries that are running for more seconds and are degrading the database performance you can kill the using killOp() operation.The necessary parameter to kill is the opid(operation id) that will be returned by the currentOp().To kill a query with the opid 3258220 issue the below command.

db.killOp(3258220)

So once done the query that has been mentioned will be killed which you can check again using the db.currentOp().

User creation in MongoDB

2015-07-08T14:01:00.003+05:30

Security is a foremost criteria in the databases. Proper authentication in the databases must be enabled to ensure high security.Users in the database paves a way for the better authentication in databases.In this post we are going to see how to create the users once you are done with the MongoDb installation.Its necesary to create the required range of the users.Each users have their own roles that they can perform.The following are the some users that are very much necessary in the mongodb databases .

Administrator user:

This user can grant themselves privileges in excess of their current privileges and even can grant themselves all privileges, even though the role does not explicitly authorize privileges beyond user administration.It is also a typical superuser in the database.

use admin

db.createUser(
{
   user: "UserAdmin",
   pwd: "password",
   roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
}
)

Root user:

It provides access to the operations and all the resources of the readWriteAnyDatabase, dbAdminAnyDatabase, userAdminAnyDatabase and clusterAdmin roles combined also this user does not include any access to collections that begin with the system. prefix.

use admin

db.createUser(
{
user: "root",
pwd: "password",
roles: [ "root" ]
}
)

Read/Write user:

This user provides the read and write permissions for a particular database that is specified.

use admin

db.createUser(

  {

    user: "rw_user",

    pwd: "password",

    roles:

      [

        { role: "readWrite",db:"test_db"}

      ]

  }

)

Read user:

This user provides only the read permissions for a particular database that is specified.

use admin

db.createUser(

  {

    user: "read_user",

    pwd: "password",

    roles:

      [

        { role: "read",db:"test_db"}

      ]

  }

)

Write user:

This user provides only the write permissions for a particular database that is specified.

use admin

db.createUser(

  {

    user: "write_user",

    pwd: "password",

    roles:

      [

        { role: "write",db:"test_db"}

      ]

  }

)

Backup User:

This user is used to perform only the backup operations in the database.So in case you have single instance its enough to have the following role:

use admin

db.createUser(

  {

    user: "backup",

    pwd: "password",

    roles:

      [

        { role: "dbAdmin",db:"admin"}

      ]

  }

)

Incase you maintain any shard or replica you can have the below role:

use admin

db.createUser(

  {

    user: "backup",

    pwd: "password",

    roles:

      [

        { role: "clusterdbAdmin",db:"admin"}

      ]

  }

)

Check user privileges:

Its very much necessary to see what are all the privileges that are given to a particular user,to find them use the below query:

db.runCommand(

              {

                usersInfo: { user: "root", db: "admin" },

                showPrivileges: true

              }

)

You can also create your own users similarly mentioned above if you have more ideas about the roles needed.To know more about the mongodb roles check this link.

Install MongoDB using yum on Linux(CentOS)

2015-07-02T12:52:00.000+05:30

MongoDB falls into NoSQL category of database also it is a document database that provides high performance, high availability, and easy scalability.Few of its advantages are high availability and scalability.In this post we are going to see how to install the mongodb using yum on linux.

Step 1:Create a repo file that is necessary to download the mongodb.

vim /etc/yum.repos.d/mongodb.repo

Step 2:Mongodb has two types of packages for 32-bit and 64-bit systems.

For 32-bit systems type the below information in repo file and save.

[mongodb-org-3.0]

name=MongoDB 3.0 Repository

baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/i686/

gpgcheck=0

enabled=1

In the first line of the file specify you own necessary version of the mongodb to be downloaded.Here i have used version 3.0.

For 64-bit systems type the below information in repo file and save.

[mongodb-org-3.0]
name=MongoDB 3.0 Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/
gpgcheck=0
enabled=1

In the first line of the file specify you own necessary version of the mongodb to be downloaded.Here i have used version 3.0.

Note:It’s always better to download 64 bit for production rather than 32 bit because 32-bit MongoDB processes are limited to about 2 gb of data.But the 64 bit doesn't have that limitation. For development both versions can be used.

Step 3:Mongodb has a set of packages as below:

mongodb-org-server-3.0.4           //Mongodb server

mongodb-org-shell-3.0.4            //Mongo Shell

mongodb-org-mongos-3.0.4           //Mongo Shard

mongodb-org-tools-3.0.4            //Mongo Tools

And it contains all the above package together as a single package as below:

mongodb-org-3.0.4

To install all the packages issue the below command:

sudo yum install mongodb-org

To install mongodb server alone issue the below command:

sudo yum install mongodb-org-server-3.0.4

To install mongo shell alone issue the below command:

sudo yum install mongodb-org-shell-3.0.4

To install mongo shard alone issue the below command:

sudo yum install mongodb-org-mongos-3.0.4

To install mongo tools alone issue the below command:

sudo yum install mongodb-org-tools-3.0.4

Step 3:Start the installed mongodb server using the below command:

sudo service mongod start

The server is now started and to login to the mongo shell issue the below command:

mongo

To check the current status of mongod issue the below command:

sudo service mongod status

To stop the running mongod server use the below command:

sudo service mongod stop

It’s finished.We have installed the mongodb and now you can start exploring :)

MySQL to MongoDB Migration in Linux

2015-07-01T18:06:00.000+05:30

In the present scenario due to the influence, Bigdata people started to move on from RDBMS to NoSQL for a better excellence. So its necessary to have a database that is good enough in performance and handling the big data. MongoDB a document-oriented database would fit into it which has high availability and scaling. In this post, we are going to see how to migrate all the data from a MySQL instance to MongoDB instance. It can be performed without downtime and the data loss. The tool used to perform the migration task is the Mongify.

Mongify tool was developed by Andrew Kalek from Anlek Consulting. This tool helps for migrating data from MySQL to mongodb.No need to worry about the IDs or foreign IDs.Allows embedding data into documents, including polymorphic associations. The problem is it can't do anything to an embedded table. Some packages are to be necessary before installing mongify in Linux so install them as below:

yum install gem

gem install bundler

bundle install

gem install json_pure

To install mongify issue the below command:

gem install mongify

Setup:

To perform the migration task it needs some requirements in the system:

There must be MySQL client running as rpm in because the tool listens only for the rpm connection
Also, have a binary or rpm instance of mongodb running in the machine
Make sure that both mysql & MongoDB servers are running

Configuration file:

It involves mainly a two files database.config and ruby file translation.rb

The database.config file contains the credentials of the mysql such as host,username,password,database and for mongodb credentials like host,database
The translation.rb is a ruby file that should contain the details of mysql tables and column values with the proper datatype mentioned

Mongify Check:

It is used to connect the both mysql and mongodb instances & checks whether connection works.It checks the configuration file database.config whether the database credentials present are correct.Once the connection is established it throws a message “SQL connection works NoSQL connection works”.This command can be performed as below:

mongify check database.config

Mongify Translation:

The is a ruby file used to translate SQL data before saving it into MongoDB.In order for Mongify to know with the data provide a translation file(translation.rb).It contains the tables and column values with the datatypes defined.This command can be performed as below:

mongify translation database.config > translation.rb

Mongify Process:

Once the translation file is setup, it tells mongify to move the data by issuing the process command. This command initiates the migration process from the MySQL to MongoDB . This command can be performed as below:

mongify process database.config translation.rb

Its very simple to migrate the data to MongoDB from MySQL.Have a great time guys meet you in another good post.

TokuMX vs MongoDB benchmark

2015-05-12T14:01:00.003+05:30

MongoDB has grown into a popular NoSQL database in a really short interval of time. With its document oriented structure and bson format storage.MongoDB has captivated the hearts of developers. We have been working on several mongoDB related benchmarks and in the recent times we came across Tokutek’s TokuMx which is a fork of mongoDB with the promising Fractal tree index.

Several of Tokutek’s benchmarks prove that the performance of TokuMx is nearly 50x faster , 90% reduction in database size. The details and the graphs given by them were quite impressive. We tried our own benchmarks and were surprised to get the below results.

Our benchmark was done on a centos 7 machine with the below configuration:

CPU no(s)       : 4

CPU speed      : 2361.140 MHz

RAM Total      : 3768708 kB

The insert benchmark was carried by inserting different number of documents and the time take for each insert was observed. The below graph depicts it.

We then pulled out the data that we inserted with different sets of rows and the time taken for each select query was noted.

We also noted down down the size of the database on disk and made a comparison.

Hope this benchmarking gives all a clear comparison between MongoDB and TokuMx.

View current user,host,port and database in mongo shell

2015-04-09T11:47:00.002+05:30

Working would be great when there is some good interface. But when your working in the mongo shell we feel little uneasy to know the user, host, port, and database we are using at present. Even though we have certain commands to find out the current user, host, port, and database it looks like a time consuming one. Its difficult to find out them the shell. How can we make all these criteria to appear in the current shell we are working on?Okay, let's have a look at this post.
The normal mongo shell looks likes this with no any criteria specified:

The below script allows displaying the current user, host, port, and database in mongo shell. All you need to do is to paste the below-shown code into your mongo shell.

 prompt = function() {
    user = db.runCommand({connectionStatus:1}).authInfo.authenticatedUsers[0]
    host = db.getMongo().toString().split(" ")[2]
    curDB = db.getName()
    if (user) {
       uname = user.user
    }
    else {
       uname = "local"
    }
    return uname + "@" + host + ":" + curDB + "> "
}

And you can see some changes over there. So now the shell turns out like this:

The shell looks so cool now right. A simple post yet useful one.Hope this helps.

Mongodb Replica Set

2015-02-20T11:44:00.002+05:30

This guide explains the concepts of configuring a two replica set.It also explains about clearly about Replica Set,Primary members,Secondary members,Arbiter,Secondary member,Priority 0 Members,Hidden Replication Members,Delayed Replication Members.Also how to configure the replica set in a shard.

Replica Set:

A replica set in MongoDB is a group of mongod processes that maintain the same data set.Replica sets provide redundancy and high availability, and are the basis for all production deployments.The main difference between the Master-slave replication and the Replica set is that Replica set has automatic failover options.The Master-slave replication has manual failover options.So the Replica set options are more preferred than the Master-slave replication.

Primary members:

The primary is the only member in the replica set that receives write operations. MongoDB applies write operations on the primary and then records the operations on the primary’s oplog. Secondary members replicate this log and apply the operations to their data sets.

Secondary members:

Secondary replicate operations from the primary to maintain an identical data set. Secondaries may have additional configurations for special usage profiles. For example, secondaries may be non-voting or priority 0.

Arbiter:

An arbiter does not have a copy of data set and cannot become a primary. Replica sets may have arbiters to add a vote in elections of for primary. Arbiters allow replica sets to have an uneven number of members, without the overhead of a member that replicates data.

Secondary member Customization:

There are instances where you may not want all of your secondary members to be beholden to the standard rules for a replication set. A replication set can have up to 12 members and up to 7 will vote in an election situation.

Priority 0 Replication Members:

There are some situations where the election of certain set members to the primary position could have a negative impact on your application's performance.For instance, if you are replicating data to a remote data center or a specific member's hardware is inadequate to perform as the main access point for the set, setting priority 0 can ensure that this member will not become a primary but can continue copying data.

Hidden Replication Members:

Some situations require you to separate the main set of members accessible and visible to your clients from the background members that have separate purposes and should not interfere.For instance, you may need a secondary member to be the base for analytics work, which would benefit from an up-to-date dataset but would cause a strain on working members. By setting this member to hidden, it will not interfere with the general operations of the replication set.Hidden members are necessarily set to priority 0 to avoid becoming the primary member, but they do vote in elections.

Delayed Replication Members:

By setting the delay option for a secondary member, you can control how long the secondary waits to perform each action it copies from the primary's oplog.This is useful if you would like to safeguard against accidental deletions or recover from destructive operations. For instance, if you delay a secondary by a half-day, it would not immediately perform accidental operations on its own set of data and could be used to revert changes.Delayed members cannot become primary members, but can vote in elections. In the vast majority of situations, they should be hidden to prevent processes from reading data that is out-of-date.

Configuring a replica set:

Let us assume to create a replica set with the following architecture:

Step 1:Create the required data directories. Creating a replica set with the name firstset,Start a mongod instance as follows for primary(10001) as follows:

./mongod --dbpath /data/example/firstset1 --port 10001 --replSet firstset --oplogSize 700 --rest

Step 2:Create a mongod instance as follows for a secondary(10002):

./mongod --dbpath /data/example/firstset2 --port 10002 --replSet firstset --oplogSize 700 --rest

Step 3:Create a mongod instance as follows for another secondary(10003):

./mongod --dbpath /data/example/firstset3 --port 10003 --replSet firstset --oplogSize 700 --rest

Step 4:Open a mongo instance for the primary(10001) as follows:

./mongo localhost:10001/admin

Step 5:Now initiate the primary(10001) replica set to the mongo instance as follows:

db.runCommand({"replSetInitiate" : {"_id" : "firstset", "members" : [{"_id" : 1, "host" : "localhost:10001"}, {"_id" : 2, "host" : "localhost:10002"}, {"_id" : 3, "host" :"localhost:10003"}]}})

Step 6:The primary interface looks as follows:

Step 7:The secondary interface looks as follows:

Step 8:To check whether the current replica is master give the command db.ismaster() from primary:

Step 9:To check whether the current replica is master give the command db.ismaster() from secondary:

Step 10:To view the replica status of the primary:

rs.status()

Thats it MongoDb replica set has been configured.

Install MongoDB on Ubuntu

2015-02-19T18:49:00.002+05:30

MongoDB is a cross-platform, document oriented database that provides,high performance, high availability, and easy scalability.This guide helps to install the mongodb on ubuntu.

Step 1: Search for the mongodb package avaliable but initially it may not be present

sudo apt-cache search mongodb

Step 2: Add the package to /etc/apt/sources.list.d because its the the latest stable MongoDB version

vim /etc/apt/sources.list.d/mongodb.list

Add the following link into the mongodb.list file

deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen

Step 3: Import the GPG key

apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

Step 4 : Update the debian package using the command:

apt-get update

Step 5: Now perform search to view the package

apt-cache search mongodb

Step 6: Now the package will be present so install it using the below command:

apt-get install mongodb-10gen

Step 7: Check mongodb has started using

ps -ef | grep mongo

ls -ls /usr/bin | grep mongo

Step 8: The mongodb control script is generated at /etc/init.d/mongodb,the MongoDB config file is at /etc/mongodb.conf add the below details:

verbose = true
dbpath = /data/db
logpath = /var/log/mongodb.log
logappend = true
port = 27017

Step 9: Start the mongodb as a service

service mongodb start

Step 10 :Check mongodb service has started using

ps -ef | grep mongodb

Step 11 : To stop/restart perform the below commands:

service mongodb stop

service mongodb restart

Thus the installation has been very simple in ubuntu.

MongoDB Master Slave Replication

2015-02-12T17:44:00.000+05:30

Master-Slave replication configuration is easy to configure. Before configuring a Master-Slave Replication in MongoDB it is necessary to know exactly what is a master, slave, replica set and their difference. The database that receives all writes in a conventional master-slave replication is called master. A read-only database that replicates operations from a master database in conventional master/slave replication is called slave. A replica set is a group of mongod processes that maintain the same data set. The main difference between the Master-slave replication and the Replica set is that Replica set has the automatic failover options. The Master-slave replication has the manual failover options. Now let's try to configure the below model Master-Slave Replication in MongoDB with a single master and three slaves.

Follow the below steps for Master-Slave Replication in MongoDB:

Step 1: Create the required data directories. To configure a master-slave deployment, start a mongod instance (30001) as master. For MongoDB installation refer here. So start an instance as follows:

./mongod --dbpath /data/example/masterdb/ --master --port 30001

Step 2: Start another mongod instance(30002) as a slave. So start an instance with the master port as a source as follows:

./mongod --dbpath /data/example/slavedb1/ --slave --port 30002 --source=localhost:30001

Step 3: Similarly start another slave(30003) with the master host and port as the source.

./mongod --dbpath /data/example/slavedb2/ --slave --port 30003 --source=localhost:30001

Step 4: Similarly start another slave(30003) with the master host and port as the source.

./mongod --dbpath /data/example/slavedb3/ --slave --port 30004 --source=localhost:30001

Step 5: Start the Mongo instance for the master with the required port(30001)

./mongo localhost:30001/local

The master interface looks as below:

Step 6: Start the Mongo instance for the slave with the required port(30002)

./mongo localhost:30002/admin

The slave interface looks as below:

Step 7: Start the Mongo instance for the slave with the required port(30002)

./mongo localhost:30003/admin

The slave interface looks as below:

Similarly, create another slave mongo instance as above of the port 30004.Now the master-slave replication has been done with a single master and three slaves.

To check use whether it is master or not use:

db.isMaster()

If it is master it returns true.

If it is a slave it returns false.

To check whether the slave synced with master or behind use:

rs.printReplicationInfo()

That's all now you have configured out a master with three slaves in your machine. Hope it helps :)

MongoDB Journal concepts

2015-02-12T13:45:00.000+05:30

People started exploring MongoDB.The one of the best feature that mongodb has unique when compared to other databases is Journal.In this section we are going to see what is a mongodb journal,how to enable or disable it and how it works.

What is a Journal in Mongodb ?

A sequential, binary transaction log used to bring the database into a valid state in the event of a hard shutdown. Journaling writes data first to the journal and then to the core data files. MongoDB enables journaling by default for 64-bit builds of MongoDB version 2.0 and newer. Journal files are pre-allocated and exist as files in the data directory.

Journal can also bring the database to a valid state even when it shutdowns.Journal first write the data into it and then to the core files.

How to enable Journal?

MongoDB version 2.0 and newer versions have journal enabled by default.To enable journaling, start mongod with the --journal command line option.For example:

./mongod --port 27017 --journal

If no journal files exist, when mongod starts, it must pre allocate new journal files. During this operation, the mongod is not listening for connections until preallocation completes: for some systems this may take a several minutes. During this period your applications and the mongo shell are not available.

How to disable Journal?

Disabling journal is not advised in the production systems.Suppose due to some power failure the mongod has gone unclean shutdown then in that case the data could be backup only if it journaling is enabled.To disable journaling, start mongod with the --nojournal command line option.For example:

./mongod --port 27017 --nojournal

Views in Journaling:

The shared view stores modified data for upload to the MongoDB data files. The shared view is the only view with direct access to the MongoDB data files. When running with journaling,mongod asks the operating system to map your existing on-disk data files to the shared view virtual memory view. The operating system maps the files but does not load them. MongoDB later loads data files into the shared view as needed.

The private view stores data for use with read operations.The private view is the first place MongoDB applies new write operations. Upon a journal commit, MongoDB copies the changes made in the private view to the shared view,where they are then available for uploading to the database data files.

The journal is an on-disk view that stores new write operations after MongoDB applies the operation to the private view but before applying them to the data files. The journal provides durability. If the mongod instance were to crash without having applied the writes to the data files, the journal could replay the writes to the shared view for eventual upload to the data files.

How mongodb works without journaling?

Generally the mongodb consists of two views such as Shared view and Private view.If the mongodb works journaling disabled it uses only the Shared view.When the mongod server starts it maps the data files to the shared view by memory mapping.If you need to read a memory at some memory address then you can get that address but you cannot actually access that address.Also if you make any changes in the memory the operating system will flush them for every 60 seconds.

How Journaling works?

The mongodb with journaling enabled has two views such as Shared view and Private view.In journaling private view is not connected to the data file so that the operating system cannot flush out any changes from private view to the disk.When a write operation is carried way then the mongod writes it to the private view.

Then the mongod will make the change to the journal with a small description what change has made within.The journal will keep the list of what all the changes made in the file.

If there is any power failure or unclean shutdown in the system then the mongod then it will move the changes to the shared view.

With a small comparative speed the shared view will flush them to the data file.Hence if there is any crash in the mongod then it is easy to repair or recover the lost data.

Finally,the remap takes place between the Shared view and Private view so that the data in them will always remain in sync.

This is how journal in the mongodb works.Hope it helps :)

Install MongoDB binary on linux

2015-02-12T11:43:00.002+05:30

MongoDB is a cross-platform, document-oriented database that provides, high performance, high availability, and easy scalability. MongoDB works on the concept of collection and document. It is also one of the leading NoSQL databases. It’s easy to install MongoDB binary file within five minutes if you follow the steps mentioned below. Here we go..!!

Step 1: Download the latest .tar.gz file from this link

Its always better to download 64 bit for production rather than 32 bit because 32-bit MongoDB processes are limited to about 2 GB of data. But the 64 bit doesn't have that limitation. For development, both versions can be used.

Step 2: Create a source directory and place the downloaded file in the source directory.For example (/usr/local)

Step 3: Now extract the downloaded file using the below command:

tar zxf mongodb-linux-i686-1.6.0.tar.gz

Step 4: Now rename the extracted(mongodb-linux-x86 64-2.6.3) file into MongoDB.

mv mongodb-linux-x86 64-2.6.3 ./mongodb

Step 5: Create a data directory using the below command

mkdir /data/db

Step 6: Create a user mongo_user using the following command

useradd mongod

This user can have all the privileges to access the MongoDB database and directories.

Step 7: Change the ownership of the files in the source and data directory using the following command

chown -R mongod.mongod /usr/local/

chown -R mongod.mongod /data/db/

Changing the ownership gives the complete access for the users to access the source and the data directory.

Step 8: Create a configuration file in any directory say vim /etc/mongod.conf

Now add the following details as shown below:

verbose = true
dbpath = /data/db
logpath = /var/log/mongodb.log
logappend = true
port = 27017

Verbose is an option that provides additional details what the computer is doing and what drivers and software it is loading during startup. Dbpath shows the data directory path. Logpath shows the path for the log file.Log append will append the log file. Port will specify which port the mongo shell will connect.

Step 9: Move to the bin folder in the source directory and now start the mongo server by the following command

./mongod --config /etc/mongod.conf

Step 10: Within the bin folder in the source directory start the mongo client by the following command

./mongo --port 27017

The interactive MongoDB prompt appears. That's it now you're done with the MongoDB binary installation. Time to play with MongoDB :)