Technical Post

MongoDB Replication set up

2019-01-30T01:00:00.002-08:00

We will have 3 instance of Mongo In order to achieve the Replication in MongoDB.

C:\Program Files\MongoDB\Server\4.0\bin>mongod --smallfiles --oplogSize 50 --replSet test --port 27017 --dbpath C:\data\db

C:\Program Files\MongoDB\Server\4.0\bin>mongod --smallfiles --oplogSize 50 --replSet test --port 27018 --dbpath D:\data\db

C:\Program Files\MongoDB\Server\4.0\bin>mongod --smallfiles --oplogSize 50 --replSet test --port 27019 --dbpath E:\data\db

C:\>cd "\Program Files\MongoDB\Server\4.0\bin"

C:\Program Files\MongoDB\Server\4.0\bin>mongo
MongoDB shell version v4.0.2
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 4.0.2
Server has startup warnings:
2019-01-24T10:26:02.197+0530 I CONTROL [initandlisten]
2019-01-24T10:26:02.197+0530 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the d
2019-01-24T10:26:02.198+0530 I CONTROL [initandlisten] ** Read and write access to data and confi
2019-01-24T10:26:02.198+0530 I CONTROL [initandlisten]
2019-01-24T10:26:02.198+0530 I CONTROL [initandlisten] ** WARNING: This server is bound to localhost.
2019-01-24T10:26:02.198+0530 I CONTROL [initandlisten] ** Remote systems will be unable to connec
2019-01-24T10:26:02.198+0530 I CONTROL [initandlisten] ** Start the server with --bind_ip <addres
2019-01-24T10:26:02.198+0530 I CONTROL [initandlisten] ** addresses it should serve responses fro
2019-01-24T10:26:02.198+0530 I CONTROL [initandlisten] ** bind to all interfaces. If this behavio
2019-01-24T10:26:02.198+0530 I CONTROL [initandlisten] ** server with --bind_ip 127.0.0.1 to disa
2019-01-24T10:26:02.198+0530 I CONTROL [initandlisten]

C:\Program Files\MongoDB\Server\4.0\bin>mongo --host 127.0.0.1:27018
MongoDB shell version v4.0.2
connecting to: mongodb://127.0.0.1:27018/
MongoDB server version: 4.0.2
Server has startup warnings:
2019-01-24T10:39:24.775+0530 I CONTROL [initandlisten]
2019-01-24T10:39:24.775+0530 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-01-24T10:39:24.775+0530 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2019-01-24T10:39:24.775+0530 I CONTROL [initandlisten]
2019-01-24T10:39:24.776+0530 I CONTROL [initandlisten] ** WARNING: This server is bound to localhost.
2019-01-24T10:39:24.776+0530 I CONTROL [initandlisten] ** Remote systems will be unable to connect to this server.
2019-01-24T10:39:24.776+0530 I CONTROL [initandlisten] ** Start the server with --bind_ip <address> to specify which IP
2019-01-24T10:39:24.777+0530 I CONTROL [initandlisten] ** addresses it should serve responses from, or with --bind_ip_all to
2019-01-24T10:39:24.777+0530 I CONTROL [initandlisten] ** bind to all interfaces. If this behavior is desired, start the
2019-01-24T10:39:24.777+0530 I CONTROL [initandlisten] ** server with --bind_ip 127.0.0.1 to disable this warning.
2019-01-24T10:39:24.778+0530 I CONTROL [initandlisten]
MongoDB Enterprise test:SECONDARY>

C:\Program Files\MongoDB\Server\4.0\bin>mongo --host 127.0.0.1:27019
MongoDB shell version v4.0.2
connecting to: mongodb://127.0.0.1:27019/
MongoDB server version: 4.0.2
Server has startup warnings:
2019-01-30T12:06:55.398+0530 I CONTROL [initandlisten]
2019-01-30T12:06:55.398+0530 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2019-01-30T12:06:55.398+0530 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2019-01-30T12:06:55.399+0530 I CONTROL [initandlisten]
2019-01-30T12:06:55.399+0530 I CONTROL [initandlisten] ** WARNING: This server is bound to localhost.
2019-01-30T12:06:55.399+0530 I CONTROL [initandlisten] ** Remote systems will be unable to connect to this server.
2019-01-30T12:06:55.400+0530 I CONTROL [initandlisten] ** Start the server with --bind_ip <address> to specify which IP
2019-01-30T12:06:55.400+0530 I CONTROL [initandlisten] ** addresses it should serve responses from, or with --bind_ip_all to
2019-01-30T12:06:55.400+0530 I CONTROL [initandlisten] ** bind to all interfaces. If this behavior is desired, start the
2019-01-30T12:06:55.400+0530 I CONTROL [initandlisten] ** server with --bind_ip 127.0.0.1 to disable this warning.
2019-01-30T12:06:55.401+0530 I CONTROL [initandlisten]
MongoDB Enterprise test:SECONDARY>

MongoDB Enterprise test:PRIMARY> rs.initiate();
{
"operationTime" : Timestamp(1548307074, 1),
"ok" : 0,
"errmsg" : "already initialized",
"code" : 23,
"codeName" : "AlreadyInitialized",
"$clusterTime" : {
"clusterTime" : Timestamp(1548307074, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}

MongoDB Enterprise test:PRIMARY> rs.add(127.0.0.1:27018);
2019-01-24T10:48:31.528+0530 E QUERY [js] SyntaxError: missing ) after argument list @(shell):1:12
MongoDB Enterprise test:PRIMARY> rs.add("127.0.0.1:27018");
{
"ok" : 1,
"operationTime" : Timestamp(1548307139, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1548307139, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}

MongoDB Enterprise test:PRIMARY> rs.add(127.0.0.1:27019);

MongoDB Enterprise test:PRIMARY> rs.conf();
{
"_id" : "test",
"version" : 2,
"protocolVersion" : NumberLong(1),
"writeConcernMajorityJournalDefault" : true,
"members" : [
{
"_id" : 0,
"host" : "localhost:27017",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
},
{
"_id" : 1,
"host" : "127.0.0.1:27018",
"arbiterOnly" : false,
"buildIndexes" : true,
"hidden" : false,
"priority" : 1,
"tags" : {

},
"slaveDelay" : NumberLong(0),
"votes" : 1
}
],
"settings" : {
"chainingAllowed" : true,
"heartbeatIntervalMillis" : 2000,
"heartbeatTimeoutSecs" : 10,
"electionTimeoutMillis" : 10000,
"catchUpTimeoutMillis" : -1,
"catchUpTakeoverDelayMillis" : 30000,
"getLastErrorModes" : {

},
"getLastErrorDefaults" : {
"w" : 1,
"wtimeout" : 0
},
"replicaSetId" : ObjectId("5c416b3d712e9a0ff1ac72b7")
}
}

MongoDB Enterprise test:PRIMARY> rs.status();
{
"set" : "test",
"date" : ISODate("2019-01-24T05:20:36.793Z"),
"myState" : 1,
"term" : NumberLong(7),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"heartbeatIntervalMillis" : NumberLong(2000),
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1548307234, 1),
"t" : NumberLong(7)
},
"readConcernMajorityOpTime" : {
"ts" : Timestamp(1548307234, 1),
"t" : NumberLong(7)
},
"appliedOpTime" : {
"ts" : Timestamp(1548307234, 1),
"t" : NumberLong(7)
},
"durableOpTime" : {
"ts" : Timestamp(1548307234, 1),
"t" : NumberLong(7)
}
},
"lastStableCheckpointTimestamp" : Timestamp(1548307194, 1),
"members" : [
{
"_id" : 0,
"name" : "localhost:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 1475,
"optime" : {
"ts" : Timestamp(1548307234, 1),
"t" : NumberLong(7)
},
"optimeDate" : ISODate("2019-01-24T05:20:34Z"),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"electionTime" : Timestamp(1548305762, 1),
"electionDate" : ISODate("2019-01-24T04:56:02Z"),
"configVersion" : 2,
"self" : true,
"lastHeartbeatMessage" : ""
},
{
"_id" : 1,
"name" : "127.0.0.1:27018",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 97,
"optime" : {
"ts" : Timestamp(1548307234, 1),
"t" : NumberLong(7)
},
"optimeDurable" : {
"ts" : Timestamp(1548307234, 1),
"t" : NumberLong(7)
},
"optimeDate" : ISODate("2019-01-24T05:20:34Z"),
"optimeDurableDate" : ISODate("2019-01-24T05:20:34Z"),
"lastHeartbeat" : ISODate("2019-01-24T05:20:35.574Z"),
"lastHeartbeatRecv" : ISODate("2019-01-24T05:20:36.616Z"),
"pingMs" : NumberLong(0),
"lastHeartbeatMessage" : "",
"syncingTo" : "localhost:27017",
"syncSourceHost" : "localhost:27017",
"syncSourceId" : 0,
"infoMessage" : "",
"configVersion" : 2
}
],
"ok" : 1,
"operationTime" : Timestamp(1548307234, 1),
"$clusterTime" : {
"clusterTime" : Timestamp(1548307234, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}

MongoDB Enterprise test:PRIMARY>

MongoDB Enterprise test:PRIMARY> use exampleDB
switched to db exampleDB
MongoDB Enterprise test:PRIMARY> for (var i = 0; i <= 10; i++) db.exampleCollection.insert( { x : i } )
WriteResult({ "nInserted" : 1 })
MongoDB Enterprise test:PRIMARY> db.exampleCollection.find().pretty();
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f5f"), "x" : 0 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f60"), "x" : 1 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f61"), "x" : 2 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f62"), "x" : 3 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f63"), "x" : 4 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f64"), "x" : 5 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f65"), "x" : 6 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f66"), "x" : 7 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f67"), "x" : 8 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f68"), "x" : 9 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f69"), "x" : 10 }
MongoDB Enterprise test:PRIMARY>

Now go to the secondary and check if the data is replicated.
MongoDB Enterprise test:SECONDARY> db.getMongo().setSlaveOk()

MongoDB Enterprise test:SECONDARY> db.exampleCollection.find().pretty();
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f5f"), "x" : 0 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f61"), "x" : 2 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f63"), "x" : 4 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f65"), "x" : 6 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f62"), "x" : 3 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f67"), "x" : 8 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f69"), "x" : 10 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f64"), "x" : 5 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f68"), "x" : 9 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f60"), "x" : 1 }
{ "_id" : ObjectId("5c494c1fa8a110e1bca27f66"), "x" : 7 }
MongoDB Enterprise test:SECONDARY>

Check on the another secondary .

To test the fail-over lets us kill the primary now. Check the rs.status() on another secondary.

Its shows the earlier primary is not available and the one running on the 27018 becomes the primary.

Let start the 27017 again.

Install MongoDB 4.0 on Windows 8

2019-01-21T03:44:00.002-08:00

1.    Now go to https://www.mongodb.com/download-center
2.    Click Download (msi) and run it.
3.    Click Next.

4. Accept the terms.

5. You choose the Complete (recommended for most users)

6. Select Install MongoD as a Service.

7. To install Compass, select Install MongoDB Compass (Default).

8. Click Install.

Run MongoDB Community Edition from the Command Interpreter

Create database directory.

From the command line

cd D:\

md “\MongoData \db”

Start your MongoDB database

To start MongoDB, run mongod.exe.

Let’s point server to your database directory.

"C:\Program Files\MongoDB\Server\4.0\bin\mongod.exe" --dbpath="c:\data\db"

If the MongoDB database server is running correctly, the Command Interpreter displays:

Connect to MongoDB.

"C:\Program Files\MongoDB\Server\4.0\bin\mongo.exe"

Achieving the Data Integrity and Incrementally Update the data on ElasticSearch Using Logstash

2018-12-31T02:48:00.001-08:00

In order to avoid the duplication issues at elasticsearch you may need to add the unique id for the documents at elasticsearch.

Modify the logstashMSSQL_Test.conf by adding the "document_id" => "%{studentid}" in the output like below.

output {
stdout { codec => json_lines }
elasticsearch {
"hosts" => "localhost:9200"
"index" => "test-migrate"
"document_type" => "data"
"document_id" => "%{studentid}"
}

Schedule the same migration if the data is continuosly been updateded on the MSSQL Sever side
This will be running after every one minute
Add the below part in order to incrementing update

schedule => "* * * * *"
statement => "SELECT * FROM Student where studentid > :sql_last_value"
use_column_value => true
tracking_column => studentid

input {
jdbc {
jdbc_connection_string => "jdbc:sqlserver://localhost:1433;databaseName=test"
# The user we wish to execute our statement as
jdbc_user => "sa"
jdbc_password => "sasa"
# The path to our downloaded jdbc driver
jdbc_driver_library => "C:\Users\abhijitb\.m2\repository\com\microsoft\sqlserver\mssql-jdbc\6.2.2.jre8\mssql-jdbc-6.2.2.jre8.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
# our query
#clean_run => true
schedule => "* * * * *"
statement => "SELECT * FROM Student where studentid > :sql_last_value"
use_column_value => true
tracking_column => "studentid"
}
}

output {
#stdout { codec => json_lines }
elasticsearch {
"hosts" => "localhost:9200"
"index" => "student"
"document_type" => "data"
"document_id" => "%{studentid}"
}
}

Add the data to Table Student one by one.

INSERT INTO Student (StudentId, FirstName, LastName, Age, Sex, City, Country, EnrolledDate) VALUES
(8,'David','Bruce', 32, 'M', 'Paris', 'FRANCE', GETDATE());

INSERT INTO Student (StudentId, FirstName, LastName, Age, Sex, City, Country, EnrolledDate) VALUES
(9,'Steve','Warne', 30, 'M', 'Paris', 'FRANCE', GETDATE()-200);

INSERT INTO Student (StudentId, FirstName, LastName, Age, Sex, City, Country, EnrolledDate) VALUES
(10,'Ajit','Patil', 32, 'M', 'Pune', 'India', GETDATE()-100);

INSERT INTO Student (StudentId, FirstName, LastName, Age, Sex, City, Country, EnrolledDate) VALUES
(11,'Steve','Scot', 32, 'M', 'Paris', 'FRANCE', GETDATE()-50);

Once you update the file, go the
Go the path C:\logstash-6.5.4\logstash-6.5.4\bin> on the command prompt.
run the below command "logstash -f logstashMSSQL_Test.conf"

C:\logstash-6.5.4\logstash-6.5.4\bin>logstash -f logstashMSSQL_Test.conf

To verify the Data on the ElasticSeach side click the link on the browser.
http://localhost:9200/student/_search?pretty=true

Check the data in descending order. You can verify the when the data is added to elastic search by checking the timestamp of it.
http://localhost:9200/student/_search?q=*&sort=studentid:desc&pretty=true

To check more properties visit https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html

Example of Migrating the MSSQL Data to ElasticSearch using the Logstash.

2018-12-31T00:05:00.000-08:00

Example of MSSQL SERVER DATABASE + LogStash + Elastic Search

Step 1 :

I assume the MSSQL server database is installed.

Let create a table and insert some data to it.

Use Test;

create table Student (StudentId int, FirstName varchar(255), LastName varchar(255), Age int, Sex char(2), City varchar(255), Country varchar(255), EnrolledDate datetime);

INSERT INTO Student (StudentId, FirstName, LastName, Age, Sex, City, Country, EnrolledDate) VALUES
(1,'Abhijit','Bashetti',33, 'M', 'Pune', 'India', '2016-05-23 16:12:03');

INSERT INTO Student (StudentId, FirstName, LastName, Age, Sex, City, Country, EnrolledDate) VALUES
(2,'Mark','Brown',31, 'M', 'Mumbai', 'India', '2017-05-23 16:12:03');

INSERT INTO Student (StudentId, FirstName, LastName, Age, Sex, City, Country, EnrolledDate) VALUES
(3,'Michell','Oak',31, 'F', 'NewYork', 'US', '2017-08-25 16:12:03');

INSERT INTO Student (StudentId, FirstName, LastName, Age, Sex, City, Country, EnrolledDate) VALUES
(4,'Peter','Samprass', 30, 'M', 'Sydeny', 'Australlia', '2017-09-30 16:12:03');

INSERT INTO Student (StudentId, FirstName, LastName, Age, Sex, City, Country, EnrolledDate) VALUES
(5,'Andre','Russel', 30, 'M', 'London', 'UK', '2017-12-01 16:12:03');

INSERT INTO Student (StudentId, FirstName, LastName, Age, Sex, City, Country, EnrolledDate) VALUES
(6,'Kevin','Peterson', 32, 'M', 'London', 'UK', '2017-12-21 16:12:03');

INSERT INTO Student (StudentId, FirstName, LastName, Age, Sex, City, Country, EnrolledDate) VALUES
(7,'Kevin','Burake', 32, 'M', 'London', 'UK', '2017-12-21 16:12:03');

select * from Student;

Step 2 :

Now Install the elastic search by downloading the same from https://www.elastic.co/downloads/elasticsearch

Go to the installed path like "C:\Program Files\Elastic\Elasticsearch\6.5.3\bin". and start the elastic search.

Step 3 :

Now Download the Logstash. https://www.elastic.co/downloads/logstash

Go to the path C:\logstash-6.5.4\logstash-6.5.4\config

Copy the logstash-sample.conf for the MSSQL DATABASE.
Rename it to logstashMSSQL_Test.conf.

Add below properties.

input {
jdbc {
# the full url string for the mssql server
jdbc_connection_string => "jdbc:sqlserver://localhost:1433;databaseName=test"
# The username and password to execute our statement as
jdbc_user => "sa"
jdbc_password => "sasa"
# The path to our downloaded mssql jdbc driver
jdbc_driver_library => "C:\Users\abhijitb\.m2\repository\com\microsoft\sqlserver\mssql-jdbc\6.2.2.jre8\mssql-jdbc-6.2.2.jre8.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
# The required query
statement => "select * from Student"
}
}

output {
#stdout { codec => json_lines }
elasticsearch {
"hosts" => "localhost:9200"
"index" => "student"
"document_type" => "data"
}
}

Go the path C:\logstash-6.5.4\logstash-6.5.4\bin> on the command prompt.
run the below command "logstash -f logstashMSSQL_Test.conf"

C:\logstash-6.5.4\logstash-6.5.4\bin>logstash -f logstashMSSQL_Test.conf

Step 4 :

Its time to verify if the data is migrated to elasticsearch.

If you have windows git go to the path "C:\Program Files\Git\mingw64\bin>"
and execute the command curl -XPOST "localhost:9200/student/_search?pretty=true"
C:\Program Files\Git\mingw64\bin>curl -XPOST "localhost:9200/student/_search?pretty=true"

or go to the browser and type http://localhost:9200/student/_search?pretty=true

Indexing and searching to ElasticSearch by Curl

2018-12-30T21:25:00.000-08:00

I have listed down the curl request for indexing the data on ElasticSearch and curl request for searching the data on ElasticSearch.

I am using the Curl provided by the Git. You may need to install Window Git in order to use the same. Once you install Windows GIT. Go to the path cd C:\Program Files\Git\mingw64\bin>

1. To create an index named "customer"

curl -X PUT "localhost:9200/customer?pretty"

Response is :

{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "customer"
}

2. To list all the indices availabale

curl -X GET "localhost:9200/_cat/indices?v"

3. To add a document

curl -X PUT "localhost:9200/customer/_doc/1?pretty" -H "Content-Type: application/json" -d "{\"name\": \"John Doe\"}"

4. To view a specific document

curl -X GET "localhost:9200/customer/_doc/1?pretty"

5. To delete a index

curl -X DELETE "localhost:9200/customer?pretty"
curl -X GET "localhost:9200/_cat/indices?v"

6. To Update a document

curl -X POST "localhost:9200/customer/_doc/1/_update?pretty" -H "Content-Type: application/json" -d "{\"doc\": { \"name\": \"Jane Doe\" }}"

7. To Delete a document

curl -X DELETE "localhost:9200/customer/_doc/1?pretty"

To verift run the below command
curl -X GET "localhost:9200/customer/_doc/1?pretty"

C:\Program Files\Git\mingw64\bin>curl -X GET "localhost:9200/customer/_doc/1?pretty"
{
"_index" : "customer",
"_type" : "_doc",
"_id" : "1",
"found" : false
}

8. Buld adding documents

curl -H "Content-Type: application/json" -XPOST "localhost:9200/bank/_doc/_bulk?pretty&refresh" --data-binary "@accounts.json"

Check if the index named bank is created
curl "localhost:9200/_cat/indices?v"

9. Search all the document in ascending order

curl -X GET "localhost:9200/bank/_search?q=*&sort=account_number:asc&pretty"

OR

curl -X GET "localhost:9200/bank/_search?pretty" -H "Content-Type: application/json" -d "{\"query\": { \"match_all\": {} }, \"sort\": [{ \"account_number\": \"asc\" }]}"

10. Search document by mentioning the limit

curl -X GET "localhost:9200/bank/_search?pretty" -H "Content-Type: application/json" -d "{\"query\": { \"match_all\": {} }, \"size\": 1 }"

11. Search Document in sorting order

curl -X GET "localhost:9200/bank/_search?pretty" -H "Content-Type: application/json" -d "{\"query\": {\"match_all\": {} }, \"sort\": {\"balance\": { \"order\": \"desc\" } }}"

12. Search all and return data of specific fields

curl -X GET "localhost:9200/bank/_search?pretty" -H "Content-Type: application/json" -d "{\"query\": { \"match_all\": {} }, \"_source\": [\"account_number\", \"balance\"]}"

13. Search for specific document by applying criteria

curl -X GET "localhost:9200/bank/_search?pretty" -H "Content-Type: application/json" -d "{\"query\": { \"match\": { \"account_number\": 20 } }}"
curl -X GET "localhost:9200/bank/_search?pretty" -H "Content-Type: application/json" -d "{\"query\": { \"match\": { \"address\": \"mill\" } }}"

14. Search for specific document by applying AND criteria with Bool with must

curl -X GET "localhost:9200/bank/_search?pretty" -H "Content-Type: application/json" -d "{\"query\": { \"bool\": { \"must\": [{ \"match\": { \"address\": \"mill\" } },{ \"match\": { \"address\": \"lane\" } }]}}}"

15. Search for specific document by applying OR criteria with Bool with should

curl -X GET "localhost:9200/bank/_search?pretty" -H "Content-Type: application/json" -d "{\"query\": { \"bool\": { \"should\": [ { \"match\": { \"address\": \"mill\" } }, { \"match\": { \"address\": \"lane\" } }]}}}"

16. Search for specific document by applying AND & NOT criteria with Bool with should

curl -X GET "localhost:9200/bank/_search?pretty" -H "Content-Type: application/json" -d "{ \"query\": { \"bool\": { \"must\": [{ \"match\": { \"age\": \"40\" } }], \"must_not\": [{ \"match\": { \"state\": \"ID\" } }]}}}"

17. Search Documents by applying some filter like greater than , lesser than to it.

curl -X GET "localhost:9200/bank/_search?pretty" -H "Content-Type:application/json" -d "{ \"query\": { \"bool\": { \"must\": { \"match_all\": {} }, \"filter\": { \"range\": { \"balance\": {\"gte\": 20000, \"lte\": 30000}}}}}}"

18. Search Document by some aggregations

curl -X GET "localhost:9200/bank/_search?pretty" -H "Content-Type:application/json" -d "{ \"size\": 0, \"aggs\": { \"group_by_state\": { \"terms\": { \"field\": \"state.keyword\"}}}}"

Generate the Boilerplate code using Lombok

2018-12-09T00:14:00.000-08:00

All the time while doing the programming we do the same thing again and again.The same code is been added by us or we use eclispe to generate the same code. This code is called as boilerplate code. Its the time to get rid of such things. There is a magic available for the same which is called as lombok. Yes, its lombok library which helps you to remove the boilerplate code and also provides some others features. If Lombok is added to your project classpath, you can can easily get rid of all the getters & setters methods, class constructors(argument constructor, no argument constructor), hashcode and equals methods and many more by just adding couple of annotations the class. For more details and other feature please refer https://projectlombok.org/features/all. The guys have created a very nice thing.

I will list down all the annotation provided by the lombok and will see some them with the examples.

@Getter/@Setter : It will generate the getter and setter methods of the class fields. All the methods genearated are by default public. You can set the access level for any field.

@NonNull : Can be used for the parameters of a methos or a constructors for null check. It can be applied to a field as well.

@NoArgsConstructor : It generates constructor with no parameters.

@RequiredArgsConstructor : It generates a constructor with one parameter of each field where it requires special handling.

@AllArgsConstructor : It generates a constructor with 1 parameter for each field in your class.

@ToString : It provides the implementation of toString method to your class where it will print the classs name and the fields of the class.

@EqualsAndHashCode : Same as toString , Lombok provides the implemenation of the equals and hashCode method. It will consider the non-static and non-transient fields. You can also incluse or exclude the fields explicitely @EqualsAndHashCode.Include or @EqualsAndHashCode.Exclude.

@Data : Its combo provided by lombok for some the commonly used annotation. Its combo of
@ToString, @EqualsAndHashCode, @Getter / @Setter and @RequiredArgsConstructor. It covers all of the above in one shot.

Here is an example where I have created a java class named Employee.java
I have added the member as empId,firstName,lastName,address
On the right side, in the outline window you can find all the code generated by lombok.

I have created the object of Employee class and shown how to use the implementation provided by lombok. In this case I have used the toString and equals method.

Split the large xml into smaller xml using xslt.

2018-12-01T22:00:00.001-08:00

Split the large xml to smaller xml using XSLT.

Here is sample of source xml.

<?xml version="1.0" encoding="UTF-8"?>
<ABC>
<END />
<Tables>
<START>
<row>
<id>111</id>
<name>abc</name>
<deptId>1</deptId>
</row>
<row>
<id>112</id>
<name>abc1</name>
<deptId>1</deptId>
</row>
<row>
<id>113</id>
<name>abc3</name>
<deptId>1</deptId>
</row>
<row>
<id>222</id>
<name>def</name>
<deptId>2</deptId>
</row>
<row>
<id>333</id>
<name>pqr</name>
<deptId>2</deptId>
</row>
<row>
<id>444</id>
<name>xyz</name>
<deptId>2</deptId>
</row>
<row>
<id>555</id>
<name>lmn</name>
<deptId>3</deptId>
</row>
<row>
<id>555</id>
<name>lmn</name>
<deptId>3</deptId>
</row>
</START>
</Tables>
</ABC>

I have a xml with the above structure.
I have to spilt the xml into 3 xml's based on the different deptId.
I have to Split the xml into smaller one based on change in tag values.
My elemement is deptId whose values is been changed after some rows.
The all elements with same deptId are in a sequence.

The required output is : Its good to have the xml name as the department id.
The first xml be with name 1.xml :

<?xml version="1.0" encoding="UTF-16"?>
<ABC>
<END />
<Tables>
<START>
<row>
<id>111</id>
<name>abc</name>
<deptId>1</deptId>
</row>
<row>
<id>112</id>
<name>abc1</name>
<deptId>1</deptId>
</row>
<row>
<id>113</id>
<name>abc3</name>
<deptId>1</deptId>
</row>
</START>
</Tables>
</ABC>

The solution to the above the problem is to use the XSLT.
Create a xsl file like below.

<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
<xsl:template match="/">
<xsl:for-each-group select="//ABC//Tables//START//row" group-adjacent="deptId">
<xsl:variable name="file" select="concat(deptId,'.xml')"/>
<xsl:result-document href="{$file}">
<ABC>
<END />
<Tables>
<START>
<xsl:copy-of select="current-group()"/>
</START>
</Tables>
</ABC>
</xsl:result-document>
</xsl:for-each-group>
</xsl:template>
</xsl:transform>

Here is the sample piece of java code that need to run.

package com.java.xml;

import java.io.File;

import javax.xml.transform.stream.StreamSource;

import net.sf.saxon.s9api.Processor;
import net.sf.saxon.s9api.SaxonApiException;
import net.sf.saxon.s9api.Serializer;
import net.sf.saxon.s9api.Xslt30Transformer;
import net.sf.saxon.s9api.XsltCompiler;
import net.sf.saxon.s9api.XsltExecutable;

public class SplitXml {

public static void main(String[] args) {
Processor proc = new Processor(false);
XsltCompiler comp = proc.newXsltCompiler();
XsltExecutable exp;
try {
exp = comp.compile(new StreamSource(new File("E://test.xsl")));
Serializer out = proc.newSerializer(new File("E://output.xml"));
Xslt30Transformer trans = exp.load30();
trans.applyTemplates(new StreamSource(new File("E://source.xml")),
out);
} catch (SaxonApiException e) {
e.printStackTrace();
}
}
}

Steps for Solr 6.1 setup with data import handler(DIH)

2017-02-27T05:45:00.003-08:00

1. Install Java Runtime Environment (JRE) version 1.8 for Solr 6.1

2. Set Java home "export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_101/"

2. Download solr 6.1 "solr-6.1.0.tgz" from http://www-eu.apache.org/dist/lucene/solr/6.1.0/

3. Extract tgz file using "tar zxf solr-6.1.0.tgz".

4. Open the port 8983 as to communicate with solr.

5. Go to the path ../solr-6.1.0/ and start the server with "bin/solr start"

============================================================================================================================================================

1. Create a config folder named myConfig under configsets directory , move the conf folder into it. Add the data-config.xml into "/myConfig/conf".

2. Fire the url to create the core.

http://localhost:8983/solr/admin/cores?action=CREATE&name=25406&instanceDir=/home/abhijit/Downloads/solr-6.1.0/server/solr/25406/&configSet=myConfig

3. Add the jars "solr-dataimporthandler-6.1.0.jar", "solr-dataimporthandler-extras-6.1.0.jar" to path "/home/abhijit/Downloads/solr-6.1.0/server/lib".

4. Add the "solr-core-6.1.0.jar" jar to the path "/home/abhijit/Downloads/solr-6.1.0/server/lib".

5. Add the "solr-solrj-6.1.0.jar" jar to the path "/home/abhijit/Downloads/solr-6.1.0/server/lib".

6. If you do the above steps you will get the error like :

"Error Instantiating requestHandler, org.apache.solr.handler.dataimport.DataImportHandler failed to instantiate org.apache.solr.request.SolrRequestHandler"
This problem occurred because dataimporthandler and requesthandler are loaded by two different class loaders. To solve it, ensure that solr loads its jars only from the same class loader.

Option to steps 3,4,5 create a folder lib inside the solr folder at path "/home/abhijit/Downloads/solr-6.1.0/server/solr"
and add the entry of the same to solrConfig.xml as "<lib dir="./solr/lib" />".
Comment the other lib entries from solrConfig.xml
the jars "solr-dataimporthandler-6.1.0.jar", "solr-dataimporthandler-extras-6.1.0.jar" to path "/home/abhijit/Downloads/solr-6.1.0/server/solr/lib".

9. Commented the updateRequestProcessorChain in solrConfig.xml.

10. HttpSolrCall null:org.apache.solr.handler.dataimport.DataImportHandlerException: Data Config problem: Entity must have a name.
Every entity defined in data-config file must have a name defined to it e.g. "<entity name="tika" , "<entity name="customProperty"

11. Error : "java.lang.NoClassDefFoundError: org/apache/tika/parser/Parser"
Add the jar "tika-core-1.7.jar" to path "/home/abhijit/Downloads/solr-6.1.0/server/solr/lib".

12. Copy all the jars from "/Downloads/solr-6.1.0/contrib/extraction/lib" to path "/home/abhijit/Downloads/solr-6.1.0/server/solr/lib".

13. Error : " java.sql.SQLSyntaxErrorException: ORA-00907: missing right parenthesis"
" ' " is missing for the date parameter passed while the full import e.g '24/05/2015'

14. Command for full import

http://localhost:8983/solr/mycore/dataimport?command=full-import&clean=true&commit=true

15. Check core status

http://localhost:8983/solr/admin/cores?action=STATUS&core=25406

16. Delete URL was not working so added the solrConfig.xml file.

http://10.55.36.19:8983/solr/25406/update?stream.body=<delete><id>25473</id></delete>&commit=true

<requestHandler name="/update" class="solr.UpdateRequestHandler">
<lst name="defaults">
<str name="update.chain">id</str>
</lst>
</requestHandler>

"o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: unknown UpdateRequestProcessorChain: add-unknown-fields-to-the-schema"

Uncommentted the "updateRequestProcessorChain" in solrConfig.xml and named it as name="id".

Change in the default port of Jetty for Solr.

2017-02-26T23:54:00.001-08:00

Change in the default port of Jetty for Solr.

Currently Jetty uses 8983 as default port.

I wanted to change it to 8080 port.

To achieve the same I have done the changes at the following places.

1. Change in the file solr.xml located at ../solr-6.1.0/server/solr/.

Change from

<int name="hostPort">${jetty.port:8983}</int>

to

<int name="hostPort">${jetty.port:8080}</int>


2. Change in the file solr.in.sh located at ../solr-6.1.0/bin/

Change from

# Sets the port Solr binds to, default is 8983
SOLR_PORT=8983

to

# Sets the port Solr binds to, default is 8983
SOLR_PORT=8080

Sqoop Learning...

2016-04-12T02:51:00.001-07:00

Sqoop is a tool designed to transfer data between Hadoop and relational databases. You can use Sqoop to import data from a relational database management system (RDBMS) such as MySQL or Oracle into the Hadoop Distributed File System (HDFS), transform the data in Hadoop MapReduce, and then export the data back into an RDBMS.

With Sqoop, you can import data from a relational database system into HDFS. The input to the import process is a database table. Sqoop will read the table row-by-row into HDFS. The output of this import process is a set of files containing a copy of the imported table. The import process is performed in parallel. For this reason, the output will be in multiple files. These files may be delimited text files (for example, with commas or tabs separating each field), or binary Avro or SequenceFiles containing serialized record data.

A by-product of the import process is a generated Java class which can encapsulate one row of the imported table. This class is used during the import process by Sqoop itself. The Java source code for this class is also provided to you, for use in subsequent MapReduce processing of the data. This class can serialize and deserialize data to and from the SequenceFile format.

Sqoop includes some other commands which allow you to inspect the database you are working with.
For example,
1.list the available database schemas (with the sqoop-list-databases tool).

sqoop-list-databases --connect jdbc:mysql://localhost:3306/ --username root --password hr;

2.List the available tables within a schema (with the sqoop-list-tables tool).

sqoop-list-tables --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr;

3.Sqoop also includes a primitive SQL execution shell (the sqoop-eval tool).

sqoop eval --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr --query "select * from employees";

Most aspects of the import, code generation, and export processes can be customized. You can control the specific row range or columns imported. You can specify particular delimiters and escape characters for the file-based representation of the data, as well as the file format used. You can also control the class or package names used in generated code.

Sqoop ships with a help tool. To display a list of all available tools, type the following command:

$ sqoop help

usage: sqoop COMMAND [ARGS]

Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information

To display help for a specific tool by entering: sqoop help (tool-name);
for example,

$ sqoop help import.

usage: sqoop import [GENERIC-ARGS] [TOOL-ARGS]

Common arguments:
--connect <jdbc-uri> Specify JDBC connect String

--connection-manager <class-name> Specify connection manager class name

--connection-param-file <properties-file> Specify connection parameters file

--driver <class-name> Manually specify JDBC driver class to use

--hadoop-home <hdir> Override $HADOOP_MAPRED_HOME_ARG

--hadoop-mapred-home <dir> Override $HADOOP_MAPRED_HOME_ARG

--password <password> Set authentication password

--password-file <password-file> Set authentication password file path

--username <username> Set authentication username

Options Files to Pass Arguments

sqoop import --connect jdbc:mysql://localhost/db --username root --table TEST

$ sqoop --options-file /users/homer/work/import.txt --table TEST

where the options file /users/homer/work/import.txt contains the following:

import
--connect
jdbc:mysql://localhost/db
--username
root

The options file can have empty lines and comments for readability purposes. Comments within option files that begin with the hash character.

sqoop-import : The import tool imports an individual table from an RDBMS to HDFS.

Table 1. Common arguments
Argument Description
--connect <jdbc-uri> Specify JDBC connect string
--connection-manager <class-name> Specify connection manager class to use
--driver <class-name> Manually specify JDBC driver class to use
--hadoop-mapred-home <dir> Override $HADOOP_MAPRED_HOME
--help Print usage instructions
--password-file Set path for a file containing the authentication password
-P Read password from console
--password <password> Set authentication password
--username <username> Set authentication username
--verbose Print more information while working
--connection-param-file <filename> Optional properties file that provides connection parameters

Table 2. Import control arguments:

Argument Description
--append Append data to an existing dataset in HDFS
--as-avrodatafile Imports data to Avro Data Files
--as-sequencefile Imports data to SequenceFiles
--as-textfile Imports data as plain text (default)
--boundary-query <statement> Boundary query to use for creating splits
--columns <col,col,col…> Columns to import from table
--delete-target-dir Delete the import target directory if it exists
--direct Use direct import fast path
--direct-split-size <n> Split the input stream every n bytes when importing in direct mode
--fetch-size <n> Number of entries to read from database at once.
--inline-lob-limit <n> Set the maximum size for an inline LOB
-m,--num-mappers <n> Use n map tasks to import in parallel
-e,--query <statement> Import the results of statement.
--split-by <column-name> Column of the table used to split work units
--table <table-name> Table to read
--target-dir <dir> HDFS destination dir
--warehouse-dir <dir> HDFS parent for table destination
--where <where clause> WHERE clause to use during import
-z,--compress Enable compression
--compression-codec <c> Use Hadoop codec (default gzip)
--null-string <null-string> The string to be written for a null value for string columns
--null-non-string <null-string> The string to be written for a null value for non-string columns

Selecting the Data to Import
--table : argument to select the table to import
--columns : argument to select a subset of columns and control their ordering
--where : control which rows are imported by adding a SQL WHERE clause to the import statement
--query : Instead of using the --table, --columns and --where arguments, you can specify a SQL statement with the --query argument.

$CONDITIONS :
If you want to import the results of a query in parallel, then each map task will need to execute a copy of the query, with results partitioned by bounding conditions inferred by Sqoop. Your query must include the token $CONDITIONS which each Sqoop process will replace with a unique condition expression. You must also select a splitting column with --split-by.

the query can be executed once and imported serially, by specifying a single map task with -m 1:

-m : Sqoop imports data in parallel from most database sources. You can specify the number of map tasks (parallel processes) to use to perform the import by using the -m or --num-mappers argument.
When performing parallel imports, Sqoop needs a criterion by which it can split the workload. Sqoop uses a splitting column to split the workload. By default, Sqoop will identify the primary key column (if present) in a table and use it as the splitting column.

--warehouse-dir : By default, Sqoop will import a table named foo to a directory named foo inside your home directory in HDFS. For example, if your username is someuser, then the import tool will write to /user/someuser/foo/(files). You can adjust the parent directory of the import with the --warehouse-dir argument.

Incremental Imports : Incremental import mode which can be used to retrieve only rows newer than some previously-imported set of rows.
--check-column (col) Specifies the column to be examined when determining which rows to import. (the column should not be of type CHAR/NCHAR/VARCHAR/VARNCHAR/ LONGVARCHAR/LONGNVARCHAR)
--incremental (mode) Specifies how Sqoop determines which rows are new. Legal values for mode include append and lastmodified.
--last-value (value) Specifies the maximum value of the check column from the previous import.

File Formats : two file formats: delimited text or SequenceFiles.
1.Delimited text is the default import format. You can also specify it explicitly by using the --as-textfile argument.
2.SequenceFiles are a binary format that store individual records in custom record-specific data types.
3.Avro data files are a compact, efficient binary format that provides interoperability with applications written in other programming languages. Avro also supports versioning, so that when, e.g., columns are added or removed from a table, previously imported data files can be processed along with new ones.

-z or --compress : compress your data by using the deflate (gzip) algorithm with the -z or --compress argument,

Examples :

1. basic import of a table named employees

sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr --table employees;

2. basic import requiring a login:

sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --username root -P --table employees;

3. Selecting specific columns from the EMPLOYEES table

sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr --table employees --columns "employee_id, first_name, last_name";

4. Controlling the import parallelism (using 8 parallel tasks):

sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr --table employees --columns "employee_id, first_name, last_name" -m 8;

5. Specifying the delimiters to use in a text-mode import:

sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr --table employees --columns "employee_id, first_name, last_name" --fields-terminated-by '\t' --lines-terminated-by '\n' --optionally-enclosed-by '\"';

6. importing data to a target directory "/abhijit"

sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr --table employees --columns "employee_id, first_name, last_name" --target-dir /abhijit;

7. Import subset of data.
sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr --table employees --where 'salary>10000' --target-dir /abhijit/subset;

8. Import Subset with specific columns

sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr --table employees --where 'salary>10000' --columns "employee_id, first_name, last_name" --target-dir /abhijit/subset;

9. IMport Subset with Specifying the delimiters

sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr --table employees --where 'salary>10000' --target-dir /abhijit/subset/terminated --fields-terminated-by '\t' --lines-terminated-by '\n';

10. An incremental import of new data

sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr --table employees --check-column employee_id --incremental append --last-value 208;

11. Import to sequenceFile format

sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr --table employees --where 'salary>10000' --target-dir /abhijit/subset/sequenceFile --as-sequencefile;

12. Import to avrodatafile format

sqoop import --connect jdbc:mysql://localhost:3306/vaibhav --username root --password hr --table employees --where 'salary>10000' --target-dir /abhijit/subset/avroFile --as-avrodatafile;

12. sqoop-import-all-tables
The import-all-tables tool imports a set of tables from an RDBMS to HDFS. Data from each table is stored in a separate directory in HDFS.
For the import-all-tables tool to be useful, the following conditions must be met:

Each table must have a single-column primary key.
You must intend to import all columns of each table.
You must not intend to use non-default splitting column, nor impose any conditions via a WHERE clause.

===============================================================================================================================

Importing Data Into Hive

Argument Description
--hive-home <dir> Override $HIVE_HOME
--hive-import Import tables into Hive (Uses Hive’s default delimiters if none are set.)
--hive-overwrite Overwrite existing data in the Hive table.
--create-hive-table If set, then the job will fail if the target hive
table exits. By default this property is false.

--hive-table <table-name> Sets the table name to use when importing to Hive.
--hive-drop-import-delims Drops \n, \r, and \01 from string fields when importing to Hive.
--hive-delims-replacement Replace \n, \r, and \01 from string fields with user defined string when importing to Hive.
--hive-partition-key Name of a hive field to partition are sharded on
--hive-partition-value <v> String-value that serves as partition key for this imported into hive in this job.
--map-column-hive <map> Override default mapping from SQL type to Hive type for configured columns.

Sqoop will by default import NULL values as string null. Hive is however using string \N to denote NULL values and therefore predicates dealing with NULL (like IS NULL) will not work correctly. You should append parameters --null-string and --null-non-string in case of import job or --input-null-string and --input-null-non-string in case of an export job if you wish to properly preserve NULL values.

Hive can put data into partitions for more efficient query performance. You can tell a Sqoop job to import data for Hive into a particular partition by specifying the --hive-partition-key and --hive-partition-value arguments. The partition value must be a string.

How does count work using apache-pig?

2016-04-06T05:42:00.001-07:00

I am learning pig and came across some question like how grouping works and how to get the count.
I thought to put some example which could be useful for others when they go for doing some grouping and count out of it.

I am not going to cover what is pig and the advantages of it. Will directly jump on with an example and try to elaborate more on to it.

Let's start with it.

Here is sample data for the example.

1,The Nightmare Before Christmas,1993,3.9,4568
2,The Mummy,1932,3.5,4388
3,Orphans of the Storm,1921,3.2,9062
4,The Object of Beauty,1991,2.8,6150
5,Night Tide,1963,2.8,5126
6,One Magic Christmas,1985,3.8,5333
7,Muriel's Wedding,1994,3.5,6323
8,Mother's Boys,1994,3.4,5733
9,Nosferatu: Original Version,1929,3.5,5651
10,Nick of Time,1995,3.4,5333

The data is related with movies as movie name, release year, rating for the movie and number of likes.

Our assignment is to get the count of movies for a year.
Below are steps to be taken to analyse the data.

1. Loading Data : Will use the Load function to read the data into Pig. Here Load is just a pointer and does not load the data. When Store or Dump is requested then only pig starts with an execution.

movies = load '/home/abhijit/Downloads/movies.txt' Using PigStorage(',');

2. To verify the output or for testing/debugging of your changes you can use DUMP.
DUMP displays the results on to the terminal.(Never use the same on production. Use the STORE instead)

DUMP movies;

The result on the terminal is :

(1,The Nightmare Before Christmas,1993,3.9,4568)
(2,The Mummy,1932,3.5,4388)
(3,Orphans of the Storm,1921,3.2,9062)
(4,The Object of Beauty,1991,2.8,6150)
(5,Night Tide,1963,2.8,5126)
(6,One Magic Christmas,1985,3.8,5333)
(7,Muriel's Wedding,1994,3.5,6323)
(8,Mother's Boys,1994,3.4,5733)
(9,Nosferatu: Original Version,1929,3.5,5651)
(10,Nick of Time,1995,3.4,5333)

3. Group the data by key year. We will use group operator for the same. It group together the tuples that have same key, which will be called group key. In our case release year is the group key.

groupByYear = group movies by $2;

$2 here is the position that represents the value of release year(We are using the position here as we have not created any schema. We will talk about schema later).

Again call DUMP to verify the result.

DUMP groupByYear ;

The result on the terminal is :

(1921,{(3,Orphans of the Storm,1921,3.2,9062)})
(1929,{(9,Nosferatu: Original Version,1929,3.5,5651)})
(1932,{(2,The Mummy,1932,3.5,4388)})
(1963,{(5,Night Tide,1963,2.8,5126)})
(1985,{(6,One Magic Christmas,1985,3.8,5333)})
(1991,{(4,The Object of Beauty,1991,2.8,6150)})
(1993,{(1,The Nightmare Before Christmas,1993,3.9,4568)})
(1994,{(7,Muriel's Wedding,1994,3.5,6323),(8,Mother's Boys,1994,3.4,5733)})
(1995,{(10,Nick of Time,1995,3.4,5333)})

4. Now from the output it is clear why we do grouping before we go for the count.

(1921,{(3,Orphans of the Storm,1921,3.2,9062)})

1921 signifies the key and (3,Orphans of the Storm,1921,3.2,9062) is the Tuple or the value of it. Here we have single tuple in the form of (3,Orphans of the Storm,1921,3.2,9062).

Tuple is an ordered set of fields.
(3,Orphans of the Storm,1921,3.2,9062).

A bag is a collection of Tuples
Representation of bag is
{(3,Orphans of the Storm,1921,3.2,9062)}

Now we want to get the count of the tuples or the values of key.

movieCountOfYear = foreach groupByYear generate $0, COUNT($1);

$0 represents the key and $1 represents the value.

To verify the result call DUMP.

DUMP movieCountOfYear;

(1921,1)
(1929,1)
(1932,1)
(1963,1)
(1985,1)
(1991,1)
(1993,1)
(1994,2)
(1995,1)

In the result set you can see by the year and the respective movie count of it.

How to update the email ids for the thousand users through database.

2015-12-06T22:49:00.002-08:00

How to update the email ids for the thousand users through database.

Updating email ids means updating only the domain part of the email.

example a user has email as bc.xyz@bajajauto.com and now has to update as abc.xyz@atulauto.com

Here you need to update the part of the email id.

I have written the function for the same.

BEGIN
FOR user_ids in (SELECT user_id FROM user usr where usr.email like '%@bajajauto.com%')
LOOP
FOR user_email_id in (
SELECT SUBSTR(t.email, 0, INSTR(t.email, '@')-1) AS part_of_email, t.user_id FROM user t where t.user_id = user_ids.user_id)
)
LOOP
update user user2 set user2.email = user_email_id.part_of_email || '@atulauto.com' WHERE user2.user_id = user_email_id.user_id;
END LOOP;
END LOOP;
END;

Hope this would be useful. Let me know if it could be improved. Also if anyone knows any other quicker solution.

Installing the MySQL workbench on Ubuntu...

2015-02-26T06:19:00.002-08:00

You can get the necessary mysql workbench deb

"http://dev.mysql.com/downloads/workbench/#downloads"

First run the following:

sudo dpkg -i mysql-workbench-gpl-5.2.40-1ubu1104-i386.deb

As you know this will fail about the dependencies. Then wget and install the following:

wget https://launchpad.net/ubuntu/+archive/primary/+files/libzip1_0.9.3-1_i386.deb

wget http://launchpadlibrarian.net/94563300/libmysqlclient16_5.1.58-1ubuntu5_i386.deb

Next install:

sudo dpkg -i libzip1_0.9.3-1_i386.deb

sudo dpkg -i libmysqlclient16_5.1.58-1ubuntu5_i386.deb

If you're using 64bit, you'll want this instead:

wget https://launchpad.net/ubuntu/+archive/primary/+files/libzip1_0.9.3-1_amd64.deb

sudo dpkg -i libzip1_0.9.3-1_amd64.deb

wget http://launchpadlibrarian.net/94808408/libmysqlclient16_5.1.58-1ubuntu5_amd64.deb

sudo dpkg -i libmysqlclient16_5.1.58-1ubuntu5_amd64.deb

Then run the following:

sudo apt-get -f install python-paramiko libzip2 libmysqlclient18 mysql-client mysql-common python-pysqlite2 libdbi-perl libdbd-mysql-perl libplrpc-perl libnet-daemon-perl mysql-client

This too will fail, and ask you to re-run, just re-run it again:

sudo apt-get -f install

Launch it once from either your terminal mysql-workbench or Unity Dash, once it's in the launcher, right click on it and select "Lock to Launcher".

Solr Revisited...

2015-01-30T03:01:00.003-08:00

Long back I had written blogs on solr setup and how I used it in my application.
Meanwhile I am getting questions/quries from the readers on the same.
And these readers make you to work on it...revisit the technologies or stuffs your had worked...

The queries are like I am searching "Solr Blog" as text but getting the results of "Solr" as well
"Solr indexing" or my blah blah search text is not getting the correct result.
After going through all these quries I realised that where it is going wrong.
Or rather how my blogs are not helping them to resolve their issues...
So I decided to write another one which may ease their job or guide them working on solr and
will brush up my knowledge on the same subject :)

Lets go with the real example, which made me to pen down this blog ...
A reader asked me a question :
"If he search for "Abhijit & his blogs", It should show the exact match as "Abhijit & his blogs" and not matches like "Abhijit" or
or "Abhijit Songs"... ".

On this what all questions comes to your mind...? What would be wrong here ...?
any idea...?

There is simple theory ...what you index, will be available for search... :)
In short you are using a wrong analyzer for indexing and quering...

Find out which is the fieldType been used for indexing and querying....
Is it the default provided by Solr or you have written your own custom field type(created by using the
available tokenizers and filtes)..

For the above example turn off the stemming ...
The solutions for the problem would be...

<types>

<fieldType name="text_no_stem" class="solr.TextField" omitNorms="false">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>

</types>

<fields>
<dynamicField name="*_nostem" type="text_no_stem" indexed="true" stored="true"/>
</fields>

Or Use solr string field whcih will do an exact value search e.g

<fieldType class="solr.StrField" name="string" omitNorms="true" sortMissingLast="true" />

Now coming to the point what I realised ..?

That is we need to know more on the role tokenizers and filtes in the analyzers.
Which all types of tokenizers and filtes are available...how to make use of it and when to use it..
How to test the same...?

Here is the brief about the three important things

1. Analyzer : They pre-process the test at the time of indexing and quering(or search). Ans yes make sure you are using same analyzers that for both index and query.
For example, if an indexing analyzer lowercases words, then the query analyzer should do the same to enable finding the indexed words.

2. Tokenizers : It splits/seperates stream of characters into a series of tokens or small chunks/words. There can be only one Tokenizer in each Analyzer.
There are different tokenizer available to use. For example..
KeywordTokenizerFactory
LetterTokenizerFactory
WhitespaceTokenizerFactory
StandardTokenizerFactory
LowerCaseTokenizerFactory

3. Filters : This takes a granular level of tokes. doing many changes to it before indexing it.
There are different filters available to use..
LowerCaseFilterFactory
ClassicFilterFactory
StopFilterFactory
EdgeNGramFilterFactory

and many more are available.

You should be very precise about how and what is the way you want your search should work.
Then its very easy to decide the fieldType ...rather very easy to create your own type and use it.

While doing any sort of analysis do use the tool given by solr i.e analysis.jsp.
It will help you a lot for resolving your problems...

If you want to read more about the solr's tokenizer's and filter's refere the solr Wiki which has a lot info
https://wiki.apache.org/solr/

Please feel free to put on your view on it...

Installing jdk 1.5 on ubantu

2014-12-07T23:43:00.006-08:00

Installing jdk 1.5 on ubantu

Here uou can keep multiple versions of Java (JRE and JDK) at a time and choose between them which one to use.
on the below link download your version of Java.

http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase5-419410.html

I have download the following file;

Java Development Kit 5.0 Update 22 42.14 MB jdk-1_5_0_22-linux-amd64.bin

Go to the termial and go the the path where you downloaded the file.

type the following command;

sudo mv jdk-1_5_0_22-linux-amd64.bin /usr/lib/jvm
cd /usr/lib/jvm
chmod +x jdk-1_5_0_22-linux-amd64.bin
sudo ./jdk-1_5_0_22-linux-amd64.bin

sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.5.0_22/bin/java" 2

sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.5.0_22/bin/javac" 2

You have installed the jdk 1.5 Successfully

If you two verions of jdk you can use following commands to switch them

sudo update-alternatives --config java
sudo update-alternatives --config javac

Confirm the version of Java currently in use with these commands:

java -version
javac -version

Cheers!!!

Storing POJOs in MongoDB Using Morphia

2014-11-14T04:01:00.000-08:00

Storing POJOs in MongoDB Using Morphia:

Storing data in any Database is always a challenge irrespective of the Datadabase.

MongoDB is a leading NoSQL Database.
We create pojo/entity and store the data in the Database using framework like Hibernate.

Similarly we have Morphia which helps to store the data in the MongoDB.
In this we will try to store the Person data in the MongoDB.
Person will have name, address and the hobbies.

To store the above mentioned person data, we will create the entities.
Will create the entities for Person Entity, Hobbies Entity and the Address Entity.

We will embed the Hobbies and Address entity into Person.

Entity Annotation : To Store the class in MongoDB thro' Morphia, add annotate to class as @Entity

To Suppress the class name from the Mongo Document use "noClassnameStored=true".

@Id : Entity require unique @Id values; these values are stored in the MongoDB "_id" field.

@Indexed : This annotation applies an index to a field.
The indexes are applied when the datastore.ensureIndexes() method is called

Example : @Indexed(value=IndexDirection.ASC, name="userName", unique=true, dropDups=true)

@Embedded : to create a class that will be embedded in the Entity.

Below is the code sample of the mentioned Entities.

package com.mongo.morphia.entity;

import java.util.List;

import org.bson.types.ObjectId;

import com.google.code.morphia.annotations.Embedded;
import com.google.code.morphia.annotations.Entity;
import com.google.code.morphia.annotations.Id;
import com.google.code.morphia.annotations.Property;

@Entity(noClassnameStored=true)
public class Person {

@Id
@Property("id")
private ObjectId id;

private String name;
private List<Hobbies> hobbies;
@Embedded
private Address address;

public ObjectId getId() {
return id;
}

public void setId(ObjectId id) {
this.id = id;
}

public String getName() {
return name;
}

public void setName(String name) {
this.name = name;
}

public List<Hobbies> getHobbiess() {
return hobbies;
}

public void setHobbies(List<Hobbies> hobbies) {
this.hobbies = hobbies;
}

public Address getAddress() {
return address;
}

public void setAddress(Address address) {
this.address = address;
}

}

package com.mongo.morphia.entity;

import com.google.code.morphia.annotations.Embedded;

@Embedded
public class Address {
private String number;
private String street;
private String town;
private String postcode;

public String getNumber() {
return number;
}

public void setNumber(String number) {
this.number = number;
}

public String getStreet() {
return street;
}

public void setStreet(String street) {
this.street = street;
}

public String getTown() {
return town;
}

public void setTown(String town) {
this.town = town;
}

public String getPostcode() {
return postcode;
}

public void setPostcode(String postcode) {
this.postcode = postcode;
}

}

package com.mongo.morphia.entity;

import org.bson.types.ObjectId;

import com.google.code.morphia.annotations.Entity;
import com.google.code.morphia.annotations.Id;
import com.google.code.morphia.annotations.Property;

@Entity
public class Hobbies {

@Id
@Property("id")
private ObjectId id;

private String name;

public String getName() {
return name;
}

public void setName(String name) {
this.name = name;
}

public ObjectId getId() {
return id;
}

public void setId(ObjectId id) {
this.id = id;
}
}

DAO class..

package com.mongo.morphia.dao;

import com.google.code.morphia.Morphia;

import com.google.code.morphia.dao.BasicDAO;

import com.mongo.morphia.entity.Person;

import com.mongodb.Mongo;

public class PersonDAO extends BasicDAO<Person, String> {

public PersonDAO(Morphia morphia, Mongo mongo, String dbName) {

super(mongo, morphia, dbName);

}

In the below class we have called the save , update , find and Delete operation. here we are using the entities to do all the operations.

package com.mongo.morphia;

import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.List;

import com.google.code.morphia.Datastore;
import com.google.code.morphia.Morphia;
import com.google.code.morphia.mapping.Mapper;
import com.google.code.morphia.query.Query;
import com.google.code.morphia.query.QueryResults;
import com.google.code.morphia.query.UpdateOperations;
import com.mongo.morphia.dao.PersonDAO;
import com.mongo.morphia.entity.Hobbies;
import com.mongo.morphia.entity.Address;
import com.mongo.morphia.entity.Person;
import com.mongodb.Mongo;
import com.mongodb.MongoException;

public class Example {

public static void main(String[] args) throws UnknownHostException,MongoException {

String dbName = new String("User");
Mongo mongo = new Mongo();
Morphia morphia = new Morphia();
Datastore datastore = morphia.createDatastore(mongo, dbName);

morphia.mapPackage("com.mongo.morphia.entity");

Address address = new Address();
address.setNumber("Q-101");
address.setStreet("M.G. Road");
address.setTown("Pune");
address.setPostcode("411001");

Hobbies hobbies = new Hobbies();
hobbies.setName("Trekking");

List<Hobbies> listHobbies = new ArrayList<Hobbies>();
listHobbies.add(hobbies);

Person person = new Person();
person.setAddress(address);
person.setName("Mr Martin d'souza");
person.setHobbies(listHobbies);

//Save Methods
PersonDAO personDAO = new PersonDAO(morphia, mongo, dbName);
personDAO.save(person);

//Find Methods
//use in a loop
for(Person personObj : datastore.find(Person.class, "hobbies.name", "Trekking")) {
System.out.println(personObj.getName());
}

//get back as a list
//List<Person> personObj1 = datastore.find(Person.class, "name", "Martin").asList();

// Get the Data Using Query Criteria
Query<Person> query = datastore.createQuery(Person.class);
query.and(query.criteria("hobbies.name").equal("Trekking"),
query.criteria("address.number").equal("Q-101"),
query.criteria("name").contains("Martin"));

QueryResults<Person> retrievedPersons = personDAO.find(query);

for (Person retrievedPerson : retrievedPersons) {
System.out.println(retrievedPerson.getName());
System.out.println(retrievedPerson.getAddress().getPostcode());
System.out.println(retrievedPerson.getHobbiess().get(0).getName());
}

//Update Methods
for (Person retrievedPerson : retrievedPersons) {

UpdateOperations<Person> updatePerson = datastore.createUpdateOperations(Person.class).set("name", "Kevin Gibbs");
datastore.update(datastore.createQuery(Person.class).field(Mapper.ID_KEY).equal(retrievedPerson.getId()), updatePerson);

//Delete Methods
//personDAO.delete(retrievedCustomer);
}
}

}

If You check the Mongo console you will find the below output ...

db.Person.find().pretty();

{

"_id" : ObjectId("5465ddf2e4b02b54a64d9fdd"),

"address" : {

"number" : "Q-101",

"street" : "M.G. Road",

"town" : "Pune",

"postcode" : "411001"

"hobbies" : [

{

"name" : "Trekking"

}

"name" : "Kevin Gibbs"

}

Java To Python.......

2014-11-13T06:22:00.001-08:00

Learning a new thing is always a fun... then whatever it is ... you enjoy it...

I am learning Python now...I knew java and use it for building the application as of now....

While learning the Python, it doesn't seems to hard with respect to the syntax..its very easy..

its like plain text sort of thing being learnt...

Python's simple and straight-forward syntax also encourages good programming habbits, especially through its focus on white space indentation, which contributes to the development of neat looking code.

We try to put out some difference between Python and Java...

Before that I am not criticizing any language here, just wanted you to enjoy this as I had...

1. If you want to print the "Hello World" in java, it looks something like this

public class TestJava {

public static void main(String[] args) {

System.out.println("Hello World");

}

and the same thing in Python is

print "Hello, Python!";

2. When it comes to some reserved words ..then yes both the languages has some keys reserved. I dint find any difference here. And the view is same that the reserved words not be used as constant or variable or any other identifier names.

3. Next is line and indentation where In Python the fact is that there are no braces to indicate blocks of code for class and function definitions or flow control. Blocks of code are denoted by line indentation, which is rigidly enforced.

The number of spaces in the indentation is variable, but all statements within the block must be indented the same amount.

Both blocks in this example are fine:

if True:

print "True"

else:

print "False"

and the below block in this example will generate an error:

if True:

print "Answer"

print "True"

else:

print "Answer"

print "False"

Thus, in Python all the continuous lines indented with similar number of spaces would form a block.

4. Quotation in Python: Python accepts single ('), double (") and triple (''' or """) quotes to denote string literals, as long as the same type of quote starts and ends the string.

Same is not the case with Java while defining the String literals.. :)

5. Commenting the code in Python : A hash sign (#) that is not inside a string literal begins a comment.

All characters after the # and up to the physical line end are part of the comment and the Python interpreter ignores them.

In java Single line commenting is done by "//" and multiple lines commenting is by /* */. Anything that comes between /* */ will be ignored.

6. Values Assigning to Variable : In Python

count = 100 # integer assignment

distance = 10.5 # floating

name = "Steve" # String

In Java

String name = "xyx";

Integer count = 100;

Difference is you need to mention the data type along with the variable name.

7. Multiple Assignment:

Python allows you to assign a single value to several variables simultaneously.

For example:

a = b = c = 1

Same thing is not possible in Java.

You can also assign multiple objects to multiple variables.

For example:

a, b, c = 1, 2, "john"

and if you print it the output is

print a,b,c

1 2 john

8. Lists : Lists are the most versatile of Python's compound data types.

A list contains items separated by commas and enclosed within square brackets ([]).

In java you have different collections like ArrayList and Hashset etc.

One difference between them is that all the items belonging to a list in Python can be of different data type.

In Java it does too but seems hard when you introduce Generics.

9. Value access from List :

In Python the values stored in a list can be accessed using the slice operator ( [ ] and [ : ] ) with

indexes starting at 0 in the beginning of the list and working their way to end -1.

list = [ 'abcd', 786 , 2.23, 'john', 70.2 ]

print list # Prints complete list

print list[0] # Prints first element of the list

print list[1:3] # Prints elements starting from 2nd till 3rd

In Java use of Iterator or new for each would be used to get the data.

10. Defining a Function

Here are simple rules to define a function in Python.

Function blocks begin with the keyword def followed by the function name and parentheses ( ( ) ).

Any input parameters or arguments should be placed within these parentheses. You can also define parameters inside these parentheses.

The first statement of a function can be an optional statement - the documentation string of the function or docstring.

The code block within every function starts with a colon (:) and is indented.

The statement return [expression] exits a function, optionally passing back an expression to the caller.

A return statement with no arguments is the same as return None.

For Example :

def printme( str ):

"This prints a passed string into this function"

print str

return

In Java :

public void test(Integer i){

System.out.println("funtion" + i);

}

I would appreciate if anyone add more if they knew and correct me if I am wrong at any point...

Till then cheers...

Generating plain xml from Jasper Irrespective of the Locale

2014-06-03T02:11:00.002-07:00

Generating plain xml from the jasper report has another problem for me.

If the locale is changed then the column names will get changed.

As I want the xml for the Data Feed, it would not work as the xml parsing
mechanism will change as the locale will change.

So to achieve the same i.e. to get the xml attribute in English all the time
irrespective of the locale, need to do some changes in the code.

Here is the modified exportText method.

public void exportText(JRPrintText text) throws IOException
{

JRStyle style = text.getStyle();
if (style != null){
String styleName = style.getName();
if(!isColumn && "TableColumnHeaderTextStyle".equals(styleName)){
columnMap.put(columnCount, text.getOriginalText());
columnCount++;
}else if(!isRow && "TableDetailTextStyle".equals(styleName)){
isColumn = true;
isRow = true;
}
}

if(isRow){
if (!isNewColumn && text.hasProperties())
{
JRPropertiesMap propertiesMap = text.getPropertiesMap();
String[] propertyNames = propertiesMap.getPropertyNames();
if (propertyNames != null && propertyNames.length > 0)
{
for(int i = 0; i < propertyNames.length; i++)
{
String value = propertiesMap.getProperty(propertyNames[i]);
if (value != null)
{
int p = value.indexOf('.');
String regex = "[_][0-9][0-9][0-9]";
if (p >= 0) {
String right = value.substring(p + 1);
String []val = right.split(regex);
String modValue = val[0];
if(newColumnMap.containsValue(modValue)){
isNewColumn = true;
}else{
newColumnMap.put(newCoulmnCount, modValue);
newCoulmnCount++ ;
}
}
}
}
}
}

if(style != null && "TableDetailTextStyle".equals(style.getName())){
isRowClosed = false;
if(rowCount == 0){
xmlWriter.startElement(ELEMENT_row);
}

if(null == text.getOriginalText()){
System.out.println("text value is null");
}

String attributName = newColumnMap.get(rowCount);
xmlWriter.addEncodedAttribute(attributName, text.getOriginalText());
rowCount++;
}

if(rowCount == columnCount){
isRowClosed = true;
xmlWriter.closeElement();
rowCount = 0;
}
}

}

XML parsing Issues

2014-06-01T22:11:00.001-07:00

While working on Generating XML output/export in jasper, I came across some xml issues.

Xml generation was successful, but parsing the same in browsers had some issues.

Listing out those issues here and the solution for the same.

Every Browser show the error or explain it differently.

Here I am showing the errors by crome and Firefox.

Issue 1:

On Crome :
This page contains the following errors:
error on line 4 at column 60: Specification mandate value for attribute Size
Below is a rendering of the page up to the first error.

Here Size is the attribute name, being used by me in the xml.
The original column name in jasper was Size(Kb)

On FireFox :
XML Parsing Error: not well-formed
Location: http://10.55.36.34:8080/xyz?id=dT01ODg5NyZwPTEzODI4JnI9VGVzdCBSZXBvcnQmaj0vOTYzNDctdXNyLWZvbC11cmkvMTM4MjgtdXNyLXdzLXJlcG9ydHMtZm9sLXVyaS9UZXN0X1JlcG9ydCZkPTE0MDA3MzkxODYwODgmYz0xNjgyMjA4NTg2
Line Number 4, Column 171:
<row Reference="abc-16316AD-2411101F000" Action_Date="18/03/2014 11:45" Title="audit.png" User_Name="SEND NO MESSAGES KEINE NACHRICHT SENDEN" Content_Id="21565941" Size(Kb)="72"

FireFox highlights the error by pointing to the attribute "Size(Kb)". The attribute contains the special character.

By removing/replacing the special character "(", the above issue gets resolved.

Issue 2:

On crome :
This page contains the following errors:
error on line 6843 at column 19: Extra content at the end of the document
Below is a rendering of the page up to the first error.

On FireFox :
XML Parsing Error: no element found
Location: http://10.55.36.34:8080/xyz?id=dT01ODg5NyZwPTEzODI4JnI9RG9jdW1lbnQgRGlzdHJpYnV0aW9uIFYxIFJlcG9ydCZqPS85NjM0Ny11c3ItZm9sLXVyaS8xMzgyOC11c3Itd3MtcmVwb3J0cy1mb2wtdXJpL0RvY3VtZW50X0Rpc3RyaWJ1dGlvbl9WMV9SZXBvcnQmZD0xNDAwNzUxODUzNDUzJmM9MzI3NjMxMDE0Mg
Line Number 6844, Column 1:

The error was because of xml root node is missing.

Issue 3:

Mozilla....

XML Parsing Error: duplicate attribute
Location: http://10.55.36.34:8080/BIReportDataFeed?id=dT01ODg5NyZwPTEzODI4Jmo9Lzk2MzQ3LXVzci1mb2wtdXJpLzEzODI4LXVzci13cy1yZXBvcnRzLWZvbC11cmkvRG9jdW1lbnRfTWV0YWRhdGEmZD0xNDAwODUyNzU5Mjg2JmM9MjUzMTIwMTg0NQ
Line Number 4, Column 42: <row Reference="GER-BRU-7" Property="" Property="" Property="" Property="" Property="" Property="" Property="" Property="" Property=""/>

This is very straight forward. Xml does not allow duplicate attributes.

Generate Xml export from Jasper for the data feed in Excel.

2014-05-21T00:13:00.003-07:00

Generate Xml export from Jasper for the data feed in Excel.

Generating xml export from jasper for the data feed.
This data feed or the xml file when imported in excel should generate a report in correct format.

The xml format is

<jasperPrint name="rt demo">
<report-results>
<row Reference="Dummy Reference 12" Date="25/03/2014 16:38" User_Name="Vaibhav joshi" Type_ID="85"/>
<row Reference="Dummy Reference 11" Date="25/03/2014 16:38" User_Name="Vaibhav joshi" Type_ID="85"/>
<row Reference="Dummy Reference 5" Date="25/03/2014 16:34" User_Name="Vaibhav joshi" Type_ID="17"/>
<row Reference="Dummy Reference 5" Date="25/03/2014 16:34" User_Name="Vaibhav joshi" Type_ID="3"/>
<row Reference="Dummy Reference 4" Date="25/03/2014 16:34" User_Name="Vaibhav joshi" Type_ID="3"/>
<row Reference="Dummy Reference 2" Date="21/03/2014 12:15" User_Name="Vaibhav joshi" Type_ID="165"/>
<row Reference="register2.JPG" Date="20/03/2014 14:03" User_Name="Vaibhav joshi" Type_ID="1"/>
<row Reference="Dummy Reference 1" Date="20/03/2014 12:43" User_Name="Vaibhav joshi" Type_ID="17"/>
<row Reference="Dummy Reference 2" Date="20/03/2014 12:43" User_Name="Vaibhav joshi" Type_ID="17"/>
<row Reference="Dummy Reference 6" Date="18/03/2014 12:48" User_Name="Vaibhav joshi" Type_ID="7"/>
<row Reference="Dummy Reference 7" Date="18/03/2014 12:48" User_Name="Vaibhav joshi" Type_ID="7"/>
........
.......
......
</jasperPrint>

The Modified XML Export code to achieve the above xml is :

public class FinalCustomExporter extends JRXmlExporter
{
private static final Log log = LogFactory.getLog(JRXmlExporter.class);
private static final String XML_EXPORTER_PROPERTIES_PREFIX = JRPropertiesUtil.PROPERTY_PREFIX + "export.xml.";
public static final String XML_EXPORTER_KEY = JRPropertiesUtil.PROPERTY_PREFIX + "xml";

public static final String PROPERTY_REPLACE_INVALID_CHARS = JRPropertiesUtil.PROPERTY_PREFIX + "export.xml.replace.invalid.chars";
protected static final String DEFAULT_XML_ENCODING = "UTF-8";
protected static final String DEFAULT_OBJECT_TYPE = "java.lang.String";
protected static final String XML_FILES_SUFFIX = "_files";
protected static final String IMAGE_PREFIX = "img_";
public static final String ELEMENT_row = "row";
public static final String ELEMENT_report_results = "report-results";

public static final XmlNamespace JASPERPRINT_NAMESPACE =
new XmlNamespace(JRXmlConstants.JASPERPRINT_NAMESPACE, null, JRXmlConstants.JASPERPRINT_XSD_SYSTEM_ID);

protected JRXmlWriteHelper xmlWriter;
protected String encoding;
protected String version;
protected VersionComparator versionComparator = new VersionComparator();

protected JRExportProgressMonitor progressMonitor;
protected Map<Renderable,String> rendererToImagePathMap;
protected Map<String,byte[]> imageNameToImageDataMap;
protected Map<String,JRStyle> stylesMap = new HashMap<String,JRStyle>();

protected boolean isEmbeddingImages = true;
protected boolean isColumn = false;
protected boolean isRow = false;
protected Integer columnCount = 0;
protected Integer rowCount = 0;
protected Map<Integer,String> columnMap = new HashMap<Integer, String>();
protected File destFile;
protected File imagesDir;

protected class ExporterContext extends BaseExporterContext implements JRXmlExporterContext
{
public String getExportPropertiesPrefix()
{
return FinalCustomExporter.this.getExporterPropertiesPrefix();
}
}

protected JRXmlExporterContext exporterContext = new ExporterContext();

public FinalCustomExporter()
{
this(DefaultJasperReportsContext.getInstance());
}

public FinalCustomExporter(JasperReportsContext jasperReportsContext)
{
super(jasperReportsContext);
}

public void exportReport() throws JRException
{
progressMonitor = (JRExportProgressMonitor)parameters.get(JRExporterParameter.PROGRESS_MONITOR);

setOffset();

try
{
setExportContext();

setInput();

if (!parameters.containsKey(JRExporterParameter.FILTER))
{
filter = createFilter(getExporterPropertiesPrefix());
}

setPageRange();

@SuppressWarnings("deprecation")
String dtdLocation = (String)parameters.get(JRXmlExporterParameter.DTD_LOCATION);
if (dtdLocation != null)
{
log.warn("The JRXmlExporterParameter.DTD_LOCATION export parameter has no effect and should no longer be used.");
}

encoding = (String)parameters.get(JRExporterParameter.CHARACTER_ENCODING);
if (encoding == null)
{
encoding = DEFAULT_XML_ENCODING;
}

setHyperlinkProducerFactory();

StringBuffer sb = (StringBuffer)parameters.get(JRExporterParameter.OUTPUT_STRING_BUFFER);
if (sb != null)
{
StringBuffer buffer = exportReportToBuffer();
sb.append(buffer.toString());
}
else
{
Writer outWriter = (Writer)parameters.get(JRExporterParameter.OUTPUT_WRITER);
if (outWriter != null)
{
try
{
exportReportToStream(outWriter);
}
catch (IOException e)
{
throw new JRException("Error writing to writer : " + jasperPrint.getName(), e);
}
}
else
{
OutputStream os = (OutputStream)parameters.get(JRExporterParameter.OUTPUT_STREAM);
if (os != null)
{
try
{
exportReportToStream(new OutputStreamWriter(os, encoding));
}
catch (Exception e)
{
throw new JRException("Error writing to OutputStream : " + jasperPrint.getName(), e);
}
}
else
{
destFile = (File)parameters.get(JRExporterParameter.OUTPUT_FILE);
if (destFile == null)
{
String fileName = (String)parameters.get(JRExporterParameter.OUTPUT_FILE_NAME);
if (fileName != null)
{
destFile = new File(fileName);
}
else
{
throw new JRException("No output specified for the exporter.");
}
}

imagesDir = new File(destFile.getParent(), destFile.getName() + XML_FILES_SUFFIX);

Boolean isEmbeddingImagesParameter = (Boolean)parameters.get(JRXmlExporterParameter.IS_EMBEDDING_IMAGES);
if (isEmbeddingImagesParameter == null)
{
isEmbeddingImagesParameter = Boolean.TRUE;
}
isEmbeddingImages = isEmbeddingImagesParameter.booleanValue();

exportReportToFile();
}
}
}
}
finally
{
resetExportContext();
}
}

protected void setHyperlinkProducerFactory()
{
hyperlinkProducerFactory = (JRHyperlinkProducerFactory) parameters.get(JRExporterParameter.HYPERLINK_PRODUCER_FACTORY);
}

protected void exportReportToFile() throws JRException
{
{
rendererToImagePathMap = new HashMap<Renderable,String>();
imageNameToImageDataMap = new HashMap<String,byte[]>();
}

Writer writer = null;
try
{
OutputStream fileOutputStream = new FileOutputStream(destFile);
writer = new BufferedWriter(new OutputStreamWriter(fileOutputStream, encoding));
exportReportToStream(writer);
}
catch (IOException e)
{
throw new JRException("Error writing to file : " + destFile, e);
}
finally
{
if (writer != null)
{
try
{
writer.close();
}
catch(IOException e)
{
}
}
}

if (!isEmbeddingImages)
{
Collection<String> imageNames = imageNameToImageDataMap.keySet();
if (imageNames != null && imageNames.size() > 0)
{
if (!imagesDir.exists())
{
imagesDir.mkdir();
}

for(Iterator<String> it = imageNames.iterator(); it.hasNext();)
{
String imageName = it.next();
byte[] imageData = imageNameToImageDataMap.get(imageName);

File imageFile = new File(imagesDir, imageName);

OutputStream fos = null;
try
{
fos = new FileOutputStream(imageFile);
fos.write(imageData, 0, imageData.length);
}
catch (IOException e)
{
throw new JRException("Error writing to image file : " + imageFile, e);
}
finally
{
if (fos != null)
{
try
{
fos.close();
}
catch(IOException e)
{
}
}
}
}
}
}
}

protected StringBuffer exportReportToBuffer() throws JRException
{
StringWriter buffer = new StringWriter();
try
{
exportReportToStream(buffer);
}
catch (IOException e)
{
throw new JRException("Error while exporting report to buffer", e);
}
return buffer.getBuffer();
}

protected XmlNamespace getNamespace()
{
return JASPERPRINT_NAMESPACE;
}

protected void exportReportToStream(Writer writer) throws JRException, IOException
{
version = JRPropertiesUtil.getInstance(jasperReportsContext).getProperty(jasperPrint, JRXmlBaseWriter.PROPERTY_REPORT_VERSION);

xmlWriter = new JRXmlWriteHelper(writer);

xmlWriter.writeProlog(encoding);

xmlWriter.startElement(JRXmlConstants.ELEMENT_jasperPrint);
xmlWriter.addEncodedAttribute(JRXmlConstants.ATTRIBUTE_name, jasperPrint.getName());

List<JRPrintPage> pages = jasperPrint.getPages();
if (pages != null && pages.size() > 0)
{
JRPrintPage page = null;
for(int i = startPageIndex; i <= endPageIndex; i++)
{
if (Thread.interrupted())
{
throw new JRException("Current thread interrupted.");
}

page = pages.get(i);

exportPage(page);
}

if(isRow){
xmlWriter.closeElement();
isRow = false;
}
}

xmlWriter.closeElement();

writer.flush();
}

protected void exportPage(JRPrintPage page) throws JRException, IOException
{
exportElements(page.getElements());

if (progressMonitor != null)
{
progressMonitor.afterPageExport();
}
}

protected void exportElements(Collection<JRPrintElement> elements) throws IOException, JRException
{
if (elements != null && elements.size() > 0)
{
for(Iterator<JRPrintElement> it = elements.iterator(); it.hasNext();)
{
exportElement(it.next());
}
}
}

public void exportElement(JRPrintElement element) throws IOException, JRException
{
if (filter == null || filter.isToExport(element))
{
if (element instanceof JRPrintText)
{
exportText((JRPrintText)element);
}
else if (element instanceof JRPrintFrame)
{
exportFrame((JRPrintFrame) element);
}
}
}

public void exportText(JRPrintText text) throws IOException
{
JRStyle style = text.getStyle();
if (style != null){
String styleName = style.getName();
if(!isColumn && "TableColumnHeaderTextStyle".equals(styleName)){
columnMap.put(columnCount, text.getOriginalText());
columnCount++;
}else if(!isRow && "TableDetailTextStyle".equals(styleName)){
isColumn = true;
isRow = true;
xmlWriter.startElement(ELEMENT_report_results);
}
}

if(isRow && style != null && "TableDetailTextStyle".equals(style.getName())){

if(rowCount == 0){
xmlWriter.startElement(ELEMENT_row);
}

if(null == text.getOriginalText()){
System.out.println("text value is null");
}

String attributName = columnMap.get(rowCount);
String replaceAttribName = attributName.replace(" ", "_");
xmlWriter.addEncodedAttribute(replaceAttribName, text.getOriginalText());
rowCount++;
}

if(rowCount == columnCount){
xmlWriter.closeElement();
rowCount = 0;
}
}

protected void exportFrame(JRPrintFrame frame) throws IOException, JRException
{
setFrameElementsOffset(frame, true);
try
{
exportElements(frame.getElements());
}
finally
{
restoreElementOffsets();
}
}

protected String getExporterPropertiesPrefix()
{
return XML_EXPORTER_PROPERTIES_PREFIX;
}

public String getExporterKey()
{
return XML_EXPORTER_KEY;
}

public JRXmlWriteHelper getXmlWriteHelper()
{
return xmlWriter;
}

protected boolean isNewerVersionOrEqual(String oldVersion)
{
return versionComparator.compare(version, oldVersion) >= 0;
}
}

Here is the output in Excel looks alike...

Generate XML Output from Jasper Reports

2014-05-20T02:26:00.001-07:00

Generate XML Output from Jasper Reports :

I was trying for generating/exporting the data in xml from Jasper.
It generates the data in xml but along with the text content it generates the other tag/layout data
in xml as which is the layout of the report...like style tag and all... which is not required.

So we modified the code to achieve xml export.

The xml ouput we were getting from the jasper was something like this

<jasperPrint xmlns="http://jasperreports.sourceforge.net/jasperreports/print" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://jasperreports.sourceforge.net/jasperreports/print http://jasperreports.sourceforge.net/xsd/jasperprint.xsd" name="rt demo" pageWidth="665" pageHeight="792" topMargin="20" leftMargin="20" bottomMargin="20" rightMargin="20" locale="en_US" timezone="Asia/Calcutta">
<property name="net.sf.jasperreports.export.xml.start.page.index" value="0"/>
<property name="net.sf.jasperreports.export.xml.end.page.index" value="2"/>
<property name="net.sf.jasperreports.export.xml.page.count" value="3"/>
<property name="net.sf.jasperreports.export.xls.collapse.row.span" value="false"/>
<property name="net.sf.jasperreports.export.html.border.collapse" value="separate"/>
<property name="net.sf.jasperreports.export.docx.frames.as.nested.tables" value="false"/>
<property name="net.sf.jasperreports.export.pdf.tagged" value="false"/>
<property name="net.sf.jasperreports.export.pdf.tag.language" value="EN-US"/>
<origin group="newSet2.CATEGORY_NAME" band="groupHeader"/>
<origin group="newSet2.CATEGORY_NAME" band="groupFooter"/>
<origin band="title"/>
<origin band="detail"/>
<origin band="summary"/>
<origin report="rt32demo_tableDataset_1400134558392_604720" group="newSet2.CATEGORY_NAME" band="groupHeader"/>
<origin report="rt32demo_tableDataset_1400134558392_604720" group="newSet2.CATEGORY_NAME" band="groupFooter"/>
<origin report="rt32demo_tableDataset_1400134558392_604720" band="columnHeader"/>
<origin report="rt32demo_tableDataset_1400134558392_604720" band="detail"/>
<style name="ReportDefault" isDefault="true" mode="Transparent" forecolor="#666666" backcolor="#FFFFFF" hAlign="Left" vAlign="Middle" fontName="DejaVu Sans" fontSize="11">...</style>
<style name="TableBaseFrameStyle" mode="Transparent">...</style>
<style name="TableFrameStyle" style="TableBaseFrameStyle">...</style>
<style name="TableColumnHeaderFrameStyle" style="TableBaseFrameStyle">...</style>
<style name="TableColumnFooterFrameStyle" style="TableBaseFrameStyle">...</style>
<style name="TableColumnHeaderTextStyle" style="ReportDefault" mode="Opaque" forecolor="#666666" backcolor="#D5DEE8" hAlign="Left" vAlign="Middle" fontName="DejaVu Sans" fontSize="11" isBold="true">...</style>
.....
......
<style name="ChartSeriesColor0" backcolor="#87C4FE"/>
<style name="ChartSeriesColor1" backcolor="#E96270"/>
......
......
<page>
<frame>
<reportElement uuid="21047d3c-c9f7-46e2-810e-3689a470fce9" style="TableFrameStyle" x="20" y="20" width="627" height="727" origin="4" srcId="3">...</reportElement>
<frame>
<reportElement uuid="1b51c819-e28b-43a1-87a3-3add5e51ac0d" x="0" y="0" width="125" height="25" origin="7" srcId="8">
<property name="net.sf.jasperreports.export.headertoolbar.cellID" value="newSet2.DOCUMENT_REFERENCE_1882018985_0"/>
<property name="net.sf.jasperreports.export.html.class" value="jrcolHeader interactiveElement"/>
<property name="net.sf.jasperreports.export.headertoolbar.columnUUID" value="93469203-1702-437d-856c-5326b63e6082"/>
<property name="net.sf.jasperreports.export.headertoolbar.tableUUID" value="21047d3c-c9f7-46e2-810e-3689a470fce9"/>
<property name="net.sf.jasperreports.export.headertoolbar.columnIndex" value="0"/>
</reportElement>
<genericElement>...</genericElement>
<text textAlignment="Left" textHeight="12.8046875" lineSpacingFactor="1.1640625" leadingOffset="-2.5942383">
<reportElement uuid="940ff028-90cf-4ca2-ae5c-864da3e50ebc" style="TableColumnHeaderTextStyle" x="0" y="0" width="125" height="25" origin="7" srcId="10"/>
<textContent>
<![CDATA[ Reference ]]>
</textContent>
<lineBreakOffsets>
<![CDATA[ ]]>
</lineBreakOffsets>
</text>
</frame>
<frame>
<reportElement uuid="6720edab-8950-4141-8120-c705e23c0082" x="125" y="0" width="125" height="25" origin="7" srcId="11">
<property name="net.sf.jasperreports.export.headertoolbar.cellID" value="newSet1.AUDIT_EVENT_DATE_TIME_806444232_0"/>
<property name="net.sf.jasperreports.export.html.class" value="jrcolHeader interactiveElement"/>
......
<frame>
<reportElement uuid="a363100e-edd8-4a0e-93dd-e05d00b25368" x="500" y="0" width="125" height="25" origin="7" srcId="20">
<property name="net.sf.jasperreports.export.headertoolbar.cellID" value="newSet1.AUDIT_EVENT_TYPE_ID_1616381145_0"/>
<property name="net.sf.jasperreports.export.html.class" value="jrcolHeader interactiveElement"/>

<reportElement uuid="7d0e99df-b27b-4d17-9642-153799495f2b" key="textField" style="TableDetailTextStyle" x="500" y="250" width="125" height="25" origin="8" srcId="27">
<property name="net.sf.jasperreports.export.html.class" value="jrcel cel_newSet1.AUDIT_EVENT_TYPE_ID_1616381145_0"/>
</reportElement>
<textContent>
<![CDATA[ 17 ]]>
</textContent>
<lineBreakOffsets>
<![CDATA[ ]]>
</lineBreakOffsets>
</text>
...........
..........
...........

this was not in readable format for us.

So we modified the code by Writing CustomXmlExporter and got the xml in proper format.

<jasperPrint name="rt demo">
<columns>
<column>
<![CDATA[ Reference ]]>
</column>
<column>
<![CDATA[ Action Date ]]>
</column>
<column>
<![CDATA[ Uploaded By Username ]]>
</column>
<column>
<![CDATA[ User Name ]]>
</column>
<column>
<![CDATA[ Action Type ID ]]>
</column>
</columns>
<rows>
<row>
<text>
<![CDATA[ Dummy Reference 12 ]]>
</text>
<text>
<![CDATA[ 25/03/2014 16:38 ]]>
</text>
<text>
<![CDATA[ xyz xyz ]]>
</text>
<text>
<![CDATA[ xyz xyz ]]>
</text>
<text>
<![CDATA[ 85 ]]>
</text>
</row>
<row>
<text>
<![CDATA[ Dummy Reference 11 ]]>
</text>
<text>
<![CDATA[ 25/03/2014 16:38 ]]>
</text>
<text>
<![CDATA[ xyz xyz ]]>
</text>
<text>
<![CDATA[ xyz xyz ]]>
</text>
<text>
<![CDATA[ 85 ]]>
</text>
</row>
..........
<row>
<text>
...........
...........
...........

</rows>
</jasperPrint>

This is been achieved by modifying the below code.

public class FinalCustomExporter extends JRXmlExporter
{
private static final Log log = LogFactory.getLog(JRXmlExporter.class);
private static final String XML_EXPORTER_PROPERTIES_PREFIX = JRPropertiesUtil.PROPERTY_PREFIX + "export.xml.";
public static final String XML_EXPORTER_KEY = JRPropertiesUtil.PROPERTY_PREFIX + "xml";

public static final String PROPERTY_REPLACE_INVALID_CHARS = JRPropertiesUtil.PROPERTY_PREFIX + "export.xml.replace.invalid.chars";
protected static final String DEFAULT_XML_ENCODING = "UTF-8";
protected static final String DEFAULT_OBJECT_TYPE = "java.lang.String";
protected static final String XML_FILES_SUFFIX = "_files";
protected static final String IMAGE_PREFIX = "img_";
public static final java.lang.String ELEMENT_columns = "columns";
public static final java.lang.String ELEMENT_column = "column";
public static final java.lang.String ELEMENT_rows = "rows";
public static final java.lang.String ELEMENT_row = "row";

public static final XmlNamespace JASPERPRINT_NAMESPACE =
new XmlNamespace(JRXmlConstants.JASPERPRINT_NAMESPACE, null, JRXmlConstants.JASPERPRINT_XSD_SYSTEM_ID);

protected JRXmlWriteHelper xmlWriter;
protected String encoding;
protected String version;
protected VersionComparator versionComparator = new VersionComparator();

protected JRExportProgressMonitor progressMonitor;
protected Map<Renderable,String> rendererToImagePathMap;
protected Map<String,byte[]> imageNameToImageDataMap;
protected Map<String,JRStyle> stylesMap = new HashMap<String,JRStyle>();

protected boolean isEmbeddingImages = true;
protected boolean isColumn = false;
protected boolean isRow = false;
protected Integer columnCount = 0;
protected Integer rowCount = 0;
protected File destFile;
protected File imagesDir;

protected class ExporterContext extends BaseExporterContext implements JRXmlExporterContext
{
public String getExportPropertiesPrefix()
{
return FinalCustomExporter.this.getExporterPropertiesPrefix();
}
}

protected JRXmlExporterContext exporterContext = new ExporterContext();

public FinalCustomExporter()
{
this(DefaultJasperReportsContext.getInstance());
}

public FinalCustomExporter(JasperReportsContext jasperReportsContext)
{
super(jasperReportsContext);
}

public void exportReport() throws JRException
{
progressMonitor = (JRExportProgressMonitor)parameters.get(JRExporterParameter.PROGRESS_MONITOR);

setOffset();

try
{
setExportContext();

setInput();

if (!parameters.containsKey(JRExporterParameter.FILTER))
{
filter = createFilter(getExporterPropertiesPrefix());
}

setPageRange();

@SuppressWarnings("deprecation")
String dtdLocation = (String)parameters.get(JRXmlExporterParameter.DTD_LOCATION);
if (dtdLocation != null)
{
log.warn("The JRXmlExporterParameter.DTD_LOCATION export parameter has no effect and should no longer be used.");
}

encoding = (String)parameters.get(JRExporterParameter.CHARACTER_ENCODING);
if (encoding == null)
{
encoding = DEFAULT_XML_ENCODING;
}

setHyperlinkProducerFactory();

StringBuffer sb = (StringBuffer)parameters.get(JRExporterParameter.OUTPUT_STRING_BUFFER);
if (sb != null)
{
StringBuffer buffer = exportReportToBuffer();
sb.append(buffer.toString());
}
else
{
Writer outWriter = (Writer)parameters.get(JRExporterParameter.OUTPUT_WRITER);
if (outWriter != null)
{
try
{
exportReportToStream(outWriter);
}
catch (IOException e)
{
throw new JRException("Error writing to writer : " + jasperPrint.getName(), e);
}
}
else
{
OutputStream os = (OutputStream)parameters.get(JRExporterParameter.OUTPUT_STREAM);
if (os != null)
{
try
{
exportReportToStream(new OutputStreamWriter(os, encoding));
}
catch (Exception e)
{
throw new JRException("Error writing to OutputStream : " + jasperPrint.getName(), e);
}
}
else
{
destFile = (File)parameters.get(JRExporterParameter.OUTPUT_FILE);
if (destFile == null)
{
String fileName = (String)parameters.get(JRExporterParameter.OUTPUT_FILE_NAME);
if (fileName != null)
{
destFile = new File(fileName);
}
else
{
throw new JRException("No output specified for the exporter.");
}
}

imagesDir = new File(destFile.getParent(), destFile.getName() + XML_FILES_SUFFIX);

Boolean isEmbeddingImagesParameter = (Boolean)parameters.get(JRXmlExporterParameter.IS_EMBEDDING_IMAGES);
if (isEmbeddingImagesParameter == null)
{
isEmbeddingImagesParameter = Boolean.TRUE;
}
isEmbeddingImages = isEmbeddingImagesParameter.booleanValue();

exportReportToFile();
}
}
}
}
finally
{
resetExportContext();
}
}

protected void setHyperlinkProducerFactory()
{
hyperlinkProducerFactory = (JRHyperlinkProducerFactory) parameters.get(JRExporterParameter.HYPERLINK_PRODUCER_FACTORY);
}

protected void exportReportToFile() throws JRException
{
{
rendererToImagePathMap = new HashMap<Renderable,String>();
imageNameToImageDataMap = new HashMap<String,byte[]>();
}

Writer writer = null;
try
{
OutputStream fileOutputStream = new FileOutputStream(destFile);
writer = new BufferedWriter(new OutputStreamWriter(fileOutputStream, encoding));
exportReportToStream(writer);
}
catch (IOException e)
{
throw new JRException("Error writing to file : " + destFile, e);
}
finally
{
if (writer != null)
{
try
{
writer.close();
}
catch(IOException e)
{
}
}
}

if (!isEmbeddingImages)
{
Collection<String> imageNames = imageNameToImageDataMap.keySet();
if (imageNames != null && imageNames.size() > 0)
{
if (!imagesDir.exists())
{
imagesDir.mkdir();
}

for(Iterator<String> it = imageNames.iterator(); it.hasNext();)
{
String imageName = it.next();
byte[] imageData = imageNameToImageDataMap.get(imageName);

File imageFile = new File(imagesDir, imageName);

OutputStream fos = null;
try
{
fos = new FileOutputStream(imageFile);
fos.write(imageData, 0, imageData.length);
}
catch (IOException e)
{
throw new JRException("Error writing to image file : " + imageFile, e);
}
finally
{
if (fos != null)
{
try
{
fos.close();
}
catch(IOException e)
{
}
}
}
}
}
}
}

protected StringBuffer exportReportToBuffer() throws JRException
{
StringWriter buffer = new StringWriter();
try
{
exportReportToStream(buffer);
}
catch (IOException e)
{
throw new JRException("Error while exporting report to buffer", e);
}
return buffer.getBuffer();
}

protected XmlNamespace getNamespace()
{
return JASPERPRINT_NAMESPACE;
}

protected void exportReportToStream(Writer writer) throws JRException, IOException
{
version = JRPropertiesUtil.getInstance(jasperReportsContext).getProperty(jasperPrint, JRXmlBaseWriter.PROPERTY_REPORT_VERSION);

xmlWriter = new JRXmlWriteHelper(writer);

xmlWriter.writeProlog(encoding);

xmlWriter.startElement(JRXmlConstants.ELEMENT_jasperPrint);
xmlWriter.addEncodedAttribute(JRXmlConstants.ATTRIBUTE_name, jasperPrint.getName());

List<JRPrintPage> pages = jasperPrint.getPages();
if (pages != null && pages.size() > 0)
{
JRPrintPage page = null;
for(int i = startPageIndex; i <= endPageIndex; i++)
{
if (Thread.interrupted())
{
throw new JRException("Current thread interrupted.");
}

page = pages.get(i);

exportPage(page);
}

if(isRow){
xmlWriter.closeElement();
isRow = false;
}
}

xmlWriter.closeElement();

writer.flush();
}

protected void exportPage(JRPrintPage page) throws JRException, IOException
{
exportElements(page.getElements());

if (progressMonitor != null)
{
progressMonitor.afterPageExport();
}
}

protected void exportElements(Collection<JRPrintElement> elements) throws IOException, JRException
{
if (elements != null && elements.size() > 0)
{
for(Iterator<JRPrintElement> it = elements.iterator(); it.hasNext();)
{
exportElement(it.next());
}
}
}

public void exportElement(JRPrintElement element) throws IOException, JRException
{
if (filter == null || filter.isToExport(element))
{
if (element instanceof JRPrintText)
{
exportText((JRPrintText)element);
}
else if (element instanceof JRPrintFrame)
{
exportFrame((JRPrintFrame) element);
}
}
}

public void exportText(JRPrintText text) throws IOException
{
JRStyle style = text.getStyle();
if (style != null)
{
String styleName = style.getName();
if(!isRow && !isColumn && "TableColumnHeaderTextStyle".equals(styleName)){
xmlWriter.startElement(ELEMENT_columns);
isColumn = true;
columnCount = 0;
}else if(isColumn && ("TableGroupHeaderTextStyle".equals(styleName) || "TableDetailTextStyle".equals(styleName))){
xmlWriter.closeElement();
isColumn = false;
isRow = true;
xmlWriter.startElement(ELEMENT_rows);
}
}
if (isColumn && text.getOriginalText() != null)
{
columnCount++;
xmlWriter.writeCDATAElement(ELEMENT_column, text.getOriginalText(),
JRXmlConstants.ATTRIBUTE_truncateIndex, text.getTextTruncateIndex());
}else if(style != null && !("TableGroupHeaderTextStyle".equals(style.getName())) && isRow){
if(rowCount == 0){
xmlWriter.startElement(ELEMENT_row);
}
rowCount++;
xmlWriter.writeCDATAElement(JRXmlConstants.ELEMENT_text, text.getOriginalText(),
JRXmlConstants.ATTRIBUTE_truncateIndex, text.getTextTruncateIndex());
}

xmlWriter.writeCDATAElement(JRXmlConstants.ELEMENT_textTruncateSuffix, text.getTextTruncateSuffix());
if(rowCount == columnCount){
xmlWriter.closeElement();
rowCount = 0;
}
}

protected void exportFrame(JRPrintFrame frame) throws IOException, JRException
{
setFrameElementsOffset(frame, true);
try
{
exportElements(frame.getElements());
}
finally
{
restoreElementOffsets();
}
}

protected String getExporterPropertiesPrefix()
{
return XML_EXPORTER_PROPERTIES_PREFIX;
}

public String getExporterKey()
{
return XML_EXPORTER_KEY;
}

public JRXmlWriteHelper getXmlWriteHelper()
{
return xmlWriter;
}

protected boolean isNewerVersionOrEqual(String oldVersion)
{
return versionComparator.compare(version, oldVersion) >= 0;
}
}

How to update multiple value of specific embedded document, inside an array, of a specific document in MongoDB?

2014-01-09T04:25:00.000-08:00

How to update multiple value of specific embedded document, inside an array, of a specific document in MongoDB?

MongoDB has a limitation of the same.

for more details please find the below link for your information.

https://jira.mongodb.org/browse/SERVER-1243

for example you have a JSON structure like this ...

{
"_id" : ObjectId("52ce2e4c9109b0e2274a388b"),
"address" : {
"street" : "Aurthor Street 1",
"area" : "Area 1",
"city" : "New York 1",
"country" : "UK"
},
"age" : 29,
"commList" : [
{
"userId" : 1,
"firstName" : "Commenter First Name",
"lastName" : "Commenter Last Name",
"commentText" : "Comment Text",
"date" : ISODate("2014-01-09T05:06:20.283Z")
},
{
"userId" : 1,
"firstName" : "Commenter First Name",
"lastName" : "Commenter Last Name",
"commentText" : "Comment Text 2",
"date" : ISODate("2014-01-09T11:59:57.050Z")
}
],
"commentCount" : 2,
"name" : "Jamse Cook"
}

Now the user name is changed and we need to update the it.

If we try using java code like :

DBCollection dbCollection = getCollection();
JacksonDBCollection<Person, String> coll = JacksonDBCollection.wrap(dbCollection, Person.class, String.class);

DBUpdate.Builder builder = new DBUpdate.Builder();
builder.set("commList.$.firstName", "Puneet");

WriteResult<Person, String> result = coll.update(DBQuery.is("commList.userId", 1), builder);
System.out.println("rows updated :: " + result.getN());

It updates the first occurence only.

you can use the below code and run it on mongoshell which will update the all occurances
var runUpdate = function(){
db.test.find({"commList.userId" : 1}).forEach( function(test) {
for(var i in test.commList){
test.commList[i].firstName = 'abhijit';
}
db.test.save(test);
});
};

db.eval(runUpdate);

Example of MongoDB with MongoJack

2014-01-07T21:44:00.000-08:00

There is need of java mapper to the mongo object and MongoJack is very helpful in the same
to achieve the objective. It uses Jackson as the Java JSON mapper.

You need to add the following jars.

1. bson4jackson-2.1.0.jar
2. commons-io-2.4.jar
3. hamcrest-core-1.3.jar
4. hamcrest-library-1.3.jar
5. jackson-annotations-2.1.2.jar
6. jackson-core-2.1.3.jar
7. jackson-databind-2.1.3.jar
8. mongo-java-driver-2.11.3.jar
9. mongojack-2.0.0-RC5.jar
10. persistence-api-1.0.2.jar

In the example you will find how add, update, updating Embedded field, find and find with some criteria using the MongoJack.

public static void main(String[] args) {
create();
findOne();
updateEmbeddedField();
update();
findOne();
findWithCriteria();
}

public static void create(){

DBCollection dbCollection = getCollection();

JacksonDBCollection<Person, String> coll = JacksonDBCollection.wrap(dbCollection, Person.class,
String.class);
Person person = getPerson();
WriteResult<Person, String> result = coll.insert(person);
Person p = coll.findOne();
Address ad = p.getAddress();
System.out.println("Create o/p ->> name :: " + p.getName() + " age :: " + p.getAge() + " area :: "+ ad.getArea());
}

public static void updateEmbeddedField(){

DBCollection dbCollection = getCollection();
JacksonDBCollection<Person, String> coll = JacksonDBCollection.wrap(dbCollection, Person.class,
String.class);

DBUpdate.Builder builder = new DBUpdate.Builder();
builder.set("address.area", "Pune");

Person result = coll.findAndModify(DBQuery.is("name", "Jamse Cook"), null, null, false, builder, false, false);
Address ad = result.getAddress();
System.out.println("updateEmbeddedField o/p ->> name :: " + result.getName() + " age :: " + result.getAge() + " area :: "+ ad.getArea());
}

public static void update(){

DBCollection dbCollection = getCollection();
JacksonDBCollection<Person, String> coll = JacksonDBCollection.wrap(dbCollection, Person.class,
String.class);

DBUpdate.Builder builder = new DBUpdate.Builder();
builder.set("name", "Abhijit");

Person result = coll.findAndModify(DBQuery.is("name", "Jamse Cook"), null, null, false, builder, false, false);
Address ad = result.getAddress();
System.out.println("update o/p ->> name :: " + result.getName() + " age :: " + result.getAge() + " area :: "+ ad.getArea());
}

public static void findOne(){

DBCollection dbCollection = getCollection();
JacksonDBCollection<Person, String> coll = JacksonDBCollection.wrap(dbCollection, Person.class,
String.class);
Person result = coll.findOne();
Address ad = result.getAddress();
System.out.println("find o/p ->> name :: " + result.getName() + " age :: " + result.getAge() + " area :: "+ ad.getArea());

}

public static void findWithCriteria(){

DBCollection dbCollection = getCollection();
JacksonDBCollection<Person, String> coll = JacksonDBCollection.wrap(dbCollection, Person.class,
String.class);
Person result = coll.findOne(DBQuery.is("name", "Abhijit"));
Address ad = result.getAddress();
System.out.println("findWithCriteria o/p ->> name :: " + result.getName() + " age :: " + result.getAge() + " area :: "+ ad.getArea());

}

private static DBCollection getCollection() {
MongoClient mongoClient = null;
try {
mongoClient = new MongoClient( "localhost" , 27017 );
} catch (UnknownHostException e) {
e.printStackTrace();
}
DB db = mongoClient.getDB( "jack" );
DBCollection dbCollection = db.getCollection("test");
return dbCollection;
}

private static Person getPerson(){

Person person = new Person();
person.setName("Jamse Cook");
person.setAge(29);

Address address = new Address();
address.setArea("Area 1");
address.setCity("New York 1");
address.setStreet("Aurthor Street 1");
address.setCountry("UK");

person.setAddress(address);

return person;
}

The output on the java console is :

Create o/p ->> name :: Jamse Cook age :: 29 area :: Area 1
find o/p ->> name :: Jamse Cook age :: 29 area :: Area 1
updateEmbeddedField o/p ->> name :: Jamse Cook age :: 29 area :: Area 1
update o/p ->> name :: Jamse Cook age :: 29 area :: Pune
find o/p ->> name :: Abhijit age :: 29 area :: Pune
findWithCriteria o/p ->> name :: Abhijit age :: 29 area :: Pune

And the output on the mongo shell is :

db.test.find().pretty();
{
"_id" : ObjectId("52cce4e19109c56d473c46eb"),
"address" : {
"area" : "Pune",
"city" : "New York 1",
"country" : "UK",
"street" : "Aurthor Street 1"
},
"age" : 29,
"name" : "Abhijit"
}

Adding JSON to MongoDB Using Java

2014-01-04T04:37:00.003-08:00

we can add JSON directly to mongo using Java.

1. In the first example will add josn String directly as shown in the code below.

public static void jsonExmple(){
String jsonString = "{'Name' : 'James Smith', 'Age' : 29," +
"'Address' : {'area' : 'Area', 'city' : 'New York', 'country' : 'USA'} }";

MongoClient mongoClient = null;
try {
mongoClient = new MongoClient( "localhost" , 27017 );
} catch (UnknownHostException e) {
e.printStackTrace();
}
DB db = mongoClient.getDB( "testJson" );
DBCollection json = db.getCollection("json");
DBObject dbObject = (DBObject)JSON.parse(jsonString);

json.insert(dbObject);

DBCursor jsonDoc = json.find();
while (jsonDoc.hasNext()) {
System.out.println(jsonDoc.next());
}
}

the output on the mongo console would be like :

db.json.find().pretty();
{
"_id" : ObjectId("52c7fead91090c7db49d1604"),
"Name" : "James Smith",
"Age" : 29,
"Address" : {
"area" : "Area",
"city" : "New York",
"country" : "USA"
}
}

2. Another case would be, convert the Java object to json and add the json to mongoDB as shown below.
I have used the google json api to convert Java object to Json.

public static void jsonExample(){

MongoClient mongoClient = null;
try{
mongoClient = new MongoClient( "localhost" , 27017 );
DB db = mongoClient.getDB( "testJson" );
DBCollection json = db.getCollection("json");

Person person = new Person();
person.setName("Jamse Cook");
person.setAge(29);

Address address = new Address();
address.setArea("Area");
address.setCity("New York");
address.setStreet("Aurthor Street");
address.setCountry("USA");

person.setAddress(address);

Gson gson = new Gson();
String jsonString = gson.toJson(person);
System.out.println(" print the json " + jsonString);

DBObject dbObject = (DBObject)JSON.parse(jsonString);
json.insert(dbObject);

}catch (Exception e) {

}
}

the output on the mongo console would be like :

{
"_id" : ObjectId("52c7fc479109fd33fdc45793"),
"name" : "Jamse Cook",
"age" : 29,
"address" : {
"street" : "Aurthor Street",
"area" : "Area",
"city" : "New York",
"country" : "USA"
}
}

How to read the array data in mongoDB, modify the array data and update the same in mongoDB.

2013-12-16T03:24:00.001-08:00

How to read the arrat data in mongoDB, modify the array data and update the same in mongoDB.

public static void mongoArrayExample(){
DBCursor cursor = null;
MongoClient mongoClient = null;
try {

mongoClient = new MongoClient("localhost", 27017);
DB db = mongoClient.getDB("school");
DBCollection students = db.getCollection("students");

BasicDBObject whereQuery = new BasicDBObject();
whereQuery.put("_id" , 0);

cursor = students.find(whereQuery);
Map<String, Double> map = new HashMap<String, Double>();
while (cursor.hasNext()){
BasicDBObject result = (BasicDBObject) cursor.next();
BasicDBList scores = (BasicDBList) result.get("scores");
// Reading the data from Array.
for(int j = 0; j< scores.size(); j++)
{
// Reading the individual records
String type = (String) ((BSONObject) scores.get(j)).get("type");
Double score = (Double) ((BSONObject) scores.get(j)).get("score");
if(map.containsKey("homework")){
map.put(type, score);
}
}
}

List<BasicDBObject> scores = new ArrayList<BasicDBObject>();
Set<String> str = map.keySet();
for (String string : str) {
BasicDBObject obje = new BasicDBObject();
obje.put("type", string);
obje.put("score", map.get(string));
scores.add(obje);
}

// Modifying the array
BasicDBObject updateQuery = new BasicDBObject();
updateQuery.append("$set", new BasicDBObject().append("scores", scores));
students.update(whereQuery, updateQuery);

}catch (Exception e) {
}
}