sparkle tech thoughts

iPad Apps helps autistic children to spread their wings

2016-09-05T22:24:00.000-07:00

Today there is app for everything, for maths, science, history, music and many more. Can these apps be useful for children with special needs? Kids with special needs cannot use the traditional approaches in learning, specially kids with autism need extra help than the rest of these children. They find it bit hard to focus on the things you teach, and process the information that is targeted. For kids like this Apps can be a life saver.

There is an app for every skill you want to impart. For example there are apps targeted:

Improve Attention Span
Communication
Eye Contact
Social Skills
Motor co-ordination
Language and literacy
Time Management
Emotional Skills and Self Care

There are many apps in the appstore that facilitate Augmentative and Alternative Communication (AAC ) which can be highly use for autistic kids who are non verbal or speech partially impaired. These apps use text to speech technology, where kids can type words, or drag and drop images to communication with others on their needs.

Here are some of the reasons why iPad apps are so attractive for the kid's with autism and why they simply love it.

Visual Learning - Children with autism tend to learn more visually or by touch. In most cases they struggle with instructions given in the traditional school setting. Therefore, ipad provides an interactive educational environment to help them learn.
Structured and Consistent - Autistic children prefer devices compared to humans mostly because they are structured and constant. The voice is constant phase is constant, and they like things to be orderly and the ipad waits, which is something most human educators lack.
Promote independent learning and gives immediate feedback - Some apps which were build for ASD gives sensory re-enforcement to the child and when giving negative feedback, it chose to give it smoothly without any loud noises to distract the child.
Diverse Therapeutic Resource - Technology evolves and expand rapidly, so every day you see new apps with lot of creativity. Kids will not get bored, and the apps they use will get updated time to time
Cost Effective - Compared to other therapeutic resources ipad is inexpensive. For example Assistive Technology Devices such as Saltillo, Dynavox, Tobil cost much more than the iPad.
Socially Desirable and Portable- Most of therapeutic devices can lead to bullying in school, for kids with special needs. But if you give a child with an iPad, this makes the child popular at school (because every other kid likes to use iPad).
Apps can be easily customised - Most apps which were developed targetting autism can be customised, you can add your unique videos, images and voice to these apps. Specially the language and communication apps, so the child can learn from the things they see in their day to day life.

Here are some of the common iPad apps which target Communication, Social, Emotional and Linguistic Skills.

Communication - Tap to talk, iComm, Look2Learn, voice4u, My Talk Tools, iCommunicate, Proloqu2go, Inner Voice.

Social Skills - Model Me Going Places, Ubersense, TherAd, iMove (these apps will mostly do video self modelling social stories and give step by step guidance for social stories such as brushing their teeth, going for a hair cut etc).

Linguistic Skills - First Phrases, iLanguage/ Language Builder, Rainbow Sentences, Interactive Alphabet, Touch and Write.

Emotional Skills - Smarty Pants.

Special Note for Educators, Teachers and Parents

Even though, there is a vast amount of apps available today, iPad is not a baby sitter, you should always be present when educating your child. The educational apps are their to help your child develop some cognitive processors or enhance their skills at reading, writing, spatial reasoning or just a way of communication. However, in order your child to grow, there should be an educator present when child is learning.. It can be a great tool and a resource for educational development but all things should come in moderation and also under supervision. It's actually quality of teaching and support that gives a positive out come, not just the device.

Autistim ..the good, the bad and the ugly

2016-09-04T01:44:00.000-07:00

Autism or Autism Spectrum Disorder is a brain based disorder which makes the brain function in a different way compared to the rest of the wold.

Autism affect three major area's of a person;

1. Social Interaction - Autistic people do not like to socialise, and they tend to be in their own world, they find it very hard to keep eye contact, and rarely shows empathy or compassion towards other human beings.

2. Communication - Most kids on the spectrum has delayed speech or non verbal (40% of autistic kids are non verbal), this can be an early sign for you to identify who has autism. Kids with autism has trouble with speech and communication and they find it hard to express what they need also they find it hard to understand what you are talking to them. This can be because they find it hard to focus on you as they are mostly distracted by their surroundings.

3. Behaviours and Interest - Behaviours and interest of autistic people can differ from person to person. Most children or grownups with Autism have repetitive movements with objects, repeated body movements such as rocking and hand flapping.

These are few of the characteristics of autism

Difficulty with verbal communication, including problems using and understanding language
Inability to participate in a conversation, even when the child has the ability to speak
Difficulty with non-verbal communication, such as gestures and facial expressions
Difficulty with social interaction, including relating to people and to his or her surroundings
Difficulty making friends and preferring to play alone
Unusual ways of playing with toys and other objects, such as only lining them up a certain way
Difficulty adjusting to changes in routine or familiar surroundings, or an unreasonable insistence on following routines in detail
Repetitive body movements, or patterns of behavior, such as hand flapping, spinning, and head banging
Preoccupation with unusual objects or parts of objects

However, autism is a spectrum disorder, therefore, children/people with autism behave very uniquely and challenges they go via differs from person to person. furthermore, the severity of the autism may change from time to time.

Autism can be a problem if it's not identified at child's early ages, autism can be identified within 18 months from child birth or even earlier. Infants with autism have no big smiles or other warm, joyful expressions by six months or later and no no back-and forth gestures such as pointing,showing reaching or waving by 12 months. Some kids with autism has no speech, babbling or social skills at any age. These are very few early signs of autism, if you see these red flags, its better not delay in asking your pediatrician or family doctor for an evaluation.

It's always good to identify autism at very early age so you can get the necessary help, with communication, language development, emotional skills and education. Autism kid's will have a hard time dealing with social interaction and traditional approaches of learning, however, they can be very smart, knowledgable and gifted children. If they were given the right set of tools for their learning, and if educators can help them understand how they can learn, they can do wonders.

Autistic Children are gifted and special, they have a really good focus on things and they see what most people miss. Due to this they have incredible skills in areas of mathematics arts music and many more. They see the world in a different manner and can help improve the world greatly. Here are few of the famous autistic people.

Albert Einstein - Very intelligent physician with many inventions

Amadeus Mozart - Talented musician

Sir Isaac Newton - Scientist with many inventions

Charles Darwin - Naturalist and geologist

Thomas Jefferson - President of United States

Michelangelo - Artist

As mention above, autistic people have un imaginable talents, and they make our world a better place.

The ugly part of autism is not the autistic people, its the society and how they look at autistic people. Due to lack of social skills, there are many instances autistic children get bullied and misunderstood. And most of them cannot understand sarcasm or phrasal words, and they only understand the literal meanings of words. Their behaviour, and how they perceive things are different compared to majority of people, due to that society tend to make autistic people feel less important or weird. According to Centres for Disease Control and Prevention, 1 in 68 people have autism. This means there is a large population of autistic people in the world today, if we can be more patient and understanding when dealing with them, we can make a better world for them as well as us.

Social Cognitive Approaches to Learning

2016-08-31T02:35:00.002-07:00

My second theory which I am going to discuss on education psychology and how it can be applied when in teaching is social cognitive approaches to Learning. This mainly focuses on how learning happen via observations. The social learning theory was proposed by Albert Bandura with his Bobo doll experiment where he demonstrated that children learn and imitate behaviours they have observed in other people. And this theory can be used in educational psychology where you can influence children and improve their learning and development via observation. In my earlier post I spoke about how behavioural theories of learning was the result of association formed by conditioning, reinforcement and punishment. In this section I would like to discuss about social learning theory where learning can also occur simply by observing the actions of others and how this can be used on classroom activities to educate children.

Social Cognitive Theory

Social cognitive theory explains that social and cognitive factors as well as behaviour plays an important learn in learning. And this plays a major role in schools and educational environments. Mostly because cognitive factors involved students brain development and success in their education and social factors include student’s observing their teachers and their behaviour.

In this example we will look at how Bandura’s social cognitive theory can be applied in a classroom environment to achieve success

Cognition influences behaviour At school a student name Alex develop strategies to solve a maths problem by thinking logically about how to solve the problem.
Behaviour influences cognition Alex working on the Math problem has led to her achieve good grades. And this will lead to self satisfaction which leads to cognition.
Environment influences Behaviour Alex will enroll on higher maths program where it gives tips on solving problems efficiently and this will improve Alex’s maths skills.
Behaviour influences environment The higher math class succeed and many student in Alex’s class join that same program so that they will get tips to succeed in maths.
Cognition influences environment The teachers and principals feel the need and expectations to have the higher maths program.
Environment influence cognition school establish a resources center where students can have books on tips on higher mathematics to improve their maths skills.

Bandura further discuss self efficacy also known as confident in doing something is also very important when it comes to social cognitive theory, as even if the students have all the resources and models students might not get motivated to observe or pay attention/study hard if he/she does not have confident in learning

Observational Learning

There are three main concepts when it comes to observational learning or social learning and I will discuss each of these concepts from a teacher’s point of view and focus more on educational environment. 1) Students can learn from observation, 2) Student’s internal mental states matter when it comes to observational learning, 3) Just because a student learn some behaviour it does not mean he/she will change his behaviour.

1. Students can learn via observation

Observation learning is the learning that involves acquiring skills, strategies and beliefs by looking at others. And observation learning happen at classrooms by student’s observing models. There are three differant types of models that can influence observational learning

a) A live model This can be a teacher or another peer student at a classroom who is an actual individual demonstrating or acting out the behaviour

b) Verbal instruction model This can be a verbal instructor, a lecturer or announcer who make announcement at classrooms verbally who does not have to present while giving it.

c) A symbolic Model This can be a real or fictional character displaying the behaviour in books or documentary videos for example characters from history books or scientist on TV programs when it comes to educational environments

2. Mental states are important to learning

When it comes to observational learning mental state of the student is also very important. if the student is not confident when it comes to learning he/she will not learn at all. Bandura mention this as intrinsic reinforcement. If a student want to do observational learning he should get internal reward such as pride/satisfaction or some sense of accomplishment when learning the observations. This emphasis on internal thoughts and cognition helps connect learning theories to cognitive developmental theories.

3. Learning does not necessarily leads to a change in behaviour

When it comes to observational learning unlike behavioural learning learning does not lead to permanent change in behaviour, it can only demonstrate new information can be learn without demonstrating new behaviours

Bandura highlighted this theory using a modeling process, Modeling process highlight the factors that make a learning successful via observation. These factors are known as Attention, Retention, Reproduction and Motivation.

Attention:

In order for a student to learn he/she must pay attention. Anything that distracts their attention is going to have a negative effect on learning. Therefore, teachers as model must make sure that students are paying attention while teaching and make sure the lesson are interesting so that students will pay more attention observing the teacher as the model

Retention:

For student to observe and learn the student must have the ability to store information. Therefore teachers must ensure that students have the capacity to store what they learn when teaching and give enough breaks when modeling.

Reproduction:

Once the student have paid attention to the teacher or the model and further retained information. As teachers, you should make sure students actually perfom the behaviour. It can be a language class or any other observation, students must practice what they learn in order to improve their skills

Motivation:

Finally, in order for observational learning to be successful, student must be motivated to pay attention, retain and reproduce. If the students are not motivated enough he/she will not learn properly.Reinforcement and punishment play an important role in motivation. While experiencing these motivators can be highly effective, so can observing other experience some type of reinforcement or punishment. For example, if you see another student rewarded with extra credit for being to class on time, you might start to show up a few minutes early each day.

How do you use observational Learning in your Classroom?

Examples of Social Learning at classroom

Even Though most of the observational learning is not done on purpose, most children learn by observing their teachers specially when it comes to kids who are at kindergarten and preschool. Therefore, its always important that teachers become good role models for their students. These are some of the outcomes in observational learning.

Students learning how to solve maths problems by looking at their teacher solving the maths problem and they learn to solve similar maths problems by using the same theory.
Student’s learn how to speak a foreign language by listening to their teacher. Therefore, the student will get the teacher’s accent.
Students learn how write by looking at how the teacher writes
Student can learn art and craft work by looking at their art teach and learn hand work
Student learn how to keep their desks clean and tidy by looking at their teachers desks

These are few examples of social learning that happen at classroom, however, teachers can use this theory to educate children.

● For example teachers can invite popular athlete to come to class room and give their student a talk, so the student get influenced by the athlete and get motivated and learn skills from the athlete further that athlete can be a role model for students who are into athletics.

● Another example would be teachers can influence weak students (students who are behind their schedule) by making them associate with brighter peers, so that weaker students can observe and learn from their peers.

● Albert Bandura’s social learning theory can be applied in classroom with the use of technology. By doing educational computer programs and educational documentaries children learn and they change their behaviour by observing the characters on movies and educational games. A good example of this can be is the popular video game “Dora the explorer” or “Sesame Streat”, most preschoolers learn language and other soft skills by watching how dora plays and they learn via observations

These are few techniques that can be used at classroom, with the use of observational learning. However, its important that the teachers make sure the models for these observations are the right and very influential.

Advantages of Observational Learning

There are many advantages of observational learning as most children learn their basics at their very early ages via observation. These are the list of advantages you can gain at classroom via observational learning. The main advantage of observational learning is it lacks objectives most of the time and it is not traditional formal learning structure and students learn unknowingly with the uses of practical and real life experience and gain life skills at very young age.

● Learning makes fun and can learn things without effort.

○ Since observational learning is not like traditional learning student learn things more naturally and it does not take an effort to learn them. It further creates a flexible learning environment if applied in classrooms and unlike behavioural approaches children are able to explore and gain creativity. As a results children learn observation skills, problem solving skills and creativity.

● Encourages social interactions

○ Since observational learning is about observing others and learning children learn language and communication skills which are building blocks of social interaction. Further, observation learning encourage pretend play which improve development skills in early childhood development. Further it will build character and self esteem so that they can interact with the society with diverse activities.

● Improve their behaviour and quality of life

○ Teachers can discipline their students via observation learning. Specially if there are students with behavioural issues, teachers can improve their behaviour by showing models that students appreciate and they will eventually fix their behaviour by looking at the models they adore (this can be fictional or real life models).

● Improve memory and brain development

○ Observation learning practices retention and retention improves memory and cognition. When students learn from their environment by retaining the information they observe it will enhance their memory capacities.

● Expands and exchange knowledge

○ Observational learning helps students expand their knowledge by observing different roles. For example if a classroom supports observational learning, the children learn via various models including their peers. This helps them gain new information and also exchange new information via their peers. Further, if technology is used on observational learning, students can gain knowledge at international level.

Disadvantages of Observational Learning

Even though, observational learning has their benefits there are some drawbacks with observational learning at classroom when it is done poorly.

● Poor Role Models Demonstrate Poor Behavior

○ If the role models gives bad examples like teachers showing aggression in front of children or if teachers show war related documentaries children learn aggression via observation (just like bandura’s bobo doll experiment). Therefore as teacher’s they must be very careful when behaving in front of students, as they tend to learn a lot from teachers.

● Undesirable Models May Reinforce Behavior

○ When students work with their peers they can learn bad behaviours from their peers and learn bad behaviours as socially acceptable. This can happen via technology as well as the staff at school. Therefore, bad behaviour and bad practices seems very desirable to children and influence them in a negative way.

● Evidence of Learning is Not Always Visible

○ In observational learning students learn by observation retaining and reproduction. However, the outcomes of learning is not always visible. It can be good or bad behaviour, even if the children learn something it will not always show until it triggers.

● Observational Learning Requires Motivation

○ Unlike behavioural tools observational learning requires motivation within themselves. If students are not motivated to reproduce what they learn they will stop at attention model itself and for some students if the role models are not powerful for them, Further for students who have various disabilities like ADHD or Autism, observational learning will not be a good tool to use

In conclusion, observational learning can be a powerful tool for educating children. Most children learn from observation therefore, its important for teachers to make sure they are exposed to powerful role models who can influence students in a positive manner. Further teachers as educators must encourage collaborative learning and incentive and the supportive environment since learning happen via the environment. Further teacher must not miss use this observational tool and use it carefully since both good and bad can be observed by children and it can lead to socially unacceptable behaviours.

Behavioural approaches to Learning

2016-08-30T22:53:00.001-07:00

Behavioural theories of educational psychology focus on learning via behavioural changes, and as a person learn, he/she alter the way they perceive their environment. Which helps them process new information. Behavioural studies focus on two main areas when it comes to education psychology. 1. Classical conditioning founded by John B Watson, and the 2. Operant conditioning founded by B.F Skinner. Classical conditioning concepts are not widely used in classroom environments therefore, I will be focusing more on operant conditioning concepts and how it can be used for educational purposes to motivate students and excel their capabilities to the maximum capacity.

Operant Conditioning used as a teaching technique

Discipline is the key to success of child’s development and learning and one of the most popular approach of disciplining a child by the use of Operant conditioning. Operant conditioning was invented by B. F skinner which is also known as instrumental learning. This is a method of learning via reward and punishment methodologies. Operant condition can be used to reward the good behaviour by reinforcement and minimize bad behaviour by punishments. In operant conditioning if an action is followed by a reinforcement (either positive or negative) it’s mostly likely to occur again in the future. For example if a student in the class clean the classroom on time and you have given him a gold star (a token as a positive reinforcement) the student will likely to clean the classroom again in order to get praised from the teacher. Conversely if an action has bad consequences and result in punishments this has a likelihood of weaken the action. For example if a student does not do his/her homework and if you give them extra hours of school (positive punishment) it;s more likely that student will do his/her homework on time next time in order to avoid punishment. This way you can motivate your students for better behaviour in classrooms using operant conditioning.

Here are some behaviours which can be improved by re enforcement

Behavior Consequences Future Behavior
Positive Reinforcement
Student ask a good questions	Teacher Praise the student	Students will continue asking more questions
Student does homework on time	Teacher gives a gold star	Student will always do the home work on time
Negative Reinforcement
When the student write a good report	Teacher gives less homework	Student will continue writing good reports
When the student come to the classroom on time	Teacher finish the lesson early	Students will continue to come to class on time

Here are some students behaviours that can be decreased by punishments

Behavior Consequences Future Behavior
Positive Punishment
Student interrupts at class	Teacher yells at the student	Student will minimize interruptions
Student gives a bad report	Teacher gives a black star	student will try to do a better report next time
Negative punishment
When student is late for class	Teacher take his play time for classroom work	Student will try to come to class on time
When student text during the class	Teacher takes away his mobile phone	Student will not use his mobile at class

Those are some of the common uses of operant conditioning at classroom, however, when applying theory of B.F skinner its important to consider the age level of the student when rewarding/punishment student’s behaviour. And there are various different types of reinforcers/punishments can be used increase or decreased the behaviour according to the desired behaviour of the teacher in a classroom setting. They can be appear in different forms such as 1. Consumable (ie Candy) 2. Social (ie praise/yelling) 3. Activity (More time on the playground) 4, Exchangeables (tokens/stars) and 5. Tangible (getting the best seat). Among these components most educational valuable and relevant can activity as they can be done with educational value such as doing an educational game or watching an educational video. However, no matter what the method is, its important to give the consequences right after the behaviour.

There are various types of reinforcement schedules can be used at classrooms, namely fixed ratio, fixed interval, variable ratio and variable interval. In an educational setting the two variable schedules are the best to maintain the desired behaviour as they are very unpredictable. For example if a students were given the chance to watch an educational movie after handing their assignments on time, they would be more motivated to do it if the outcome is always not the same (variable ratio) or the time during which they hand over the assignment and the reward has unpredictable gaps (variable interval).

Advantages of using operant conditioning in classroom

When a child is rewarded for his/her good behaviour the child will get motivated and keep on doing that behaviour and he/she will eventually absorbed good behaviour
When the child is punished for his bad behaviour he/she will stop doing it.
It will be effective when it comes to shaping and discipline preschool and kindergarten children
Internal reinforcement benefits students with self satisfaction, when doing a math problem right will motivate them to do more and more maths and eventually the student will be a pro at it (provide students an incentive to learn)
Deliberate use by teachers provides students with the ability to respond to antecedents without fully realising it is influencing their behaviour.
Can be easily used as a tool for kids with disabilities such as ADHD/Autism/Conduct disorder to discipline and get attention

Disadvantages of using operant conditioning in classrooms

It will decrease child’s creativity and to think from his own as he/she will be scared of punishment
It will not work for for everyone, and some children might take punishment as a reinforcement (ie a student who does not get enough attention will do bad things to get attention from the teacher and will enjoy getting punished)
The student can pretend to stop the behaviour just to get the reward or avoid the punishment. Once the reward is given they could go back to bad behaviour
If rewards or punishments are misused it would give bad outcomes
Operant conditioning does not take cognitive factors into account

In conclusion operant conditioning is a very effective tool for teachers, specially when it comes to shaping the children for good behaviour and also to help them learn. This can also be used for kids with disabilities such as autism or ADHD to discipline them in order to get their attention. However, it must be used cautiously and consistently. Since behaviour management will be different for different students, punishment and rewarding must be done at individual level at individual circumstances.

Technology For Education ..

2016-08-30T21:02:00.001-07:00

For the next few years, I will be focusing more on technology for education and going to discuss how technology can play a major role in early childhood education. Also I will be talking about education psychology, technology for children with special needs and a whole lot about Autism.

The main reason for this change is because, as educators/care givers/human beings, we need to be aware of children with special needs and see the world from their point of view and always help them to achieve their best. For the next couple of years I will be researching more about educational platforms that can improve cognition, social and communication, emotional intelligence and linguistic skills in special needs child. Also how they can learn how to learn and see how we can improve the traditional education system for the betterment of our children.

Real time log event analysis with WSO2 BAM(CEP features)

2014-01-29T03:54:00.001-08:00

One of the major problems when it comes to managing/monitoring distributed systems is not being able to detect when your system is giving issues all of a sudden. Lets say you have like 100 servers and you need to detect when a fatal error appears in the system and you want to act right away. This is nearly impossible if you do not have proper monitoring tools.
WSO2 BAM with WSO2 CEP features has the perfect mechanism to monitor real time logs and alert it to relevant parties when an expectation/ suspicious error occurs.

In this sample I will be using WSO2 Appserver and WSO2 BAM in order to demonstrate the monitoring and alerting capabilities of WSO2 BAM 2.4.0.

WSO2 Appserver will be used to send log events to BAM.
WSO2 BAM will do all the monitoring and alerting when logs are captured.

Send Logs to BAM

To send logs all you need to is go to WSO2_AS_HOME/repository/conf/log4j.properties and add LOGEVENT to the root logger. And start Appserver. (please make sure your BAM server is up and running before starting the Appserver)

To test if the logs are successfully sent to BAM, you can log in to cassandra explorer and see if there is a new column family created under EVENT_KS keyspace

ie log.AS.0.2013.1.12

Configuring WSO2 BAM for REAL TIME analytics

In this demo, I will be sending an email if an ERROR log occurs in WSO2 Appserver. Since we will be using mail transport in BAM to send email alerts to recipients, we have to enable mail transport in BAM. To do that go to repository->conf->axis2->axis2-client.xml and add email configurations.

<parameter name="mail.smtp.from">wso2esb.mail@gmail.com</parameter>

<parameter name="mail.smtp.host">smtp.gmail.com</parameter>

</transportSender>

Here you can give your own email configurations.

Restart/Start the BAM server.

Assuming the logs are getting published to BAM, lets see how we can capture these log events for real time analytics.

Step 1 - Creating Event Adapters

In order to do real time analytics we need to create an execution plan. For that we need two input adapters.

Input adapter - to capture log events coming to BAM (in this case it will be a wso2event).
Output adapter - to send emails

To create adapters you need to login to BAM management console. Under configurations, there will be "Event Processor Configs" and add input event adapters and output event adapters as shown below.

Input Event adapter

Output Event Adapter

Step 1 - Creating Stream Definitions

Now that we have created event adapters, we need to create a stream definition to capture the LogEvent in order to do complex event processing. In the appserver log event, these are the attributes we have.

Meta Data

clientType {String}

Payload Data

tenantID {String}
serverName {String}
appName {String}
logTime {Long}
priority {String}
message {String}
logger {String}
ip {String}
instance {String}
stacktrace {String}

To create the stream definition, go to main tab and under create stream definition you can create the log event stream as shown below.

Step 1 - Creating the Execution Plan

In the execution plan we will be specifying the input stream and writing a CEP Query (SQL Like Query) for the event stream.

Go to Create Execution Plan, and give suitable name for the execution plan. Select the needed stream and give an alias. Click on import after selecting the stream. In our CEP Query we will analyze events and if an event has an error, we will send it to an output stream.

CEP QUERY

from LogEvents[priority == "ERROR"]

select message,stacktrace,serverName

insert into ExceptionStream

After creating the Query, we need to add the exported stream (which is the stream that we are sending Error logs) according to our CEP querry exported stream name should be ExceptionStream give the value of the exported stream name and select "Create Stream Definition" to create the ExceptionStream. This stream will be auto generated by looking at the CEP as shown below.

Once we create the Exception stream select the Exception stream as exported stream and create a new formatter. This formatter is used to specify the email body and email related information like the subject, to address ect. Give the output mapping type as text so we can give the content of the email message body inline.

Email Body

Error Occurred in {{serverName}} – {{message}}

In this body we are taking message, stacktrace and server name from the OutputStream (ExceptionStream) and adding a readable message for the email message body.

Add the event formatter and save the execution plan. Now we have successfully created the event trigger to monitory error logs for wso2 appserver. You can test this by invoking a service with an error.

If you want more in depth information on real time log event analytics you can follow the following screen cast for more details

[Screencast] Monitoring system logs with WSO2 Business Activity Monitor

2014-01-15T00:39:00.000-08:00

This screencast explain how you can do your enterprise monitoring using BAM and CEP features.

Overview of the screencast

* Send Log Events to BAM/CEP

* Create analytics to analyse logs daily( can be scheduled accordingly)

* Create gadgets with log statistics (charts/tables)

* Monitor logs real time with CEP (send alerts to relevant parties when an error log appears.)

Eventing using WSO2 DSS - Send data to a JMS Queue for a given event trigger

2014-01-09T07:01:00.001-08:00

If you want to trigger an event depend on an SQL related service call this is a great example on how you can do it using WSO2 Data Service Server.

In this sample I am going to explain how you can handle event triggers in WSO2 Data service server. In WSO2 DSS, you have two types of event triggers.

1. Input event triggers - which trigger's an event looking at the request.
2. Output event triggers - which trigger's an event looking at the response.

For both of these trigger's you can specify an ex-path expression to trigger the event.

To demonstrate this functionality I am going to write a small data service which insert Student information. And every time an insertion happen, primary key of the inserted record will be send to a JMS Queue.

Prerequisites

In order to try out this sample you need to have
-> WSO2 Data Services Server Downloaded
-> You need to have a database of your choice
-> Add related drivers to DSS_HOME/repository/component/lib folder
-> ActiveMQ downloaded and started

Step 1 - Enable JMS senders in WSO2 DSS.

In order to send JMS messages from data services, you need to enable the JMS sender in data services server.
To enable JMSSender -> Go to DSS_HOME/repository/conf/axis2/axis2.xml and un-comment the following line.

<transportSender name="jms" class="org.apache.axis2.transport.jms.JMSSender"/>

Click on Add Data Source and give database configuration appropriately and click on next.

Next section we will discuss how we can write a data service Query which can invoke a trigger according to the query result.

In this Query my SQL is simple insert

INSERT INTO Student(name,address,country,phone,major,gpa,tutorid) VALUES(:name,:address,:country,:phone,:major,:gpa,:tutorid)

Click on "Generate Input Mappings", and this will generate the input mappings for the input parameters.

Since we need a response back (ideally it will be the primary key of the newly inserted record) we can click Return generated key which will auto generate the response.

To add the event trigger, scroll down a little bit. Under Events, click on Manage Events. There, we give the eventing configurations.

Event ID - student_addition_trigger

Xpath - //*[local-name()='ID' and namespace-uri()='http://ws.wso2.org/dataservice']>0

Target Topic - student_insertion_topic

Event Sink URL ( JMS URL according to your activeMQ)

jms:/student_insertion_topic?transport.jms.DestinationType=queue&transport.jms.ContentTypeProperty=Content-Type&java.naming.provider.url=tcp://localhost:61616&java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&transport.jms.ConnectionFactoryType=queue&transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory

Go to Main Configuration and add the Output trigger as shown below.

Now we have done with our Query Section. Click on Save to save the Query and Click next to add the operation.
Add New Opperation ->
Operation Name* - InsertStudent
Query ID* - insertQ

Save and finish.

Now we can test this functionality by using the try it function provided by Data Service Server

My Request

<body> <p:InsertStudent xmlns:p="http://ws.wso2.org/dataservice">
<p:name>Amani</p:name>
<p:address>Soysa</p:address>
<p:country>SL</p:country>
<p:phone>1923123111</p:phone>
<p:major>SE</p:major>
<p:gpa>4.2</p:gpa>
<p:tutorid>12</p:tutorid>
</p:InsertStudent>
</body>

My Response

<GeneratedKeys xmlns="http://ws.wso2.org/dataservice"> <Entry>
<ID>5</ID>
</Entry>
</GeneratedKeys>

If you go to your active MQ console, You would see number of pending messages as shown below. You can go inside your Queue and explore more!!!!

How to Receive Emails to WSO2 ESB

2013-10-12T20:52:00.001-07:00

In one of my previous blogs, I have explained how we can send emails using WSO2 ESB, in this post I am going to explain how we receive emails to WSO2 ESB, so that you can collect data from in coming emails and do some processing and send them to data storage or any other end point.

Before we start ESB server, you need to enable mail transport listeners in ESB. In order to do that you need to go to WSO2_ESB_HOME/repository/conf/axis2/axis2.xml and un comment the following.

<transportReceiver name="mailto" class="org.apache.axis2.transport.mail.MailTransportListener" />

Then start the server.

Creating the proxy service

To create the proxy, go to add -> Proxy service -> Custom Proxy service. There we'l give the name as MyMailProzy.
And you need to give set of service parameters as listed below.

 <parameter name="transport.PollInterval">5</parameter>

<parameter name="mail.pop3.host">pop.gmail.com</parameter>
<parameter name="mail.pop3.password">wso2mail</parameter>
<parameter name="mail.pop3.user">wso2esb.mail</parameter>
<parameter name="mail.pop3.socketFactory.port">995</parameter>
<parameter name="transport.mail.ContentType">text/plain</parameter>
<parameter name="mail.pop3.port">995</parameter>
<parameter name="mail.pop3.socketFactory.fallback">false</parameter>
<parameter name="transport.mail.Address">wso2esb.mail@gmail.com</parameter>
<parameter name="transport.mail.Protocol">pop3</parameter>
<parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>

And also under Transport Settings you need to enable mailto transport as shown in the below diagram.

Click on next and we will configure the in sequence.

There we will log the sender's email address by getting the sender email address to a property from the transports (get-property('transport', 'From')) and then using log mediator to log it

<inSequence xmlns="http://ws.apache.org/ns/synapse">

Click on next to configure the out sequence.

In the out sequence we will be having a simple send mediator which will send the response back to the client.

The full ESB Configuration

<?xml version="1.0" encoding="UTF-8"?>

<proxy xmlns="http://ws.apache.org/ns/synapse"
name="MyMailProzy"
transports="mailto"
statistics="disable"
trace="disable"
startOnLoad="true">
<target>
<inSequence>
<property name="senderAddress" expression="get-property('transport', 'From')"/>
<log level="full">
<property name="Sender Address" expression="get-property('senderAddress')"/>
</log>
<drop/>
</inSequence>
<outSequence>
<send/>
</outSequence>
</target>
<parameter name="mail.pop3.host">pop.gmail.com</parameter>
<parameter name="transport.PollInterval">5</parameter>
<parameter name="mail.pop3.password">wso2mail</parameter>
<parameter name="transport.mail.ContentType">text/plain</parameter>
<parameter name="mail.pop3.socketFactory.port">995</parameter>
<parameter name="mail.pop3.user">wso2esb.mail</parameter>
<parameter name="mail.pop3.socketFactory.fallback">false</parameter>
<parameter name="mail.pop3.port">995</parameter>
<parameter name="transport.mail.Address">wso2esb.mail@gmail.com</parameter>
<parameter name="mail.pop3.socketFactory.class">javax.net.ssl.SSLSocketFactory</parameter>
<parameter name="transport.mail.Protocol">pop3</parameter>
<description/>
</proxy>

Sending the EMAIL

I have used my gmail to send the email to ESB, my to address will be wso2esb.mail@gmail.com, and you can customize your own subject, and the message body.

Once we send the email, the email will be received to ESB, and you will see the following logs

[2013-10-13 09:06:45,038]  INFO - LogMediator To: , From: mailto:wso2esb.mail@gmail.com, WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: <CAMJOoU667WqnFbx-cCG=hgOy3E2i2qRqKaAUoAcETJE8a-2b+g@mail.gmail.com>, Direction: request, Sender Address = Amani Soysa <amani.soysa@gmail.com>, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">--001a11c233ea18b53d04e8970f15&#xd; Content-Type: text/plain; charset=ISO-8859-1&#xd; &#xd; Sending emails to  WSO2 Enterprise Service Bus!!!!&#xd; &#xd; --001a11c233ea18b53d04e8970f15&#xd; Content-Type: text/html; charset=ISO-8859-1&#xd; Content-Transfer-Encoding: quoted-printable&#xd; &#xd; &lt;div dir=3D"ltr">Sending emails to=A0&lt;span style=3D"color:rgb(68,68,68);fon=&#xd; t-family:arial,sans-serif;line-height:16px" >=A0&lt;/span>&lt;span style=3D"font-w=&#xd;eight:bold;color:rgb(68,68,68);font-family:arial,sans-serif;line-height:16p=&#xd;x" >WSO2 Enterprise Service Bus!!!!&lt;/span>=A0&lt;/div>&#xd;&#xd; --001a11c233ea18b53d04e8970f15--&#xd; </text></soapenv:Body></soapenv:Envelope>

How to call multiple soap operations using call out mediator - WSO2 ESB

2013-10-12T07:38:00.003-07:00

In this post I will explain how we can call multiple operations inside the same proxy service using the call out mediator in WSO2 ESB. For this sample I have used my distributed transaction sample where we need to call multiple operation in the same transaction.

In order to do this you need to have WSO2 ESB, as well as sample web service with multiple operations. For that I will be using WSO2 DSS distributed transaction data service which I have explained in my previous blog post. Where we will insert customer information, in to multiple data bases in the same transaction.

Sample input

<customer>

<id>2</id>
<name>Smith</name>
</customer>

Step 1 - Creating the proxy service.

To create the proxy, you need to start the ESB server under services -> add -> proxy service -> Custom Proxy service. Give an appropriate name. For this sample I am calling it MyTransactionProxy.

In the in sequence I will be calling several mediators.

Log Mediator - To log the incoming request
Property Mediators - To store the input parameters
Payload Factory Mediator - to arrange the payload for each operation
Call out mediator - to call each operation of the given service

Sample In sequence

<inSequence xmlns="http://ws.apache.org/ns/synapse">

Here I have created payload factory mediator and a call out mediator to call each operation. In the call our service URL I have given the end point for my service. For example I have an operation called my_insert. So I created a payload factory mediator and defined the payload format inline as shown below (parameters for the operation is given as $1,$2 which are later replaced by the id, and the name properties.

<payloadFactory xmlns="http://ws.apache.org/ns/synapse" media-type="xml">

<format>
<p:my_insert xmlns:p="http://ws.wso2.org/dataservice">
<xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>
<xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>
</p:my_insert>
</format>
<args>
<arg expression="get-property('id')" evaluator="xml"/>
<arg expression="get-property('name')" evaluator="xml"/>
</args>
</payloadFactory>

Likewise, I have called each operation with the same format.

In the out sequence I have used an empty send mediator which will send back the response to the same service.

Complete proxy xml is attached below.

<proxy xmlns="http://ws.apache.org/ns/synapse"
name="TransactionProxy"
transports="http"
statistics="disable"
trace="disable"
startOnLoad="true">
<target>
<inSequence>
<log level="full">
<property name="M1"
value="*************HITTING Transaction PROXY*************"/>
</log>
<property name="OUT_ONLY" value="true"/>
<property name="id" expression="//id/text()"/>
<property name="name" expression="//name/text()"/>
<payloadFactory media-type="xml">
<format>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:dat="http://ws.wso2.org/dataservice">
<soapenv:Header/>
<soapenv:Body/>
</soapenv:Envelope>
</format>
</payloadFactory>
<callout serviceURL="https://localhost:9445/services/DTPDS/"
action="urn:begin_boxcar">
<source type="envelope"/>
<target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
</callout>
<payloadFactory media-type="xml">
<format>
<p:my_insert xmlns:p="http://ws.wso2.org/dataservice">
<xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>
<xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>
</p:my_insert>
</format>
<args>
<arg expression="get-property('id')" evaluator="xml"/>
<arg expression="get-property('name')" evaluator="xml"/>
</args>
</payloadFactory>
<callout serviceURL="https://localhost:9445/services/DTPDS/"
action="urn:my_insert">
<source xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
<target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
</callout>
<payloadFactory media-type="xml">
<format>
<p:pos_insert xmlns:p="http://ws.wso2.org/dataservice">
<xs:id xmlns:xs="http://ws.wso2.org/dataservice">$1</xs:id>
<xs:name xmlns:xs="http://ws.wso2.org/dataservice">$2</xs:name>
</p:pos_insert>
</format>
<args>
<arg expression="get-property('id')" evaluator="xml"/>
<arg expression="get-property('name')" evaluator="xml"/>
</args>
</payloadFactory>
<callout serviceURL="https://localhost:9445/services/DTPDS/"
action="urn:pos_insert">
<source xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
<target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
</callout>
<payloadFactory media-type="xml">
<format>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:dat="http://ws.wso2.org/dataservice">
<soapenv:Header/>
<soapenv:Body/>
</soapenv:Envelope>
</format>
</payloadFactory>
<callout serviceURL="https://localhost:9445/services/DTPDS/"
action="urn:end_boxcar">
<source type="envelope"/>
<target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
</callout>
</inSequence>
<outSequence>
<send/>
</outSequence>
</target>
<description/>
</proxy>

Calling the proxy service

Here I am calling the proxy service using the curl tool.

curl -v --request POST -d '2Smith' -H Content-Type:"text/xml" http://amani-ThinkPad-T520:8280/services/MyTransactionProxy

Distributed Transaction with WSO2 Data Services

2013-10-09T04:59:00.002-07:00

What is distributed transaction ?

Distributed database transaction means executing multiple related actions/operations in a coordinated way. This is also known as global transaction. Distributed transaction can occur in the same database level however, in most cases distributed transaction happens in different databases (typically different RDBMS types) and often in different locations. Distributed transactions are often described as ACID -- atomic, consistent, isolated, and durable. Meaning changes made to the database during the transactions are tentative, if any of the operation fails then none of the other changes will get affected. In a typical distributed transaction, you have to make sure if one operation fails the whatever previously executed operation should roll back, undoing all the changes as if the transactions never took place. Even if your application crashes in the middle of the transaction, when it restarts, transaction recovery should roll back the open transaction.

A typical Distributed transaction example would be moving data from one data base to another database. For example lets say you want to delete customer from one location and add that same customer to the another location.You would not want either transaction committed without assurance that both will complete successfully. Therefore, for these kind of instances its important to have distributed transaction feature.

The above transactions involve the following steps:

Begin a transaction.
Add Customer
Delete Customer
End transaction

Distributed Transaction with WSO2 Data Services Server

WSO2 data services server provide distributed transaction using Java Transaction API (JTA) which enables global level transaction across multiple X/Open XA resources in java environment.

When you use XA functionality, the transaction manager uses XA resource instances to prepare and coordinate each transaction branch and then to commit or roll back all each of individual transaction appropriately.

For each RDBMS type there is a specific XA-Datasource class and set of configuration properties. Therefore you need to know the XA-Datasource class and their configurations before creating the data service.

Lets see how we can create a data service for the above transaction.

For this example I will have two databases in postgres and mysql. For this demo I will be using a very simple customer table which has id and name and we will see how we can inset values to these two tables in each database in a coordinated manner

=========================== SQL script ==============================

CREATE TABLE customer (
cust_id int NOT NULL,
name varchar(255) NOT NULL,
PRIMARY KEY (cust_id)
)

========================data service configuration file======================

<data disableStreaming="true" enableBoxcarring="true" enableDTP="true" name="DTPDS">
   <config id="pos_ds">
      <property name="org.wso2.ws.dataservice.xa_datasource_class">org.postgresql.xa.PGXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="ServerName">localhost</property>
         <property name="PortNumber">5432</property>
         <property name="DatabaseName">MyDB</property>
         <property name="User">postgres</property>
         <property name="Password">root</property>
      </property>
   </config>
   <config id="my_ds">
     <property name="org.wso2.ws.dataservice.xa_datasource_class">com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="URL">jdbc:mysql://localhost:3306/MyDB</property>
         <property name="User">root</property>
         <property name="Password">root</property>
      </property>
   </config>
   <query id="pos_q" useConfig="pos_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <query id="my_q" useConfig="my_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <operation disableStreaming="true" name="pos_insert" returnRequestStatus="true">
      <call-query href="pos_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
   <operation disableStreaming="true" name="my_insert" returnRequestStatus="true">
      <call-query href="my_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
</data>

You need to add the configuration file to WSO2DS_HOME/repository/deployment/server/dataservices in order to deploy this file

Step by step explanation

Data source configuration

I have created two data source configuration in the data service descriptor file

 <config id="pos_ds">
      <property name="org.wso2.ws.dataservice.xa_datasource_class">org.postgresql.xa.PGXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="ServerName">localhost</property>
         <property name="PortNumber">5432</property>
         <property name="DatabaseName">MyDB</property>
         <property name="User">postgres</property>
         <property name="Password">root</property>
      </property>
   </config>
   <config id="my_ds">
     <property name="org.wso2.ws.dataservice.xa_datasource_class">com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</property>
      <property name="org.wso2.ws.dataservice.xa_datasource_properties">
         <property name="URL">jdbc:mysql://localhost:3306/MyDB</property>
         <property name="User">root</property>
         <property name="Password">root</property>
      </property>
   </config>

There we have specified the XA datasource classes "org.wso2.ws.dataservice.xa_datasource_class" in a property along with it's parameters. For each XA datasource class configuration properties may differ.

Query Configuration

  <query id="pos_q" useConfig="pos_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>
   <query id="my_q" useConfig="my_ds">
      <sql>INSERT INTO customer VALUES(?,?)</sql>
      <param name="id" sqlType="INTEGER"/>
      <param name="name" sqlType="STRING"/>
   </query>

I have created two queries pointing each data source, and also two distinct opperations mapping to each query.

<operation disableStreaming="true" name="pos_insert" returnRequestStatus="true">
      <call-query href="pos_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>
   <operation disableStreaming="true" name="my_insert" returnRequestStatus="true">
      <call-query href="my_q">
         <with-param name="id" query-param="id"/>
         <with-param name="name" query-param="name"/>
      </call-query>
   </operation>

After you deploy this data service. You will see the deployed data service under the data service list. You can invoke this service using the try-it tool (or using your own class)

Invoke the operations in this order

begin_boxcar
my_insert
pos_insert
end_boxcar

Please make sure you have set the "max_prepared_transactions" to a non zero value in "/etc/postgres/postgres.conf in oorder this sample to work

In my next post, I will be explaining how we can call all these four operations using a single proxy service with the use of of WSO2 ESB.

Configure HTTP Access Logging in WSO2 products

2013-07-25T18:38:00.001-07:00

If you want to analyze your application's usage activities such as who is accessing your page, number of hits, errors ect, you can use the HTTP access log files monitor and analyze the above measures. WSO2 products uses embedded tomcat as it's primary runtime therefore, you can use apache tomcat access logger to monitor your activity and performance of the server as well as any errors that may be occurring. This post explain how you can configure HTTP access logging in WSO2 products in order to get efficient monitoring.

In WSO2 products you can customize the http access log configuration by editing the catalina-server.xml which is located {CARBON_HOME}/repository/conf/tomcat directory, which is the server descripter file for the embedded tomcat integration. In the catalina-server.xml, under valves you have the HTTP access log configuration as shown below

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs"

prefix="http_access_" suffix=".log"

pattern="combined" />

The server access log records all requests processed by the server. You can modify the what to log and what not to log by customizing the pattern attribute.

In the pattern attribute we can define formatting layouts. A formatting layout identifying the various information fields from the request and response to be logged, or the word "common" or "combined" to select a standard format.

Values for the pattern attribute are made up of literal text strings, combined with pattern identifiers prefixed by the "%" character to cause replacement by the corresponding variable value from the current request and response. The following pattern codes are supported:

%a - Remote IP address

%A - Local IP address

%b - Bytes sent, excluding HTTP headers, or '-' if no bytes were sent

%B - Bytes sent, excluding HTTP headers

%h - Remote host name

%H - Request protocol

%l - Remote logical username from identd (always returns '-')

%m - Request method

%p - Local port

%q - Query string (prepended with a '?' if it exists, otherwise an empty string

%r - First line of the request

%s - HTTP status code of the response

%S - User session ID

%t - Date and time, in Common Log Format format

%u - Remote user that was authenticated

%U - Requested URL path

%v - Local server name

%D - Time taken to process the request, in millis

%T - Time taken to process the request, in seconds

%I - current request thread name (can compare later with stacktraces)

In addition, the caller can specify one of the following aliases for commonly utilized patterns:

common - %h %l %u %t "%r" %s %b

combined - %h %l %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i"

Please note that the optimized access does only support common and combined as the value for this attribute.

If you want to change the format you can modify the as shown below

 <Valve className="org.apache.catalina.valves.AccessLogValve" directory="${carbon.home}/repository/logs"

prefix="http_access_" suffix=".log"

pattern="%h %l %u %t '%r' %s %b" />

You can further change the log file location by changing the directory, prefix is the name of the log file. By default logs are rotated daily and the date is appended to the log file name. However, you can disable the log rotation by adding the following attribute rotatable="false". If you want to not have the current date appended in the log file name however, need the date to the rotated log file you can add renameOnRotate="true" attribute the the valve configuration. Please refer Access Log Valve Attributes for the full list of supported attributes.

How to enable tenant aware service logging using WSO2 ESB

2013-05-08T21:07:00.002-07:00

Unlike in other WSO2 products (WSO2 Appserver, WSO2 Data Services, ect) we will not be able to see the application/service level logs in ESB by default in ESB. Mainly because in the current carbon logging framework is tenantaware and synapse is not multi tenanted and it runs on its own worker thread pool and does not know aboout the carbon context. Since it uses an independent thread pool from the carbon context logging does not work as expected with NHTTP. Therefore, to overcome this we wrote a handler call TenantDomainSetter which will set the tenant infomation/application infomation in the carbon context of each request.

To enable this you need to go to CARBON_HOME/repository/conf/axis2/axis2.xml and add the TenantDomainSetter in the prefetch of the request path (PreDispatch).

Once you do that restart WSO2 ESB then you will be able to see service specific logs in LogViewer (monitor -> View Application Logs) which will show the logs of ESB proxy service logs. However application log viewer DOES NOT filter out Sequances, APIs, which are components inside a proxy service ... it will ONLY list down the proxy services.

Application Logs

System Logs

Please note since we are setting the tenant domain for every request there will be an aditional overhead in ESB.. if you want to minimize the overhead its recomended to switch of TenantDomainSetter and use ESB.

Kick start on Openstack with Devstack {Folsom}

2013-04-28T07:07:00.000-07:00

Recently I was told to install openstack controller-compute node set ... and my first impression on it was piece of cake!! because you get couple of scripts and all you need to just run the script.. !! I WAS SOO WRONG!! yes it is just couple of script but you need to know how things work and there are soo many places where it can totally messed up your machine and your set up. Meaning you can easily get messed up!!! If you don't go in the right path it will take ages .. and you might even have to format your machine several times to get things clean and start all over again which is very time consuming.. So I ended up installing openstack for almost a week spending day and night on my precious development time!!!

So I decided to write a post on focusing on HOW TO INSTALL OPENSTACK? So other people want do the same mistake I did or take ages to get things done. If you know what you are doing then installing devstack only take one hour's time all you need to do is make your configuration right and run the scrip!!!

So before we begin I would like to give a small introduction to OpenStack to make things more clearer.

What is OpenStack ?

In simple terms OpenStack is an opensource cloud operating system ... It is like the fabric of the cloud. OpenStack was initially founded by rackspace and NASA in 2010 and its being growing ever since. The best way to get what OpenStack is by reading their mission which is very simple and direct

"To produce the ubiquitous Open Source Cloud Computing platform that will meet the needs of public and private clouds regardless of size, by being simple to implement and massively scalable."

OpenStack is written in purely written in python, true it use other systems like MYSQL and rabbitMQ but the core is written in python which makes things very efficient and fast. And also initially it had two main component 1) Nova - which is the compute side of it and the other one 2) swift - the data storage side of it.

Current OpenStack stable version {Folsom} have few core components

Openstack object storage (swift)
OpenStack image registry and delivery (Glance)
Openstack compute (nova)
Horizon (web UI)
Keystone (authentication)
Quantum/Melange (Network)

In future OpenStack is going to provide other compoents such Cellometer (Billing service), Cinder (block storage) and load balancer services. So that's enough about OpenStack lets see how we can install it. I will be installing OpenStack using devstack..

What is devstack ?

Well devstack is openstack development project (there are quite a few :) ) but I think devstack is the easiest :). Devstack provides a documented shell script (which is written in bash) to build a complete OpenStack development environments. Originally created by Rackspace now maintained by OpenStack community. This supports both ubuntu and fedora (but today we will be focusing how we can install openstack in ubuntu 12.04).

Installing OpenStack

In this installation I will be explaining how we can install OpenStack in Multi-Node environment.

Prerequisites

Install ubuntu 12.04 and

Install git
apt-get install -y git sudo

Network setting on each node

You need to have a stack ip and also decide on what your Floating IP range (public IP range ) and Fixed IP range (private IP range) are.

This is my network configurations

Static IP (controller ) - 10.100.0.30

Floating (Public) IP range - 10.100.0.128/25

Fixed (Private) IP range - 192.168.16.0/25

Go to /etc/network/interfaces and configure your network accordingly.

Controller node

auto eth0
iface eth0 inet static
address 10.100.0.30
netmask 255.255.252.0
gateway 10.100.1.254

Compute Node

auto eth0
iface eth0 inet static
address 10.100.0.29
netmask 255.255.252.0
gateway 10.100.1.254

Before we begin we need to create separate user to do things in a cleaner way.

groupadd stack
useradd -g stack -s /bin/bash -d /opt/stack -m stack
echo "stack ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

From here onward we will be doing all the operation from the stack user.

Once you create the user log out and login from the the stack user and download openstack using git. When you are downloading make sure you download the stable version (in my case its folsom) so you will not have any surprises. :)

sudo su stack # login from the user
cd # go to user home
git clone git://github.com/openstack-dev/devstack.git -b stable/folsom #download devstack stable version

Once you download devstack other things are pretty straight foward. Go to devstack folder and create localrc file and add the configuration accordingly in each node.

Configuring and Installing OpenStack in Controller Node

vi localrc

HOST_IP=10.100.0.30
FLAT_INTERFACE=eth0
FIXED_RANGE=192.168.16.0/25
FIXED_NETWORK_SIZE=126
FLOATING_RANGE=10.100.0.128/25
MULTI_HOST=1
LOGFILE=/opt/stack/logs/stack.sh.log
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=root
RABBIT_PASSWORD=rabbitmq
SERVICE_PASSWORD=password
SERVICE_TOKEN=poohdedoo

Once you create the file you can run the setup script using the following command.

./stack.sh

After a successful installation you will get the following output

Horizon is now available at http://10.100.0.30/
Keystone is serving at http://10.100.0.30:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: openstack
This is your host ip: 10.100.0.30
stack.sh completed in 3443 seconds.

Configuring and Installing OpenStack in Compute Node

Just like the controller node you need to have the basic infrastructure in order to install the computer. Only change is there is the content of the localrc file in the compute node.

vi localrc

SERVICE_HOST=10.100.0.30
HOST_IP=10.100.0.29
FLAT_INTERFACE=eth0
FIXED_RANGE=192.168.16.0/25
FIXED_NETWORK_SIZE=126
FLOATING_RANGE=10.100.0.128/25
MULTI_HOST=1
LOGFILE=/opt/stack/logs/stack.sh.log
ADMIN_PASSWORD=openstack
MYSQL_PASSWORD=root
RABBIT_PASSWORD=rabbitmq
SERVICE_PASSWORD=password
SERVICE_TOKEN=poohdedoo
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
Q_HOST=$SERVICE_HOST
ENABLED_SERVICES=n-cpu,n-net,n-api,c-sch,c-api,c-vol,rabbit

That's it !!. Just like the controller node you need to run the setup script to execute the installation.

./stack.sh

Once you successfully install openstack in both nodes you can login to openstack using the horizon dashboard

http://10.100.0.30/ and login according to the user credentials you have provided in your setup configuration.

You can run the exercise.sh run provided by devstack in order to test your set up which includes uploading/launching instances and creating user groups/keypairs and user rules.

If you want to upload images manually using the script you can use the following commands

glance image-create --name mysql-cartridge-amd64 --is-public true --container-format ami --disk-format ami < /home/wso2/wso2s2-openstack-1.0.0/cartridges/mysql-cartridge-amd64.img

To see what you have uploaded

source openrc demo demo

glance image-list

You can also see these images using the horizon dashboard and further do task like launching the instances, assigning public IPs create key pairs and assign rules to security groups ect. I am not going to details since its a very straight forward dashboard and you can easily do everything by following the dashboard.

But I would like to proudly attach my openstack dashboad :) for references sake :)

On a final note restarting openstack is pretty tricky however, you can use the devstack script to restart openstack.

Restarting OpenStack using Devstack

Run the following in order to own the shell

script /dev/null

And execute the following script
./rejoin-stack.sh

inside devstack folder in order to restart openStack.

To get out of the screen use ctrl-a , ctrl-d

Thats it folks !!
Once you set it up its like having a data center in your basement you can test load balancers and you can do all those caching stuff and experiment new stuff its really a cool thing to have. All the things you do which was in a private cloud and you can point it to a public cloud very easily!!!

Carbon Logging framework - Conversion Characters for TenantAwarePatternLayout

2013-04-26T23:18:00.000-07:00

Usually when users use a logging framework one of their primary requirement is to customize the log output format. In Log4j framework this is done using the layout components.

There are couple of predefined layouts comming from the log4j api however, if you want more customize layout with extra set of attributes you need to write your own custom layout extending the org.apache.log4j.PatternLayout class.

In WSO2 Products we do not use the default PatternLyout we have written our own layout to make sure our log format is service aware and multi-tenanted. Therefore we have our own set of conversion characters defined in our TenantAwarePatternLayout class.

Below table explains characters used in the TenantAwarePatternLayout and all other characters (which are comming from PatternLayout) which you can use in your custom pattern.

Conversion Character	Description
Coming from custom carbon layout
D	used to output the name of the tenant domain which is currently logged in to the server (super tenant domain is carbon.super)
T	used to output the tenant id - to uniquely identify each tenant tenant id can be used in the layout (super tenant id is 0)
S	used to output the server key - to uniquely identify each carbon server a server key can be used (ie as,esb,dss )
U	used to output the user name - current user who is logged into the server
A	used to output the application name - service name or web application name of the current running applicaiton
H	used to output the host name
I	used to output the carbon instance id- uuid which is generated for each carbon instance to uniquely identify each instance
Coming from the log4j patternlayout
c	used to output category(logger); a.b.c → %c{2} = a.b (this can be also used as fully qualified name)
d	used to output the date of the logging event. for example, %d{hh:mm:ss,sss} or %d{dd mmm yyyy hh:mm:ss,sss}.
F	used to output the file name where the logging request was issued.
l	used to output location information of the caller which generated the logging event.
L	used to output the line number from where the logging request was issued.
m	used to output the application supplied message associated with the logging event.
M	used to output the method name where the logging request was issued.
n	used to output the platform dependent line separator character or characters.
p	used to output the priority of the logging event.
r	used to output the number of milliseconds elapsed from the construction of the layout until the creation of the logging event.
t	used to output the name of the thread that generated the logging event.

Please note %C, %F, %l, %L, %M slow down program run!

To create your own ConversionPattern you can go to the log4j.properties file and edit ConversionPattern of the appender of your choice.

log4j.appender.CARBON_LOGFILE.layout.ConversionPattern=TID: [%T] [%S] [%d] %P%5p {%c} - %x %m {%c}%n

Relational Storage Solution using WSO2 Storage as a Service

2013-04-26T01:34:00.002-07:00

Lets say you want to create a relational database on the cloud and you want to expose these data to other services?
Its very easy simple and very straight forward. All you need to have is a stratoslive account and then login to WSO2 Storage Server.

WSO2 Stroage Server provides almost all the data administrator level functionality when it comes to creating and manipulating data storage. Basically it provides features such as creating and manipulating data storage for relational data bases, Cassandra data storage and hadoop/hdfs file system.

Wondering How to create databases in the cloud ??

It only takes 4 clicks ... to Create and secure your database on the Cloud

In this article I am going to demonstrate on how you can create a database, and assign users and special privileges to that data base within few clicks

Step 1 - Creating Users

Assuming you already have a stratoslive account and logged in to storage as a server, lets first try to create a user for the database we are going to create.

Under Manage -> RSS Manager (relational storage service manager), you have the three basic functions to manipulate relational data bases.

To Create users click on Database Users. This will show the list of users (if you have not created users before you will not see any) along with "Add Database User "

Select Database Server Instance Name as WSO2_RSS (which is the default server), and give the user credentials as your choice and click save to save the user.

Step 2 - Creating Privilege Group

Once you create the user you can create a template which separate out set of functions which can be performed on a given database, so you can assign roles.

Go to RSS Manager -> Privilege Templates -> Add Database Privilege Template.

Lets say you only want this template to have INSERT UPDATE DELETE CREATE DROP. Select the above fields along with a suitable name and save.

Step 3 - Creating Database

Now lets create a database in the cloud and assign the created users and privilege template to that database.

Go to RSS Manager -> Databases -> Add Database

Give the RSS instance name, database name and save.

Once you create the database it will go under Databases list, where you have the option to manage databases by adding users and privileges to that database.

Go to RSS Manager -> Databases -> database name -> Manage -> Attach Database User

Assign the user and role we created in step1 and step2 to the database as shown below.

Once you attach this user and role you can use this database with the user and the jdbc url shown in the databases page.

That's It .. now you have created your very own database in the cloud!!!

If you want view this database using a mysql client.. you can use WSO2 Data Services Server DB explorer.

Go to WSO2 Data Services Server -> Tools -> Database Explorer.

Give the database information accordingly along with our newly created database and user. And log in to your database.

And you can create your tables and manage data accordingly

If you want to expose your data as services you can use the following guide which explains how to expose your data as a service within few clicks.

How to setup Cassandra Cluster using WSO2 Storage Server

2013-03-21T09:15:00.000-07:00

In my previous article I have explain how to configure multi-tenant logging in WSO2 stratos, in this post I am going to explain how we can create an external carbonized cluster and point it to BAM so that our logs will be stored in an external cassandra cluster. If you are having a production deployment and if you want to store big data for example daily logs of all servers, then you need to have an external cassandra cluster for high available and for high perfomance.

So to have a carbonized cassandra cluster.. we are going to use WSO2 storage server. WSO2 Storage server provides rich set of tools to create and manipulate storages such as relational data storages, cassandra and hdfs file systems.

To start you need to download WSO2 Storage Server.

Before we begin you need to have a basic understanding on cassandra in order to get a clear understanding on what we are trying to do. Unlike most master-slave deployments .. cassandra does not use master/slave architecture, it uses a peer-to-peer implementation which avoids the pitfalls, latency problems, single point of failure issues and perfomance hits assosiated with master/slave setups. Which makes cassandra more high available and efficient.

So basically what happens when we write to cassandra is client write to any node in the cassandra cluster controller node replicate to nodes and zones and nodes return acknowledgement to coordinator. Then coordinator return ack to client and data is written to internal commit log disk. If a node goes offline hinted handoff completes the write when the nodes come back up/

So lets begin to configure carbonized cassandra ..

Storage Server Management node deployment steps

cassandra-component.xml -

cassandra-component.xml point the backend Cassandra cluster.

<Name>SSCluster</Name>

</Cluster>

</Cassandra>

cassandra-auth.xml

User has to create a system user with admin privilege to communicate with the cassandra backend and

configure the cassandra-auth.xml with that user and the remote shared key publisher service URL.

<EPR>https://cassandra.cluster.backend.ip:9443/services/CassandraSharedKeyPublisher</EPR>

<User>admin</User>

<Password>admin</Password>

</Cassandra>

For the EPR you can give the IP of the first backend server node, you also need to consider the offset when providing the port as it will not automatically change it.

Since we are doing many hdfs/hadoop functions (to complete the bam story) using relational data storage we also need to configure rss-config.xml

rss-config.xml

There you need to give the datasource properties (jdbc url of the mysql server and user credentials) accordingly.

<property name="URL">jdbc:mysql://mysql.stratos-local.wso2.com:3306/rss_db</property>

</dataSourceProps>

Storage Server Cassandra cluster deployment steps

All the nodes in the Storage Server Cassandra cluster should configure to use common carbon user base as any other carbon server. User has to update the user-mgt.xml and registry.xml with correct
configurations.

Node Configuration

cassandra.yaml

Start the node with default seed configuration

- seeds: "127.0.0.1"

Edit the cluster listening address

listen_address:cassandra.node.ip

Edit the thrift listening address

rpc_address::cassandra.node.ip

Both these ip addresses are the ip address of the machine it self which cassandra recedes on.

If you are having another node as the seed node, then you can seed node to boostrap

rpc_address:ip.of.cassandra.api

Configuring BAM to talk to External Cassandra

Once we configure the external carbonized cassandra we need to tell the BAM we are no longer using the internal cassandra inbuilt in BAM but to use the external cassandra. In order to do that you need to start the BAM server with the following system property.

-Ddisable.cassandra.server.startup=true

And also you need to configure the following configuration files.

cassandracomponent.xml

<Cassandra>
<Cluster>
<Name>ClusterOne</Name> 
<Nodes>node0.cassandra.com:9160,node1.cassandra.com:9160,node2.cassandra.com:9160</Nodes> 
<AutoDiscovery disable="false" delay="1000"/>
</Cluster>
</Cassandra>

cassandra-auth.xml

<Cassandra> <EPR>https://cassandra.cluster.backend.ip:9443/services/CassandraSharedKeyPublisher</EPR>

<User>admin</User>

<Password>admin</Password>

</Cassandra>

rss-config.xml

</dataSourceProps>

<property name="URL">jdbc:mysql://mysql.stratos-local.wso2.com:3306/rss_db</property>

</dataSourceProps>

Also if you are using BAM analytics you need to configure the hive-site.xml as well

hive-site.xml

<name>javax.jdo.option.ConnectionURL</name>

<value>jdbc:mysql://mysql.server.url/hive_db</value>

<description>JDBC connect string for a JDBC metastore</description>

</property>

<name>fs.default.name</name>

</property>

<name>mapred.job.tracker</name>

<value>hdfs.url:hdfs_job_tracker_port</value>

</property>

Introducing Leamo - To LEarn And MOre

2013-03-06T05:52:00.002-08:00

Today I thought of introducing a cool cutting edge technology, which can help young minds to learn educate and have fun. Leamo, is an interactive learning platform which uses an iphone and bunch of flash cards and brings a magical environment to home/pre schools for kids under all ages.

Why not traditional learning ?

Most traditional education tools/applications uses the sit and learn approach, where educators have to talk and interact with kids while kids sit and listen. Most kids get soo bored with this traditional learning approach, and sometimes they get tired of the whole education system. Kids are more energetic than adults they need to run around poke things to nurish their curious minds. They cant just listen to you all the time and agree on what you say rather they like to experiment with new things and then learn how things work.

Why Leamo ?

Kids like to play with toys, flash cards, play-dough and also interact with things without just listening and watching. They like to touch and feel things rather than look and listen. In the modern days kids are more attracted to digital screens, such as iphone, ipads, tablets and smartphones. Even though these ipad/ipones apps are very attractive for kids, there are major draw backs when it comes to kids education. I agree there are good educational applications provided by these gadgematics, however, it will not make your child develop the basic skills such as creativity, motor skills and hand-eye coordination ect. Most of these skills are develop when they interact with physical objects. So the solution is to mix the digital world with the physical world and give a hybrid educational platform for kids. So they wont get bored, or get distracted while learning and give a fulfilled learning experience.

This is the base of leamo, leamo provides a novel way of educating your child, using flashcards and digital surface. Where there are several education apps focusing mathematics, science and language skills along with history,arts music which will give them a balanced and rich education and develop both sides of their brain.

How Leamo Works?

Leamo runs on ipad, iphones and ipod (at the moment however, it will improve for android devices as well), all you need to do is connect your apple device to an LCD screen or a projector, and run the application. Leamo will come with bunch of flash cards and physical toys so the kids can interact with the digital world through the physical objects, by placing them in the correct place at the correct time. Leamo will guide you with all the kid's activities, parental supervision is not needed, kids can learn independently at their own space. Leamo will provide bunch of educational applications such as number games, language, fun with arts and music and story telling which will take the kids to a magical world where learning becomes addictive and fun. This is the best educational tool for energetic kids where they can run around, talk with it and make imagination come to reality.

Educating young minds ....

From my experience as a technology for education enthusiastic and a technology provider for kids, there are major advantages using Leamo.One of them is making them eager and addicted to learning new things through technology. And it improves their creative thinking skills, psycho-motor skills and reactive skills compared to other learning tools. Last but not least you don't need much parental guidance with leamo, you can easily set it up and make them learn independently.

For more information on Leamo, please visit get-leamo and be amazed!!!!. Leamo is a proud product of Kreyonic, An Interactive Digital Media Solutions provider specialize in providing interactive experiences, mobile solutions, that augment your perception of reality to create a new dimension in multimedia experience, taking digital media to the next level.

Introduction to GIT with GIT commands

2013-02-23T23:05:00.003-08:00

Recently I had some experiance working with git and I thought of sharing most basic and mostly used set of git commands. Before we begin I will give a quick introduction to "git".

What is "git"?

Git is an open source distributed version control system designed to manange source code, system which designed for speed and efficiency. So the important factor here to remember is git is a distributed version control system unlike the old fasion centralized version control system such as SVN. In a centralized VCS you have a server which has all the source code and clients, and these two parties can be distringuishly identified. If a client wants to use the code to do any modification, they have to checkout the source from the server and comit back to the server. So the entire code base stays in a single data store.If you loose the data store then you are screwed!!!. Git on the other hand is a distributed version control system, what it means is you dont check out a version of a project to start working but you clone it. So you just clone it to the local file system, this way is far efficient and you can work offline (You dont have to be online for most operations) no network issues,you can push and pull directly to peers. This means almost everything is local, so its very fast, every clone is a backup (so everyone working in that project has a backup for the system), and you can work offline.

Why use "git"?

I asked a friend why he thinks a project/company should adopt "git", his simple answer was less build breaks :D (which of course a good enough answer) and you can give restricted commit rights to team members so no careless glitches :). But of course there's more to that story, So I listed down few advantages which I could think of "git" and Why projects should move to "git"

Its very fast (since almost everything is local)

You want lose your code (Every clone is a backup to the system)

You can work offline to perfom each of the mostly used task such as:

performing a diff,

viewing file history,

committing changes,

merging branches,

obtaining other revision of a file,

switching branches

Which means you can work anywhere in the world even when you are up in the sky !!!!

Its imutable (it never removes data) -"git" will not re write your history, it will always write a new history (you will have a pointer to your new history) you can always go back so you want lose data.

Before I begin I should warn you, if you are used to old fasion version control system like SVN you are going to hate "git". You will start hating everything about "git" and get tired of it very easily. Because its very differant from the centralized version control systems for example most version control systems are file based delta storage its mainly file based operations on the other hand "git" thinks about data as snapshots,it looks at the content (ignores the filename) and put that content in the database as key value pairs and return the key.That is, instead of thinking about and storing commit points as file based patches or changes, it stores it as a simple snapshot of what your project looked like when you committed.. So the easiest way to get you hands on these cool stuff, you need to foget all you know about centralized version control systems, youve been using for the past years and start thinking differantly and it will blow your mind :)

Before I bore you off with the conceptual infomation on "git" I will start giving the most commonly used "git" commands to start off with "git" and I will continue explaining interesting stuff about "git" in my next blog posts.

init - This will initialize a brand new git repository in a project directory.

git init

clone - This will clone an exact copy of an existing project.

git clone http://git.stratos.com/amani.com/poo

add - Adding files

To add a single file:

 git add info.php

To add multiple files:

 git add info.php README.txt

To add all the files in the directory:

git add .

Status - This tells you what files have been modified since the last time they were committed.

git status

commit - Commit changes to head (but not yet to the remote repository):

git commit -m "Committing my changes"

Push - Send changes to the master branch of your remote repository

git push origin master

Blame - To check who screwed it up :)

git blame hello.java

If you screwed it up :( ?

reset - Revert the uncommited changes from last commit

git reset --hard HEAD

checkout - Undo local changes

git checkout -- myFile.txt

* If you mess up, you can replace the changes in your working tree with the last content in head:

Changes already added to the index, as well as new files, will be kept.

fetch - fetch the latest history from the server and point your local master branch at it.

git fetch origin

git reset --hard origin/master

Grep - Search the working directory for isService():

git grep "isService()"

Thats it for now to get a quick start on git try GitHub. GitHub made git more easy, you can try it out and boost up on git with GitHub. :)

How to configure a hadoop cluster

2013-02-19T06:13:00.003-08:00

Prerequisites

Before we begin there are couple of softwares you need to install along with the hadoop user, before installing hadoop .

Java - Install Java into a location where all the user groups can access.
Eg: opt/java/jdk-1.6_29

rsync - Install rsync using apt-get (This is to copy the Hadoop distribution of Name Node across all the other nodes)

Create hadoop user - Navigate to /home. Create the user “hadoop”

To login as user hadoop using the command.

su - hadoop

bash

PS: Above steps need to be performed (software should be installed) on all the other Hadoop nodes as well

Setup public key login from master to slave nodes

Create ssh public keys for each user (ssh-keygen -t rsa -b 2048) and added the public key (*.pub) to the authorized_keys file in master and slave nodes.

Key Exchange for Passphraseless SSH

1. We need to have password / passphraseless SSH to communicate with other Hadoop nodes in the
cluster.

Try to SSH to another node
   ssh hadoop@amani26.poohdedoo.com

2. Generate a key for the Name Node using the following command.

   ssh-keygen

This will generate an output similar to below.

Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
8207:9e:1e:60:37:28:03:a6:18:b3:b6:f1:e4:2f:ef hadoop@bam01

3. This will create a .ssh directory inside the ‘hadoop’ user account. Navigate and to .ssh directory. It will contain a file with the generated public key. Inspect the public key stored in the ‘id_rsa.pub’ file

with the command,
cat id_rsa.pub

It will display the public keys …

4. This public key of the Name Node should appended to the ‘authorized_keys’ file in the other Data
Nodes. Execute the following command and copy the id_rsa.pub file into the other nodes.

   scp id_rsa.pub root@amani276.poohdedoo.com:/root

5. Login to the second Hadoop node’s ‘hadoop’ user account. Try to SSH to another node from this.

   ssh hadoop@amani26.poohdedoo.com

This will create the .ssh directory in the hadoop account.

6. Append the copied public key to the ‘authorized_key’ file in the hadoop account of this Data node.
Execute the following commands.

      cat /root/id_rsa.pub > authorized_keys
   chown hadoop:hadoop authorized_keys
   chmod 600 authorized_keys

7. Now you can ssh to this Data node from the earlier configured Master node. Login to the Master
node. From the hadoop account, login to the Data node with the following command.

ssh -i id_rsa hadoop@amani27.poohdedoo.com
   or
ssh hadoop@amani27.poohdedoo.com

Setup Hadoop

Download and extract hadoop (tar xvfz hadoop-x.x.x.tar.gz -C /mnt/)
Change the permission of the extracted directory if necessary (chown -R user:user /mnt/hadoop-x.x.x)
[optional] If IPv6 is not used disable it.

- add 'net.ipv6.conf.all.disable_ipv6 = 1'

Configure Hadoop

Configuration files $HADOOP_HOME/conf/

Set JAVA_HOME in $HADOOP_HOME/conf/hadoop-env.sh (Add export JAVA_HOME=/path/to/javahome)

eg: export JAVA_HOME=/opt/java/jdk1.6.0_29

Edit he HADOOP_HOME/conf/core-site.xml as follows:

<name>fs.default.name</name>

<value>hdfs://hadoop0.poohdedoo.com:9000</value>

</property>

<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>

</property>

<name>hadoop.tmp.dir</name>

<value>/mnt/hadoop_tmp</value>

</property>

</configuration>

Edit the $HADOOP_HOME/conf/hdfs-site.xml as follows:

<name>dfs.replication</name>

</property>

<value>/mnt/hadoop_data/dfs/name</value>

</property>

<value>/mnt/hadoop_data/dfs/data</value>

</property>

</configuration>

Edit the $HADOOP_HOME/conf/mapred-site.xml as follows

<name>mapred.job.tracker</name>

<value>hadoop0.poohdedoo.com:9001</value>

</property>

<name>mapred.system.dir</name>

<value>/mnt/hadoop_data/mapred/system</value>

</property>

</configuration>

Edit the $HADOOP_HOME/conf/hadoop-policy.xml

By default the value for 'security.job.submission.protocol.acl' is *

change it to a user group or a name

<name>security.job.submission.protocol.acl</name>

<value>adminuser</value>

* Change the 'masters' and 'slaves' files (Master node only; slave machines does not need this configurations)

- $HADOOP_HOME/conf/maseters (masters file contain secondary namenode servers)

hadoop0.poohdedoo.com

- $HADOOP_HOME/conf/slaves (slaves file contain slave servers ; datanodes and task trackers)

hadoop1.poohdedoo.com

hadoop2.poohdedoo.com

Setting up hadoop cluster

Format the namenode before starting the cluster

$HADOOP_HOME/bin/hadoop namenode -format

start the services

$HADOOP_HOME/bin/start-all.sh

It will start namenode,jobtracker, secondarynamenode in master node and datanode and tasktracker on slave nodes. (To check the services run $JAVA_HOME/bin/jps)

stop the services

$HADOOP_HOME/bin/stop-all.sh

It will stop namenode,jobtracker, secondarynamenode in master node and datanode and tasktracker on slave nodes. (To check the services run $JAVA_HOME/bin/jps)

How to Create Analytics and gadget Dashboard for Logs

2013-01-14T07:10:00.001-08:00

Assume you have configured logs according to my Previous Article, you will get logs from different nodes to BAM which will resides in cassandra data store. If you wont to analyzed those log data, a very simple use case would be to get number of logs by the priority.

Lets see how we can do it

Step 1 – Write an hive script to retrieve data from cassandra and store it in a relational database.

First we will use hive query language to retrieve the needed data and we’ll store it in relational database so we can create gadgets against relational data.

My simple query is retreive all log infomation to mysql database. (You can write your own logic accordingly)

Default keyspace is - EVENT_KS

My needed CF is – log_0_AS_2013_01_07

My Mysql database - MYBAMDB

Go to BAM management console -> Analytics -> Add-> and copy paste the above query. Please change the necessary parameters accordingly ( cassandra.cf.name,mapred.jdbc.url', mapred.jdbc.username, mapred.jdbc.password)

CREATE EXTERNAL TABLE IF NOT EXISTS LogEventInfo (key STRING,
tenantID INT,serverName STRING,
appName STRING, priority STRING,logTime DOUBLE,logger STRING,message STRING) STORED BY
'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' WITH SERDEPROPERTIES ( "cassandra.host" =
"localhost",
"cassandra.port" = "9160","cassandra.ks.name" = "EVENT_KS",
"cassandra.ks.username"
= "admin","cassandra.ks.password" = "admin",
"cassandra.cf.name" = "log_0_AS_2013_01_07",
"cassandra.columns.mapping" =
":key,payload_tenantID,payload_serverName,payload_appName,
payload_priority,payload_logTime,payload_logger,payload_message"
);

CREATE EXTERNAL TABLE IF NOT EXISTS Logs(tenantID INT,serverName STRING,
appName STRING, priority STRING,logTime DOUBLE,logger STRING,message STRING)
STORED BY 'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler' TBLPROPERTIES (
'mapred.jdbc.driver.class' = 'com.mysql.jdbc.Driver',
'mapred.jdbc.url' =
'jdbc:mysql://localhost:3306/MYBAMDB',
'mapred.jdbc.username' =
'root','mapred.jdbc.password' = 'root',
'hive.jdbc.update.on.duplicate' = 'true',
'hive.jdbc.table.create.query' =
'CREATE TABLE LogEvent(tenantID INT,serverName
VARCHAR(200),
appName VARCHAR(200), priority VARCHAR(200),logTime DOUBLE,logger VARCHAR(800),message VARCHAR(3800))');
insert overwrite table Logs select tenantID, serverName, appName, priority, logTime,
logger, message from LogEventInfo;
select tenantID, serverName, appName, priority, logTime,
logger, message from LogEventInfo;

In the first part of hive query it will map the cassandra data store to a storage handler which can extract data from cassandra. The second part we map the cassandra storage handler data to relational database storage handler to have the pipe between cassandra and relational database. The third part is where you write your logic for the analytics.

You can test your script by clicking execute. And if no errors occur go to your mysql database and check for the data.

Once you have done writing the hive script you can schedule it and save the query.

Step 2 – Writing gadgets

Once we have written our analytics, we can write gadgets using gadget gen tool.

Go to Management Console -> Tools-> Gadget gen tool

Give your database credentials as shown below

Write your SQL script with your analytical logic.

In my case I am going to get a count of number of log messages per priority.

My SQL would look like

select count(tenantID), priority From LogEvent group by priority;

Click on next and select the type of UI element you wont and fill the necessary information.

Fill the gadget Details and Click on generate to generate the gadget.

Step 3 – Add gadgets to gadget dashboard

Once we create the gadget we can go view the gadgets by adding the gadgets to the BAM dashboad.

Before we do that we need to add the gadget to the gadget repository from the registry. When we create the gadgets, gadgets will be saved in the registry repository therefore , we need to add them to the gadget repo.

Go to gadget Repository under gadgets in BAM management console. And click on add gadget.

Give the needed infomation as shown below and add the gadget to gadget repository.

Then you can go to view portal -> Add gadget from the gadget repo and view your gadgets.

How to configure MT-Logging with WSO2 Products

2012-10-11T02:57:00.002-07:00

Prerequisites

Carbon 4.0.0 and above product
WSO2 BAM 2.0.0 and above
Apache Server installed

Introduction

Stratos MT-Logging architecture provides a logging frame work to send logs to BAM. Which opens wide variety of possibilities when it comes to monitoring logs. In my previous article I have explained the architecture of Distributed Logging with WSO2 BAM, this tutorial explains how you can set up logging effectively for any WSO2 Product and how you can analyze and monitor logs effectively.

Architecture

Setting up Hadoop Server to host archived log files

Once the logs are sent to BAM, we analyse the logs daily and send them to a file system. For better performance for archive logs analytic, we send archive logs to HDFS file system. So we can analyze archive logs using map reduce task (big data, long term data analysis).

Please refer How to Configure Hadoop to see how we can configure a hadoop cluster, Once you have hadoop cluster you can give your hdfs information in summarizer-config.xml. So it will automatically analyse your daily logs and send them to HDFS file system

Summarizer Configuration for log archiving.

<cronExpression>0 0 1 ? * * *</cronExpression>
<tmpLogDirectory>/home/usr/temp/logs</tmpLogDirectory>
<hdfsConfig>hdfs://localhost:9000</hdfsConfig>
<archivedLogLocation>/stratos/archivedLogs/</archivedLogLocation>
<bamUserName>admin</bamUserName>
<bamPassword>admin</bamPassword>

cronExpression - The schedule time, that summarizer runs daily

hdfsConfig - hdfs file server intimation
archivedLogLocation - HDFS file patch which the archived logs should be saved

Setting up Log4jAppender - Server Side (AS/ESB/GREG/etc)

To publish log events to BAM, log4j appender should be configured in each server. In order to do that you need to add LogEvent to the root logger and configure the LogEvent credential accordingly.

Add LogEvent to the root logger in log4j

Go to Server_Home/repository/con -> log4j.properties and LOGEVENT to log4j root logger (or replace the following line)
log4j.rootLogger=INFO, CARBON_CONSOLE, CARBON_LOGFILE, CARBON_MEMORY, CARBON_SYS_LOG,LOGEVENT

Add Data publishing URLs and credentials
Go to Server_Home/repository/con -> log4j.properties. Modify LOGEVENT appender’s LOGEVENT.url as BAM Server thrift URL,LOGEVENT.userName, .LOGEVENT.password

log4j.appender.LOGEVENT=org.wso2.carbon.logging.appender.LogEventAppender

log4j.appender.LOGEVENT.url=tcp://localhost:7611

log4j.appender.LOGEVENT.layout=org.wso2.carbon.utils.logging.TenantAwarePatternLayout

log4j.appender.LOGEVENT.columnList=%T,%S,%A,%d,%c,%p,%m,%H,%I,%Stacktrace

log4j.appender.LOGEVENT.userName=admin

log4j.appender.LOGEVENT.password=admin

Enabling the Log Viewer

When the log viewer is not enable to take logs from cassandra the default behaviour of the log viewer to take logs from the carbon memory. It will only display the most recent logs of the carbon server. To get persistence logs (logs which are coming from the current date) you need to enable isLogsFromCassandra true so that you can view persistance logs through the management console of any carbon server (ESB/DSS/AS etc) . And also you need to give the user credentials of the cassandra server as shown below.

Change Logging-Config.xml to View Logs from BAM.

Got to Server_Home/repository/con/etc-> Logging-config.xml

Enable isDataFromCassandra

Give cassandra url of BAM Server

<cassandraHost>localhost:9160</cassandraHost>

Give BAM Server user credentials to access Cassandra Server in BAM

<userName>admin</userName>
<password>admin</password>

Give hadoop hdfs hosted url for the logs viewer

<archivedHost>hdfs://localhost:9000</archivedHost>
<archivedHDFSPath>/stratos/logs</archivedHDFSPath>

Setting up Logging Analyzer - WSO2 BAM Side

Setting up BAM

Bind IPs for cassandra {This is not logging related, this is just to bind an ip address to cassandra so that cassandra will not start in localhost}

Copy cassandra.yaml from {WSO2_BAM_HOME}/repository/components/features/

org.wso2.carbon.cassandra.server_4.0.1/conf/cassandra.yaml to repository/conf/etc. Change the

IP address (localhost) to the correct ip address of BAM of listen_address and rpc_address

Copy cassandra-component.xml from {WSO2_BAM_HOME}/repository/components/

features/org.wso2.carbon.cassandra.dataaccess_4.0.1/conf/cassandra-component.xml to

repository/conf/etc. Change the IP address (localhost) to the correct ip address of BAM of

192.168.4.148:9160

Installing Logging Summarizer

Download P2 Profile which will contain Logging summarizer features. Install logging.summarizer feature through Management Console (Go to Configure -> Features and Click on Add Repository). Once you add the repository you will be redirected to a page which contains available features. Select bam summarizer feature and install it.

Change the logging config.xml

Change log rotation paths, give the log directory as apache log rotation directory, and give BAM username password credentials

<publisherURL>tcp://localhost:7611</publisherURL>
<publisherUser>admin</publisherUser>
<publisherPassword>admin</publisherPassword>

<logDirectory>/home/usr/apache/logs/</logDirectory>

Point BAM to external hdfs file server

ow the logging is configured in both publisher and receiver, and you can view your logs by log-in into
In order to point the analyzers to the hdfs file system you need to update BAM_HOME/repository/conf/advanced/hive-site.xml to point it to your hdfs file system.

<property>

<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>

Now the logging is configured in both publisher and receiver, and you can view your logs by log-in into management console. And view System Logs - This will view the current logs as well as archived logs taken from the apache server. Logs are daily archived to the apache server through a cron job.

If you want to analyze logs using hive analytics, and display in dashboards you can use bam analytics tools and dashboard tool kits to customize Logging KPIs for system administration.

How Distributed Logging Works in WSO2 Stratos.

2012-09-26T19:51:00.001-07:00

Why we need distributed Logging ???????

Stratos is a distributed clustered setup where we have several applications such as ESB Servers,Application Servers, Identity Servers, Governance Servers, Data Services Sever etc deployed together to work with each other to serve as Platform as a Service. Each of these servers are deployed in a clustered environment, where there will be more than one node for a given server and depending on the need, there will be new nodes spawned dynamically inside this cluster. And all these servers are fronted through an Elastic load balancer and depending on the request the load balancer will send requests to a selected node in a round robin fashion.

What would you do when there is an error occurs in a deployment like above where there are 13 different types of servers running in production and each of these servers are clustered and load balanced across 50+ servers?. This would be a nightmare for the system administrators to log-in into each server and grepping for the logs to identify the exact caused of the error. This is why distributed application deployment's needs to keep a centralized application logs. These centralize logs should also be kept in a high scalable data storage in an ordered manner with easy access.So that the users (administrators,developers) can easily accesses logs, whenever something goes unexpected, with the least amount of filtering to pinpoint the exact cause of the issue.

When designing a logging system like above, there are several things you need to consider.

Capturing the right information inside the LogEvent – You have to make sure all the information you need in order to monitor your logs is aggregated in the LogEvent. For example in a cloud deployment setup you have to make sure not only the basic log details(logger,date,log level) are not enough to point a critical issue. You further needs tenant information (user/domain), Host information (to identify which node is sending what), Name of the server (from which server you are getting the log) etc. These information is very critical when it comes to analyzing and monitor logs in an efficient way.
Send logs to a centralized system in a nonblocking asynchronous manner so that monitoring will not affect the performance of the applications.
High availability and Scalability
Security – Stratos can be deployed and hosted in public clouds therefore, its important to make sure the logging system is high secured.
How to display system/application logs in an efficient way with filtering options along with log rotation.

Those are the 5 main aspects which were mainly concerned when designing the distributed logging architecture. Since Stratos support multitenancy we made sure that logs can be separated by tenants, services, and applications.

MT-Logging with WSO2 BAM 2.0

WSO2 BAM 2.0 provide a rich set of tool for aggregation, analyzing and presentation for large scale data sets and any monitoring scenario can be easily modeled according to the BAM architecture. We selected WSO2 BAM as the backbone of our logging architecture mainly because it provides high performance with non intrusiveness along with high scalability and security. Since those are the crucial factors essential for a distributed logging system WSO2 BAM became the idol candidate for MT-Logging architecture.

Publishing Logs to BAM

We implemented a Log4JAppender to send LogEvents to bam. There we used BAM Data agents get Log Data across to BAM. BAM data agents send data using thrift protocol which gives us high performance message through put as well as it is non blocking and asynchronous. When publishing Log events to BAM we make sure the Data Stream is created per tenant, per server, per date. When the data stream is initialized there will be a unique column family created per tenant, per server per date and the logs will be stored in that column family in a predefine keyspace in cassandra cluster.

The Data stream defines the set of information which needs to be stored for a particular LogEvent and can be modeled into a Data Model.

Data Model which is used for Log Event.

{'name':'log. tenantId. applicationName.date','version':'1.0.0', 'nickName':'Logs', 'description':'Logging Event',

'metaData':[{'name':'clientType','type':'STRING'} ],

'payloadData':[

{'name':'tenantID','type':'STRING'},

{'name':'serverName','type':'STRING'},

{'name':'appName','type':'STRING'},

{'name':'logTime','type':'LONG'},

{'name':'priority','type':'STRING'},

{'name':'message','type':'STRING'},

{'name':'logger','type':'STRING'},

{'name':'ip','type':'STRING'},

{'name':'instance','type':'STRING'},

{'name':'stacktrace','type':'STRING'}

] }

We extend org.apache.log4j.PatternLayout a in order to capture tenant information, server information and node information and wrap it with log4j LogEvent.

Log Rotation and Archiving

Once we send the log events to BAM the logs will be saved in a Cassandra cluster. WSO2 BAM provides a rich set of tools to create analytic and schedule task. Therefore, we used these hadoop task to rotate logs daily and archive them and store it in a secure environment. In order to do that we use a hive query which will run daily as a cron job. It will read Cassandra data store, retrieve all the column families per tenant per application and archive them in to gzip format.

The hive Query which is used to rotate logs daily

set logs_column_family = %s;

set file_path= %s;

drop table LogStats;

set mapred.output.compress=true;

set hive.exec.compress.output=true;

set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;

set io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec;

CREATE EXTERNAL TABLE IF NOT EXISTS LogStats (key STRING,

payload_tenantID STRING,payload_serverName STRING,

payload_appName STRING,payload_message STRING,

payload_stacktrace STRING,

payload_logger STRING,

payload_priority STRING,payload_logTime BIGINT)

STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'

WITH SERDEPROPERTIES ( "cassandra.host" = %s,

"cassandra.port" = %s,"cassandra.ks.name" = %s,

"cassandra.ks.username" = %s,"cassandra.ks.password" = %s,

"cassandra.cf.name" = ${hiveconf:logs_column_family},

"cassandra.columns.mapping" =

":key,payload_tenantID,

payload_serverName,payload_appName,payload_message,

payload_stacktrace,payload_logger,payload_priority,

payload_logTime" );

INSERT OVERWRITE DIRECTORY 'file:///${hiveconf:file_path}'

select

concat('TID[',payload_tenantID, ']\t',

'Server[',payload_serverName,']\t',

'Application[',payload_appName,']\t',

'Message[',payload_message,']\t',

'Stacktrace ',payload_stacktrace,'\t',

'Logger{',payload_logger,'}\t',

'Priority[',payload_priority,']\t'),

concat('LogTime[',

(from_unixtime(cast(payload_logTime/1000 as BIGINT),'yyyy-MM-dd HH:mm:ss.SSS' )),']\n') as LogTime from LogStats

ORDER BY LogTime

Once we archived the logs we will send these archived logs to HDFS file system. The archived logs can be further analysed using map-reduce jobs, for long term data analytics

Advantages of sending Logs to WSO2 BAM

Asynchronous and None Blocking Data publishing
Receives and Stores Log Events Cassandra Cluster which is high scalable and a big Data Repository
Rich tools set for analytics
Can be shared with CEP for real time Log Event analysis.
Can provide Logging tool boxes and dashboards for system administrators using WSO2 BAM
High performance and non-intrusiveness
Big data analysis

Daily log information analytic - Analyse cassandra data storage
Long term log information - Analyse HDFS file system using map-reduce

Monitoring and Analyzing System Logs

Using the Log Viewer

Both application and system logs can be displayed using the management console of a given product. Simply log-in to Management console and under monitor there are two links 1. System logs which has system logs of the running server 2) Application Logs which has application level logs (this can be services/web applications) for a selected application. This makes it easy for users to filter logs by the application they develop monitor logs up to application level.
Dashboards and Reports

System administrators can log-in to WSO2 BAM and create their own dashboards and reports, so the can monitor their logs according to their Key performance Indicators. For example if they want to monitor number of fatal errors occur per given month for a given node.
SMS Alerts and Emails
Not just dashboards and Reports ... Combining WSO2 BAM with WSO2 CEP you can get real time alerts like trigger emails, SMS so that System administrators can instantly get to know when your system is going through an unexpected behavior.

View Logs Using the Log Viewer - Current Log

View Logs Using the Log Viewer - Archived Logs

All these rich set of monitoring capabilities can be in built into your deployment using Stratos Distributed Logging system. Where you don’t have to worry about always going to the system administrators for logs whenever something goes wrong in your application :).

Sneak Peek at WSO2 BAM 2.0 & How to Install BAM Data Agents in WSO2 Products

2012-09-11T19:42:00.001-07:00

In a fast growing company, enterprise data plays a major role when it comes for decision making and other top level business activities. When I say enterprise data, it can be anything which is an asset to your company, for example

It can be the data coming into your system (Employee Information, Product Information Sales data etc)
It can be mediation data (who is accessing your services/application, when and how)
Request,response and faulty count for your services.

These row data can be crucial and also it is very important to analyze and convert these data into information in order provide business intelligence for decision makers.

WSO2 BAM is mostly used in SOA environments because when it comes to SOA environment monitoring your data means monitor your services. Most of your business functionalites are exposed as services, for example if you want to insert set of records to your data base you will expose set of data services to do it or if you want to mediate some services you need to create proxy service. You can monitor these kind of data using WSO2 Buisness activity monitor Server.

If you look at BAM highlevel architecture BAM can be modelled according to three major components

Aggregation – BAM Data Agents which publish data in to BAM Data storage
Analytics - Analyzers which analyzed Data in BAM storage
Presentation – Gadgets which displays key performance indicators of analyzed information

You can monitor any business modeling scenario according to these components and BAM architecture is modelled around these three components.

Aggregation – basically this means getting your data into BAM (capturing/collecting data into BAM). BAM data are sent using events, you can capture important set of data and make it as an event stream. I will give more details on events stream in my next posts :) but for the moment all you need to know is capture many data as possible when you are sending data to BAM more data means more business intelligence.

Analytics – After you capture your data you need to make your data meaningful (basically converting data into information) by analyzing your storage data. In order to do that you need to do some data operation (aggregation, merging, sorting ordering) and ultimately build some key performance indicators which will be needed for business intelligence. In BAM you will be able to write your own analyzers (custom analyzers) using hive/hadoop and also these queries can be saved and scheduled accordingly. We will dig deep into how to write hive queries using bam and how to analyze data and create KPIs using data in future.

Presentation – Once you analyze your data, you might need to visualize your KIPs (using bar charts tables) and build your own dashboards in order to visualize your information to decision makers. And not just visualizing you might need to send these analyzed data to needed parties using reporting alerts daily to the topper management etc. WSO2 BAM will provide rich set of tools to do those task without much coding effort. It provides a gadget editor tool which give you a drag and drop kind of way to create your gadgets from your analyzed data or you can connected to reporting o do some complex event processing and send emails etc just by using set of configurations.

Referance http://mackiemathew.files.wordpress.com/2012/01/bamarchitecture.png

Now that we discussed the main architecture behind BAM and also bit about the functionalities of BAM (you can go through BAM samples and install bam samples in order to get in depth knowledge on BAM)

Lets see how we can use BAM to monitor your services.

For this I am going to use WSO2 ESB and we will see how we can monitor service statistics using BAM.

First we need to install BAM data agents inside ESB.

How to install BAM data agents in ESB?

Download P2 Profile which will contain BAM Data agent features. Download WSO2 ESB Latest packs. Go to ESB_HOME/bin and start up the server.

Go to Configure -> Features and Click on Add Repository.

Click on add Repository, You need to give a meaningful name for the repository name, I am going to give it as My_REPO and give the path to your P2 Profile and click on add.

Once you add the repository you will be redirected to a page which contains available features. You need to install BAM data agent features therefore, type BAM on Filter by feature name and untick group feature by category and click on find.

Tick on following features BAM Mediation Data Agent Aggregate, BAM Mediator Aggregated,BAM Service Data Agent Aggregate and click install.

Read and accept License Agreement and click next. Once the installation is finished restart the server.

Once you have installed the BAM data agents you will see newly installed features under Configure.
1. Service Data Publishing, 2. Mediation Data Publishing 3. BAM Server Profiling.

How to Configure Service Data Agent ?

You can click on Service Data Publishing to send your Service Data to WSO2 BAM. In order to do that you need to Download and Start up BAM (if you are running ESB on the same machine you need to change the offset{BAM_HOME->repository->conf->carbon.xml set offset to 1}.

Go to ${WSO2ESB_HOME}/repository/conf/etc and open bam.xml file. Please make sure that you enable the ServiceDataPublishing.
The bam.xml should have the following configuration:

   <BamConfig>
   <ServiceDataPublishing>enable</ServiceDataPublishing>
   </BamConfig>

NOTE: By default the this is disabled, and the BAM publishing won't occur even though you proceed the below steps and changed in the mentioned UI. Therefore please enable the ServiceDataPublishing to use with WSO2 BAM.
Then in the ESB side go to Service Data Publishing, Give the BAM Server URL which will be the thrift ports started in BAM server (tcp://localhost:7611), give BAM credentials (default is admin,admin) and click on update.

Now if you invoke some services from ESB side the service data will get published to bam. Now to check to confirm whether your data is in BAM you can go to BAM Management Console Check it by connecting to Cassandra Explorer.

In this post I have only explained an intro to BAM 2.0 and how to send service data to bam From my next session I will explain how to send your enterprise data to BAM and how you can do analytics and presentation using WSO2 BAM tool kits.

How to send mails using WSO2 ESB (sending your payload data to e-mail)

2012-08-24T07:07:00.000-07:00

Suppose you have a web service which returns some data and you want to send that data into a mail... WSO2 ESB provides an easy mechanism to do this by creating a proxy service.

First get your endpoint (web service endpoint which returns data ).

Step 1 - Enable mail transport in axis2.xml.

Go to your axis2.xml (under WSO2ESB_HOME/repository/conf/axis2/), and uncomment mailto transportSender as shown below.

 <transportSender name="mailto" class="org.apache.axis2.transport.mail.MailTransportSender">

   <parameter name="mail.smtp.host">smtp.gmail.com</parameter>
   <parameter name="mail.smtp.port">587</parameter>
   <parameter name="mail.smtp.starttls.enable">true</parameter>
   <parameter name="mail.smtp.auth">true</parameter>
   <parameter name="mail.smtp.user">synapse.demo.0</parameter>
   <parameter name="mail.smtp.password">mailpassword</parameter>
   <parameter name="mail.smtp.from">synapse.demo.0@gmail.com</parameter>
  </transportSender>

Step 2 Creating the proxy Service.

Go WSO2 ESB, start up the server. Under Main menu -> Axis2 Services -> Add -> Proxy Service. Click on Custom Proxy Service. Give an appropriate Proxy Service Name. (Give the WSDL URL as your requirement)

Step 3 - Define Endpoint

In the define endpoint section click on define inline, click on create -> Address Endpoint and give the address end point of your service which you get data from. I am going to invoke the Data Service which i created in my previous blog. And my Address endpoint will be

Step 4 - Creating the Out Sequence. Go to Out sequence -> Define Inline

In the In Out sequance We need to add a
A Log Mediator - Which logs the incoming message
Three Property Mediators
   1. Subject - Which will be the subject of our mail
   2. MessageType - Message Type of our mail
   3. ContentType - Content Type of our mail
   4. Send mediator - Send the mail to our email address

To add the log mediator, Click on add child -> Core -> Log. And in the log mediator give the Log level as full.

To add the three properties,  Click on add child -> Core -> Property. And add three properties one by one.
property name="Subject" value="CEP Event" scope="transport" type="STRING"
property name="MessageType" value="text/html" scope="axis2" type="STRING"
property name="ContentType" value="text/html" scope="axis2"  type="STRING"

To add the send mediator Click on add child -> Core -> Send. And Select Endpoint Type -> Define inline -> Address Endpoint -> mailto:amani.soysa@gmail.com

Once you add the three properties and the log mediator your out sequance editor should look like below.

Click save and close and finish creating your proxy Service.

<proxy xmlns="http://ws.apache.org/ns/synapse" name="MyMailProzy" transports="https,http" statistics="disable" trace="disable" startOnLoad="true">

You can invoke your proxy service by going to try it in the service list, and you should get a mail according to your payload. My next blog post will show how we can receive emails to WSO2 ESB using a standard email client.