Cloud Bazaar

Design consideration for novice mobile app developers

2019-08-04T07:04:00.001-07:00

In this blog I want to share an experience where I had to switch to a micro-service architecture for a mobile app. The requirement was to create an appointment scheduler for the business. Being a huge fan of ionic I with one of my friend started with the same, and in few days our first version was out in the market. During the initial discussion with the customer, the requirements which came forth had simple static pages and one dynamic appointment scheduling page. For the appointment scheduling page we used Firebase in the backend.

Things were going fine after the release and the customer started marketing the app. The initial customer reviews/downloads were very inspiring. After the release of the first version we started getting bugs and change in requirements. We got a tricky bug filed during this time.

The customer initially told that his shop was scheduled to open at 9 am and will continue till 6:30pm. Our appointment timings were not having this restriction, so a person can book an appointment at 8:00 am and expect the owner to serve him/her then. This proved to be a great issue as the owner would have to call the customer and inform him/her about the timings.

The fix was simple just have a restriction in the UI which allows the customer to chose a time slot between 9-6:30. We implemented this and pushed the fix. As soon as we were about to celebrate the success with a chilled beer, the customer again informed us that the bug was still not fixed. On further investigation we saw that few customers were having the old version, and asking them to upgrade to a new version was not a good approach.

Our chilled beers went back in the fridge and we started scratching our head again.

The issue was we were novice mobile app developers without having a real life project experience. During our design phase we never considered version upgrades as a issue. We developed everything in ionic and had scripts talking to firebase which led to this issue.

Time was running out for us, and we had to do something quick to calm our infuriated customer. We started doing spike and started going through such scenarios. Surprisingly there were so many cases where version upgrades proved to be a challenge for the businesses.

We happened to see a way to fix this issue, luckily for us the solution was in firebase. Firebase provided a way to create functions which can help in creating small microservices

Firebase functions helps to create triggers on CRUD operations. For us this was the optimal solution as we thought to create a trigger on insertion action, where a customer who is booking an appointment before 9am or after 6:30pm will get notified that he/she has been rescheduled to the working hours.

Now we were able to notify the users who were having older version of the app. We were also able to verify the logs and monitor how many users were booking out of business working hours

That's it now we were able to relax with our chilled beer as things went back to normal.

Summary

If you are starting with your mobile app development do consider version upgrade issues. Firebase functions can provide micro service architecture which can prove extremely useful for your app development.

Code review guidelines for GRPC

2018-05-31T21:39:00.000-07:00

This blog we are going to discuss some nuances one should be aware of while reviewing GRPC related code. GRPC stands for (gRPC Remote Procedure Call. This is primarily getting developed by google as a platform for binary serialized messaging protocol. gRPC has made api development seamless because of its ability to support multiple coding languages, api versioning, high performance achieved due to binary data serialization etc.

This blog is not related to introduction of gRPC, for that we have many other links like this one gRPC Intro

In this blog I want to share my learnings during my recent project in gRPC. In my project I am having a file getting streamed from one service to another and vice-versa. A sample scenario will be conversion of JPG to PNG where a JPG file is streamed to the server which converts it and sends back a PNG file.

1. Which one will you choose

First question that you need to get answer while you are reviewing someone's code is what type of gRPC service flavor did they opt for and why. Take a minute to think about our scenario, what do you think should be an ideal choice?

For our scenario we need to send file bytes and receive them so preferred way should be bi-directional streaming. This will give you a flexibility to break the file into chunks of bytes and send it to the server. You can either monitor the network transmission time or performance through such mechanism.

So having a chat with the author regarding the choice of service and getting to know why he/she didn't choose the other types is a good way to start code reviewing.

2. Follow TDD

An industrial way to write code in production is begin with unit tests. TDD (Test data driven) development has proven not only to be efficient in producing less bugs but also has proven to be a good way of understanding the code for a new pair of eyes.

This can help you as a code reviewer to understand the problem in hand and give reviews accordingly.

In order to write efficient unit tests, note we are creating a channel to talk to the grpc service from client. A test should not be complicated in the sense of creating a channel and stub , processing it and eventually cleaning it. gRPC provides few ways through which you can create mocks of such channel and stubs seamlessly.

Take a look at this example provided by them.

/*
* This rule manages automatic graceful shutdown for the registered servers
and channels at the end of test.
*/
@Rule
public final GrpcCleanupRule grpcCleanup = new GrpcCleanupRule();

Creating a stub for tests should use such channel in order to prevent the hassle of graceful shutdown handling code

GreeterGrpc.GreeterBlockingStub blockingStub = GreeterGrpc.newBlockingStub(
   // Create a client channel and register for automatic graceful shutdown.grpcCleanup.register(InProcessChannelBuilder.forName(serverName).directExecutor().build()));

Talking about the stub takes us to the next point which one should keep in mind while code reviewing

3. Deadlines

Indeed one should not be missing deadlines :) In world of gRPC deadline is considered to be timeouts. There is an emphatic statement at the start of this blog which I am sure will keep you awake during those sleepless nights of gRPC :)

In the earlier example we saw how to create a stub, if the author is creating the stub in a similar way there should be a comment for him/her to think about deadlines

response = blockingStub.withDeadlineAfter(deadlineMs, TimeUnit.MILLISECONDS).sayHello(request);

Monitoring deadlines often can help understand various issues which are due to external factors like network, instance shutdown etc.

4. Stubs

We have been talking about stubs, and this is the next thing as a code reviewer you should consider. gRPC provides 2 ways to create stubs blocking and non/blocking. Details at here . If you happen to see a blocking gRPC stub ask the author from where it should get invoked. If that is a UI feature then there is a big red flag waving in front of that code.

As we already know Asynchronous actions are the preferred way to do UI flow creation unless there is a dire need to have a blocking action.

While code reviewing this is a critical observation that you need to make and discuss the consequences with the author.

5. onError

@Overridepublic void onError(Throwable t) {
logger.log(Level.WARNING, "Encountered error in routeChat", t);

}

gRPC StreamObserver response comes with onNext, onComplete and onError scopes. onNext and onComplete is having a strict binding with your custom logic. But onError scope is interesting and should be considered for code reviewing.

This gets invoked whenever there is an error in server side. While debugging errors should be logged both on client and server side, but there are some scenarios when you don't throw specific exception to client because of information disclosure (security reason). In such cases it is very imperative to understand how the user is handling the error and how can you monitor for such exceptions.

6. How to monitor for such exceptions

Monitoring is the next crucial thing which the author should be thinking about. In order to have good monitoring one should be logging some kind of metric counter in scenarios like onError or onCompleted.

Prometheus provides an excellent way to create metrics which can help track errors on success and can help project in a dashboard. Take a look at the different types of counters they provide at here

Here is a typical dashboard which I have for my image conversion service

Dashboards like this not only can help monitor your service but also can help you create alert notifications in case something goes bad.

Conclusion

I still consider myself newbie to the world of gRPC, I am capturing the learnings which I got during my project. Will look forward to the community to highlight any points I have missed which you feel can be crucial while reviewing for gRPC code.

Hope to get your feedback on my understanding.

Sayonara for now....

Miracles can happen with Watson and Salesforce

2018-04-19T21:50:00.004-07:00

After a long time I am writing this blog to share my experience with some ML stuff. Over past few days I made myself busy in learning about IBM Watson apis and how can they help Salesforce customers in their businesses.

Thanks to Salesforce trailhead module getting ramped up on IBM watson SDK for Salesforce was a piece of cake. During the trail one can learn about how to setup federated search in Salesforce in order to search through the files in IBM Watson.

Below is a screenshot of my IBM Watson collection. Note all the ML stats that comes readily baked from IBM Watson

I have 34 documents uploaded in my collection as soon as the documents are uploaded IBM Watson massages the data and finds out top topics getting discussed in the document, overall sentiments in those documents, finds out the context(concepts) that the document is talking about, classification of those documents upto 5 levels etc.

All the mentioned above aspects are already explained in multiple blog sources. What I am interested in sharing with you all is the use case scenarios where we can leverage this feature.

Salesforce has 3 primary standard Objects - Accounts, Opportunities and Cases. Most of the time there is a need of getting relevant informations. This implies a strong need of recommendation services. Among the recommendation features, files recommendation is an important aspect. Netflix leverages this aspect a lot. Based on context of what you are watching it tries to recommend you the next probable movie.

Similarly in Salesforce, based on the opportunity and the stage that you are in, recommendation feature can help the customer with the relevant file that is going to help him/her close that opportunity or deal.

Earlier it was difficult because Salesforce still don't provide Smart search feature for files. But with IBM watson NLP search feature we can create an intelligent component for the same.

Use Case Scenario: As a customer I should get document recommendation for closing my deal/opportunity, case or account

Acceptance Criteria:

If I change the opportunity stage from prospect to negotiation/review the component should be able to recommend me all the relevant documents which are similar to this opportunity and were used in such similar stage.
If a support person searches for something like "Show me relevant files to fix login issues" the component should respond back with all the relevant examples

There are such n number of examples like these which can be solved with such component.

Such scenarios can be achieved using the IBM Watson Discovery api for Natural Language Processing. Let me show you the API call and then discuss about that briefly.

Coding Time = (2hrs Trailhead + 3 hour Apex code + 1 hour testing)

It took me around 5-6 hours to get this thought into reality.

Step 1: We need to create an apex controller which will be invoking the IBM Watson Discovery API through Heroku Service.

@AuraEnabled

public static String getSearchResults(String query) {

IBMDiscoveryV1 discovery = new IBMDiscoveryV1('2017-11-07');

IBMDiscoveryV1Models.QueryOptions options

= new IBMDiscoveryV1Models.QueryOptionsBuilder('<<environmentId>>',

'<<collectionId>>')

.naturalLanguageQuery (query)

.returnField(new List<String> {'url','extracted_metadata','enriched_text'})

.build();

IBMDiscoveryV1Models.QueryResponse response = discovery.query(options);

System.debug(response.toString());

return response.toString();

}

With this small code snippet we can create our own intelligent component. In here we are query a NLP string which goes as a NLPQuery. In response we ask Watson to give us extracted metadata like file name, title etc. along with enriched text like Top Topics getting discussed, Context of those documents, Classification categories etc.

Step 2: Below is the javascript sample where we can create an NLP query such as "Give recommended files for <opportunity name>> in stage <<Opportunity Stage>> Such query can help us mine through our vast data and retrieve other similar opportunity documents where files were uploaded or created by other teams.

getQuery : function(cmp, id) {

var self=this;

var action = cmp.get('c.getSearchResults');

var acctAction = cmp.get('c.getOpportunity');

acctAction.setParams({recordId: id});

acctAction.setCallback(this, function(response) {

var result = response.getReturnValue();

var query = "Get recommended files for " + result.Name+" in stage "+ result.StageName;

console.log(query);

action.setParams({name: query});

action.setCallback(this, function(response) {

var state = response.getState();

let fileNames = new Set();

if(cmp.isValid() && state === 'SUCCESS') {

var resp = response.getReturnValue();

var respArr = JSON.parse(resp)["results"];

respArr.forEach(function(file){

var fileName = file['additional_properties']["extracted_metadata"]["filename"]

fileName=fileName.replace(/\.[^/.]+$/, "");

fileNames.add(fileName);

});

self.getUrl(cmp, Array.from(fileNames));

}

});

$A.enqueueAction(acctAction);

$A.enqueueAction(action);

}

Note there are 2 nested actions. First action will get the opportunity Id and do a SOQL query to get the opportunity name and stage

@AuraEnabled

public static Opportunity getOpportunity(Id recordId) {

Opportunity oppt = [SELECT Name, StageName FROM Opportunity WHERE Id=:recordId LIMIT 1];

return oppt;

}

Second action creates the NLP query with the results of the first action and send it to IBM Watson to get the response. In the javascript controller I am just getting the filename from the extracted metadata, this can be enhanced to get top topics for that document, sentiment analysis and classification categories.

Conclusion

In this blog I wanted to share my experience of creating a lightning component within hours which is equipped with AI features in order to break the apprehension that comes along with AI requirements. I am trying to enhance this component with more Watson responses and hope to share those experiences with you all pretty soon. As always thanks for inspiring me by giving your feedback on my blogs. If you need more details regarding this, do let me know in the comment section.

Unleash the power of Google Vision and Metamind apis in Salesforce

2017-10-06T21:51:00.000-07:00

POC

Create an easy #AI enabled Lightning component in #Salesforce using #Gensim

2017-09-27T00:32:00.000-07:00

<< If you like to see the component live in my na30 org, please post your email id and Name in this blog comment and I will send you the org details>>

Salesforce is a major player in CRM industry. Among its major key features are Marketing cloud, Sales cloud, Service cloud, community cloud etc. Across all the clouds resides a platform called files. There is a huge potential of improving the way we use files though Artificial Intelligence.

Chatter is another prevalent aspect of Salesforce. This looks like a corporate Facebook platform. Here you share your views, ask questions, update your groups, Follow your peers etc. This is another platform which is having exposed to variant amount of data and can help us infer lots of hidden meaning.

So what I am trying to infer? I mentioned about Artificial intelligence, so definitely we are going to talk about that. In this blog I want to show how we can easily create a Lightning component which can help you do some interesting cool things related to Files and Chatter Feed.

Problem Statement

First and foremost I wrote down the issues which were affecting me on a daily basis. Out of which I ranked 2 scenarios which I felt if solved, will have a beneficial impact on the customer.

1. First was File Summarization. We tend to upload multiple files on a daily basis but how to make the user interested to read them was an issue. If I was @mentioned in a feed with a big pdf as an attachment, normally I would skip reading it unless I felt it is really important. Even if it was important most of the time I felt reading first few paragraphs from the document. I always felt a need to file summarization which would have highlighted only the high priority sentences. I felt if this problem was solved it will not only help the user to save time and effort but also will have a great impact in recommending files.

2. Second similar problem was in Chatter feeds. Most of the time we have seen in a question and answer feed, the comment posts grow beyond 5 or 10 threads. In such threads it is time consuming for a reader to go through each and every post and figure out the answer from them. The solution that I thought was to have a summarized view of a feed which can highlight the main feeds and can also suggest the user which post to refer.

Both the above problems were flavors of a single text summarization feature. I felt like if I was able to get a solution of these 2 problems then it will suffice me for my needs.

After finalizing the Problem statement my next intent was to figure out the tool or utility which I will use for this solution.

Why Gensim?

Nowadays thanks to the open source world there are numerous tools and utilities which can help you do machine learning without any detailed prior knowledge about the same. Among these tools one of them is Gensim.

Gensim has been used by many corporates because of its ease to use and free licenses (Please note not all features in Gensim are having free licenses, please go through the documentation before you opt for it)

Gensim is written in python language. Because of it modular designs it is easy to install and leverage the apis. During this learning phase I did a little bit of research with Metamind and ML libraries for Apache Spark.

Metamind is a best opted tool for this purpose because
a. It is Salesforce product
b. Its apis easily bind with Salesforce.
But there was a blocker in terms of text related analysis. Metamind has just beta released a sentiment analysis feature. Apart from this it is not having any other text related model.

ML Lib in Apache spark comes with a set of libraries which can be trained and used for different machine learning use cases. There was no con for this, but the fact that you need to have a dataset on which you need to run your model was cumbersome. Moreover the effort quadruples itself when we consider multi language scenarios.

Gensim proved to be the best among the 3 tools based on following aspects
1. It is open sourced and has free commercial license
2. It can easily be deployed in Heroku. Heroku has been owned by Salesforce and normally customers don't have issues sharing their data with Heroku services

3. It is having support for fasttext. FastText as you might know it is a research lab for Facebook which has pre-trained models for 90+ languages. So all the goodness of that comes with Gensim too.
4. You get a generic model which can be used right away instead of making the model learn from scratch with your data. For a hackday this felt as a boon.

Gensim leverages Text2Rank algorithm where it ranks each sentences respecting their occurrence of appearance in the document. It then shows top ranked sentences based on weighing factor which will be discussed later. You can read more about it at here.

After finalizing the tool, the next item was to design the system.

Architectural Design

Gensim can be easily ported as a microservice. The intent was to have a microservice which talks to a custom lightning component. The custom lightning component can be easily integrated in the user's page and can help the user retrieve the summary details.

In order to port Gensim in Microservice I began to explore all the options and figured out that we can easily host gensim in Heroku.

Following diagram gives more details

Details

1.Lightning Component talks to Heroku Service.

2.Heroku Service talks to Salesforce Rest apis.

3.Through Rest Api we fetch Salesforce File content or Chatter Feed data based on request

4.Data is sent back to Heroku Service

5.Data sent to Gensim which summarizes the content and sends back the result

6.Summary saved in custom object

7. Heroku Service fetches summary data from custom object

8. Heroku Service returns the results to Lightning component

Coding Time

Let's start with the microservice first.

Note we need will be deploying the python application on Heroku using gunicorn and Flask. There are numerous example in internet which can show you how. Here is a link of such approach that you can refer.

As per the documentation we need a requirements.txt file which will have all the pre-requisites. For our case we need the following softwares to be installed as a setup process. Note we are having gensim 0.12.4 and simple-salesforce=0.72.2. Gensim will help us to do the summarization while simple-salesforce will help us connect to Salesforce

asn1crypto==0.22.0

certifi==2017.4.17

cffi==1.10.0

chardet==3.0.4

click==6.7

cryptography==1.9

enum34==1.1.6

Flask==0.12.2

gunicorn==19.7.1

idna==2.5

ipaddress==1.0.18

itsdangerous==0.24

Jinja2==2.9.6

MarkupSafe==1.0

pycparser==2.17

pyOpenSSL==17.0.0

requests==2.18.1

six==1.10.0

urllib3==1.21.1

Werkzeug==0.12.2

gensim==0.12.4

Cython==0.24

simple-salesforce==0.72.2

Here is the Python code for the same

import sys

import requests

from flask import Flask, request

from simple_salesforce import Salesforce

from gensim.summarization import summarize, keywords

We need to import Flask which will help us create the get and post api endpoints. Note with summarization from gensim I am also importing keywords. Keywords api helps in doing topic classification from within the content.

app = Flask(__name__)

reload(sys)

sys.setdefaultencoding('utf-8')

Flask needs to be initialized which will give the app object. We have to set the default encoding value to utf-8 else it will throw exception during GET and POST transactions.

@app.route('/getfilesummary/',methods=['GET'])

def getFileSummary():

docId = request.args.get('docId')

print docId

summaryObj = Summary()

content =summaryObj.getcontentfromtextfile(docId)

summary=summarize(content, word_count=50)

topics=keywords(content, ratio=0.01)

if summaryObj.verifyDocEntryExist(docId) is False:

#Insert the summary with document in custom object

summaryObj.insertSummaryInFileSummaryObject(docId, summary, topics)

return summary

Believe me when I say you just need to do this for summarization logic. Now you will realize how easy it is to integrate gensim with Salesforce. Let me explain what is done in this module. First and foremost the endpoint is named as getfilesummary and it is invoked over GET protocol. Ex: http://herokuserviceurl/getsummary

The endpoint accepts you to send a docId (which starts with a prefix 069). We are invoking a Summary Class object which we have defined later which will help us with the handle of the object. it has a method called getcontentfromtextfile which gets the text content of that file. Then with the content we just need to invoke the summarize method which was imported from gensim. While calling summarize method we can mention a weight which says how much word should the resultant summary has. In my case I wanted to keep it short so made 50 words.

Once we get the summary we are calling for the topics for the same file by invoking the keywords method. Note here there is a different weight given to the method and that is ratio. Ratio helps you define a percentage which you want to skim from the top as the output. Precisely you can play around with these 2 weighing factors. After getting the summary we need to insert the same in our custom object.

The intent is if the summary is already present in the custom object then skip insertion else insert the data. Note: we should also have another scope to update the summary. As it was a hackday project I didn't felt the necessity to make it full proof for all scenarios. Eventually you return the summary

@app.route('/getfeedsummary/', methods=['GET'])

def getFeedSummary():

feedId = request.args.get('feedId')

summaryObj = Summary()

summary = summaryObj.getdatafromfeed(feedId)

if summaryObj.isFeedEntryExist(feedId) is False:

#Insert the summary with feed in custom object

summaryObj.insertInFeedSummaryObject(feedId, summary)

print summary

return summary

Similarly we can have the same flow for feed summary too, where you get the feed data from Salesforce, send the text to gensim summary and get the summary. Eventually insert the details in the Salesforce custom object

@app.route('/getsummary/', methods=['GET','POST'])

def getSummary():

text = request.values.get('text')

summary = summarize(text, word_count=50)

print "SUMMARY: " + summary

return summary

I made a generic method just for fun where if you send a text as an argument it should respond back with the summary text.

Now comes the Summary class definition

class Summary:

def getcontentfromtextfile(self, docId):

sf=self.__getSalesforceObject()

sessionId = sf.session_id

versionobject = sf.query_all("SELECT VersionData from ContentVersion WHERE ContentDocumentId='%s'" % (docId))

versionDataUrl=''

for record in versionobject['records']:

versionDataUrl = record['VersionData']

url = 'https://na30.salesforce.com'+versionDataUrl

response = requests.get(url,

headers={ 'Content-Type': 'application/json', 'Authorization': 'Bearer %s' % sessionId })

content = response.text

return content

The first method of Summary class is getcontentfromtextfile. In here our intent is to get the text content of a file. We will try to execute sobject query using simple_salesforce.

Firstly we need to get the salesforce object, this is nothing but initialization of simple salesforce object. docId is sent to the sobject query as an argument. Note to get the text content we need to construct an url which will have the versionData from Content Version pertaining to the content document Id.

Once the url is constructed , querying the same should give you the text content

def insertSummaryInFileSummaryObject(self, docId, summary, keywords):

sf = self.__getSalesforceObject()

sf.FileSummary__c.create({'ContentDocumentId__c':docId, 'FileSummaryText__c':summary, 'Topics__c':keywords})

def insertInFeedSummaryObject(self, feedId, summary):

sf = self.__getSalesforceObject()

sf.FeedSummary__c.create({'FeedId__c':feedId, 'FeedSummaryText__c': summary})

def deleteSummaryObject(self, docId):

sf=self.__getSalesforceObject()

sf.FileSummary__c.delete(docId)

def verifyDocEntryExist(self, docId):

totalRecords=''

sf=self.__getSalesforceObject()

try:

fileSummaries = sf.query_all("SELECT FileSummaryText__c from FileSummary__c WHERE ContentDocumentId__c='%s'" % (docId))

totalRecords=fileSummaries['totalSize']

except ValueError:

print 'I see an exception' + ValueError

if totalRecords == 0:

return False

else:

return True

def updateDocEntryExist(self, docId, summary):

sf=self.__getSalesforceObject()

record = sf.query_all("SELECT FileSummaryText__c from FileSummary__c WHERE ContentDocumentId__c='%s' LIMIT 1" % (docId))

record.FileSummaryText__c= summary

sf.FileSummaryText__c.upsert(record)

All the above methods are related to the CRUD operations for custom object. These are pretty simple to understand and can be interpreted from the Sobject query.

#Get the salesforce object

def __getSalesforceObject(self):

sf = Salesforce(username=‘<<username>>’, password=“<<password>>“, security_token=‘<<token>>’)

return sf

Finally documenting the method which was helping us to create the simple_salesforce object. In here I was using the username/password authentication approach as it was easy to implement and less risky for a hackathon. If you want to implement this in production, I would suggest you follow the oauth approach.

Conclusion

Note I have not documented the feed summarization method here as it is pretty similar to files summarization.

Well that's it folks this is the only code snippet you need to follow for your heroku service. I am not documenting the implementation of Lightning component as there are multiple examples available on net. In order to query this heroku service endpoint from your lightning component, setup a named credential and then create a http request from inside the controller.

There is a huge opportunity to leverage this solution in multiple arenas of Salesforce

Above figure highlights a few

1. New Sales representatives can quickly digest files and feed information

2. Better file search if the indexing happens on summary field

3. Machine can now suggest which could be the correct answer in a question answer feed. Humans can just vote up or down the particular answer

4. Support representative has the ability to connect documents with cases without going through the entire document and just by following the summary.

Screenshots

File Summary

For the above summary here is the text for the actual document

Pomegranate stimulates fat reduction and prevent insulin resistance

Pomegranate is a rich source of punicic acid.

A study conducted Leiden University Medical Centre, Netherlands demonstrated that obese and insulin resistant mice after feeding with pomegranate seed oil for 12 weeks showed lower body weight, significant decrease in body fat and improved peripheral insulin sensitivity.

A research study was conducted at The Department of Biological Sciences and Biotechnology, Tsinghua University, China to investigate the anti-obesity effects of pomegranate leaf extract (PLE).

Obese mice were treated with this extract for 5 weeks. It was observed that obesity and hyperlipidaemia were inhibited by the PLE. Scientists concluded that PLE can be a novel appetite suppressant too.

Another study conducted to investigate the benefits of pomegranate seed oil (PSO) on hyperlipidaemic subjects showed that PSO has favourable effects on the lipid profiles.

Insulin resistance is associated with weight gain and obesity.

Pomegranate is a good source of anthocyanins. A scientific study demonstrated that anthocyanins reduce dyslipidaemia, enhances antioxidant capacity and prevents insulin resistance in diabetic patients.

A review conducted in the King Saud University, Saudi Arabia mentions pomegranate as a preventive agent against obesity. This is mainly because of the tannin, anthocyanin, antioxidants and flavonoid content in them.

What it means? Pomegranate through various mechanisms is found to promote fat reduction and prevents insulin resistance thereby averting the risk of obesity.

2. Pomegranate enhances satiety, appetite control and prevent over-eating

Dietary fiber has a great physiological effect on satiation because of its properties of bulk addition and viscosity impartment.

These fibers prolong the intestinal phase of digestion and absorption. This provides a good time for macronutrients to trigger the signals of satiation and thus prevent in over eating. This property of dietary fiber is highly beneficial in weight management.

Pomegranate is a great source of dietary fiber. They consist of around 11.3g per fruit providing 45% of the daily need of dietary fiber.

Dietary fiber withdraws water and holds on to it. This results in expanding of dietary fiber in stomach enhancing the satiety. As a result, food is absorbed slowly preventing the intake of unnecessary calories.

What it means? Pomegranate is a great source of dietary fiber. Dietary fiber enhances satiation, appetite control and thus helps in controlling body weight.

Creating Lightning component for Salesforce using Machine Learning

2017-07-08T21:42:00.002-07:00

Excited to share my new Salesforce Lightningcomponent which takes in a File content and shows the File Summary and Top File Topic

Firebase authentication with Ionic creator

2017-02-20T21:28:00.001-08:00

Why Ionic creator?

Mobile first is the new mantra of today's applications. Building mobile based apps has also seen a huge growth from early 90s. Along with languages like swift and Java, which are focussed on specific environments like IOS and Android respectively, we have seen good traction on platforms like phonegap and cordova.

Today's startup are aiming mostly on business logic which are compatible with desktop and mobile devices. Investing on individuals for each aspects might be expensive. For ex: Instead of having 3 developers, 1 for android 1 for IOS and 1 for web is far too expensive than having 1 for all 3. Here platforms like Phonegap and Cordova play a vital role in binding one code for all 3 platforms.

Cordova is just a fork of Phonegap, but as per my research if someone ask you which one you will chose, I suggest go with Cordova. Phonegap is a adobe project compared to Cordova which is an apache open source project. Being a part of Apache community it seems to be more reliable and maintainable in future by the community.

You might be wondering where Ionic is going to come?

Well here it is, Ionic plays a critical role in binding cordova plugins with angularjs components. Ionic framework exists since past few years, but what we want to talk about here is Ionic creator which is a web version of creating mobile based app with easy to use utilities.

It took me just 3 hours to have a basic app up and running without going through Android or IOS based native coding tutorials.

Ionic creator gives you a web view where you can not only preview your changes real time but also can create beautiful designs and give that standard native look and feel which will make your app great for the consumer

Why Firebase authentication?

Now after getting the knowledge of ionic creator, which is the UI look and feel, let's now focus on the backend stuff. When we are talking about backend we should be considering couple of questions which will help us design our app

1. App performance should be good, Database related actions should not prove to be a poor performant.

2. App should be able to handle offline data. This becomes incredibly important for mobile based apps. Offline data should sync seamlessly when it gets network back.

3. App authentication can be personified as a backbone to your app work. Authentication should be secure and managed by certified vendor.

4. It should work in scale

The list of design consideration goes on and on. But let's focus on a solution based on the above listed aspects.

Firebase is a relatively new product released by Google. Yeah Google! that answers you scaling requirement.

Performance is relatively fast, as the data transmission is done in form of json objects. Being a no sql type , fields can be added or deleted on the fly.

Firebase has support for offline data also, however I have not explored this, but as per documentation feel this can be achieved with minimal effort

There are tons of other advantages for this. But the most striking advantage is with Ionic creator Firebase comes as a plugin. Believe me when I say it will take 15 min for you to setup an app and get it authenticated from Firebase.

Why Ionic creator with Firebase authentication?

Let's jump to the other topic which is the crux of this blog. Let's first talk about the motivation behind this blog.

If you search youtube there are samples where Ionic creator team has shown how to create an app in minutes. If you checkout the authentication part link you will see they used ionicauth for the same.

Now when I started working with this, the first question I had that where I want to store my data. I wanted to go with Firebase, now my question was how will I authorize my app. Examples showed by integration with ionic auth. Honestly this did not make sense to me.

I didn't like the fact that ionic auth will authorize the app and the data then will go and get stored in firebase. This was weird because Firebase itself has an authentication piece where user can signup/signin with their email and password

So I started exploring Firebase auth with Ionic creator. Now comes the exciting part. The code which was working with ionic auth in case of error was not working with Firebase. In error scenario, ionicauth was able to update the ui element with the corresponding error message. Here {{errorBox}} is the ui element.

People have posted on forums (https://forum.ionicframework.com/t/basic-firebase-error-handling/65798 ) but I couldn't find any solution for this. Note there are examples with ionic framework and firebase auth but none with ionic creator.

The reason for this blog is to share the fix with you

Enough of whys now let's answer how?

Refer to https://github.com/driftyco/creator-weekly-workshops/tree/master/04-ionic-auth-and-gravatar for the ionic auth integration with ionic creator, video is available at https://www.youtube.com/watch?v=McBS4fbPt5Q

The code which is handling the auth and on success helping the user to navigate to the home page is as following

$ionicAuth.signup($scope.data).then(function() {

// `$ionicUser` is now registered

$ionicAuth.login('basic', $scope.data).then(function(){

$state.go('menu.home');

});

}, function(err) {

var error_lookup = {

'required_email': 'Missing email field',

'required_password': 'Missing password field',

'conflict_email': 'A user has already signed up with that email',

'conflict_username': 'A user has already signed up with that username',

'invalid_email': 'The email did not pass validation'

}

$scope.errorBox = error_lookup[err.details[0]];

});

Note on success the ionicAuth.login is going to home and on error it should update a paragraph component populating it with the error details. This error scenario fails when you try to get Firebase auth. Why? because the firebase auth action is asynchronous and we need a mechanism to invoke the update action on the ui paragraph element in case of error so that it is updated properly.

To solve this we can have the following

$scope.errorBox='';

const promise = firebase.auth().createUserWithEmailAndPassword($scope.data.email,

$scope.data.password);
promise.then(resp => {
    
        $state.go('menu.home');
    })
    .catch(err => {
        $scope.$apply(function(){
        //Set error box message        $scope.errorBox = err.message;
    });
})

In case of firebase, when there is an error one should do $scope.$apply and then update the element with the message

Note the intent of this blog is to save that one hour which I spent learning about firebase async transaction and angularjs $apply function. Once this was done everything worked fine.

Hope this blog will help someone to save his/her few hours of investigation. Let me know if you see any issues.

Part 2 - (This section has been updated with the latest)

Thanks all for liking the earlier analysis. As few of you wanted to know the implementation I am pasting the code snippet which helped me create the login and signup feature. In my app I had a restaurant signup page and a user signup page. The login page was common for both.

Will request you to post any questions/concerns if you have regarding the following. I will try to help to the best of my knowledge.

Login method:

.controller('loginCtrl', ['$scope', '$stateParams', '$firebaseArray', '$state', function ($scope, $stateParams, $firebaseArray, $state) {

$scope.data = {

'email': '',

'password': ''

}

$scope.login = function(){

$scope.errorBox = '';

const promise = firebase.auth().signInWithEmailAndPassword($scope.data.email, $scope.data.password);

promise.then(resp => {

$state.go('myHome')

})

.catch(err => {

$scope.$apply(function(){

$scope.errorBox = err.message;

});

}

}])

Here is the corresponding template

<div>

</div>

<ion-list id="login-list1">

<span class="input-label">Email</span>

</label>

<span class="input-label">Password</span>

</label>

</ion-list>

<a ui-sref="signup" id="login-button2" class="button button-positive button-block">Or create an account</a>

</form>

Signup:

.controller('signupCtrl', ['$scope', '$stateParams', '$firebaseArray', '$state', // The following is the constructor function for this page's controller. See https://docs.angularjs.org/guide/controller

// You can include any angular dependencies as parameters for this function

// TIP: Access Route Parameters for your page via $stateParams.parameterName

function ($scope, $stateParams, $firebaseArray, $state) {

$scope.data = {

'email': '',

'password': '',

'usertype':''

}

$scope.signup = function(){

$scope.errorBox='';

const promise = firebase.auth().createUserWithEmailAndPassword($scope.data.email, $scope.data.password);

promise.then(resp => {

createUser();

if($scope.data.usertype == "individual"){

$state.go('individualSignup');

} else if($scope.data.usertype = "restaurant"){

$state.go('restaurantSignup');

}

})

.catch(err => {

$scope.$apply(function(){

//Set error box message

$scope.errorBox = err.message;

});

})

}

//By default check the individual option

$scope.activeSection = 1;

$scope.data.usertype = "individual";

//Choose Occupation type as individual or business

$scope.choseType = function(option){

$scope.activeSection = option;

if(option == 1){

$scope.data.usertype = "individual";

}

else if(option == 2){

$scope.data.usertype = "restaurant";

}

function createUser(){

firebase.auth().onAuthStateChanged(function(user) {

if (user) {

var uid = user.uid;

var email = user.email;

var userType = $scope.data.usertype;

firebase.database().ref('users/' + uid).set({

email: email,

user_type : userType

});

} else {

// No user is signed in.

}

});

}

}])

Here is the corresponding template

<div>

</div>

<button id="signup-button30" ng-class="{'active':activeSection==1}" ng-click="choseType(1)" class="button button-positive button-block">Individual</button>

<button id="signup-button31" ng-class="{'active':activeSection==2}" ng-click="choseType(2)" class="button button-positive button-block">Restaurant</button>

</div>

<ion-list id="signup-list4" ng-show="activeSection==1">

<span class="input-label">Email</span>

</label>

<span class="input-label">Password</span>

</label>

</ion-list>

<ion-list id="signup-list5" ng-show="activeSection==2">

<span class="input-label">Restaurant Email</span>

</label>

<span class="input-label">Password</span>

</label>

</ion-list>

<p style="margin-top:0px;color:#000000;">{{errorBox}}</p>

</div>

<button id="signup-button25" ng-click="signup()" class="button button-assertive button-block button-positive">Signup</button>

<a ui-sref="login" id="signup-button26" class="button button-positive button-block">Already have an account</a>

</form>

Challenges faced with Runc

2017-01-27T23:49:00.004-08:00

Introduction and motivation

I started working on Runc since past 2 weeks and it has indeed helped me articulate the infrastructure I wanted to have. The use cases for Runc stands out as it helps to create a docker container with ease. A docker container can help you add more security to your app or solution. As an attacker I will always look for secure system related information access like /etc/passwd. Docker solution can provide a jail environment which will not allow the attacker to get access of anything outside jail scope. The use case is potent enough for any application scope.

Creating a docker container is not at all a rocket science, and to add a cherry to the cake, now we have runc. Runc is a wrapper around docker which in turn spawns up a container without you installing docker on your machine. Let me give a brief about the environment setup before diving into the challenges I faced during this exploration.

Pre-requisites

I verified all my experiments in CentOS so will be discussing the same

1. Make sure Kernel OS is 2.6.32-642.6.2 or higher

2. Install Go from here

https://golang.org/doc/install

3. Build Runc by following documentation at here

https://github.com/opencontainers/runc

4. Make sure you have gcc if not execute the following

sudo yum install gcc gcc-c++

5. Runc will get installed in /usr/local/sbin , make sure this is in your path

Troubleshoot

I am not going to talk about the execution aspect of Runc because the intention of this blog is to save time for others who might get stuck in situations I have faced

Troubleshoot #1

Issue Link : https://github.com/opencontainers/runc/issues/1270

I started getting this broken pipe error pretty early in the stage. Things began to get worse when I was getting solution related to docker and not related to runc. Runc is pretty new player, so we don't see enough discussion threads around it. I was struggling to get response from Runc forum too.

During my exploration I happen to hit this thread where people were discussing about the related services. This led me to search for cgroup services.

Here cgroup means control groups, you can define max min caps for cpu processes, memory utilization etc in these cgroups. For me this service was turned off. I need to explicitly execute

Service cgconfig start

to make this service start which solved the issue.

Troubleshoot #2

In config.json you can have mounted section for cgroup

{
"destination": "/sys/fs/cgroup",
"type": "cgroup",
"source": "cgroup",
"options": [
"nosuid",
"noexec",
"nodev",
"relatime",
"ro"
]
}

This section means that there will be a mounted path created on container which is noexecutable and read only path in the target container

The issue we are talking here is similar to broken pipe issue which we discussed in troubleshoot #1, whose solution is also the same. Because of service unavailable you might see the broken pipe issue. Service cgconfig start , solves the issue.

Troubleshoot #3

Issue Link: https://github.com/opencontainers/runc/issues/1299

I should mention the solution of this issue is a sloppy hack. Runc is currently built with glibc 2.14. Another blocker arose when I figured out that my machine had glibc.2.12 and till date I have not found any solution where I can build runc using libc 2.12

Still I am trying to find a permanent solution to this but to unblock you here is the sloppy hack I mentioned earlier

sed s#"2.14"#"2.12"#gi /path/to/runc > runc_new

This will rename all 2.14 versions to 2.12, and will help you to get unblocked.

Troubleshoot #4

When you are building runc you get couple of BuildTags with which you can build it

Build Tag	Feature	Dependency
seccomp	Syscall filtering	libseccomp
selinux	selinux process and mount labeling
apparmor	apparmor profile support	libapparmor
ambient	ambient capability support	kernel 4.3

Quoting from https://github.com/opencontainers/runc/blob/master/README.md

Each Build Tag has its importance. Initially I was having selinux from my tutorial while building runc

make BUILDTAGS='selinux'

I figured out this was an issue for my runc as SeLinux was not a compiler support in my case. In such situation I will suggest you include BUILDTAGS with empty params like

make BUILDTAGS=''

Conclusion

The primary intent to capture these findings in a blog is not only for future reference but also is an approach of validating my observation. If you feel I have captured something wrong please do mention it.

I will continue capturing the issues with associated learnings. Hope this helps.

Create Salesforce Contacts from Quip Contacts

2017-01-22T23:03:00.002-08:00

Quoting Wikipedia: "Quip is a collaborative productivity software suite for mobile and the Web. It allows groups of people to create and edit documents and spreadsheets as a group, typically for business purposes"

Motivated by Quip Apis mentioned at here I started exploring use cases where we can leverage the apis to solve a particular problem. One use case which prevailed on all others was to Create Salesforce Contacts from Quip Contacts

Use Case: Assuming there is a sales document which was shared with some key stake holders who may or may not be associated with Salesforce org. These contacts may make sense to associate with a lead or opportunity or account. As a Salesforce Administrator I should be able to import contacts from Quip documents to Salesforce orgs.

In order to do this my plan was to create a lightweight utility which can easily be used. As a part of my study I figured out that Quip came with 2 kind of api wrappers one was written in python and another one in nodejs. The node.js library does a rest api call in the backend.

I chose nodejs because I had some experience with Jsforce which is a Salesforce nodejs library.

Before starting I played around with the nodejs api for quip contacts. Interestingly I got following response

Note the API not only sends the name but also sends a weightage factor called affinity.

Interestingly this affinity seems to be accurate as it somehow recommends who are the users which might be more related to this document in comparison with others. Somehow I feel this is a potent entity which can be used in many use cases to filter out those users who might not be interested in the document.

After getting comfortable with Quip Api it was time for importing Quip contacts and bridging the gap by creating those contacts in salesforce. For this I used Jsforce which gives an easy access to play around with SOQL queries

Following is the nodejs script which does this

var quipaccessToken = "<<Quip Access Token>>";
var loginUrl = "<<Salesforce Login Url>>";
var loginUserName = "<<Salesforce login username>>";
var loginPassword = "<<Salesforce login password>>";
var quip = require('./quip.js');
var client = new quip.Client({accessToken: quipaccessToken});

importQuipContactsAndCreateSalesforceContacts(client);

/*

Get Quip Contacts and create Salesforce contacts

*/

function importQuipContactsAndCreateSalesforceContacts(client) {
    /*

    Get contacts with affinity     Result: name, id, affinity

*/

    client.getContacts(function (err, user) {
        if (err != null) {
            console.log("err: " + err);
        }

        for (var i = 0, len = user.length; i < len; i++) {
            nameArr = user[i].name.split(/\s+/);
            console.log("FirstName: "+nameArr[0]+" LastName: "+ nameArr[1]);

            //Verify if Salesforce Contact exists and then Create it

            verifyAndCreateSalesforceContact(nameArr[0], nameArr[1]);
        }

    });
}


/*

Create a Contact in salesforce

*/

function createContactsInSalesforce(firstname, lastname){
    var jsforce = require('jsforce');
    var conn = new jsforce.Connection({
        loginUrl: loginUrl    });
    conn.login(loginUserName, loginPassword, function(err, res) {
        if (err) {
            return console.error(err);
        }

        //Verify if the user exists. If the user is not there then insert it.

        console.log("Inside Create");
        conn.sobject("Contact").create({FirstName: firstname, LastName: lastname},

        function (err, ret) {
            if (err || !ret.success) {
                return console.error(err, ret);
            }

            console.log("Created record id: " + ret.id);
        });
    });
}

/*

Verify if the contact exists

*/

function verifyAndCreateSalesforceContact(firstName, lastName){
    var jsforce = require('jsforce');
    var conn = new jsforce.Connection({
        loginUrl: loginUrl    });
    conn.login(loginUserName, loginPassword, function(err, res) {
        if (err) {
            return console.error(err);
        }
        conn.sobject("Contact")
            .find(
                {
                    FirstName: firstName,
                    LastName: lastName
                },
                {
                    Id: 1,
                    Name: 1,
                    CreatedDate: 1                }
            )
            .execute(function (err, records) {
                if (err) {
                    return console.error(err);
                }
                if(records.length === 0){
                    createContactsInSalesforce(firstName, lastName);
                }
            })
    });
}

Note here we are getting Quip Access token from this link

The above script imports QuipContacts in importQuipContactsAndCreateSalesforceContacts function.

In verifyAndCreateSalesforceContact we verify if the contact is existing in salesforce or not.

In createContactsInSalesforce we create Salesforce contacts.

The app gives following response

Created record id: 003R0000001WuFEIA0
Created record id: 003R0000001WuF4IAK
Created record id: 003R0000001WuFOIA0
...

where it responds back with the newly created userId in Salesforce.

Hope this information helps. I feel the design for the above script can be improved,

but here the intent was to quickly come up with a utility for import contacts. Let me know

if you have any feedback for this.

Topic Classification using Latent Dirichlet Allocation(LDA) ML Library

2016-08-10T21:38:00.001-07:00

import org.apache.spark.sql.SparkSession
val sparkSession = SparkSession.builder
  .master("local")
  .appName("my-spark-app")
  .config("spark.some.config.option", "config-value")
  .getOrCreate()
​
val df = spark.read.json("dbfs:/mnt/JSON10/JSON/sampleDoc.txt")


import org.apache.spark.sql.SparkSession
sparkSession: org.apache.spark.sql.SparkSession = org.apache.spark.sql.SparkSession@351b4b37
df: org.apache.spark.sql.DataFrame = [filename: string, id: string ... 1 more field]

> 

display(df)

06930000002Uik0AAC_v003.txt06930000002Uik0Getting started is wonderful here
069300000033HZFAA2_v001.txt069300000033HZFre: december fy13 wonderfully explained �
06930000001cVBkAAM_v001.txt06930000001cVBkfitness reimbursement ?? slide number 1

filenameidtext


06930000002Uik0AAC_v003.txt	06930000002Uik0	Getting started is wonderful here
069300000033HZFAA2_v001.txt	069300000033HZF	re: december fy13 wonderfully explained �
06930000001cVBkAAM_v001.txt	06930000001cVBk	fitness reimbursement ?? slide number 1

> 

df.drop(df.col("_corrupt_record"))
val splits = df.randomSplit(Array(0.8, 0.2), seed = 11L)
val train = splits(0).cache()
val test = splits(1).cache()
​

org.apache.spark.sql.AnalysisException: Cannot resolve column name "_corrupt_record" among (filename, id, text);

> 

display(df)

06930000002Uik0AAC_v003.txt06930000002Uik0Getting started is wonderful here
069300000033HZFAA2_v001.txt069300000033HZFre: december fy13 wonderfully explained �
06930000001cVBkAAM_v001.txt06930000001cVBkfitness reimbursement ?? slide number 1

filenameidtext


06930000002Uik0AAC_v003.txt	06930000002Uik0	Getting started is wonderful here
069300000033HZFAA2_v001.txt	069300000033HZF	re: december fy13 wonderfully explained �
06930000001cVBkAAM_v001.txt	06930000001cVBk	fitness reimbursement ?? slide number 1

> 






import org.apache.spark.ml.feature.RegexTokenizer
​
// Set params for RegexTokenizer
val tokenizer = new RegexTokenizer()
.setPattern("[\\W_]+")
.setMinTokenLength(4) // Filter away tokens with length < 4
.setInputCol("text")
.setOutputCol("tokens")
​
// Tokenize document
val tokenized_df = tokenizer.transform(df)


import org.apache.spark.ml.feature.RegexTokenizer
tokenizer: org.apache.spark.ml.feature.RegexTokenizer = regexTok_05b9b2c90cb1
tokenized_df: org.apache.spark.sql.DataFrame = [filename: string, id: string ... 2 more fields]

> 

display(tokenized_df)

06930000002Uik0AAC_v003.txt06930000002Uik0Getting started is wonderful here["getting","started","wonderful","here"]
069300000033HZFAA2_v001.txt069300000033HZFre: december fy13 wonderfully explained �["december","fy13","wonderfully","explained"]
06930000001cVBkAAM_v001.txt06930000001cVBkfitness reimbursement ?? slide number 1["fitness","reimbursement","slide","number"]

filenameidtexttokens


06930000002Uik0AAC_v003.txt	06930000002Uik0	Getting started is wonderful here	["getting","started","wonderful","here"]
069300000033HZFAA2_v001.txt	069300000033HZF	re: december fy13 wonderfully explained �	["december","fy13","wonderfully","explained"]
06930000001cVBkAAM_v001.txt	06930000001cVBk	fitness reimbursement ?? slide number 1	["fitness","reimbursement","slide","number"]

> 

display(tokenized_df.select("tokens"))

["getting","started","wonderful","here"]
["december","fy13","wonderfully","explained"]
["fitness","reimbursement","slide","number"]

tokens


["getting","started","wonderful","here"]
["december","fy13","wonderfully","explained"]
["fitness","reimbursement","slide","number"]

> 

%sh wget http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words -O /tmp/stopwords

--2016-08-11 04:48:37--  http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words
Resolving ir.dcs.gla.ac.uk (ir.dcs.gla.ac.uk)... 130.209.240.253
Connecting to ir.dcs.gla.ac.uk (ir.dcs.gla.ac.uk)|130.209.240.253|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2237 (2.2K) [text/plain]
Saving to: ?/tmp/stopwords?

     0K ..                                                    100%  350M=0s

2016-08-11 04:48:37 (350 MB/s) - ?/tmp/stopwords? saved [2237/2237]

> 

%fs cp file:/tmp/stopwords dbfs:/tmp/stopwords

res53: Boolean = true

> 

val stopwords = sc.textFile("/tmp/stopwords").collect()

stopwords: Array[String] = Array(a, about, above, across, after, afterwards, again, against, all, almost, alone, along, already, also, although, always, am, among, amongst, amoungst, amount, an, and, another, any, anyhow, anyone, anything, anyway, anywhere, are, around, as, at, back, be, became, because, become, becomes, becoming, been, before, beforehand, behind, being, below, beside, besides, between, beyond, bill, both, bottom, but, by, call, can, cannot, cant, co, computer, con, could, couldnt, cry, de, describe, detail, do, done, down, due, during, each, eg, eight, either, eleven, else, elsewhere, empty, enough, etc, even, ever, every, everyone, everything, everywhere, except, few, fifteen, fify, fill, find, fire, first, five, for, former, formerly, forty, found, four, from, front, full, further, get, give, go, had, has, hasnt, have, he, hence, her, here, hereafter, hereby, herein, hereupon, hers, herself, him, himself, his, how, however, hundred, i, ie, if, in, inc, indeed, interest, into, is, it, its, itself, keep, last, latter, latterly, least, less, ltd, made, many, may, me, meanwhile, might, mill, mine, more, moreover, most, mostly, move, much, must, my, myself, name, namely, neither, never, nevertheless, next, nine, no, nobody, none, noone, nor, not, nothing, now, nowhere, of, off, often, on, once, one, only, onto, or, other, others, otherwise, our, ours, ourselves, out, over, own, part, per, perhaps, please, put, rather, re, same, see, seem, seemed, seeming, seems, serious, several, she, should, show, side, since, sincere, six, sixty, so, some, somehow, someone, something, sometime, sometimes, somewhere, still, such, system, take, ten, than, that, the, their, them, themselves, then, thence, there, thereafter, thereby, therefore, therein, thereupon, these, they, thick, thin, third, this, those, though, three, through, throughout, thru, thus, to, together, too, top, toward, towards, twelve, twenty, two, un, under, until, up, upon, us, very, via, was, we, well, were, what, whatever, when, whence, whenever, where, whereafter, whereas, whereby, wherein, whereupon, wherever, whether, which, while, whither, who, whoever, whole, whom, whose, why, will, with, within, without, would, yet, you, your, yours, yourself, yourselves)

> 






import org.apache.spark.ml.feature.StopWordsRemover
​
// Set params for StopWordsRemover
val remover = new StopWordsRemover()
.setStopWords(stopwords) // This parameter is optional
.setInputCol("tokens")
.setOutputCol("filtered")
​
// Create new DF with Stopwords removed
val filtered_df = remover.transform(tokenized_df)


import org.apache.spark.ml.feature.StopWordsRemover
remover: org.apache.spark.ml.feature.StopWordsRemover = stopWords_783ff1555eb7
filtered_df: org.apache.spark.sql.DataFrame = [filename: string, id: string ... 3 more fields]

> 

display(filtered_df)

06930000002Uik0AAC_v003.txt06930000002Uik0Getting started is wonderful here["getting","started","wonderful","here"]["getting","started","wonderful"]
069300000033HZFAA2_v001.txt069300000033HZFre: december fy13 wonderfully explained �["december","fy13","wonderfully","explained"]["december","fy13","wonderfully","explained"]
06930000001cVBkAAM_v001.txt06930000001cVBkfitness reimbursement ?? slide number 1["fitness","reimbursement","slide","number"]["fitness","reimbursement","slide","number"]

filenameidtexttokensfiltered


06930000002Uik0AAC_v003.txt	06930000002Uik0	Getting started is wonderful here	["getting","started","wonderful","here"]	["getting","started","wonderful"]
069300000033HZFAA2_v001.txt	069300000033HZF	re: december fy13 wonderfully explained �	["december","fy13","wonderfully","explained"]	["december","fy13","wonderfully","explained"]
06930000001cVBkAAM_v001.txt	06930000001cVBk	fitness reimbursement ?? slide number 1	["fitness","reimbursement","slide","number"]	["fitness","reimbursement","slide","number"]

> 






​
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.sql.Row
​
​
import org.apache.spark.ml.feature.CountVectorizer
​
// Set params for CountVectorizer
val vectorizer = new CountVectorizer()
.setInputCol("filtered")
.setOutputCol("features")
.fit(filtered_df)


import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.sql.Row
import org.apache.spark.ml.feature.CountVectorizer
vectorizer: org.apache.spark.ml.feature.CountVectorizerModel = cntVec_663ad4846e39

> 

vectorizer.transform(filtered_df).select("id", "text","features", "filtered").show()

+---------------+--------------------+--------------------+--------------------+
|             id|                text|            features|            filtered|
+---------------+--------------------+--------------------+--------------------+
|06930000002Uik0|Getting started i...|(11,[2,3,6],[1.0,...|[getting, started...|
|069300000033HZF|re: december fy13...|(11,[5,7,8,10],[1...|[december, fy13, ...|
|06930000001cVBk|fitness reimburse...|(11,[0,1,4,9],[1....|[fitness, reimbur...|
+---------------+--------------------+--------------------+--------------------+

> 

import org.apache.spark.mllib.linalg.Vector;
import org.apache.spark.mllib.linalg.Vectors;
​
val countVectors = vectorizer.transform(filtered_df)
​

import org.apache.spark.mllib.linalg.Vector
import org.apache.spark.mllib.linalg.Vectors
countVectors: org.apache.spark.sql.DataFrame = [filename: string, id: string ... 4 more fields]

> 

countVectors.show
​

+--------------------+---------------+--------------------+--------------------+--------------------+--------------------+
|            filename|             id|                text|              tokens|            filtered|            features|
+--------------------+---------------+--------------------+--------------------+--------------------+--------------------+
|06930000002Uik0AA...|06930000002Uik0|Getting started i...|[getting, started...|[getting, started...|(11,[2,3,6],[1.0,...|
|069300000033HZFAA...|069300000033HZF|re: december fy13...|[december, fy13, ...|[december, fy13, ...|(11,[5,7,8,10],[1...|
|06930000001cVBkAA...|06930000001cVBk|fitness reimburse...|[fitness, reimbur...|[fitness, reimbur...|(11,[0,1,4,9],[1....|
+--------------------+---------------+--------------------+--------------------+--------------------+--------------------+

> 






import org.apache.spark.ml.clustering.DistributedLDAModel
import org.apache.spark.sql.{Row, SQLContext}
import org.apache.spark.sql.types.{StructField, StructType}
​
val lda = new LDA()
  .setK(3)
  .setMaxIter(10)
  .setFeaturesCol("features")
​


import org.apache.spark.ml.clustering.DistributedLDAModel
import org.apache.spark.sql.{Row, SQLContext}
import org.apache.spark.sql.types.{StructField, StructType}
lda: org.apache.spark.ml.clustering.LDA = lda_f977d0e6a90e

> 

​
​
val transformed = model.transform(countVectors)
​

transformed: org.apache.spark.sql.DataFrame = [filename: string, id: string ... 5 more fields]

> 






​
​
val ll = model.logLikelihood(countVectors)
val lp = model.logPerplexity(countVectors)
​
// describeTopics
val topics = model.describeTopics(10)
​
// Shows the result
topics.show(false)
transformed.show(false)


+-----+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|topic|termIndices                    |termWeights                                                                                                                                                                                                       |
+-----+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|0    |[2, 5, 7, 9, 4, 10, 1, 0, 3, 8]|[0.1062041440811267, 0.10583753784271348, 0.10442749933639424, 0.09682858305200294, 0.09351340983989168, 0.09131890650704068, 0.08714867387466924, 0.08070231248484354, 0.0802804983696842, 0.07701228167379913]  |
|1    |[1, 6, 2, 5, 0, 3, 8, 9, 4, 7] |[0.10195178268898618, 0.09823618268931927, 0.09637471483627483, 0.09537821409096363, 0.09317671877781553, 0.08903558963685265, 0.08890389418467605, 0.08631694410851917, 0.08621391878142007, 0.08438517259595037]|
|2    |[1, 9, 4, 2, 5, 10, 3, 8, 6, 7]|[0.10612398716703186, 0.09757374578910046, 0.09660179632115497, 0.09375920815595054, 0.09318237101826143, 0.09050611023493967, 0.08978021671904904, 0.08546034203857293, 0.08355203917046891, 0.08177244097296857]|
+-----+-------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

+---------------------------+---------------+-------------------------------------------+----------------------------------------+----------------------------------------+---------------------------------+------------------------------------------------------------+
|filename                   |id             |text                                       |tokens                                  |filtered                                |features                         |topicDistribution                                           |
+---------------------------+---------------+-------------------------------------------+----------------------------------------+----------------------------------------+---------------------------------+------------------------------------------------------------+
|06930000002Uik0AAC_v003.txt|06930000002Uik0|Getting started is wonderful here          |[getting, started, wonderful, here]     |[getting, started, wonderful]           |(11,[2,3,6],[1.0,1.0,1.0])       |[0.10318703949282229,0.7938521983720901,0.10296076213508765]|
|069300000033HZFAA2_v001.txt|069300000033HZF|re: december fy13 wonderfully explained   �|[december, fy13, wonderfully, explained]|[december, fy13, wonderfully, explained]|(11,[5,7,8,10],[1.0,1.0,1.0,1.0])|[0.835109741017235,0.08292735466870861,0.08196290431405635] |
|06930000001cVBkAAM_v001.txt|06930000001cVBk|fitness reimbursement   ??   slide number 1|[fitness, reimbursement, slide, number] |[fitness, reimbursement, slide, number] |(11,[0,1,4,9],[1.0,1.0,1.0,1.0]) |[0.08429553958886034,0.08695919972860706,0.8287452606825326]|
+---------------------------+---------------+-------------------------------------------+----------------------------------------+----------------------------------------+---------------------------------+------------------------------------------------------------+

ll: Double = -46.015729151104544
lp: Double = 4.183248263936846
topics: org.apache.spark.sql.DataFrame = [topic: int, termIndices: array<int> ... 1 more field]

For developers only - Runu Apis

2016-07-31T11:52:00.002-07:00

Site: http://runu.herokuapp.com/

Introduction

Runu has been running live from past 3 weeks and is getting traction on how to interact with cognitive apis. This has been a platform to showcase the learning algorithms of Google and Microsoft, thereby helping any individual to decide if he/she wants to leverage these features in their app or not.

Runu currently comes with 8 kinds of cognitive Analysis

Image detection - Talking to Google Vision
Logo Detection - Talking to Google Vision
Face detection - Talking to Google Vision
Location detection - Talking to Google Vision
Celebrity detection - Talking to Microsoft's Cognitive APIs
Text Detection - Talking to Google Vision
Emotion Detection - Talking to Microsoft's Cognitive APIs
What is this image (Caption detection)-Talking to Microsoft's Cognitive APIs
Sentiment Analysis for text - Talking to Google NLP

The app has an hybrid mechanism to talk to both Google and Microsoft apis in order to leverage both the features in analyzing an image.

What's new in Runu?

Runu has been updated with a REST api layer for each features mentioned above. This will help any developer to leverage the apis easily with his/her own features. (Please note, this is an experimental version and can be changed in future). If you want to integrate this feature drop a mail to sengupta.nabarun@gmail.com so that we can make you aware of any kind of disruption in service.

These features will be available without any cost of payment.

Currently all REST Apis are giving JSON responses.

1. Image Detection

API call: http://runu.herokuapp.com/getlabels?url=http://runu.herokuapp.com/Images/runu-imagedetection_owl.jpg

Request Type: GET

Response Format: JSON

Example: ["bird","owl","wildlife","beak","nature"]

2. Logo Detection

API call: http://runu.herokuapp.com/getlogo?url=http://runu.herokuapp.com/Images/runu-logodetection_cocacola.png

Request Type: GET

Response Format: JSON

Example: ["Coca-Cola"]

3. Face Detection

API call: http://runu.herokuapp.com/getface?url=http://runu.herokuapp.com/Images/runu-facedetection.png

Request Type: GET

Response Format: STRING

Example: "Found 1 face"

4. Location Detection

API call: http://runu.herokuapp.com/getlandmark?url=http://runu.herokuapp.com/Images/wunu-locationdetection_tajmahal.jpeg

Request Type: GET

Response Format: JSON

Example: ["Taj Mahal"]

5. Celebrity Detection

API call: http://runu.herokuapp.com/getcelebrity?url=http://runu.herokuapp.com/Images/runu-celebritydetection.jpg

Request Type: GET

Response Format: JSON

Example: ["Brad Pitt","Angelina Jolie"]

6. Text Detection

API call: http://runu.herokuapp.com/gettext?url=http://runu.herokuapp.com/Images/runu-ocrdetection.jpg

Request Type: GET

Response Format: JSON

Example: ["Imagination is\nGG\nmore important\nthan knowledge.\nAlbert Einstein\n","Imagination","is","GG","more","important","than","knowledge.","Albert","Einstein"]

7. Emotion Detection

API call: http://runu.herokuapp.com/getemotion?url=http://runu.herokuapp.com/Images/runu-emotiondetection.jpg

Request Type: GET

Response Format: JSON

Example: {"Anger":0.999675453,"Contempt":0,"Disgust":0,"Fear":0,"Happiness":0,"Neutral":0,"Sadness":0,"Surprise":0}

8. What is this image (Caption Detection)

API call: http://runu.herokuapp.com/getAnswers?url=http://runu.herokuapp.com/Images/runu-answerme.jpg

Request Type: GET

Response Format: STRING

Example: "there is a star"

9. Sentiment Analysis for text

API call: http://runu.herokuapp.com/getentities?text=Stop%20human%20killing.%20This%20is%20not%20helping%20anyone.%20We%20want%20peace.

Request Type: GET

Response Format: STRING

Example: {"documentSentiment":{"polarity":-0.9,"magnitude":1.8},"language":"en"}
Details: https://www.linkedin.com/pulse/runu-sentiment-analysis-nabarun-sengupta

Conclusion

The intent of these apis are to provide easy rest api calls to the cognitive apis in backend. These features can be leveraged in creation of demo POCs. For further enquiry regarding the transaction drop a mail to sengupta.nabarun@gmail.com.

Happy Coding!!!

Project Runu - Try it out yourself

2016-07-10T11:50:00.000-07:00

Ever imagined how robots in future will understand object.

Well, here it is... Introducing Runu vision which works with Google Vision apis to understand what is there in an image.

Find out exciting stuff where the system not only understand the images but also learn about them.

URL: http://runu.herokuapp.com/
Demo: https://youtu.be/PL8ewCu79gU

Copy and paste any image url and click on Analyze button and see how the system understands what is there in the image.

For further questions ask sengupta.nabarun@gmail.com

Please do share if you like this and give your feedback for improvement.

Project Runu- Giving eyes to machine

2016-06-04T10:27:00.002-07:00

Project Runu is interacting with Microsoft cognitive apis to analyze an Image and tell the user the details about it.

Motivation

The motivation of project runu came from Microsoft cognitive api demo. In here it showed a blind person leveraging the power of cognitive api and learning about his surroundings.



This led me think of an alternate solution for security cameras, face detection badge swapping machines, tourist helper etc. I will mention about these ideas later.

Preparation Time

I am a great fan of Node.js. With Node.js you can easily device a lightweight javascript application for your system. The increasing popularity of Node.jsmade Technical Machine team to device an unique IOT device which we know as Tessel.

Coding with IOT is no longer a challenge for any node.js enthusiast like me as deployment of code on IOT has never got so easy. In this project I am using Tessel 2, which is having support for io.js. Tessel2 comes with USB support, so any USB supported camera can easily marry with it.

Action Time

To start with I explored the npm modules for tessel camera, when I got this resource . Tessel-AV helps you interact with image, video and audio streams easily.

The next thing was to marry this with the Microsoft cognitive api. For that one need to register in Microsoft cognitive service. On registering it will give a API key. For our project we just need this api key.

Here is the node.js script which is interacting with USB supported camera, capturing a picture, sending it to cognitive api, and coming back with the analysis

var fs = require('fs');
var av = require('tessel-av');
var os = require('os');
var http = require('http');
var port = 8082;
var camera = new av.Camera();
null
//Create a http server and capture the image from tessel
//supported usb camera
http.createServer((request, response) => {
 response.writeHead(200, { 'Content-Type': 'image/jpg' });
 var capture = camera.capture();
 capture.pipe(response);
 capture.on('data',function(data){
       saveimage(data);
       analyzeimage();
 });
}).listen(port, () => console.log(`http://${os.hostname()}.local:${port}`));
null
//Analyze the captured image by sending it to the cognitive
//api for description
function analyzeimage(){
       fs.readFile(__dirname+'/captured-image.jpg',function(err,data){
       if(err)
               console.log(err);
       else{
               parseURL = require('url');
               parsedURL = parseURL.parse("https://api.projectoxford.ai/vision/v1.0/analyze?visualFeatures=Description");
               console.log(parsedURL.host);
               var options = {
                       host: parsedURL.host,
                       path: "/vision/v1.0/analyze?visualFeatures=Description,Tags",
                       headers: {
                               'Content-Type':'application/octet-stream',
                               'Content-Length':data.length,
                               'Ocp-Apim-Subscription-Key':'<>'
                       },
                       method:'POST',
                       };
               callback = function(response){
                       var str ='';
                       response.on('data', function(chunk){
                               str+= chunk;
                       });
                       response.on('end', function(){
                               var obj = JSON.parse(str);
                                
                                //Print out the caption
                               console.log("I think the caption of this object should be " + obj.description.captions[0].text+". I am "+obj.description.captions[0].confidence+"sure.");
                       });
               }
               var post_req = http.request(options, callback);
               post_req.write(data);
               post_req.end();
       }
   });
}

This simple 100 approx line of code helped me create this low cost solution which can act as a mobile image identification system.

Here is the demo POC for the same, where I am analyzing a vase of flower and a table clock. The technology not only gives the correct observation but also it considers the environment, like table clock sitting on a desk

POC



Project Runu

Conclusion

This technology can prove to be having immense potential in few fields. Some of the ideas are as follows

1. Security cameras: Security cameras can talk to cognitive apis and send the video stream. As soon as there is any kinds of violence detected by the api, a security alarm can be sent in order to prevent it. The detection alarm can work on captions which have text like gun, knife, bomb etc. A captured image is sent to the officials who can decide whether or not to respond to the alert. Face detection is another story for the same

2. Badge swapping systems: Currently badge surfing is quiet common. There need to be another system which detects the person face and then allow him/her. To strengthen the system, we can use this technology for verification.

These are few of my ideas where I see this technology useful apart from visual impaired support system. If you have any such ideas let us know.

Thanks for reading through this!!

CRMTest - Understanding the Github Repo

2016-04-30T11:11:00.001-07:00

This video is created to walk you through the github project for CRMTest. In this video we took an example of a scenario - Interaction with public cloud. The intent was to show the starting point of understanding the whole flow. Hope you guys will like it. Please post your feedback if you have.

Salesforce + Azure = Happy Customers :)

2016-04-08T01:38:00.001-07:00

Salesforce Rendition + Microsoft Cognitive API = Happy Customers - Part 2

2016-04-05T00:59:00.001-07:00

Yesterday I blogged about the idea of including microsoft's cognitive api to analyze image getting generated from Salesforce content rendition at http://nabaruns.blogspot.com/2016/04/salesforce-content-microsoft-cognitive.html.

However yesterday I was facing challenge in connecting the Salesforce file preview image to cognitive apis.

The major challenge was that Salesforce was not allowing me to get an insecure http image url for the preview renditions. Because of this I was not sure how the image url will be scanned by the cognitive apis.

But wait! there was another way to provide the content

Supported input methods: Raw image binary in the form of an application/octet stream or image URL.

The documentation said it can take raw image binary. This was a life saver for me as the rendition apis in Salesforce https://developer.salesforce.com/docs/atlas.en-us.chatterapi.meta/chatterapi/connect_resources_files_rendition.htm was capable of giving an InputStream for the generated thumbnail.

Now the only gap that need to be bridged was to convert this InputStream to InputStreamEntity and send it to the cognitive apis. Wallah! it worked seamlessly. :)

Here is the node.js client call that has been added after making the changes

Upload a File in Chatter feed
```
response = crmtest.insertChatterFeedFile("/Users/Documents/pdf.pdf", "File Name","Title", accessToken);
var fileId = crmtest.parseresponse(response,"id");
```
- In order to upload a file in chatter insertchatterfeedwithfile method is invoked
- First parameter is the file location
- Second parameter will be the file name
- Third will be the file title
- Fourth will be the oauth accesstoken
- Once the response is recieved it can be parsed through and corresponding file id of the uploaded file can be retrieved.

Send the generated thumbnail preview from file rendition to Microsoft's cognitive api and get image analytics

response = crmtest.analyzeimage(fileId, "<<Microsoft cognitive api key>>", accessToken);
console.log(response+'\n');

Once file preview are generated, in order to get more analytics out of that send the fileId to the analyzeimage call with the Microsoft's cognitive api and accessToken
Raw Image is read from the chatter api and passed to the Microsoft's cognitive api. Note: Microsoft is not going to store the images as per its documentation

Execution time

I uploaded the below seminar pic in my chatter feed

Here is the output

-------------------------------------------------------------

-----------Analyze the generated file preview----------------

-------------------------------------------------------------

{"tags":[{"name":"indoor","confidence":0.98557054996490479},{"name":"auditorium","confidence":0.23898179829120636}],"description":{"tags":["indoor","room","large","building","sitting","table","auditorium","computer","people","man","group","laptop","crowd","standing"],"captions":[{"text":"a crowd of people in a room","confidence":0.26782138202681804}]},"requestId":"a23ecf3e-c0da-42b2-8d48-367848a97b93"}

Note tags are nothing but the suggested topics in salesforce terms.

For more details refer to https://www.npmjs.com/package/crmtest which is updated with these changes. In order to contribute towards crmtest refer at https://github.com/Nabarun/CrmTest/

Hope this will help you in getting more out of Salesforce content.

Salesforce Content + Microsoft Cognitive API = Happy Customers :)

2016-04-04T01:15:00.002-07:00

I cannot hold myself writing a blog on a most amazing video demonstration of cognitive api from Microsoft at https://www.youtube.com/watch?v=R2mC-NUAmMk. This video shows a blind person who is getting assisted with emotions through a cognitive api.

Wow!! first of all hats off to this thinking. I bet whoever is seeing this is having the same impression as I have.

They say innovation triggers innovation, and here it is. After watching this amazing video I went through the Microsoft cognitive api at https://www.microsoft.com/cognitive-services/

This to me is a goldmine to solve many real life problems.

The problem that I am focussed now is on driving topic suggestion out of generated thumbnail preview in chatter file post. The idea is as soon as the chatter post with file attachment is made, based on preview thumbnail generation, the salesforce user should be able to see the topics recommendation/suggestion.

Following step helped me to bring out a partial story around this. (I say partial as still I am trying to bridge the gap between salesforce rendition image url and microsoft's cognitive api.)

1. Signup to Microsoft cognitive api https://www.microsoft.com/cognitive-services/ Register for the free edition

2. After successful registration, click on "My Account" and make a note of key1

3. Follow the instructions at https://www.npmjs.com/package/crmtest and install crmtest by executing the following

npm install crmtest

4. Observe crmtestclient.js and note the cognitive api calls in the bottom

//Suggest topics in preview image

console.log("-------------------------------------------------------------");

console.log("-----Suggest Topics in generated file preview----------------")

console.log("-------------------------------------------------------------");

response = crmtest.suggesttopicsinimage("<<image url>>", "<<cognitive api key>>");

console.log(response+'\n');

//Analyze preview image

console.log("-------------------------------------------------------------");

console.log("-----------Analyze the generated file preview----------------")

console.log("-------------------------------------------------------------");

response = crmtest.analyzeimage("<<image url>>", "<<cognitive api key>>");

console.log(response+'\n');

5. Note the first call is for getting the suggested topics from cognitive api, and the second call is for getting an analysis on the caption of the image .

I verified it on a sample image url at https://cdn1.getyourguide.com/niwziy2l9cvz/1QSurmYr9OmqaMOcOYysq4/1a330fafe9e96da25259c5f004756f36/rome-Vatican-Museum-1112x630.jpg

First impression for a human eye will be what?

Execution time

This is a group of people standing in front of a building and watching an object. Sounds reasonable for me. When I executed the following script I was amazed to see the analysis.

var crmtest = require('./crmtest.js');

//Suggest topics in preview image

console.log("-------------------------------------------------------------");

console.log("-----Suggest Topics in generated file preview----------------")

console.log("-------------------------------------------------------------");

response = crmtest.suggesttopicsinimage("<<image url>>", "<<cognitive api key>>");

console.log(response+'\n');

//Analyze preview image

console.log("-------------------------------------------------------------");

console.log("-----------Analyze the generated file preview----------------")

console.log("-------------------------------------------------------------");

response = crmtest.analyzeimage("<<image url>>", "<<cognitive api key>>");

console.log(response+'\n');

Here is the output

-------------------------------------------------------------

-----Suggest Topics in generated file preview----------------

-------------------------------------------------------------

Topic Name: outdoor

Topic Name: road

Topic Name: building

Topic Name: street

Topic Name: front

Topic Name: large

Topic Name: man

Topic Name: riding

Topic Name: walking

Topic Name: people

Topic Name: sidewalk

Topic Name: city

Topic Name: standing

Topic Name: table

Topic Name: group

Topic Name: bus

Topic Name: train

Topic Name: holding

Topic Name: air

Topic Name: parked

Topic Name: board

Topic Name: track

Topic Name: trick

Topic Name: doing

Topic Name: white

-------------------------------------------------------------

-----------Analyze the generated file preview----------------

-------------------------------------------------------------

Category: building_ Confidence Value: 0.38671875

Category: others_ Confidence Value: 0.0078125

Category: outdoor_ Confidence Value: 0.01953125

Whoever eager to contribute to this framework, please note the github link for the develop branch is at https://github.com/Nabarun/CrmTest/tree/develop. Will really look forward for innovative enthusiast for contribution.

Life is not accumulation, it is about contribution.-Stephen Covey

Continued at http://nabaruns.blogspot.com/2016/04/salesforce-rendition-microsoft.html

Observer desgin pattern in Service Fabric Actor

2015-08-07T15:15:00.000-07:00

First of all I want to thank Service Fabric Team to help me overcome and understand the actor logic with clarity.

In this blog I aim to create an Observer design Pattern of a real world sample using Service Fabric as a backbone. Apart from reliable service creation Service fabric also allows an author to create actor instances. Details are at here. These actors can be imagines as entities with specific business logic. These have proven to be fast and highly reliable in gaming solutions where response is the prime factor.

Today we are going to discuss of a widely used design pattern - Observer pattern in Service Fabric Actor to accomplish a threat analyzer for our test framework.

Problem Statement

Every company big or small emphasizes on testing frameworks. Usually these are typical dashboard based web solution which have compute role scheduling and triggering tests, monitoring them while they execute and storing the result in database. Test Dashboard shows a report view of the stored data.

We have often come across statements like

1. How many tests are failing due to infra issue?

2. How many tests are failing due to real code issue?

3. What are the Azure Infra Issue type? etc

In my case we have a parser which goes and reads the database and do the bucketing based on different regex expressions. The challenge was the service was not reliable. This include service failures, connectivity issues etc. for our analyzer system, so we thought to try out Service Fabric and create a reliable service which will not only help us bucket the issue types but also will help us maintain the state for later analysis through dashboard. This was the time when we started going through service fabric actor model.

We heavily rely on Observer design pattern for our service. The intent was all the different issue types will be listening to a subject. The subject will at an interval of 30 min or so will query the database and push the list to all the observers. Based on issue type observers will add the object to itself.

When we tried to visualize this with Service Fabric, we had a picture which is described below:

In the above picture there is a stateless Fabric Actor which is our subject which will talk to the database at an interval of 30 min, and will push the objects to the observers. Observers will add the data based on its exception type.

Stateless Actor will push the data to the stateful actors, which will maintain the state as a list of issues. (Refer to the diagram below)

Later during analysis the user can directly call the actors and get their corresponding data. (Refer to the pic below)

Coding Time

Now let’s deep dive into how an actor works. As per my understanding author should keep these following things in mind. I will map the pointers to my example for better clarification

1. Create a Serializable class which the Actor interface can call.

In this case my class was

[DataContract]

public class ThreatState

{

[DataMember]

public string ExceptionMessage;

[DataMember]

public string ExceptionType;

[DataMember]

public string TestRunId;

[DataMember]

public string TestId;

}

2. Define a Subject Interface. Following OOD for implement to interface instead of concrete implementation, here is my ISubject interface

public interface ISubject : IActor

{

//Register Observer to Subject

Task<bool> RegisterObserver(IIssueObserver observer);

//Remove Observer from Subject

Task<bool> RemoveObserver(IIssueObserver observer);

//Set the Observable class state

Task<bool> SetState(ThreatState state);

}

3. Define a Observer Interface

public interface IIssueObserver: IActor

{

//Update the Observer object

Task<bool> Update(ThreatState t);

}

Note: In both these interfaces we have implemented IActor which makes them actor interface. One primary thing to keep in mind is all the actor interfaces should return Task results.

4. The Stateless Observable class will implement Actor and ISubject

public class ThreatObservable : Actor, ISubject

{…}

5. For Stateful Observers here is the only change we need to make from Actor point of view

[ActorService(Name="AzureIssue")]

class AzureInfraIssueObserver : Actor<ThreatState>, IIssueObserver

{…}

[ActorService(Name="MachineIssue")]

class MachineIssueObserver : Actor<ThreatState>, IIssueObserver

{…}

[ActorService(Name="ProductIssue")]

class ProductIssueObserver : Actor<ThreatState>, IIssueObserver

{…}

[ActorService(Name = "TestIssue")]

class TestIssueObserver : Actor<ThreatState>, IIssueObserver

{…}

6. Now in your Service Code during initiation add the different observers to the subject

public static void Main(string[] args)

{

try

{

using (FabricRuntime fabricRuntime = FabricRuntime.Create())

{

fabricRuntime.RegisterActor(typeof(ThreatObservable));

fabricRuntime.RegisterActor(typeof(AzureInfraIssueObserver));

fabricRuntime.RegisterActor(typeof(MachineIssueObserver));

fabricRuntime.RegisterActor(typeof(ProductIssueObserver));

fabricRuntime.RegisterActor(typeof(TestIssueObserver));

var machineIssue = new MachineIssueObserver();

var testIssue = new TestIssueObserver();

var codeIssue = new ProductIssueObserver();

var azureIssue = new AzureInfraIssueObserver();

ThreatObservable observable = new ThreatObservable();

var registerMachine = observable.RegisterObserver(machineIssue);

var registerTest = observable.RegisterObserver(testIssue);

var registerCode = observable.RegisterObserver(codeIssue);

var registerAzure = observable.RegisterObserver(azureIssue);

if (!registerMachine.Result || !registerAzure.Result || !registerTest.Result || !registerCode.Result)

{

Trace.WriteLine("Registeration failed");

}

Thread.Sleep(Timeout.Infinite);

}

catch (Exception e)

{

ActorEventSource.Current.ActorHostInitializationFailed(e);

throw;

}

7. This is what needs to be done from Service point of view.

8. I tested the service with the following client

public static void Main(string[] args)

{

var proxy = ActorProxy.Create<ISubject>(new ActorId("ThreatObservable"),new Uri("fabric:/ThreatObserverApplication/SubjectActorService"));

var lstThreats = GetListOfThreats();

var result1 = proxy.SetState(lstThreats[0]).Result;

var result2 = proxy.SetState(lstThreats[1]).Result;

var result3 = proxy.SetState(lstThreats[2]);

var result4 = proxy.SetState(lstThreats[3]);

//Assert all the results to true and verify

}

private static List<ThreatState> GetListOfThreats()

{

var listThreats = new List<ThreatState>();

listThreats.Add(new ThreatState() {ExceptionMessage = "Azure Storage Exception", ExceptionType = "Azure", TestRunId = "1234",TestId = "2345"});

listThreats.Add(new ThreatState() { ExceptionMessage = "Bad Machine Exception", ExceptionType = "Machine", TestRunId = "1234", TestId = "2345" });

listThreats.Add(new ThreatState() { ExceptionMessage = "Test Exception", ExceptionType = "Test", TestRunId = "1234", TestId = "2345" });

listThreats.Add(new ThreatState() { ExceptionMessage = "Code Exception", ExceptionType = "Code", TestRunId = "1234", TestId = "2345" });

return listThreats;

}

That is it!!

As I wanted to keep this short, I wanted to focus on major interactions of your code with service fabric actor model. If you want to know more I have pushed a sample app at https://github.com/Nabarun/ServiceFabricSample.git. Let me know if there are any feedback or suggestion.

Somebody clean my log container, it is growing exponentially

2015-07-29T23:12:00.002-07:00

Who has the time to keep a check of log container size. Nowadays most organization rely on cloud services for storage and maintenance. One of the most common artifact which gets uploaded in these storage account is Logs.

Let me give an example from my experience. I have seen teams using Azure blob storage for storing logs files, dumps, manifest files etc. During investigation these files are downloaded and rendered at user's box. Similarly alert traces go to Azure Table storage.

Now who has the time and effort to keep a check on these storage capacity. I once wrote a simple powershell command which helped me to delete all artifacts older than 60 days as of that day.

Cool! but after 2 months, here comes a mail from one team member saying he is not able to see the containers from CloudXplorer or Azure Explorer.

What happened?? I forgot to rerun my script again and the container grew exponentially again.

OK so now I added a task scheduler which will invoke my script on daily basis.

After another 2-3 months later I got to see a similar mail from another team member. What happened this time?

My machine got formatted because I wanted to deploy a new OS, and everything got deleted. Again I thought I need to write the script and enable the task scheduler.

But Wait!! What if my account stops working? What if again I format my machine??

It will not only be a re-work but also loss of time and effort.

In a midst of confusion one of my team member told me why don't you write a stateless azure fabric service for this and deploy it on one of the azure cluster and forget about it?

Cool! why did not I think about this....

So I started writing this service. Problem statement was delete azure container and azure table which are 60 days older, so Ideally the service will trigger itself once in a day and find out all the container and table older than 60 days and delete it.

Now comes designing of this app. There are 2 actions performed by this service DeleteAzure container and Delete Azure Table. Both depends on time limit of 60 days. The user can also delete a Azure Table or Azure Container by its name.

As per me this resembled a Strategy design pattern. Following is a simple conceptual diagram

Now I hope I will never ever get any issues for cloud storage clean :)

Does my organization need AzureServiceFabric?

2015-07-11T14:25:00.004-07:00

Azure Service Fabric is a technology to provide high reliable stateful/stateless/stateful persisted services. In this blog I want to relate few of the solution which helped me envisage answers for simple day to day problem at work. Microsoft is trying to promote this service currently for both private and public clouds. In Open networking summit of 2015 Azure chief Technology officer Mark Russinovich presented the roadmap of this service which encompasses future release support for Linux and Java containers.

Support for Java can open horizon of comparison between existing technologies prevailing in distributed computing field. Based on same Leader Election and Replication concept, technologies like Hadoop, HDFS, Zoo Keeper already dominate the space. Deep Diving into Azure Service Fabric pops out many similar concepts.

Similarity with existing technologies

1. Data locality in Hadoop = Hot data in Azure Service Fabric (ASA)

Hadoop is having this concept of Data Locality where it tries to co-locate the data with the compute nodes, so data access is faster. This is the reason for it good performance and which makes it at the heart of data computing.

Azure Service Fabric term the local data as hot data (ref: link). The intent of this is same high performance. These hot data features are provided by Reliable actors or Reliable services. Service Fabric replicates the data to other instances for data reliability.

This concept of hot data is also debatable as some say it is good to have this for higher performance and in some guidelines it is not preferred. Let me point to the link at Cloud foundry. Cloud Foundry avoids writing to local file system primarily because the disk might be taken. It always suggest you to go with Cold Data which can be stored in external storage. Again it depends on the architecture and design of replication implemented. Discussion on this might be another blog of comparison.

Organization which are new to big data analysis, which is thinking around the terms of hot and cold data might want to leverage Azure service Fabric for their requirement. Here Cold data is termed as that set of data that can be stored into external storage like database, or storage containers for future reference. Here going with Hadoop or small micro-service on ASA is a point of debate. It depends on what is the business need. If you are a c# guy your best bet is to try ASA.

2. Reliable Application development using Zookeeper in Hadoop or Reliable Service/Actors in ASA

Zookeeper is about building general distributed applications. It is also termed as Hadoop’ distributed coordination system. Zookeeper deals with some common terminologies in distributed computing world- Distributed Queues, Distributed Locks and leader election. At Yahoo, Zookeeper has been benchmarked at over 10000 operation per seconds.

Azure Service Fabric comes up with a similar feature which it calls as Reliable Service and Reliable Actors. Reliable Services/Actors offers reliable data structures like Reliable dictionaries and Reliable Queues. These are based on same replication concepts. One can create an application and define x number of partition, each partition may have n number of replicas which fulfil the purpose. Hot data in Service Fabric Cluster get replicated across other instances for high reliability.

Here Zookeeper is having an advantage over ServiceFabric and that is termed as HBase. Hbase which is based on ZooKeeper has proved its ability to geo replicate. Microsoft on other hand has still not mentioned anything about geo replication in Service Fabric.

Organization can really leverage the reliable data structures provided by Service Fabric. One simple implementation that I envisage is machine management where in organizations are having huge number of machines and bucketing them in different scenarios like health, test failures, suspicious activities can easily be done using reliable dictionaries. Reliable Queues on other hand can provide a good implementation of priority queues.

Conclusion

Still Service Fabric is in its infancy and this blog needs a lot of meat, but this is a start. Compared to Hadoop a huge opportunity lies in terms of feature development in ASA. Once Microsoft decides to release this in Java and Linux containers it will open more opportunities to explore. As reported this is also a back bone for some of upcoming technologies Microsoft plan to release like Cortana along with existing technologies like Skype, Intune and Azure SQL, so this will encourage developers to create real time, high reliable applications. The intent is to provide a one button solution for all the needs.

3 lines of Node.JS code to run a Web Server on ServiceFabric

2015-06-24T16:51:00.002-07:00

Since Service Fabric got publically released we saw many enthusiasts trying creating their apps with this. One of the proven strong case study is having a website hosted on Service Fabric.

If one goes through the documentation of Service Fabric he/she will find the tutorial on OWIN hosted webapp at https://azure.microsoft.com/en-us/documentation/articles/service-fabric-reliable-services-communication-webapi/. This is pretty descriptive and explains how quickly you can host a website on Service fabric.

I started this blog taking reference of this tutorial. The intent was to reduce the code and try to come up with a story around this. This was the time when I thought to see if we can figure out something with Node.JS. My early experience with Node.Js was pretty exciting wherein we created a Azure Mobile Service pinger to Service fabric Cluster. http://nabaruns.blogspot.com/2015/05/azure-mobile-service-service-fabric.html. The only challenge was how to call Node.JS script from C#.

With a little bit of research I found that this issue has been solved by Tomasz Janczuk. Acknowledging Tomasz let me now touch a little bit of Edge.JS.

Edge.JS GIT: https://github.com/tjanczuk/edge

Pre-requisite of this exercise is you have Node.JS and Service Fabric installed in your machine.

Step By Step : How to create a web host from Service Fabric

1. In my example I have created a Stateful Service Fabric Service. Stateful is just by choice as I plan to explore some more stuff with Node.JS and at a later stage have a stateful story around that. You can start with stateless or actor services too

2. Install the Nuget package of Edge.js from NPM (Tools -> Nuget Package Manager)

3. Now let's write our 3 line code for creating the webserver

public static void Main(string[] args)
{
            try
            {
                using (FabricRuntime fabricRuntime = FabricRuntime.Create())
                {
                    // This is the name of the ServiceType that is registered with FabricRuntime.
                    // This name must match the name defined in the ServiceManifest. If you change
                    // this name, please change the name of the ServiceType in the ServiceManifest.
                    fabricRuntime.RegisterServiceType("WebHostStaefulSFType", typeof(WebHostStaefulSF));
                    ServiceEventSource.Current.ServiceTypeRegistered(Process.GetCurrentProcess().Id, typeof(WebHostStaefulSF).Name);
                    CallWebServer("Thanks for creating this server").Wait();
                    Thread.Sleep(Timeout.Infinite);
                }
            }
            catch (Exception e)
            {
                ServiceEventSource.Current.ServiceHostInitializationFailed(e);
                throw;
            }
}
private async static Task CallWebServer(string greeting)
{
            const string jSFunction = "return function createServer (data, callback){ var http = require(\"http\"); var server = http.createServer(function(request, response) { response.write(\"Hello World!\"); response.end();});server.listen(8081);}";
            var func = Edge.Func(jSFunction);
            WriteLine(await func(greeting));
}

That's it. That is what we need to spawn a http web server using node.js

As per my understanding, from Service Fabric point of view it does not care how the web server is hosted. Here it just considers the Primary replica is ready or not. Once ready it executes the code.

It will be great to know other views on this.

Azure Mobile Service + Service Fabric = Easy Monitoring

2015-05-14T02:02:00.003-07:00

I want to thank all my readers who appreciated my previous blogs on Service Fabric. It is indeed a motivation which drives an tech-enthusiast like me. While driving home, I was thinking of my next blog, when all of a sudden I remembered of a POC which involved Service Fabric Rest API and Azure Mobile Service.

Intent of the POC

The intent of POC was to create a quick efficient way of monitoring Service Fabric cluster without writing tons of code and making things complex. The POC ask was from a client who wanted a feasible solution to monitor his Service Fabric Cluster. The ETA for this was decided to be 7 days, but believe me the first POC was ready by end of second day.

Why Azure Mobile Service?

Azure Mobile Service is an awesome tool for developers who want to write lightweight services, which runs continuously without incurring much effort and cost. I was aware of AMS as I wrote couple of scripts interacting with facebook api and Linkedin in past. (If interested, you can take a look into my netme blog). I thought it will be efficient if the same scripts are used for Service Fabric monitoring. As Service Fabric was exposing REST APIs , it was easy to perform a call from AMS and output the result into Azure Table Storage. A MVC app was created to read the Azure Table storage and show the output in a graphical chart.

Acknowledgement

I want to acknowledge Balaji who mentored me through out this process and made the simple blatant script turn out to be a generic utility.

Let's get our hands dirty

Before starting the problem statement was broken down to major 2 chunks
1. Do Rest API calls to Service Fabric
2. Store the output in table storage

Performing above 2 operations were pretty simple as AMS provide 2 Node.JS packages
a. azure
b. request

If you are new to Node.Js no issues, following script section will help one understand what is the usage for them

Pre-requisite

Ensure that the Http gateway is enabled in Service Fabric Cluster. This will be a section in Cluster Manifest

<Section Name="HttpGateway">
<Parameter Name="IsEnabled" Value="true"/>
</Section>

Also in your ServiceDefinition.csdef under Endpoints, we should have an endpoint for HttpGateway

<InputEndpoint name="HttpGatewayListenAddress" protocol="http" port="19007" />

In my case the endpoint port is 19007

Do Rest API calls to Service Fabric

Service Fabric provides multiple REST Apis which are documented at https://msdn.microsoft.com/library/azure/dn707692.aspx

For this POC we will discuss few of them

1. In order to start with the POC first we need to create a AMS service

2. Once the mobile service is created one should go to the Scheduler section and create a new job

3. Once the scheduler job is created one can click the AMS scheduler job and go to the Scheduler section, where we will get an area to write our script

4. Once the job is created one will see an empty function with the job name. Consider this to be the first entry point to the script. As my job name is smurfjob , I got a function name with the same name

function smurfjob() {
console.log("Starting My App Monitor");
console.log("This will monitor Node, Application, App Services , Sestem Services, Partition and Replica");
var dataTypes = [
"Node"
, "Application"
, "AppServices"
, "SystemServices"
, "Partition"
, "Replica"
];
dataTypes.forEach(getData);
}

As mentioned in the above log statements we are going to monitor Service Fabric Node, Application, Application Services, System Services, Partition and Replica

5. Our next module will be saying what is the input and what should be the output. For instance if input is AppServices, it should go to MyAppCurrTable storage table

function getData(dataType, index, array) {
if (dataType == "Node") {
getNodesData();
} else if (dataType == "Application") {
getApplicationData();
} else if (dataType == "SystemServices") {
fetchAndUpdateServicesDataForAppName("System");
} else if (dataType == "AppServices") {
getChildDataForParent("MyAppCurrTable", fetchAndUpdateServicesData);
} else if (dataType == "Partition") {
getChildDataForParent("MySvcCurrTable", fetchAndUpdatePartitionData);
} else if (dataType == "Replica") {
getChildDataForParent("MyPrtnCurrTable", fetchAndUpdateReplicaData);
}
}

6. Now let's have the modules for different datatypes

Nodes

Let me now take a single module and explain what we want

function getNodesData() {
//Import the request node.js and azure node.js module
var req = require('request');
var azure = require('azure');

// Url for fetching the Node Data
var nodeUrlToMonitor = "http://serviceFabricClusterName.cloudapp.net:19007/Nodes?api-version=1.0";

console.log("Url to Monitor" + urlToMonitor);

// fetch Node data
req.get({
uri:urlToMonitor,
timeout : 200000,
headers:{'content-type': 'application/x-www-form-urlencoded'},
},
function (error, response, body) {

//This is where we fetch the response body
if (!error && response.statusCode == 200) {

//Parse the json body
var items = JSON.parse(body);

//Now create the azure storage Table
var tableService = azure.createTableService("azure storage account name", "azure storage account key");

tableService.createTableIfNotExists(TableName, function (error) {
if (!error) {

//Parse the items in json and insert them in Azure Table Storage
tableService.insertEntity( parse relevant info)
}
else {
console.log("CreateTableIfNotExists for " + currentTableName + " returned error: " + error);
}
})
}
else {
console.log("ERROR: GET " + urlToMonitor + " returned error: " + error + "and statusCode " + response.statusCode);
}
})

}

Hope in the above section we got the actual intent of the idea mentioned earlier. Following is a snapshot of tabular structure of Application service

PartitionKey	RowKey	ServiceKind	Name	TypeName	ManifestVersion	HasPersistedState	HealthState	ServiceStatus	IsServiceGroup	AppName
System_ClusterManagerService	5VbGMVNM2EZ7NFYu81gAfO	Stateful	fabric:/System/ClusterManagerService	ClusterManagerServiceType	4.0.99.9490	TRUE	Ok	1	FALSE	System
System_FailoverManagerService	GSc38KxoFFXFThEJbP+vQ0	Stateful	fabric:/System/FailoverManagerService	FMServiceType	4.0.99.9490	TRUE	Ok	1	FALSE	System
System_ImageStoreService	YcPTmeiqsR0Vh33CpC+7nC	Stateful	fabric:/System/ImageStoreService	FileStoreServiceType	4.0.99.9490	TRUE	Ok	1	FALSE	System
System_InfrastructureService	uEvwax1amAIoal0LvoRL0y	Stateful	fabric:/System/InfrastructureService	InfrastructureServiceType	4.0.99.9490	FALSE	Ok	1	FALSE	System
System_NamingService	x+vT3v9YgNundefinedYvZJPgbTKvo	Stateful	fabric:/System/NamingService	NamingStoreService	4.0.99.9490	TRUE	Ok	1	FALSE	System
System_RepairManagerService	1bj5zeS2ReeKwbPVSQ1dOE	Stateful	fabric:/System/RepairManagerService	RepairManagerServiceType	4.0.99.9490	TRUE	Ok	1	FALSE	System

Now one can easily create a MVC app or use PowerBI to read this data , and create a simple dashboard

7. Once all this is done you can save the script and enable it. AMS will act as a background task manager script, and will keep uploading the results as required

Here is the full Azure Mobile Script

function smurfjob() {

console.log("Starting My App Monitor");

console.log("This will monitor Node, Application, App Services , Sestem Services, Partition and Replica");

var dataTypes = [

"Node"

, "Application"

, "AppServices"

, "SystemServices"

, "Partition"

, "Replica"

];

dataTypes.forEach(getData);

}

function getData(dataType, index, array) {

if (dataType == "Node") {

getNodesData();

} else if (dataType == "Application") {

getApplicationData();

} else if (dataType == "SystemServices") {

fetchAndUpdateServicesDataForAppName("System");

} else if (dataType == "AppServices") {

getChildDataForParent("MyAppCurrTable", fetchAndUpdateServicesData);

} else if (dataType == "Partition") {

getChildDataForParent("MySvcCurrTable", fetchAndUpdatePartitionData);

} else if (dataType == "Replica") {

getChildDataForParent("MyPrtnCurrTable", fetchAndUpdateReplicaData);

}

function getNodesData() {

var req = require('request');

var azure = require('azure');

var nodeUrlToMonitor = "http://servicefabricname.cloudapp.net:19007/Nodes?api-version=1.0";

var nodeHistoryTableName = "NodeHistoryTable";

var nodeCurrentTableName = "NodeCurrentTable";

var nodeEventsTableName = "NodeEventsTable";

var nodePrimaryKeyName = "Name";

fetchAndUpdate("Node", nodeUrlToMonitor, nodePrimaryKeyName, nodeCurrentTableName, nodeEventsTableName, nodeHistoryTableName, 15, null);

}

function getApplicationData() {

var req = require('request');

var azure = require('azure');

var appCurrentTableName = "MyAppCurrTable";

var appEventsTableName = "MyAppAppEventsTable";

var appHistoryTableName = "MyAppAppHistoryTable";

var applicationURL = "http://servicefabricname.cloudapp.net:19007/Applications?api-version=1.0";

var appPrimaryKeyName = "Id";

fetchAndUpdate("Application", applicationURL, appPrimaryKeyName, appCurrentTableName, appEventsTableName, appHistoryTableName, 15, null);

}

function getChildDataForParent(parentTableName, fetchAndUpdateChildData) {

var azure = require('azure');

var tableService = azure.createTableService("storageaccountname", "storageaccountkey");

var getMatchingEntities = azure.TableQuery

.select()

.from(parentTableName);

tableService.queryEntities(getMatchingEntities, function (error, entities) {

if (!error) {

entities.forEach(fetchAndUpdateChildData);

}

else {

console.log("ERROR: queryEntities in " + parentTableName + " returned error " + error);

}

});

}

function fetchAndUpdateServicesData(item, index, array) {

fetchAndUpdateServicesDataForAppName(item.PartitionKey);

}

function fetchAndUpdateServicesDataForAppName(appName) {

var svcCurrentTableName = "MySvcCurrTable";

var svcEventsTableName = "MyAppSvcEventsTable";

var svcHistoryTableName = "MyAppSvcHistoryTable";

var serviceURL = "http://servicefabricname.cloudapp.net:19007/Applications/" + appName + "/$/GetServices?api-version=1.0";

var svcPrimaryKeyName = "Id";

var additionalData = {"AppName": appName};

console.log("Fetching svc data for app " + appName);

fetchAndUpdate("Service", serviceURL, svcPrimaryKeyName, svcCurrentTableName, svcEventsTableName, svcHistoryTableName, 15, additionalData);

}

function fetchAndUpdatePartitionData(item, index, array) {

var svcName = item.PartitionKey.toString();

var appName = item.AppName;

fetchAndUpdatePartitionDataForServiceName(appName, svcName);

}

function fetchAndUpdatePartitionDataForServiceName(appName, svcName) {

var prtnCurrentTableName = "MyPrtnCurrTable";

var prtnEventsTableName = "MyAppPrtnEventsTable";

var prtnHistoryTableName = "MyAppPrtnHistoryTable";

var updatedSvcName = svcName.toString().replace('_', '/');

console.log("Fetching partition data for app " + appName + " and svc " + updatedSvcName);

var partitionURL = "http://servicefabricname.cloudapp.net:19007/Applications/" + appName + "/$/GetServices/" + updatedSvcName + "/$/GetPartitions?api-version=1.0";

var prtnPrimaryKeyName = "PartitionId";

var additionalData = {"AppName": appName, "ServiceName": updatedSvcName};

fetchAndUpdate("Partition", partitionURL, prtnPrimaryKeyName, prtnCurrentTableName, prtnEventsTableName, prtnHistoryTableName, 15, additionalData);

}

function fetchAndUpdateReplicaData(item, index, array) {

var svcName = item.ServiceName.toString().replace("_", "/");

fetchAndUpdateReplicaDataForPartitionId(item.AppName, svcName, item.PartitionKey, item.ServiceKind);

}

function fetchAndUpdateReplicaDataForPartitionId(appName, svcName, partitionId, serviceKind) {

var replCurrentTableName = "MyAppReplCurrTable";

var replEventsTableName = "MyAppReplEventsTable";

var replHistoryTableName = "MyAppReplHistoryTable";

var updatedSvcName = svcName.toString().replace('_', '/');

console.log("Fetching replica data for app " + appName + " and svc " + updatedSvcName + " and prtnId " + partitionId);

var replicaURL = "http://servicefabricname.cloudapp.net:19007/Applications/" + appName + "/$/GetServices/" + svcName + "/$/GetPartitions/" + partitionId + "/$/GetReplicas?api-version=1.0";

var replicaPrimaryKeyName = "NodeName";

if (serviceKind == "Stateless") {

replicaPrimaryKeyName = "InstanceId";

}

var additionalData = {"AppName": appName, "ServiceName": svcName, "PartitionId": partitionId};

fetchAndUpdate("Replica", replicaURL, replicaPrimaryKeyName, replCurrentTableName, replEventsTableName, replHistoryTableName, 15, additionalData);

}

function fetchAndUpdate(dataType, urlToMonitor, primaryKeyName, currentTableName, eventsTableName, historyTableName, frequency, additionalData)

{

var date = new Date();

var current_min = date.getMinutes();

var fUpdate = current_min % frequency;

var req = require('request');

var azure = require('azure');

console.log("fetchAndUpdate " + urlToMonitor);

// fetch data

req.get({

uri:urlToMonitor,

timeout : 200000,

headers:{'content-type': 'application/x-www-form-urlencoded'},

function (error, response, body) {

if (!error && response.statusCode == 200) {

var items = JSON.parse(body);

// console.log(items);

var tableService = azure.createTableService("storageaccountname", "storageaccountkey");

tableService.createTableIfNotExists(currentTableName, function (error) {

if (!error) {

// for each row

items.forEach(updateTables.bind(null, dataType, primaryKeyName, urlToMonitor, currentTableName, eventsTableName, historyTableName, fUpdate, additionalData));

// items.forEach(getHealth.bind(null, dataType, primaryKeyName, urlToMonitor, additionalData));

}

else {

console.log("CreateTableIfNotExists for " + currentTableName + " returned error: " + error);

}

})

}

else {

console.log("ERROR: GET " + urlToMonitor + " returned error: " + error + "and statusCode " + response.statusCode);

}

})

}

function getHealth(dataType, primaryKeyName, primaryKeyValueToUse, urlToMonitor, additionalData, item

// , index, array

) {

// Modify url to add data and GetHealth query

// console.log("getHealth " + dataType + " item " + item[primaryKeyName]);

var primaryKeyValue = item[primaryKeyName];

if (dataType == "Partition") {

primaryKeyValue = primaryKeyValueToUse;

}

var primaryKeyValue = item[primaryKeyName];

if (dataType == "Partition") {

var partitionInformation = item["PartitionInformation"];

primaryKeyValue = partitionInformation["Id"];

}

var healthUrl = urlToMonitor.replace("?api-version=1.0", "/" + primaryKeyValue +"/$/GetHealth?api-version=1.0");

var req = require('request');

req.get({

uri:healthUrl,

timeout : 200000,

headers:{'content-type': 'application/x-www-form-urlencoded'},

function (error, response, body) {

if (!error && response.statusCode == 200) {

var items = JSON.parse(body);

// console.log(items);

var entityName = primaryKeyValue;

if (dataType == "Service") {

entityName = primaryKeyValue.toString().replace('/', '_');

}

else if (dataType == "Replica") {

entityName = additionalData["PartitionId"] + "_" + primaryKeyValue;

}

items["HealthEvents"].forEach(updateHealthEventsTable.bind(null, dataType, entityName));

}

else {

console.log("ERROR: GET health for " + healthUrl + " returned error: " + error + "and statusCode " + response.statusCode);

}

})

}

function updateHealthEventsTable(dataType, entityName, item, index, array) {

var healthEventsTable = "MyAppHealthEventsTable";

var azure = require('azure');

var tableService = azure.createTableService("storageaccountname", "storageaccountkey");

tableService.createTableIfNotExists(healthEventsTable, function (error) {

if (!error) {

item["PartitionKey"] = entityName;

item["RowKey"] = item["SequenceNumber"];

item["DataType"] = dataType;

item["HealthState"] = mapEnum("HealthState", item["HealthState"]);

tableService.insertOrReplaceEntity(healthEventsTable, item, function(error) {

if (!error) {

console.log(item.PartitionKey + " Health event updated");

}

else {

console.log("ERROR while updating health event for " + item.PartitionKey + " : " + error);

}

})

}

else {

console.log("ERROR: CreateTableIfNotExists for " + healthEventsTable + " returned error: " + error);

}

})

}

function updateTables(dataType, primaryKeyName, urlToMonitor, currentTableName, eventsTableName, historyTableName, fUpdate, additionalData, item, index, array) {

var azure = require('azure');

var primaryKeyValue = item[primaryKeyName];

if (dataType == "Service") {

primaryKeyValue = item[primaryKeyName].toString().replace('/', '_');

}

else if (dataType == "Partition") {

var partitionInformation = item["PartitionInformation"];

primaryKeyValue = partitionInformation["Id"];

item["PartitionInformation"] = null;

}

else if (dataType == "Replica") {

primaryKeyValue = additionalData["PartitionId"] + "_" + item[primaryKeyName];

}

for (var prop in additionalData) {

item[prop] = additionalData[prop];

}

var getMatchingEntities = azure.TableQuery

.select()

.from(currentTableName).where('PartitionKey eq ?', primaryKeyValue);

var tableService = azure.createTableService("storageaccountname", "storageaccountkey");

tableService.queryEntities(getMatchingEntities, function (error, entities) {

if (!error) {

if (entities.length > 0) {

for (var prop in item) {

// console.log("item[" + prop + "] = " + item[prop] + " ; eProp " + entities[0][prop]);

if (prop != "id" && prop != "Id" && prop != "link" && prop != "updated" && prop != "etag" && prop != "PartitionKey" && prop != "RowKey" && prop != "Timestamp") {

// if different from currenttable, then update events table

if (dataType == "Application" && prop == "Parameters") {

item[prop] = JSON.stringify(item[prop]);

}

item[prop] = mapEnum(prop, item[prop]);

var iProp = (item[prop] == null) ? null : item[prop].toString();

var eProp = (entities[0][prop] == null) ? null : entities[0][prop].toString();

if (iProp != eProp) {

writeToEventsTable(eventsTableName, primaryKeyValue, prop, item[prop]);

entities[0][prop] = item[prop];

}

// update row of existing entity

item["PartitionKey"] = primaryKeyValue;

item["RowKey"] = entities[0].RowKey;

tableService.updateEntity(currentTableName, item, function (error) {

if (!error) {

// console.log(item.PartitionKey + " Entity updated");

}

else {

console.log("ERROR while updating entity " + item.PartitionKey + " : " + error);

}

});

}

else {

item["PartitionKey"] = primaryKeyValue;

item["RowKey"] = randomString(128);

for (var prop in item) {

item[prop] = mapEnum(prop, item[prop]);

if (dataType == "Application" && prop == "Parameters") {

var jsonValue = JSON.stringify(item[prop]);

console.log("App Parameters = " + jsonValue);

item[prop] = jsonValue;

}

tableService.insertEntity(currentTableName, item, function (error) {

if (!error) {

console.log(item.PartitionKey + " " + dataType + " Entity inserted in current table");

}

else {

console.log("Error while inserting entity " + item.PartitionKey + " in current table : " + error);

}

});

}

// Update history table if frequency is met

if (fUpdate == 0) {

tableService.createTableIfNotExists(historyTableName, function (error) {

if (!error) {

item["RowKey"] = randomString(128);

tableService.insertEntity(historyTableName, item, function (error) {

if (!error) {

// console.log(item.PartitionKey + " Entity inserted in history table");

}

else {

console.log("ERROR while inserting entity " + item.PartitionKey + " in history table : " + error);

}

});

}

});

}

else {

console.log("queryEntities returned error: " + error);

}

});

getHealth(dataType, primaryKeyName, primaryKeyValue, urlToMonitor, additionalData, item);

}

function mapEnum(prop, value) {

var retVal = value;

var NodeStatus = {

"0": "Invalid"

, "1": "Up"

, "2": "Down"

, "3": "Enabling"

, "4": "Disabling"

, "5": "Disabled"

}

var HealthState = {

"0": "Invalid"

, "1": "Ok"

, "2": "Warning"

, "3": "Error"

, "65535": "Unknown"

}

var ReplicaRole = {

"0": "Invalid"

, "1": "None"

, "2": "Primary"

, "3": "IdleSecondary"

, "4": "ActiveSecondary"

}

var ReplicaStatus = {

"0": "Invalid"

, "1": "InBuild"

, "2": "Standby"

, "3": "Ready"

, "4": "Down"

, "5": "Dropped"

}

var ServiceKind = {

"0": "Invalid"

, "1": "Stateless"

, "2": "Stateful"

}

switch (prop) {

case "NodeStatus":

retVal = NodeStatus[value];

break;

case "HealthState":

retVal = HealthState[value];

break;

case "ReplicaRole":

retVal = ReplicaRole[value];

break;

case "ReplicaStatus":

retVal = ReplicaStatus[value];

break;

case "ServiceKind":

retVal = ServiceKind[value];

break;

default:

retVal = value;

break;

}

return retVal;

}

function writeToEventsTable(eventsTableName, partitionKey, propertyName, propertyValue)

{

var azure = require('azure');

var tableService = azure.createTableService("storageaccountname", "storageaccountkey");

tableService.createTableIfNotExists(eventsTableName, function (error) {

if (!error) {

var rowKey = randomString(128);

var task = {

PartitionKey: partitionKey

, RowKey: rowKey

, NotifyStateName: propertyName

, NotifyStateValue: propertyValue

}

tableService.insertEntity(eventsTableName, task, function (error) {

if (!error) {

// console.log('Event table updated for ' + partitionKey + ' property ' + propertyName);

}

else {

console.log('ERROR while Event Table update: ' + error + ";pk = " + partitionKey + "; rk = " + rowKey + "; pname = " + propertyName + "; pval = " + propertyValue);

}

});

}

});

}

function randomString(bits){var chars,rand,i,ret

chars='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+'

ret=''

// in v8, Math.random() yields 32 pseudo-random bits (in spidermonkey it gives 53)

while(bits > 0){

rand=Math.floor(Math.random()*0x100000000) // 32-bit integer

// base 64 means 6 bits per character, so we use the top 30 bits from rand to give 30/6=5 characters.

for(i=26; i>0 && bits>0; i-=6, bits-=6) ret+=chars[0x3F & rand >>> i]}

return ret

}

Hope this blog will give readers confidence of creating a service fabric monitor with less cost and effort. Any suggestion or views are always welcome.

ServiceFabric + ASA = Crime Detection, Part2

2015-05-07T15:05:00.001-07:00

This blog is a continuation of http://nabaruns.blogspot.com/2015/05/servicefabric-asa-crime-detection-smurf.html

In this blog we want to highlight the PowerMap Analysis and the Database schema which we already discussed.

Thanks to one of my colleague, who quickly got the PowerMap skills, I am keeping this short by showing the analytics fetched by him.

The data is getting simulated while I am writing this blog. It is fetching the results from Database as follows

The latitude and longitude data are helping us to point it in PowerMap, thus giving a real time tracking analysis of the stolen car, hence proving the POC.

ServiceFabric + ASA = Crime Detection, Smurf Part2

2015-05-07T01:11:00.000-07:00

This blog captures a story of an idea which was in my mind from a long time, but did not prove to be feasible in past. With Service Fabric and Azure Stream Analytics this has now become possible for us to perceive the potential of technology to prevent crime.

Scenario

Here in this story we have tried to emulate a scenario wherein there are multiple traffic cameras capturing the vehicle number plates and sending it to ASA(Azure Stream Analytics) service. Suppose there is a vehicle that got stolen and has been reported just now. Law agencies need to add this vehicle in lookout table. Now the stolen vehicle happens to pass from Camera3. Camera3 which is continuously emitting data to Event Hub which in turn goes to ASA, captures the number plate data. ASA queries the input against the lookout table and filters the vehicle data as soon as it is scanned by the ASA query into the database. PowerMap sheet provides a analysis based on the database content and shows a pointer at Camera3 location, thus providing the law agencies a quick mechanism to track the vehicle

Problem Breakdown

Broadly for our POC we need to see these 2 paths

1. Simulate the traffic cam scenario which is sending continuous data to event hubs

2. Input is read by ASA and stolen vehicle details with the Camera details are sent to output DB

Technical details

Input

In order to have input. We created a Stateless Service Fabric Actor project. As Service Fabric Actor has a compute and a state , this is feasible to simulate a traffic camera.

Broadly we are looking for following data structures. StolenCarModel is a datastructure that is holding the Car details such as SerialId (Any identification, we are considering this as unique id, and this will help us to create the clustered index in our db), Vehicle Number(Captured from image), CameraSerial(Identification Id of Camera), CaptureTime(Timestamp when it was captured, which will help in tracking investigation)

public class StolenCarModel
{
public int SerialId { get; set; }

public string VehicleNumber { get; set; }

public string CameraSerial { get; set; }

public string CaptureTime { get; set; }
}

Another datastructure that needs to be maintained is CameraDetails , which will have the CameraID, Latitude and Longitude where Camera is positioned (This will help us to point in powermap), and Area which will have the name of the area

    public class CameraDetails
    {
        public string CamID { get; set; }

public string Latitude { get; set; }

public string Longitude { get; set; }

public string Area { get; set; }
}

Service Fabric Details

First and foremost is the interface Actor for our service

public interface IStolenCarStatelessActor : IActor
{

Task<bool> StartSendingCarDetails(List<StolenCarModel> stolenCarList);
}

Corresponding to the interface the Actor Class should have the implementation

public async Task<bool> StartSendingCarDetails(List<StolenCarModel> cameraDetails)
        {
            ActorEventSource.Current.ActorMessage(this, "Start Sending Data");
            EventHubLayer hub = new EventHubLayer();
            var sendDetails = await hub.SendEventsToEventHub(cameraDetails);
            return sendDetails;
        }

Smurf Details

We are using Smurf core libraries in order to send data to event Hubs. Please refer to my previous blog on Smurf Core api at http://nabaruns.blogspot.com/2015/03/smurfan-asa-based-tool-in-making-part-1.html

In order to populate the StolenCarDetails class, we are using this function

/// <summary>
        /// Get List of sample StolenCarModel details which can be sent later to the event hub
        /// </summary>
        /// <returns>Returns the list of stolen car model</returns>

private static List<StolenCarModel> GetMetricDataList(string camera)
{

Random rand = new Random();
var metricDataList = new List<StolenCarModel>();

Console.WriteLine();
Console.WriteLine("#####################{0}####################", camera);

            for (var loop = 0; loop < 5; loop++)
            {
                var date = DateTime.Now;
                var serialId = rand.Next(0, 50);
                var nowTime = date.Ticks.ToString();

                Console.Write("Camera:{0} SerialId:{1} TimeStamp:{2} Vehicle:{3}", camera, serialId, nowTime, "ADC01" + serialId);
                Console.WriteLine();

var info = new StolenCarModel() { CameraSerial = camera, SerialId = serialId, CaptureTime = nowTime, VehicleNumber = "ADC01" + serialId };

metricDataList.Add(info);
}

Console.WriteLine("##############################################");
Console.WriteLine();

return metricDataList;
}

In our POC at time t=0, we are assuming there are 5 events getting generated by each camera. We are currently simulating the camera details as mentioned in the marked line

This is giving us the following input

Now moving ahead we need to simulate multiple actors as cameras.

var camera1 = ActorProxy.Create<IStolenCarStatelessActor>(ActorId.NewId(), "fabric:/StolenCarStatelessActorApplication");
            var camera2 = ActorProxy.Create<IStolenCarStatelessActor>(ActorId.NewId(), "fabric:/StolenCarStatelessActorApplication");
         ...
            var camera10 = ActorProxy.Create<IStolenCarStatelessActor>(ActorId.NewId(), "fabric:/StolenCarStatelessActorApplication");

            int loop = 0;
            do
            {
                var tollboothList = GetMetricDataList("Camera1");
                var sendSuccess = camera1.StartSendingCarDetails(tollboothList);
                Console.WriteLine("Sending Camera1 Details {0}: {1}", camera1.GetActorId(), sendSuccess.Result);
                Thread.Sleep(5000);

                tollboothList = GetMetricDataList("Camera2");
                sendSuccess = camera2.StartSendingCarDetails(tollboothList);
                Console.WriteLine("Sending Camera2 details {0}: {1}", camera2.GetActorId(), sendSuccess.Result);
                Thread.Sleep(5000);

...

                tollboothList = GetMetricDataList("Camera10");
                sendSuccess = camera10.StartSendingCarDetails(tollboothList);
                Console.WriteLine("Sending Camera10 details {0}: {1}", camera10.GetActorId(), sendSuccess.Result);
                Thread.Sleep(5000);
                loop++;
            } while (loop < 10);

In above code we are creating 10 actor objects calling the Stolen Car service and then sending random data under them

Now having done the input part next is ASA query which will help us populate the data to db

ASA Query

In order to have a POC for the idea, our ASA query is very simple

SELECT SerialId, VehicleNumber, CameraSerial, CaptureTime FROM CallStream WHERE VehicleNumber='ADC0116';

In here we are saying that the stolen vehicle is ADC0116, if I get any data related to that , populate the data in my database.

SQL Database

In order to create a database which will keep a track of the stolen vehicles we need to create a table similar to the StolenCar Schema

Keeping it very simple I create the table as follows

Create table StolenCardTable (SerialId varchar(255), VehicleNumber varchar(255), CameraSerial varchar(255), CaptureTime varchar(255));

CREATE CLUSTERED INDEX StolenCardTable
ON StolenCardTable(SerialId ASC);
GO

Action Time

Till now I have all pieces in place to verify my app. Here is a screenshot which verifies the stateless service actor app deployed on my localhost cluster

Now after starting the client first point of verification is event hub dashboard and logs which should start receiving the events

Second verification will be ASA dashboard and logs

If till now you have seen everything is fine most probably you will get the data in database

16 ADC0116 Camera7 NULL
16 ADC0116 Camera9 NULL
16 ADC0116 Camera7 NULL
16 ADC0116 Camera10 NULL
16 ADC0116 Camera3 NULL
16 ADC0116 Camera2 NULL
16 ADC0116 Camera2 NULL
16 ADC0116 Camera9 NULL
16 ADC0116 Camera2 NULL

I have got this sample output in my db which says ADC0116 was first seen in Camera7 then 9 then again 7 and so on , eventually ending in 2. This can be now easily highlighted in PowerMap based on camera details and location which will be my next blog.

I think by now I am having a story to share with you all. The intent of this blog is to utilize the potent technologies and try to make this world a better place. Hope you will like it. Please feel free to comment your views or suggestions.

Continuation blog is at http://nabaruns.blogspot.com/2015/05/servicefabric-asa-crime-detection-part2.html

Why Service Fabric?

2015-05-05T08:42:00.002-07:00

This is a brief article which is not related to What is Service Fabric. The intent is to add more pointers to this as we explore Service Fabric.

Here in this article the focus is to study Why Service Fabric is needed. Following is a comparison study that I manage to collect talking to some Service Fabric devs and PMs. Having said that the intent of this blog is to open debate points.

Why we want to use Service Fabric as Hadoop Map Reduce

Suppose we have 1 TB of data size, and the data is divided in many sections. Considering one section , if there is a failover during the execution. The processing start again from the beginning of the section in case of Hadoop Map Reduce. If we implement map reduce in Service Fabric, as the states are saved this will not be an issue.

Traditional Map Reduce is having mapper, combiner and reducer. Data gets copied from mapper to combiner and then Reducer. In case of Service Fabric we save the data from getting copied multiple times (Copy happens from Mapper directly to Reducer)

Distributed Data Structures

Service Fabric is providing Distributed data structures like Distributed Queue and Distributed Dictionary. These in built data structure helps one to quickly develop stateful services on windows fabric. This also ensures that your service never loses data inspite of cluster wide failures.

Unlike Azure Redis cache , Azure Table Store, Azure Queue the state is kept locally in service instance. Advantages are all reads are local, all write incur minimum number of network.

	// `$ionicUser` is now registered
	$ionicAuth.login('basic', $scope.data).then(function(){
	$state.go('menu.home');
	});
	}, function(err) {

	var error_lookup = {
	'required_email': 'Missing email field',
	'required_password': 'Missing password field',
	'conflict_email': 'A user has already signed up with that email',
	'conflict_username': 'A user has already signed up with that username',
	'invalid_email': 'The email did not pass validation'
	}

	$scope.errorBox = error_lookup[err.details[0]];
	});