Petabyte

Attention not needed!

2022-08-01T09:13:00.004-07:00

Attention, Attention

For as long as we have been using Transformer models, there has been a question about how to put it in production because it needs a lot of computing power. Google took about a year from when BERT (Bidirectional Encoder Representations from Transformers) was introduced to being incorporated into their search algorithm. It is not just the advancement in understanding searches using BERT; they also needed new hardware.

“But it’s not just advancements in software that can make this possible: we needed new hardware too. Some of the models we can build with BERT are so complex that they push the limits of what we can do using traditional hardware, so for the first time, we’re using the latest Cloud TPUs to serve search results and get you more relevant information quickly. “(https://blog.google/products/search/search-language-understanding-bert/)

One of the complexities in BERT is the ‘self-attention’ modules, these are neural network layers that compute a compatibility value from the original input to the model. BERT’s initial architecture used multiple attention layers, which introduced complexities that are quadratic in space and time. But why use ‘self-attention’ if it is that complex? Self-attention yields in a more interpretable model. These attention layers exhibit behavior related to the syntactic and semantic structure of sentences [1] which we know are helpful when doing various language tasks.

Addressing Self-Attention Complexity

To address the inefficiency of self-attention, a few approaches have been used to reduce the quadratic complexity to linear via low-dimensional projection and row selection.

Here are some of the examples of the approaches to address the complexity of self-attention in transformer models:

Linformer - added linear projection matrices to compute the self-attention using smaller vector dimensions, reducing the computation to linear complexity with almost the same performance.[2]

Informer – instead of calculating with quadratic complexity using all the inputs to compute the attention values, select inputs that will contribute major attention [3].

Most of the strategies will make a selection on the attention input matrices and make sure that it does not significantly affect the transformer’s performance. Here are some of those strategies for further exploration.

Sparse Transformer - https://arxiv.org/pdf/1904.10509.pdf

Log Sparse Transformer - http://arxiv.org/abs/1907.00235

Longformer - http://arxiv.org/abs/2004.05150

NAACL 2022 – Paper

Two papers caught my attention in NAACL 2022 that might have potential upsides other than addressing the complexity of self-attention in Transformer models.

Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences

In this paper, they introduce Skeinformer, which accelerates self-attention and improves the selection process on the attention input matrices. It discusses how the strategies in Informer and Linformer from the perspective of matrix approximation, specifically by using a ‘sketching’ method.

What is sketching?

A common way to represent data is as a matrix. A matrix will have ‘n’ rows and ‘d’ columns, which can be thought of as a n x d matrix. This nxd matrix (let’s call this matrix ‘A’) is approximated with a more compact matrix r x e (let’s call this matrix ‘B’) or a product of a few smaller matrices, so that matrix ‘B’ preserves the properties of ‘A’ up to some approximation ratio. The ‘r’ dimension of matrix ‘B’ should be sufficiently small so that any downstream computations can be carried out using that smaller dimension, like in the case of self-attention in the transformer models.

This kind of approximation can be applied to any matrix computation to take advantage of the lower computational cost using a reduced dimension matrix.

Additional references to understand sketching

A brief survey on the literature about the topic - https://www.cs.utah.edu/~jeffp/teaching/cs7931-S15/cs7931.html

Here is a good introduction to sketching - http://arxiv.org/abs/1411.4357

In the paper, they have evaluated the sketch matrices to the original self-attention. The paper claims that the approximation tends to outperform the original. The paper speculates that good approximation can recover the main signals from the original self-attention.

FNet: Mixing Tokens with Fourier Transforms

In this paper, the whole self-attention layer of a Transformer model was replaced with an unparameterized (no parameters are learned) Fourier Transform. The new model achieves the same performance as the original model.

What is Fourier Transform?

The Fast Fourier Transform (FFT) is at the center of this paper. The Fourier transform is one of the most ubiquitous transformations in all mathematical physics and engineering. The FFT is useful for representing data, images, and solutions for partial differential equations. The FFT has been central in how we analyze and process data that we collect – pretty much all of our modern digital communication is built on the FFT (e.g., audio sequences, video images). Sending pictures, audio clips, or any data efficiently would not have been possible without the FFT.

The Fourier Transform is an approximation by using the sum of sines and cosines in increasing frequencies. The sine and cosine functions are very useful functions that can model various natural phenomena.

In the paper, the authors removed the attention layer in the transformer model and substituted an FFT. Substituting an FFT means that there are no learned parameters which means it will also save some cycles during backpropagation. The paper claims the FNET model almost performs the same as the BERT base models but trains 80% faster and matches the same accuracy of other transformer models in the Long-Range Arena benchmark (documents with longer sequence texts)

What Next?

Both Papers above introduce really good techniques that can be used beyond the transformer architecture models. Both sketching and FFT deserve a second look for applications. Here is a promising paper - (https://www.cs.purdue.edu/homes/spa/papers/feature20.pdf)

[1] Vaswani et al., “Attention Is All You Need.”

[2] Wang et al., “Linformer.”

[3] Zhou et al., “Informer.”

[4] Brunton et. al., “Data Driven Science & Engineering: Machine Learning, Dynamical Systems, and Control”

[5] Bracewell, “Computing with the Hartley Transform.”

58th Annual Meeting of ACL (Association for Computational Linguistics) 2020 July 5 - July 8

2020-07-25T13:59:00.006-07:00

Unlike the NAACL (North American Chapter of the Association for Computational Linguistics which is the regional arm of ACL) conference last year, held in Minneapolis, ACL 2020 becoming virtual was very convenient. There were no issues with parking spots, which allows everybody to watch more sessions than usual. One essential item to consider with virtual conferences is time zones. Presenters are around the globe, so make sure the presentation time is in local time. Here is my account of ACL 2020 and a quick framing of what I think the effect and impact of sessions in ACL 2020 would be.

ACL 2020 - Effect and Impact

BERT (Bidirectional Encoder Representations from Transformers) is not going away anytime soon.

Bidirectionality in deep neural networks gives better representations of words because it takes into consideration the left and the right context. The built-in bidirectionality in BERT shows that context helps in most NLP (Natural Language Processing) tasks. That is why BERT is being used and explored by NLP practitioners. BERT has been around for almost two years now and was formally submitted to NAACL in 2019. Here is the list of every session in ACL 2020 that explicitly included BERT in the name of the paper: MobileBERT, ExpBERT, DeeBERT, GAN-BERT, exBERT, BERTRAM, SentiBERT, CamemBERT, CluBERT, tBERT, TaBERT, SenseBERT, FastBERT, schuBERT and there are several more papers that did not explicitly included BERT in the name but used BERT. The general approach of the sessions that used BERT are the following:

1 .pre-train BERT on a new dataset

2. fine-tune BERT on a specific task

3. fine-tune BERT on a specific task then benchmark it using a standard dataset like GLUE,

4. include BERT in the system pipeline for classification or generating embeddings

5. exploring BERT thru ablation or adding external features for task performance improvement.

Effect and Impact:

Using deep neural models like BERT for NLP tasks is becoming a standard approach to getting a decent performance in the shortest amount of time. As an organization, we should be looking at how we would be able to include such models in our product platforms. The reason is that most of our products are knowledge-based solutions that rely heavily on NLP based tasks for improvement (e.g., Question Answering, Summarization, Text Classification).
The downside to using deep neural models for NLP tasks is that it is more like a trial and error approach. The ACL community is well aware of this dilemma, which is why the BERT bandwagon that we see in ACL is accompanied by studies into understanding deep neural models for NLP tasks, as discussed below.
The number of papers submitted using BERT was overwhelming and clearly showed its influence on the research that is happening for NLP.

Deep Neural Network NLP models need to be understood.

Deep neural network models have been pushing the state of the art performance for several NLP tasks. The tutorial for "Interpretability and Analysis in Neural NLP," as well as several papers, are looking into these deep neural networks.

"Showing your work doesn't always work"

"Contextual Embeddings: When are they worth it?"

"Understanding Attention for Text Classification"

Effect and Impact:

The ACL community is doing studies to understand deep neural NLP models thru probing, testing, or measuring mutual information. The interest in understanding neural network NLP models are giving rise to visualization tools like LSTMVis , Captum , AllenNLP Interpret as well as give us a toolbox of interpretability for deep neural network NLP models.
As of now, the studies surrounding the understanding of neural network NLP models give us practical insights on:
- how neural networks acquire syntactic knowledge
- how neural networks aid the scientific study of language
- creating better systems.
- accountability
- user's right to explanation
Here are some of the papers that explore deep neural NLP models

Question Answering is being explored using transformer models, data augmentation, distant supervision, and new approaches for retrieval.

With BERT achieving higher accuracy against SQuAD (Stanford Question Answering Dataset) thru the power of contextual embeddings, transformer models are being explored for Question Answering.

Here are some of the papers that used BERT based models for Question Answering in ACL 2020:

Effect and Impact:

We can see the impact of large pre-trained models (e.g. BERT) using a large corpus (e.g. Wikipedia) on Question Answering as it captures knowledge
Leveraging transformer models (e.g. BERT) in Question Answering can be combined with new unsupervised retrieval approaches:
- Unsupervised Alignment-based Iterative Evidence Retrieval for Multi-hop Question Answering
- Template-Base Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering

Augmenting training data and distant supervision can also improve on Question and Answering tasks:

Transformers models are good at getting answers, but some answers to questions can be found in different places like Knowledge graphs and sometimes visual images:

Conclusion

Deep Neural Networks (e.g. BERT) for NLP tasks are here to stay. We should look into understanding deep neural networks because it becomes less of a trial and error if we understand deep neural networks. If we use deep neural networks in our system, a better understanding of them can help us create better, accountable, and interpretable systems. Deep Neural Networks models impact the research direction of NLP tasks, for example, Question Answering is being explored with such models.

MLOPS: Emerging best practices for deploying ML Models in production

2020-06-24T15:51:00.001-07:00

It comes as no surprise that machine learning models are in production systems right now. How a machine learning algorithm got there is a little bit complicated story. So in the recently concluded MLOps (Machine Learning and IT Operations), that was held virtually from June 15th to 18th, one thing is clear, the proliferation of machine learning algorithms in production systems is a challenge to deploy. The theme of most of the talks in the conference is centered around taking software development practices and DevOps (software development and IT operations) and applying them to deploying machine learning algorithms.

The workshop gave meaningful insights into how an organization can adapt some of the platforms presented at the conference. If we want to leverage machine learning algorithms in software products, we need a platform to consistently develop, test, and deploy machine learning algorithms in production.

The workshop presented some of the platforms that can be used right now, like Domino https://www.dominodatalab.com/, MLFlow, KubeFlow. These platforms are mature enough to be used in production. I hope that we would not build our half-baked version of some of these platforms in our organization. We will be years behind in functionality, and it can be a bottleneck for getting machine learning in our products. Creating your MLOps tool can be uneconomical in the long run too. Besides adopting one of these platforms for your MLOps needs (or if we ever build one - which I hope not), there would also be work to ensure that product platforms can accommodate the delivery mechanism from these platforms.

Another issue that most of the sessions address using an MLOps platform to manage the process of production algorithm delivery is "model drift." A model's performance will eventually dip or drift because of the dated data used to train it. A model can encounter cyclical drift, which is seasonal (e.g., activities tied to the season), gradual drift the model meets new data, or a sudden drift because of model actions. The critical concept to mitigate model drift is to do continuous improvement or delivery, using AutoML techniques as well as architecture that allows multiple versions of the model in production. Reflecting on some of the models we have in production, a need to establish how to address model drift is necessary.

Highlights of sessions that I attended from the conference:

Walkthrough: Utterance Generator Fast Tracks Chatbot Training

The session is one of the bonus workshops that you need to get a ticket to reserve a seat. The presentation is about how the presenters trained a chatbot with utterances of the same meaning. At first, I was a bit confused about why you would train your chatbot with statements that mean the same thing, that would have been a boring chat. A chatbot is now synonymous with command-line parsing, which now makes sense to train with utterances of the same meaning. A client using the chatbot will now be able to submit phrases of the same meaning that have the chatbot take action on.

The utterances of the same meaning were found through agglomerative clustering using cosine distance as the metric. Once the utterances are clustered, a quick check of the quality and quantity of the utterances was performed to ensure that the phrases are reflective of real-user utterances. These clustered utterances were used to train a language model so that the model would be syntactically diverse. The language model is now used in the chatbot to generate quality paraphrases for the chatbot to consume.

Automating Production Level Machine Learning Operations on AWS

Speaker was not able to connect - turned into a Q and A with a quick impromptu presentation.

Simplify ML Pipeline Automation

A presentation about MLRun

The point was to present one of the most straightforward ML pipeline automation

https://github.com/mlrun/mlrun

DevOps for Machine Learning and other Half-Truths

As most of the sessions in the conference, the fundamental question is, "How to deploy a model ?"

We should follow DevOps best practices. Following DevOps best practices means there will be many parallels between SDLC and the emerging Machine Learning Life Cycle. The Machine Learning Life Cycle will have unique situations where a typical SDLC might not have dealt with before, like including data versioning when training models. Data is useless without algorithms, and models will not perform well without data. Model training requires a lot of resources and long compute cycles, which might not be familiar in a traditional SDLC.

Making a long story short, deploying ML is economically challenging, demanding the need for best practices. Some of the challenges are the following:

Model and Algorithms can be stuck in the lab - researchers might never deploy a model but will have to spend time and resources experimenting with the model.
Disconnected teams - along with a data scientist, organizations need a data engineer, a DevOps engineer, and a software engineer to bring models to production. Asking a data scientist to build infrastructure is just not reasonable. The first step of creating a model can be done on a laptop, but getting it in production is another story.
Technology mismatch - technology stack to use for deploying machine learning in production. This is one point that resonates deeply in our organization. We do have a defined stack and best practices in our current product platforms, but it is severely mismatched with the current technology that leverages machine learning. We need to consider adding Python services to our product platforms.

Smart Data Products – From Prototype to Production

The highlight of this session is the mistakes of taking an experiment notebook to a production software artifact. A notebook is a suitable proof of concept, but a lot of work is needed before it can be used in production.

Some of the grave mistakes that should be avoided:

Not incorporating data science artifacts as part of the CI/CD chain. Labeled and curated data is the necessary foundation of a machine learning model. This point is evident but not easily implemented. Our organization should consider looking into the best practices for this.
Assumptions that data will remain the same Hence the need for number 1. Variables change, and a model needs to be refreshed because of drift.

ML Monitoring ML: Scalable Monitoring of ML Models in Production

Well, what can go wrong in production once we successfully deploy a model in a product? There are a lot of things that can go wrong. Mistakes in production can be costly, so monitoring your model in production is an essential infrastructure that needs to be in place.

What do we need to monitor? Model metrics and unexpected changes.

Here is the tool that was introduced in this session:

https://github.com/anodot/MLWatcher

The philosophy behind this tool is that dashboards don't scale really well; that is why the tool is designed to detect anomalies at scale.

After all of that, I found a session

Building an AI Capability in the United States Army – Isaac Faber which I attended because of an organization as big as the United States Military, I was interested in how they are rolling AI capabilities in different platforms with high reliability in such a large scale. The United States Military is one huge enterprise that any organization can look upon for best practices.

So the first thing they did is launch an AI Task Force - which is the central organization, to direct the effort of rolling out AI capabilities in a much broader organization. Let us mimic this task force in our organization. The task force lets the military units dictate the AI initiative but intervene to develop AI frameworks at scale. The task force also reviews policies that impede the deployment of AI capabilities in the smaller units.

Another session that I found interesting was

Lessons Learned Building – NLP Systems in HealthCare

NLP in healthcare is a little bit like NLP in legal documents. The vocabulary and syntax are unique to the domain that out of the box solutions trained on general language corpora like news does not work. Sentence disambiguation and named entity recognition are more challenging in the healthcare and legal domains because of the characteristics of the field. The speaker enumerated the difficulties encountered for working in the domain. By applying state-of-the-art methods in reading comprehension, the speaker concluded that it is better to train NLP models in the healthcare domain with your proprietary data. The reuse of an NLP model doesn't work for the task the speaker was trying to solve.

This kind of resolve towards off-the-shelf NLP models are reflective of what we encounter when we use such models in the legal domain. So to effectively solve NLP tasks in such fields, the general approach was to use multiple sources for the same document taking into consideration both structured and unstructured data.

Natural Language Processing (NLP): A Key to Human Advancement

2020-05-13T13:40:00.001-07:00

Abstract

In this paper, we will explore why NLP (Natural Language Processing) techniques contribute to Human Advancement. We will revisit how the interpretation of a text, learning from writing, and interaction with NLP based technologies affect our lives, decision, and behaviors. We will also see how NLP affects our community and social structure.

1 Introduction

NLP technique is key to our human advancement. It is crucial now for interpreting a corpus of text. It will be critical as more sources of text are digitized. It is also the reason why we need to advocate digitization adaptation. NLP techniques are how we are going to sustain the information needs of our society. NLP is also a key technology in our learning. It is becoming a necessity for our human development as we shift our social interactions online. As we produce more text that can be analyzed, we can look into a person’s profile and information without actually revealing it. Profiling can be used to target and influence people. We should have strong guidelines for its proper application. In this paper, we explore some of the highlights on where NLP technology has brought us.

2 Interpreting Text

One goal of NLP (Natural Language Processing) is to interpret text so that we can use the information, meaning, and intention in the text. To interpret text means to explain why words do various things through the way they are interpreted. (Umberto, 1990).

2.1 Human Approach To Interpretation of Texts

We, as humans, interpreted text early on without using advanced NLP techniques to find meaning in the text. Early interpreters often do it for the benefit of others because decoding the written text is not accessible to everybody. A text is an open-ended universe where the interpreter can discover infinite interconnections (Umberto, 1990). The early interpreters interpret text often under the lens of philosophical thinking that gave rise to legal and social contracts. Different interpretations become foundations of ethics and belief systems. With this newfound ethics, we as humans started building social institutions that govern behavior. Now, social order is a foundation in any modern society.

2.2 Same text different meaning

The text we interpreted is unable to capture preexisting meaning. (Umberto, 1990) That is why different interpretations lead to a different meaning of the text. Does text have a limit in its interpretation? Does text only capture a single sense? Text, when interpreted separate from the concrete circumstance of its utterance or writing, could be taken in a different context (Umberto, 1990), therefore, loosing any referential power that we think we have but is still a text that can be analyzed.

2.3 Using NLP To Interpret

The way we interpret text now using NLP techniques is far removed from the philosophical underpinning of early interpreters of text. We interact with the text rather than deriving the text’s original intent. Using NLP techniques to interpret a text, we get entities, facts, sentiments, and different types of features. We routinely apply some of the states of the art approaches to different kinds of text to extract knowledge, meaning, and context. An example of this, we often use word embedding trained on news articles into different types of text. We use semiotics to find information in the text and often use the apparent elements of the text as significant to what the text means. We associate sentiments with the text based on the terms used in the text. But as we all know, different types of texts like literary text, news articles, social media texts often have different levels of intent and should, therefore, have different levels of interpretation. That is why most of the common NLP techniques do not necessarily apply to most bodies of text we analyze.

2.4 No more misinterpretation

Since we apply common NLP techniques to understand unstructured text. Does it mean that our analysis is more objective by using NLP techniques and not tainted by human bias? Human interpretation often “beats the text into a shape which will serve his own purpose” (Rorty, 1982). Are we able to identify the truth of what the text implied obscurely and understand what is just beneath the surface of the text? Our NLP techniques are based on mathematics, logic, science, and computer programming, which are still dominated by human rationalism. The words that we use today may not mean the same things when used in the future. The attempt to look for a final, unattainable meaning leads to the acceptance of a never-ending drift or sliding of meaning.(Umberto, 1990) Beware of misinterpretation.

2.5 NLP Interpretation

As we analyze more and more text, we should be careful of any misinterpretation because We also have more text to interpret, which can be prone to errors. Using NLP techniques to interpret text may not be easily accessible to everybody. We might discover new meanings for the same old body of text. Our interpretation might be the beginning of a new social construct that might guide new social behavior in the future. Our social behavior is seeing a text-based shift as we interact a lot more using social media. We can not help but extend our NLP techniques for analysis to this form of text. From this, we can instantly identify behaviors and tendencies almost immediately.

3 Digitization of Text

When we analyze text now, we always get it in digitized form. Even if we have the original text in printed, audio, or graphic form, advances in computer technology can easily transform it into its digitized version. We have greater access to any form of text and more advanced tools to interpret those texts. This same ease of using digitized text makes it easier for that text to be distant to its original context. The digitized text gave us a stable object on which we can perform our interpretation countless of times. A text could now be operated upon as a stable physical object felt as somehow distinct from the living, moving thought, and speech performing the hermeneutic operation. Hermeneutics refers to the resulting systematized interpretation of text.(Ong, 1995) The digitization of text allowed us to perform complex ways of interpreting text like word embeddings. The digitized text also transformed us to an information society and information of itself says nothing unless it is interpreted.(Ong, 1995)

3.1 Breakdown of digitized text

As we all know, computers work by turning any information to 1 and a 0 in order to operate on it. The original text is taken further away from its original context and form. This break down is necessary in order to perform interpretation using advanced computing power but also is highly prone to identify words with the text and only the text unless we supplement it with additional information.

3.2 Human digitization

Digitization means the treatment of data in terms of numerically distinct units. A digit today commonly means a numerical unit such as digitization employs—in computer programs 0 and 1.(Ong, 1995) Computers are human tools that are extensions of us. They are artificial but we continually put a little bit of ourselves in it. All the information stored away in the digitized form is concrete representations of human thought that we can analyze over and over again. We need to keep capturing text because eventually, it will give use of unresolved contradictions and stale information, and, outdated word embeddings. We can not totally put ourselves in something as artificial as the computer but we have something good enough in the data, text, and images that big social media companies capture. The data captured helps us discover human tendencies by analyzing text using NLP techniques in the context of social media postings and the immediate community. Could we also influence human tendencies this way?

3.3 Digitization Sustainability

If we are going to sustain our society that is deeply rooted in information nowadays, we must keep capturing a lot of digitized information like text, utterances from videos, discourses, speeches in order to supply new and relevant answers. Whatever happened to the idea of digitizing all books? (Jennifer, 2017) Digitization also makes all that information easily accessible to anyone in the world. Imagine having access to a rare book’s text to analyze? That would be impossible for virtually everybody without digitization. Imagine if newspapers did not adopt digitization? You would not be evaluating some of your NLP tasks using the WSJ corpus.(Paul and Baker, 1992) NLP practitioners should advocate for digitization because it would give us new sources of insights and help support the advancement of NLP techniques. Advances in NLP techniques would bring about more accurate information that our information society would consume. Accurate information leads to better decisions and outcomes.

4 NLP in Learning

Natural Language Processing (NLP) approaches help students in better understanding educational material and curriculum.(Alhawiti, 2014) NLP techniques are almost universal that the methods can be applied to different languages now. NLP techniques help us consume information regardless of language. The language barrier is being chipped away little by little in social and education settings by NLP techniques. With this, NLP will give us unprecedented access to information in a lot of areas like science, education and learning, books in other languages. Understanding new content will give us new perspectives and insights to us, as humans. The transfer of knowledge will abound and the context of our learning becomes universal.

4.1 Social Learning and NLP

One of our adaptations as humans is a relatively long childhood. (Remmel, 2008) We need a lot of time to develop our complex brains. Some of the areas that drive this long childhood are social intelligence and language. Language, in particular, is learned by being a child a lot faster because of the limited information processing capacity that forces a child to focus on constituent components of language and build from there. (Remmel, 2008) With this extended childhood, we learn about social relations and competence. Social complexity is a required ingredient in human cognitive evolution. (Remmel, 2008) Our learning is highly correlated with our social interactions. Our social interactions are dominated by language.

4.2 Social Learning through Social Media

The context of our social interactions is now shifting to an online setting. Our communications are readily captured text where we can perform advanced NLP techniques. We can use NLP techniques to analyze our social interactions online quickly. Pair NLP techniques with advanced AI (Artificial Intelligence) techniques, then artificial interactions that teach and influence behavior is possible. We can significantly improve communications, shape culture, and help accelerate learning through NLP based on our online interactions.

4.3 Platform of Influence

A lot of information flow through social media. Members of these social networks compete for attention and influence. (Romero et al., 2011) A lot of companies that are on these platforms develop social media strategies to reach target audiences. (Hanna et al., 2011). One of the most effective strategies that NLP offers is to extract valuable intelligence and tracking sentiment from Social Media. Sentiment analysis involves discerning subjective material and extracting various forms of attitudinal information: sentiment, opinion, mood, and emotion.(Sattikar and Kulkarni, 2012). Sentiment analysis helps us understand conversations and determine appropriate actions to influence either an individual or a community. Most of those individuals are young people. Social Media is where most of our young people are. Using Social Media is among the most common activity of today’s children and adolescents. Engaging in various forms of Social Media is a routine activity that research has shown to benefit children and adolescents by enhancing communication, social connection, and even technical skills. (O’Keeffe and Clarke-Pearson, 2011) Social media will play a big part in the social and emotional development of young people. Employing NLP techniques in social media is a critical technology that can bring advancement to our future generations if we so choose to use it carefully.

5 Quantitative Analysis, Revealing You

One of the interesting parts in NLP is Quantitative Text Analysis. Quantitative text analysis is a set of techniques stemming from the social sciences where either a human judge or a computer extracts semantic or grammatical relationships between words in order to find out the meaning or stylistic patterns of a casual personal text for the purpose of psychological profiling etc.(Sattikar and Kulkarni, 2012) There has been a lot of research into profiling authors personality and demographic traits as well. Revealing information about the author of the text just by analyzing the text is a task that is of growing importance for national security, criminal investigations, and market research.(Litvinova et al., 2016) As we communicate more online, the need to identify authorship is a growing need as well because we can assume an online identity through the use of assumed names making it hard to trust online communications. Applying quantitative analysis to our everyday communication online is now becoming commonplace on most social media platforms. They can use it to target us with advertising, recommendations, and influence us to adapt to certain buying behaviors.

6 Discussion

When we, as humans, interpreted text without using advance NLP techniques to find the meaning in the text, to using advanced analysis of our daily communications to profile conjecture information, indeed, NLP has come a long way to shape us. Our early interpretations of text formed the basis of our modern social contracts and laws. NLP techniques that get a sense of who you are like quantitatively analyzing your text in social media imply that NLP has a certain level of influence in our daily lives. We are a society that is hungry for information all because of NLP.

Digitization of text makes NLP much more critical now. Without NLP techniques, how can we possibly extract all that information from digitized text. We need to sustain our societies’ hunger for information coming from a different context and different languages. Using NLP techniques on a lot of digital text helps us in understanding and learning materials quicker.

Social media platforms that have most of our young people’s attention captured are being analyzed and influenced by using NLP techniques applied in those platforms. They might even begin aiding their cognitive and emotional development.

As more advanced techniques like quantitative analysis that can reveal information about us just by analyzing the text we have written, NLP is becoming a cornerstone in influencing and behavior targeting. If we can influence people on a scale that we have in social media platforms, then NLP techniques will be a crucial technology on how we shape our future society. NLP will be an essential technology either to our advancement or decline.

7 Conclusion

We should be careful and ethical on how we leverage NLP techniques. We can be impacting communities, demographics, and segments of people, mainly if used on a social media platform. Young people on social media are especially vulnerable with targeting strategies by using NLP techniques on their social media posting.

Interpreting and analyzing text should be done without our bias. We, as humans, are naturally biased. The text that we produce is an extension of our prejudices. NLP techniques that use bias data would be biased. We should be aware of this when applying NLP techniques.

There will be more digitized text in the future. We should rally behind efforts in digitizing text sources, for it will benefit us, and we need this to discover new insights and knowledge. NLP expertise will be in demand if there are more digitized text sources for us to explore.

Use the quantitative analysis for author profiling with care because it might not paint a whole picture.

There is more to NLP than just looking at text. It is one of the critical technologies that will work in our advancement, if we use it carefully. All of the following are possible because of where we are right now. Carry out conversations with computers, win in Jeopardy, understand a different language that we don’t speak or write, deduct logic from a series of sentences, influence young consumers, know your sentiment, and reveal you, using a piece of text that you produced.

References

Dr Khaled M Alhawiti. 2014. Natural language processing and its use in education. Computer Science Department, Faculty of Computers and Information technology, Tabuk University, Tabuk, Saudi Arabia.

Richard Hanna, Andrew Rohm, and Victoria L Crittenden. 2011. We’re all connected: The power of the social media ecosystem. Business horizons, 54(3):265–273.

HOWARD Jennifer. 2017. What happened to google’s effort to scan millions of university library books.

Tatiana Litvinova, Pavel Seredin, Olga Litvinova, and Olga Zagorovskaya. 2016. Profiling a set of personality traits of text author: what our words reveal about us. Research in Language, 14(4):409–422.

Gwenn Schurgin O’Keeffe and Kathleen ClarkePearson. 2011. The impact of social media on children, adolescents, and families. 127(4):800–804.

Walter J Ong. 1995. Hermeneutic forever: voice, text, digitization, and the” i”.

Douglas B Paul and Janet M Baker. 1992. The design for the wall street journal-based csr corpus. In Proceedings of the workshop on Speech and Natural Language, pages 357–362. Association for Computational Linguistics.

Ethan Remmel. 2008. The benefits of a long childhood. American Scientist, 96(3):250–252.

Daniel M Romero, Wojciech Galuba, Sitaram Asur, and Bernardo A Huberman. 2011. Influence and passivity in social media. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 18–33. Springer.

Richard Rorty. 1982. Consequences of pragmatism: Essays, 1972-1980. U of Minnesota Press.

AA Sattikar and RV Kulkarni. 2012. Natural language processing for content analysis in social networking. International Journal of Engineering Inventions, 1(4):6–9.

Eco Umberto. 1990. Interpretation and overinterpretation: World, history, texts. The Tanner Lectures on Human Values. Delivered at Clare Hall, Cambridge University March, 7:141–202.

Are you tired of waiting ? - AI can help!

2020-03-05T07:11:00.000-08:00

Are you tired of waiting?

Almost every day of your life will be spent waiting. You will be waiting in traffic. You will be waiting to buy or acquire a service. You will be waiting on the internet for your movie, music, or social media update. For the longest time, the human race has been inventing a lot of things for us to save time. But why are we still waiting?

We live in only one Earth, where we are bound within finite space and resources. More human beings are getting introduced to this home of ours. There are only enough resources that we can share. So, the future will bring us longer waiting times for us to share these resources because there will be more of us. There will be longer wait times because there are a lot more of us (humans) vying for the same resources. Roadways, Food, Water, Air, Internet, and Services (e.g., Healthcare, Legal, Social) will need to be efficiently used and scheduled. We have systems in place to access these resources. Our current policies are constrained and inefficient. Our overburdened systems are the reason why we see our waiting times increase for almost anything.

Not only Physical Resources - most importantly services

How long have you waited in your local coffee shop? And that is just coffee, which can be a little bit tolerable if we wait for it more than usual. Imagine if that service is a legal service like immigration or incarceration, well a delay of a year to a couple of years might be fine - but around a decade or more, say about 50 years? (https://www.washingtonpost.com/immigration/the-employment-green-card-backlog-tops-800000-most-of-them-indian-a-solution-is-elusive/2019/12/17/55def1da-072f-11ea-8292-c46ee8cb3dce_story.html) Do you think that is a reasonable wait time? That is just an example of wait times in the legal service industry. That is why it almost becomes a necessity to provide faster legal research using AI so that lawyers using our products are empowered to service more clients who are waiting.

What about other essential services that we need like healthcare, government, and social services. There is only a lot that an employee of these services can do.

AI Can Help!

Finding inefficiencies in resources and services management is one of the areas where AI/ML technology can be an advantage. Let AI/ML make predictions when a resource is needed or even ultimately decide on the next steps for faster access to resources and services. Assisting humans in determining to grant requests for resources and services is an excellent opportunity for AI/ML to thrive.

Example of Trends that AI/ML is helping provide faster access to resource and services:

Autonomous vehicles can drive much closer together at higher speeds, therefore, making use of roads much more efficiently. (https://www.sciencedaily.com/releases/2019/05/190519191641.htm)
Quick Check - lawyers can quickly check briefs instead of taking hours to file motions. It helps speed up a lawyer's time to service clients.
Support Chat Bots - you no longer need an actual person to help you with a specific product issue. Help speeds up to find a resolution to common product issues.

AI/ML becomes a necessity

The trend of competing for limited resources and services will only continue to be an issue in the future; that is why AI/ML becomes a necessity for us humans. Without AI/ML, it will just be more waiting time for us. We should be clamoring for AI/ML to be explored and studied like what Canada is doing. https://vectorinstitute.ai/

I hope that when in the future when we develop AI/ML in its fullest, that it might not conclude to solve the root cause of the problem of overburdened resources on Earth (like a science fiction movie) that it decides to get rid of us

AI Speeds Up Our Rendezvous With Complexity

2019-10-21T11:52:00.000-07:00

Intelligent systems are augmenting our ability to make decisions. This ability to make decisions is based on machine learning models or deep learning models that use large amounts of data that are non-parametric that fit into non-linear functions. These models are hard to comprehend and explain. We are plugging a lot of these models into business processes in every industry. Soon it will be hard to monitor and justify the root of a decision made because of these models.

Take, for example, a step in a process where a human decides. We can ask the human about the reason for the decision. There is also discretion on the decision based on the ethics of the one who decides. The decision might not be as consistent or efficient to maximize any benefit after that decision, but at least somebody can explain the details about the decision and understand the reason behind it. Now let's take the data behind past human choices and create a model using AI in an intelligent system. We optimize it, update it, and train it so that it will maximize any benefit after that decision. We lose the discretion of a human as we defer to an intelligent system to make that decision; we might be able to explain the decision-making process a little bit. There is always a part in there that we might not be able to explain because it might be sophisticated enough for us not be able to understand it as humans or the data behind it is just not humanly possible to understand. Imagine having an ensemble of these models in intelligent systems. The decision process will now be dependent on these models, and we might not be able to explain all the steps that happened for that decision.

AI in Intelligent Systems Will Increase The Rate of Our World's Complexity

Do you remember AlphaGo move 37? (https://www.wired.com/2016/03/two-moves-alphago-lee-sedol-redefined-future/) Even experts were unable to explain the move. Lee Sedol has to counter that move with a move that only an expert can make. That is what an Intelligent System can do, and we will be able to find the niches, mundane areas of a domain because we won't discover them with our human abilities. Take the practice of law, for example, citing other cases in a brief or complaint has been the norm. We want good law to be the foundation of a brief or complaint. Law is a complex landscape to navigate because of the increase in the complexity of our social structures. We have to create new laws to accommodate complex situations to bring about social order. Add AI in an intelligent system to the mix, and we will discover the niches of the law that are either right, inadequate or lacking. The citation graphs of lawyer's brief or complaint will increase in complexity as they try to win arguments using niches of law undiscoverable before intelligent systems. How about finance? An area that is highly regulated. Decisions can be hard to evaluate because of the complexity of a global economic landscape. What if an intelligent system in finance made a similar move like move 37 by AlphaGo? A strategic move that no expert in finance can explain, how would we even fathom the effects of such a move to the global economy?

Complexity Should Motivate Us to Explain AI Models in Intelligent Systems

The world we move in grows more complex every day (Entropy increases over time). Intelligent systems can simplify some of the processes through automated decision making, but to mimic human judgment, we need complex models. Complex models might be partially explainable. We need transparency and explainability from this AI and ML models deployed in intelligent systems so that we can manage the complexity that it brings to the domains where it is applied. As well as being explainable, we need to be able to trust an Intelligent System's judgment and how it will affect us. Explainability of the models in an intelligent system will aid us in learning new insightful things through all the complexity. Like what happened in AlphaGo, the effect of move 37 was the infamous move 78 by Lee Sedol. If an intelligent system's decision affects our daily lives, we should be able to appeal a wrong decision made by an intelligent system. We can only evaluate the wrong decision if we can explain the process of the decision.

Embrace Complexity

Ok, so now let's imagine we do not use AI in intelligent systems. We will slowly march towards complexity, whether we like it or not because our world grows increasingly complex. We have to adapt to our processes and make decisions from data from a complex environment. AI in intelligent systems will help us manage those complexities but at the same time, exposes us to those complexities early rather than later and at a much faster rate than we could manage as humans. We are finite humans in a world with finite resources. Discovering complexities early so that we can make better decisions with our resources is extremely helpful to our condition, but at the same time, we might not be ready to deal with such complexity because we are not capable of solving issues about it yet.

Deal With Complexity

We will need AI in intelligent systems to deal with the increasing complexity of our business process in different industries. To counter the effect of the complexity brought by AI to these processes, we should be able to explain AI in Intelligent Systems for the sake of managing complexity in such a system. The explainability of intelligent systems should be everyone's concern to build social confidence and navigate the complexity of such technology. At the same time, we should diligently evaluate the decisions made by such systems so that we can have a handle on its effects and how it might introduce new problems.

AI's Part In Our Evolution

2019-10-16T20:05:00.000-07:00

For the longest time, our ancestors have transferred a lot of knowledge through social learning. Social learning in a family or community setting builds the foundation for adaptability as an adult. An extended childhood with a lot of opportunities for social learning is one of our adaptation. (https://www.americanscientist.org/article/the-benefits-of-a-long-childhood). All humans learn socially and individually. We use whatever we learn to guide our behavior or decision to what we think would be beneficial to us.

The context of social learning is an ever-changing landscape. The setting of our social learning and the influences that we have from people are now shifting online. With just a few clicks of the button, you can learn from professors from ivy league universities, consult with a doctor, or even cook dinner with a world-class chef.

Gone are the days that you learn your trade from your father or immediate family. You no longer are bound by the tradition of your ancestry of what you can learn because at your fingertips are the expertise of millions of people. When was the last time you called your Mom or Aunt for how to bake a cookie? When was the last time when you have to call your Dad how to fix a faucet? I think where at a point that a child will be learning more online than what their parents be able to impart. With this shift in our social learning and our influences, there is also a shift in our behaviors. We can now see the emergence of some of these, like trolling, catfishing, more toxic comments, and reactions become amplified. We see this because the context of our social interactions is much less base on face to face conversations and teachings. Even traditional religious interactions are being affected by this shift. Just take for example the rise of screen preaching and wearable prayer articles (Screen Preaching - https://www.thegospelcoalition.org/article/why-i-object-to-screen-preaching/) (Click and pray - https://www.cnn.com/2019/10/16/world/vatican-click-to-pray-erosary-trnd/index.html)

A New Platform of Influence

Since our behaviors are a result of our learnings, most of our actions will be influenced by the most significant contributor to our learning. Those contributors will be the most dominant entities in our online world. Online gaming, social media, shopping are but a few entities that will influence human behavior. Evidence of this has been apparent in how a social media platform been used to win an election. For online shopping, you look at reviews of products. The most popular one with a 5-star rating usually becomes your choice. Now imagine this: the influence on your behavior does not come from another human with ethics. What if it comes from a pre-trained intelligent system?

Social Implications

AI has now a say in your social behavior and decision.

It is becoming hard for a human to decide without assistance from a smart system. There is now a lot of data that we need to analyze, Can a human be able to judge the quality of the data? It is becoming more normal to buy, retweet, repost, copy an online interaction with an intelligent system. Gone are the days where humans will make a rational choice base on their own analysis of the merit of that choice. No wonder following viral trends, no matter what the consequences are, becomes acceptable social behavior. (Fire Challenge - https://www.usatoday.com/story/life/parenting/2019/10/04/fire-challenge-michigan-boy-treated-burns-social-media-stunt/3862622002/) What was the last challenge you did?

Evolution

Whether we like it or not, AI is becoming part of our social and cultural evolution or maybe even our natural evolution by being part of our learning that influences our behavior. Is it something to be scared of, or something that we should accept and try to put a strong framework around it?

Ethics and Fairness in Artificial Intelligence

2019-10-14T09:52:00.003-07:00

We are at a time where leveraging Artificial Intelligence becomes commonplace. Every business organization is now looking into its data and building intelligent agents that predict, recommend, and even decide on organizational transactions. Automated intelligent systems are under scrutiny because of the recent biases uncovered from systems rolled out by some of the giants in tech right now. It is being proposed that such intelligent agents are regulated, developed, and implemented ethically. Equality, non-discrimination, accountability, and safety are being suggested as guiding principles when building such intelligent agents. Regulating automated intelligent systems is going to be huge in impact in every industry.

Need To Be Accountable

Most intelligent systems are an ensemble of Machine Learning Algorithms or Deep Learning Neural Networks. Which fundamentally is fitting a function over past data. Some are from a math formula developed through simulation or past data. Ultimately, we are deferring to complex math functions to make decisions for us. As we try to automate some of our tasks, we blindly defer some of our decisions to math functions. Using math to make decisions has been around for a long time. Ancient civilizations that have a firm understanding of math for decision making have us in awe of their ancient creations and inventions. Take, for example, the pyramids of ancient Egypt. Early mathematicians decided the shape and structure of the pyramids because of a good understanding of a function that precisely calculates a ratio (e.g., https://www.goldennumber.net/great-pyramid-giza-complex-golden-ratio/). Humans decided to go to the moon because we understand how the Earth and Moon interact because of math functions. (e.g., https://www.nasa.gov/pdf/377727main_Lunar_Math.pdf). Math functions are also being used for trading stocks in high frequency since the 1970s. Is math itself unethically biased? (e.g., https://www.k12.wa.us/sites/default/files/public/socialstudies/pubdocs/Math%20SDS%20ES%20Framework.pdf) (That is why the Seattle Public school system have come up with this Math Ethnic Studies Framework). An understanding of functions that give us beautiful aesthetic ratios does not necessarily mean that we have to enslave people to create the pyramids. Being able to decide quickly using a math function to do high-frequency trading does not necessarily that we should enrich ourselves using that strategy. If you can break encryption using a math function, would you read all private messages? The functions that we model using AI and Machine Learning are not inherently unethical nor biased; in fact, it is atypical. It is up to the people and industry that uses AI and ML to have a moral guide so as not to use AI and ML that might be a detriment to society. But most automated decisions do not involve a personal moral guide once we deploy our AI and ML functions in an intelligent system.

Need For A Framework on Ethics and Fairness in Algorithms

To make sure that our AI and ML algorithms are fair, ethical, and unbiased. We should have a framework to follow while developing it so that we make sure that fairness is built in from the training to deployment. There is ORCAA (http://www.oneilrisk.com/) that provide services that audit algorithms are fair and unbiased. ORCAA is a private group that serves as a consultant. There is a need for an open-source framework or guide that helps AI and ML developers on how to develop fair and unbiased algorithms.

Challenges of Ethics and Fairness in Algorithms

Most AI and ML algorithms are inherently discriminative. There are discriminative and generative algorithms. (https://medium.com/@mlengineer/generative-and-discriminative-models-af5637a66a3) Both types of algorithms use data to come up with a model. When we classify something, we need to compare it with previous examples to group it accordingly. Our classifier needs to be discriminative, to group similar examples. The underlying data would tell us how things would be grouped. We need to find the bias (the characteristics of the group) in the data to group things effectively. Finding the bias is counter indicative of being unbiased. So how should we make a model fair when the objective is to be discriminative? This is the dilemma that most AI and ML practitioners are facing. There have been explorations for removing bias (debiasing methods) - but these methods do affect performance. (https://www.aclweb.org/anthology/N19-1062.pdf)

Parting Thoughts

Every AI and ML practitioner should be aware of how easily an algorithm be biased and unfair when developing a model. Practitioners should also be aware of the pitfalls in unfairness and bias when a model is used in an intelligent system that makes automated decisions. We should treat unbiasedness and fairness in algorithms like we treat security when developing software.

Predicting Helpful Posts

2019-07-15T11:18:00.000-07:00

Original paper: https://www.aclweb.org/anthology/N19-1318
Here is a quick summary:

The research purpose is to identify helpful posts from discussion threads in forums, especially long-running discussions.
The approach is to model the relevance of each post concerning the original post and the novelty (not presented in the earlier posts of the discussion thread)of a post based on a windowed context.
To model, the 'relevance' the original post and the target post are encoded using an RNN (GRU). The encoded sequences are then element-wise multiplied.
As for modeling of the 'novelty,' the target post and the past K posts (where K is the number of past posts taken into context. A 'K' between 11 to 7 worked best for the Reddit dataset used in the experiment - performance stops improving after a certain number of posts taken into context) are also encoded using the same RNN text encoder. Once the 'K' posts are text encoded it is then fed thru another RNN encoder to encode the sequence of the posts before combining it with the target post encoding thru element-wise multiplication.

The output of both the 'relevance' and 'novelty' are then concatenated then fed thru a fully connected layer. (Please see network diagram)

This approach determines if a new post to a discussion is helpful because of the two factors considered: 1. relevance of the new post to the discussion 2. the novelty of the information introduced to the discussion.
The approach outperforms current methods of sequence classification because of this approach.

Abusive Language Detection

2019-07-15T11:14:00.000-07:00

Original paper - https://www.aclweb.org/anthology/N19-1221
Here is a quick summary:

In the paper submitted by Facebook AI, London to the recent NAACL (North American Chapter of the Association for Computational Linguistics) conference held in Minneapolis, they presented a novel approach using Graph Convolutional Networks to outperform some the best ways to detect abusive language on the internet.
The approach made use of a heterogeneous graph that contains an authors community network and tweets. The graph is then used to predict the class and generate an embedding. In the paper's experiments, the researchers used embeddings from node2vec (sample implementation here https://snap.stanford.edu/node2vec/) and a 2-layer Graph Convolutional Network.

The Graph Convolutional Network that represents the author's profiles and tweets were used to predict the author's tweet into three classes using a softmax layer as the output layer of the network. To extract the embedding from the Graph Convolutional Network, the researchers set the size of the hidden units to 200 (to be comparable to the node2vec method) then processed each graph input using a cross-entropy loss. The representation for each graph input is the first layer (the layer before the output layer) of the network (without activation).

The classification performance of the GCN (Graph Convolutional Network) somewhat improved the current best method for detecting the abusive language in two of the three classes ((based on F1 Score)) it is predicting.

Along with the node2vec and GCN generated embedding the authors tried to combine it with the best Logistic Regression (LR) classifier. The performance of the LR+node2vec (the researchers labeled it in the paper as LR + EXTD (Extended Graph - for the heterogeneous graph used)) was a little bit better in two of the three classes (based on F1 Score) it is predicting.
The performance of LR+GCN (embedding) outperforms the LR in three of the three classes (based on F1 Score).

Overall, an authors community network combined with the author's tweets helps to detect abusive language on the internet. The direction the researchers took makes sense as people with the same views form a community. What if the researchers could have derived echo chamber measurements ? or tried some feature related to echo chamber measurements.

Fakes News and A New Life Philosophy

2018-12-19T12:35:00.001-08:00

When I worked on Reuters Tracer for a little bit, one of the hot item topics during that time is detecting Fake News. Fake News is a real issue for people wanting reliable information and data. People who want reliable information and data are often decision makers, who need to take action.

I tried to look into what is "Fake News" which lead me to a new life philosophy. This new philosophy - is a moral imperative to check every information we consume before we believe because beliefs are the basis of our action. In this day and age that everyone is streamed with information all day thru social media, adoption of this philosophy becomes a necessity.

Why does anyone want to create "Fake News"?

The short answer to this is to win people over. "Fake News" mostly appeal to the target audience emotions. You can't win people over with logic, facts, and data. Have you ever been riled up to work harder when presented with charts and graphs of our earnings?

Majority of "Fake News" is communicated in order to evoke two major emotions FEAR or HOPE.

Sample Fear: "Muslim Refugee Kills Texas County Deputy"

Sample Hope: "Homeless man"

Evoking emotions from people is a great way to lead people to believe and act for your cause. Engaging this way for your cause results in highly devoted and dedicated people. This strategy is employed by some terrorist organizations. Why would a person act for an organization's cause that will lead to certain death? It is not logical, not based on facts. We can also see this from new technology startups. Startup employees who are emotionally involved with the company's cause that they would almost work for nothing. In short, is this how you mobilize people to act.

Knowing this, it is really important to detect "Fake News", convince people that it is "Fake" and ultimately have people not believe "Fake News". We need to prevent people from acting on false information. It also does not help that "Fake News" is a growing industry (The fake news machine: Inside a town gearing up for 2020). We better get a handle on it soon.

So we come to this question: "Would we be able to detect Fake News?",

"Once detected, How would we convince people not to believe "Fake News?

Would we be able to detect Fake News?

In order to detect "Fake News" we need evidence. Where would we get evidence?

- Search Engines

Search engines like Google are known to have bias results. It is based on what they wanted you to see (for commercial gain - Google Products).

- Social Media

Can we rely on the number of times that people share a news story? Can sharing indicate that people believe a story to be true? Does that mean a lot more people looked at evidence about the story? Or is this a good indication to look for more evidence about the shared story?

- Reliable News Sources

Organizations whose main business is to gather the truth about a news story. CNN? WP? NYT? - Do they have their own biases?

Once we have collected evidence. We know for a fact that people would still believe "Fake News" even though the evidence suggests otherwise - why?

- They are not compelled to look at the evidence

- Not motivated to find out the truth

- Established belief systems

So even if we have evidence people would still believe "Fake News" - why not approach it from the news story and the people who read it? Collect data about the news then collect data from people who read the fake news.

But what kind of data should we collect from the people who read "Fake News"?

and why collect data about the people reading "Fake News"?

"Fake News" is usually targeting an audience with an established belief system. The fake news reinforces this belief, with reinforcement, comes action.

Data collected about our activities online might be objective and blind to our established belief systems. A little bit of analysis will show established activities that we do online. What we do paints what we believe, so once we have that picture somebody can reinforce it or have us question our own belief. Now, this makes the business of collecting user online data lucrative.

Once we have a picture of a persons belief system we need to re-align those beliefs that are based on facts and evidence. but how?

I guess going down this path lead me to more questions. Exploring "Fake News" lead me to believe that it is everyone's moral imperative to check evidence before we believe because our beliefs are the basis of our action. Emotion plays an important role to influence people to act not facts and data. Even if we have enough evidence to discredit a fake news story it should be presented to counter the emotional effect of the "Fake News" in order to neutralize action based on the "Fake News".

Now go check the facts on what you just read.

Graph Algorithms - Strongly Connected Components in Spark 2

2017-08-12T14:20:00.000-07:00

Ever since I generated doc2vec (Word Embeddings) for our documents, we found interesting things by doing computations and comparisons to these vectors. For example, we try to find similar documents using cosine similarity and other similarity measures. These representations of the document give us the flexibility of doing a lot of stuff. We tried using the vectors in an ANNOY index to find near neighbors for a document quickly.

Now I am exploring these same vectors in finding documents that are repeatedly written and discusses the same topic. If I want to find these documents I figured that these documents would be closely similar. Documents that address the same topic will probably have a set of standard vocabulary. What if I want to find the most influential document among these related documents? To do that we need to define the connections between these documents. When we say "connections," I can't help but think of a Graph (or Network). Another approach is to use a clustering algorithm to group together related documents base on the doc2vec vectors and other features. Some of the unsupervised machine learning algorithms for clustering will need an arbitrary number of cluster (a good guess of how many clusters you might want to find) as a parameter. The unsupervised machine learning algorithm feels somewhat a trial and error until the somebody figures out the right number of clusters that groups together the documents as expected.

GraphFrames

So enter GraphFrames (GraphFrames - Databricks), a spark library that supports general graph processing. This library was introduced March 2016 - more info here (GraphFrames). The GraphFrames library promotes several graph algorithms. There are several algorithms that I wanted to try against the document vectors that we have. One of the algorithms in the library is PageRank (PageRank). You mean I can use Google's search algorithm just like that. Well, the algorithms in the library are only tools, the real clincher is how would we define the graph that will help us search for what we are trying to find. In the context of what I was doing looking for documents that talk about the same topic, PageRank would just turn up a lot of similar documents on the top but will not define a grouping of documents that discuss the same topic. So I tried Strongly Connected Components (Strongly Connected Components), this algorithm will find the subgraphs of vertices that have a connection going both directions. Here is a quick discussion on Wikipedia. SCC is what exactly I need. So we just need to define the vertices and the edges of the graph we are going to use.

Since we already have the vectors for each document, I ran a spark job that will calculate all the cosine similarity between two documents as long as they are part of search result coming from Elasticsearch if I do a More Like This query. This approach saved me from doing a cartesian join of about 30 million documents and eliminated the need for calculating similarity for un-related documents. So with the vertices being each document and the edges are cosine similarity I ran the Strongly Connected Components algorithm. It was quick. I thought it would take a day for 2 million vertices and 140 million edges but to my surprise, it only ran in less than 20 minutes. I initially thought that I might have missed a step or did something wrong. After verifying, I indeed have found groups of documents that are strongly connected. Not only that I discovered that the library provided the number of degrees for each vertex, so in a group, I can find the document that has the most number of incoming edges.

Conclusion

If you find that unsupervised clustering algorithms might be too loose of an approach to find clusters, I recommend using graphs as long as you define the proper graph about your problem context. Strongly Connected Components algorithms will find the connections easily.

Word Embeddings - Vectors that represents words

2017-06-06T11:56:00.002-07:00

Ever since I started being a part of the R & D in our company. I have been dealing with understanding neural networks that generate representations of words and documents. Representing words and document as vectors allow us to carry out Natural Language Processing Tasks mathematically.An example, we can see how similar two documents are (using cosine similarity), a quick analogy (vector operations), and ranking documents.

But how do you produce the vectors that will represent the words?
Well, there are many ways. There are traditional NLP approaches that can still work very well like Matrix Factorization (LDA, GloVe) and newer methods many of which uses neural networks (Word2Vec). I have been producing document vectors using the gensim doc2vec (which uses Word2Vec). I have been using the hierarchical softmax skip-gram model (Word2Vec.py Neural network). When I was reading this part of the code, I thought that if this is a neural network (shallow - not a deep learning model) won't I be able to use Tensorflow to generate my embeddings. The creators of Tensorflow are way ahead of me on that.

Background on Word2Vec (before we try doing embeddings in Tensorflow)
Although Word2Vec is using a neural network, we don't use the neural network for the trained task, but the goal is to learn the weights of the hidden layer of the neural network. We are going to save these weights, and this is the "vectors" that we are trying to learn.

There are several ways to train the neural network and several ways the input layer is represented. For this blog, let us stick with the skip-gram model because I am familiar with it.

But there is an issue with learning the "word vectors" just using a standard neural network. The neural network being used is predicting the next word given a window of words. Predicting the next word is just like predicting a class, but in this context, each class is a word. Doing a multi-class classifier using a standard neural network would require having the output layer have the same number as the classes you are predicting. Training with a large body of text with a lot of different unique words can get out of hand quickly using a standard neural network to generate the word vectors..

Softmax may not entirely cut it for a large vocabulary size
Here is a quick jsfiddle why a softmax will be bad for multiclass classification - (look at the "for loop" - it is a summation of all the elements of the vector - Softmax in javascript)

To deal with this issue, the neural network used to generate word vectors uses a technique called noise contrastive estimation. The idea behind NCE (noise contrastive estimation) is to convert the multi-class classification problem into binary classification.

Here is a sample of how embeddings can be generated in Tensorflow that I wrote following the sample in Learning Tensorflow. The code in here uses a skip-gram model - Tensorflow embeddings

The code sample is base on Vector representations of words
and Learning Tensforflow (Early Release)

References:
NCE Loss
Skip-gram Word2Vec Tutorial
Learning Word Embeddings efficiently using NCE

RE WORK - Boston Deep Learning Summit

2017-06-05T09:30:00.000-07:00

I have recently attended the Deep Learning Summit in Boston. The event was organized by RE WORK. RE WORK was founded in London. The team is all women. The mission of the RE WORK team is to encourage conversations around entrepreneurship, technology, and science to shape the future.

This is a quick recount of the event from my perspective.

First of all, I have never been to Boston. The public transportation that I took from the airport to the place of the conference was really easy to navigate (In short, I did not get lost). This is probably a result of the effort put in by the local government to make Boston a premier conference venue. Traffic congestion is another story.

Schedule of Talks

The conference schedule is packed. The speakers are researchers from some of the top tech companies. Facebook, Google, Amazon, Ebay, and Spotify are all represented. I was excited about two topics in the schedule. Here are some of the papers presented. The papers I chose below are some of the research which I thought was significant to some of the efforts we are doing for our business partners:

Autonomous Understanding of Stories and Videos ([1506.06726] Skip-Thought Vectors , MovieQA ) - presented by Sanja Fidler, assistant professor from the University of Toronto. One of the founders of the newly commissioned AI institute by the Canadian government (Vector Institute for Artificial Intelligence ).

The end product of this research is the MovieQA dataset which contains several sources of information – video clips, subtitles, scripts, plots, and DVS. One of the representations that were used to build the MovieQA data set is SkipThoughts. SkipThoughts uses a recurrent neural network to capture the underlying sentence semantics. The SkipThought model [2] was inspired by word vector learning using the skip-gram model. Instead of just using words to predicting the context around it, SkipThought encodes sentences to predict sentences around it. The model is based on an encoder-decoder architecture. The encoder maps the English sentence into a vector. The decoder then conditions on the encoded vector to generate a translation for the source English sentence.

Domain Separation Networks ([1608.06019] Domain Separation Networks, models/domain_adaptation at master · tensorflow/models · GitHub )- presented by Dilip Krishnan, a Research Scientist at Google.

Domain Separation Networks lets you adapt your model to a target dataset which you did not train on. Domain adaptation's goal is to train on a "source" dataset then apply the model to a "target" data set generated from a different distribution [3]. The motivations behind domain adaptation are the following: 1. Create high-quality datasets for training is costly and time-consuming. 2. Certain annotation tasks are impossible to scale. Domain adaptation allows you to train on synthetic data then apply the model to the real domain.

References

[1] Distributed representations of words and phrases and their Compositionality, https://arxiv.org/abs/1310.4546
[2] Skip-Thought Vectors, http://arxiv.org/abs/1506.06726

[3] Domain Separation Networks, https://arxiv.org/abs/1608.06019

Gensim Doc2Vec on Spark - a quest to get the right Vector

2017-03-31T19:59:00.001-07:00

Ever since I joined the R & D group we have been doing a lot of cool things, like trying IBM Watson (see previous blog entry). Now we are doing a lot of Natural language processing. We wanted to compare the similarity of two documents. There is this excellent project Gensim (doc2vec) that easily allow to you to translate large blocks of text to a fixed length feature vector to make comparisons. Here is the link to the original paper from some people from Google that explains the approach. In essence, they wanted to find a representation that will overcome the weaknesses of the bag of words model.

The doc2vec approach proves to be a reliable approach to comparing similarities of documents because it takes into consideration the semantics and the order of the word in context. So with that, we wanted to use it for a corpus of 26 million documents. Calculating the doc2vec for 26 million documents is not a small task, so we need to process it in Spark. The problem is that there is no implementation of Gensim for Spark. So I tried the canonical version of gensim in pyspark.

So using a pre-trained model we needed to infer the vector of all the documents in our corpus. Using pyspark, I passed in the models using the --files option then inferred the vector in the mapping function. After all of that I got some vectors, BUT when I tested the vectors, they are useless because comparing similar documents the vectors are orthogonal. (Not related similar at all using cosine similarity) Tried a few things but nothing is changing those vectors to closer for related documents. I have been attempting reading thru the gensim code on how the vectors are generated to make sense of what is happening. I also have looked at several projects that used gensim in spark https://github.com/yiransheng/gensim-doc2vec-spark and http://deepdist.com/. Nothing worked, as a workaround getting inspiration on how deepdist uses flask to distribute the training. I created a standalone Flask service that calculates the doc2vec and calls it from spark. Calling as service from Spark works BUT at the rate of how the doc2vec is being calculated my spark job is set to finish all 26 million in 2.7 days - which is a better choice that generating all the doc2vec for all 26 million locally using several threads which might take up to 30 days. Still stuck on this problem - I might debug a spark process to see what is happening to the weights in gensim.

Spark DataFrame - Array[ByteBuffer] - IllegalAurmentException

2016-12-22T08:44:00.001-08:00

IllegalArgumentException - ByteBuffer - Spark DataFrame

I was processing a several million documents (~ 20 million) in which we need to extract the NLP features using NLP4J, OpenNLP, and WordNet. The combination of the three NL features blows up each record to 11 times its original size. We are using all three because we do not know yet what feature sets will be helpful to us.
The original dataset is in parquet files in HDFS (16 partitions). I thought that was convenient just use withColumn and pass a UDF (User Defined Function) on the column where it needs those features. withColumn adds the calculated column back to the DataFrame. So I created the spark job (I am on Spark 1.5.2-cdh5.5.2)for the above, and things started to get nasty. I am blowing up the ByteBuffer array on the in-memory columnar storage. This is the exception that I am getting. There seems to be no reference in my code in this stack trace.

       

       java.lang.IllegalArgumentException
 at java.nio.ByteBuffer.allocate(ByteBuffer.java:334)
 at org.apache.spark.sql.columnar.ColumnBuilder$.ensureFreeSpace(ColumnBuilder.scala:142)
 at org.apache.spark.sql.columnar.BasicColumnBuilder.appendFrom(ColumnBuilder.scala:72)
 at org.apache.spark.sql.columnar.NativeColumnBuilder.org$apache$spark$sql$columnar$NullableColumnBuilder$$super$appendFrom(ColumnBuilder.scala:87)
 at org.apache.spark.sql.columnar.NullableColumnBuilder$class.appendFrom(NullableColumnBuilder.scala:62)
 at org.apache.spark.sql.columnar.NativeColumnBuilder.org$apache$spark$sql$columnar$compression$CompressibleColumnBuilder$$super$appendFrom(ColumnBuilder.scala:87)
 at org.apache.spark.sql.columnar.compression.CompressibleColumnBuilder$class.appendFrom(CompressibleColumnBuilder.scala:78)
 at org.apache.spark.sql.columnar.NativeColumnBuilder.appendFrom(ColumnBuilder.scala:87)
 at org.apache.spark.sql.columnar.InMemoryRelation$$anonfun$3$$anon$1.next(InMemoryColumnarTableScan.scala:142)
 at org.apache.spark.sql.columnar.InMemoryRelation$$anonfun$3$$anon$1.next(InMemoryColumnarTableScan.scala:120)
 at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:278)
 at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:171)
 at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:78)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:262)
 at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
 at org.apache.spark.rdd.MapPartitionsWithPreparationRDD.compute(MapPartitionsWithPreparationRDD.scala:63)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
 at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
 at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
 at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
 at org.apache.spark.scheduler.Task.run(Task.scala:88)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)

So I googled this - and there seems to be no definite answer to this issue.
So here is what I did -
1. Looked at the ColumBuilder source code - There is a default size for each column type which is about ~ 128 MB (if my math is right)
2. I looked how columnar storage works and how it relates to the ByteBuffer - the only thing I can find out there is this blog - (oopsoom (Chinese - blog- get ready to translate)) which with the help of picture in the blog gave me a good idea on how this columnar storage works (in terms of the data structure that it uses)
3. I have to review my code again and see if I made any mistakes in coding this for Spark - so just a refresher I looked at this slides again - op-5-mistakes-to-avoid-when-writing-apache-spark-applications. In this presentation, it reminded me that no spark shuffle block can be greater than 2 GB and this is especially problematic for spark SQL.

So armed with this knowledge I have to pull out several tricks from up my sleeve. Here is what I did.
1. First increased the driver memory and executor memory - memory is cheap!
2. Increased the number of partitions (from the original 16 to 2500 (Spark uses a compressed data structure if the partitions are more than 2000 - (Line 48 on MapStatus.scala))) before processing each row. This is to have each executor deal with smaller chunks in which the dataset can grow to 11 times its original size
3. Of course, I used KryoSerializer and here are the settings that I used

     conf.set("spark.kryoserializer.buffer.max","2000m")
     conf.set("spark.kryoserializer.buffer","512m")
     conf.set("spark.rdd.compress","true")
     conf.set("spark.serializer.objectStreamReset","10")

4. In the SQLContext, I have used the following settings as well

   sqlContext.setConf("spark.sql.parquet.compression.codec", "snappy")
   sqlContext.setConf("spark.sql.inMemoryColumnarStorage.compressed", "true")
   sqlContext.setConf("spark.sql.inMemoryColumnarStorage.batchSize", "1000000")
   sqlContext.setConf("spark.sql.shuffle.partitions", )

So after doing all that - that exception is gone
I can start removing the changes that I made one by one to determine which of the things I did directly address the issue but I am keeping all my settings because from a 4-hour job, it now only runs in about 1.5 hours.

I think that in the next version of spark, there should be a setting to define the column size for any columnar type. In short there should be a legal argument to pass in the size of the column.

Watson - The mystery after jeopardy

2016-11-21T08:19:00.005-08:00

We have been deep diving in cognitive computing. One of the best platforms that a business can leverage to hit the ground on cognitive computing is IBM Watson. Watson has a lot of capabilities especially with the acquisition of Alchemy's API as well. (Alchemy Acquisition - IBM). You get a language translator, language classifier, retrieve and rank, text to speech, tone analyzer, and a lot more. It is just a matter of how these capabilities can be integrated to your business use cases. As part of "the answer" company we have a tremendous and diverse use case for searching - and giving you answers in a way that makes sense, relevant and make a user decide better is at the heart of what makes us "the answer" company. I was a part of the team given the freedom to explore IBM Watson (no matter what the cost). So we have tried the different APIs in a span of a few weeks. Of course, we have to take a look at the Watson's retrieve and rank (IBM Watson;s Retrieve and Rank). Learning to rank have been researched extensively(Learn to Rank) and we have been trying out a few things ourselves so we are quite versed on the retrieving and ranking. Here are the three things that I have observed with Watson's Retrieve and Rank

1. Data! Data! Data! I can't make brick without clay (Can't make bricks without clay)

- Well before you can train IBM's retrieve and rank. The data that you will need to use for your ground truth should meet a minimum quality standard which is well described in the documentation here (Retrieve and Rank documentation). If you have a lot of data, meeting the requirements in the documentation is easy enough. Once, you have met the data requirements you will probably need a translation from the original labels in your data to the relevance label that retrieves and ranks needs. This can be a subject of misinterpretation especially if you are evaluating performance. Evaluating performance might be tricky again because you might need to translate the relevance label to yet another mapping - especially if you are using a third party library like TREC Eval (TREC Eval). I think that IBM should provide all those testing capabilities in Bluemix.

2. Everyone loves a mystery to be solved

- One of the things that we wanted to uncover was how can we turn the knobs/levers on how to the re-rank portion of retrieve and rank behave to better suit our needs. The rank portion is IBM's secret sauce - there is no quick turning of knobs. It does some magic in mysterious ways, that no one using the API will ever know (unless you work for IBM =). The way to add a little bit of spice to IBM's secret sauce is to use a proxy, which is described here (Retrieve and Rank - Proxy). I think the disconnect here is that our team has the expertise in turning the knobs especially in machine learning and language processing. Other businesses might not have this expertise. So having the magic happen in the background and have IBM solve it is good enough for most organizations.

3. Watson will never have the same deductive skills as Sherlock

- Since our company have been doing language processing for a long time. We discovered that we have been doing a little more advanced things than what IBM is doing in the retrieve and rank space. This is good to know because we know we have the capability to address the businesses needs to become "the answer" company.

So there is no mystery in IBM's retrieve and rank API. We know that we are not eager to use it. Right, now it doesn't suit our needs. Maybe if IBM will add more capabilities and make it easier for us to tweak - so that we can have a little bit of control on how it ranks - we might use it. The biggest hindrance I think to its adoption in our organization is probably the cost of running the API in Bluemix. We probably don't want to be charged a fee every time a user does a search. We racked up a few thousand dollars of the bill while exploring in a span of two weeks (probably because we have a lot of data). Hit some limits on the API. One more thing, you have to transport your data in IBM's server in order to use retrieve and rank - well that might not suit organizations whose core business is the data. Especially with legislation like this (Personal Data for Russian Citizens), you might want to control where you transport your data.

2016 - Movies Data Analysis - Linear Regression Modelling

2016-10-14T21:46:00.001-07:00

Java 1.8 Migration - Performance and Garbage Collection

2016-06-04T21:21:00.000-07:00

Java 7 to Java 8 - that is easy!

I have been working on migrating our web application from Java 1.7 to Java 1.8. Migrating our web app is a lot of challenge. What makes it more challenging is that our web application has a really unique process footprint (well that can be said for all web application). You have to know your application like the back of your hand especially if you want to tune garbage collection for it. When I accepted the challenge of changing our web application from Java 1.7 to Java 1.8. I thought that it was going to just a breeze considering that from 1.7 to 1.8 was not that far of a version. It turned out that I was totally wrong. Here are some of the major challenges that I encountered:

1. Permanent Generation turned into MetaSpace

Before Java, 1.8 class metadata is located in the permanent generation of the java heap. This can be set using the -XX:PermSize option. This was removed in Java 1.8 (Remove Permanent Generation). The reason why it was removed is because of the convergence effort between the Oracle JRockit JVM and the HotSpot JVM. The JRockit JVM does not have a permanent generation. So the specification proposed the MetaSpace. The MetaSpace is not part of the heap. It will be allocated as part of native memory. So you need to account for the memory size that you use for the MetaSpace. By default, the MetaSpace is unlimited in how much it can grow in native memory. To limit the MetaSpace use the -XX:MaxMetaspaceSize option. The option -XX:MetaspaceSize is the size when you want to trigger GC on this memory space. I am not sure why you might want to have the values for the previous options to be different. So if you are migrating from Java 1.7 to Java 1.8. You can just practically grab the -XX:PermSize setting and put it in the -XX:MaxMetaspaceSize setting but just take note that the MetaSpace is an additional memory you need on top of whatever your heap size setting is.

2. TieredCompilation is on by default in Java 1.8

This was one of the challenging issues that I faced. In Java 1.7 our application has -XX:ReservedCodeCacheSize set to 64 MB (Default size of code cache in 64-bit Java 1.7 JVM without TieredCompliation is 48 MB). So like the MetaSpace I copied this setting. The default size of the code cache for Java 1.8 with TieredCompilation is 240 MB. My smaller code cache setting did not become apparent when I was tuning garbage collection using G1GC from CMS and I was looking at the CPU usage. The CPU Usage is about 50% to 60% higher than usual. The higher CPU usage is caused by the fact that the JVM is not able to optimize more code thru compilation. So what is basically doing is doing more Just-In-Time compilation like an interpreted language. It took me a while to hunt this down but once I set my -XX:ReservedCodeCacheSize to about 256MB that CPU spike went away.

3. Garbage Collection Tuning - the new G1GC

Coming from Java 1.7 we are using Concurrent Mark Sweep GC with Parallel Remark and compaction. I have to start from scratch in order to determine the settings that we will need for our web app with G1GC. After all the testing that we did we were able to settle with these set of parameters:

-XX:+UseG1GC -XX:G1HeapRegionSize=32m -XX:+ParallelRefProcEnabled -XX:MaxTenuringThreshold=6 -XX:InitiatingHeapOccupancyPercent=45 -XX:G1HeapWastePercent=5 -XX:G1RSetUpdatingPauseTimePercent=5

The biggest issue that I have encountered here is the -XX:G1HeapRegionSize - which the default value is determined dynamically based on the heap size. The region size is how G1 will try and manage the memory blocks in terms of allocation and compaction. When I was looking at our GC logs there were many humongous allocations in our logs which will trigger a lot of GC activity to find space for our big objects (our biggest allocation is about ~20 MB). Once I set this region size to 32 MB that humongous allocation went down and GC activity went down as well.

Conclusion:

Migrating from Java 1.7 to 1.8 will be unique for each application. Just make sure that you separate issues with Garbage collection (if you are tuning garbage collection) and issues with performance (just in time compilers). I suggest grouping the change in JVM options together if it is for performance (might want to change them first then do testing) and grouping the options for garbage collection (change them once you nailed the performance options already).

Tuning garbage collection was a real challenge too and here is what I have learned:
The most important factor affecting garbage collection it totals available memory. So if you have a chance not to tune garbage collection then don't. It will be cheaper to buy memory and add it to your server than spending the amount of time tuning garbage collection. But if your application really needs a fine tune garbage collection, available memory is still the most important factor affecting GC.
You need to increase your memory as you increase the number of processor on the machine you are running your JVM. Another thing that I have learned is that in Java 1.8 an Out of Memory error can be thrown by GC activity.

Agile is not a process!

2016-05-30T10:07:00.002-07:00

Agile - What is it?

I have recently reinforced my understanding of what Agile is in relation to software development.
One of the things that I realized is that the Agile manifesto does not dictate a process you have to follow but is more like a culture of what you need to value.

One of the most popular agile methodologies is Scrum. Scrum Guide (The last time I read the guide was in 2009 - they have released a new guide in 2013. They will probably release a new one soon)- If you follow Scrum, you need to follow everything otherwise you are not doing Scrum. You can do stand ups but if that is only you are doing then it is not Scrum. Scrum will ensure that an Agile culture is developed with the team - and the person to see that through is the Scrum Master.

Since Agile is in its teen years - we probably need to reconnect with it and revisit where it is being taken.
Agile in its teen year!

Speech to Text - HTML 5

2016-05-24T20:48:00.002-07:00

Technology Can Help

Recently there was a news in Good Morning America were a deaf cashier is taking orders and customers patiently writing out there orders. Here is a link Deaf Cashier

Well with a computer in every pocket there is technology to help.
Here is a quick demo that i have created to use HTML 5 technology to create a speech to text web page.

Here is a link to the working page
http://petabyte.github.io/textToSpeech.html

https://www.youtube.com/watch?v=27SZISZAPEA

Internet of Things Project - Music Teacher

2016-05-24T20:24:00.001-07:00

Why ?

Music Education usually involves a student using several instruments. A student needs to log practice time which should meet a certain amount of time spent practicing with the instrument. This is usually done using a log book. The logbook usually needs to be initialed by the parent. ( Sometimes the parent does not really know if the student practiced or not.). So, the logbook usually gets signed by the parent but the student did not really practice. How can you verify this ? - You can record every practice via video (but you are going to need to watch the video in order to determine the amount of practice time) There must be a better way of logging music practice.

Music Practice Logbook

How to solve this issue ?

Good thing is that there is a lot of makers out there connecting things to the internet. We can create a device that will log music practice so that we can see a summary of the length of practice every day.

Using a mic sensor, led lights , and a particle wifi board - I created a prototype that will log music practice. Here is a high level fritz of the circuit board I have created - the RGB led will light up RED - when the practice time is not met YELLOW - met half of the practice time GREEN- met the amount of practice time for the day. The other LED just indicates if the sound is being picked up by the sensor. The device code is here https://github.com/petabyte/IonT

Simple IOT Device

High Level Architecture of how the device talks to the server

As you can see in the diagram above I used a cluster of elasticsearch instances. Since we are logging a lot of events and only trigger a feedback using an aggregation of certain events, we need to find those events.

Technical Difficulties

Isolating sounds became an issue - specially an instrument like the piano which has wide spectrum can be difficult to identify

I can't tell if somebody is gaming the device - again based on sound isolation

Where can this fit?

I think a metronome manufacturer can easily fit this feature but they will need the backend service to log the practice time

Here is a video demo:

https://www.youtube.com/watch?v=hyLx62t5s3I

Messaging for the Internet of Things

2016-02-19T22:36:00.000-08:00

Messaging for the Internet of Things

As we embed processing power to a lot of our everyday things. The thing will need to communicate with a server to send data or get data . The thing needs data in order to make something happen. The thing needs to send data so that we can be smarter on how we use the thing. But How do we send data to a central server that is robust, asynchronous and reliable ?

MQ, what is old is new Again

MQ or Message Qeueing is new again in the age of the Internet of Things. Messaging Patterns for the enterprise has been around for ages. What were the major reasons that messaging addresses ?
1. Networks are unreliable - data needs to be transported over the network. These networks are a conglemeration of different types of network. Since we do not have control of over most of the network that we use to access the internet. The network will present a lot of problem specially when you are sending messages across the internet.
2. Networks are slow - The network that we are using to use the internet is mostly aging. Again, not having control how some of the networks that we might use to send data over the internet it can have performance implications.
3.Application are different - Using messaging any application can listen or subscribe to the message as long as it can be parsed or consumed by the application subscribing to that message.
4. Change is inevitable - in the internet of things, we can put more processing power in more things. There will probably be a need to add more devices that will send data over time. We might need to change the servers where it send data to as well as it gets updated.

These are the same issues that the Internet of Things will encounter.

Messaging Patterns

Since messaging allows high-speed, asynchronous, program to program communication with reliable delivery it is only but natural to be adopted in the internet of things. But why use messaging for the Internet of Things ?
1. Asynchronous communication - Messaging enables a send and forget approach to communication. Since most of the connected devices would be somewhere remote. These devices does not have to wait for the receiver to receive the message. 2. Throttling - The issue with synchronous communication or call to a server like HTTP. A lot of connected devices can
overload the receiver which can cause performance degradation on the server. Messaging will help control the rate at which the receiver can consume the requests.
3. Disconnected operation - most of the time an internet of thing device will be used in a low-power autonomous setting. Low power consumption can be achieved by not doing a lot of communication with the server. During those time that the device tries to communicate and the network is not available it can queue up the messages and synchronize when the network becomes available.

Asynchronous Communication

Since messaging using MQ can be done asynchronously. A different kind of approach is required for designing applications like the use of callbacks and other result notification patterns.

Since messaging has been around for a while we can adapt certain patterns that have been used in the enterprise to the internet of things. Sample patterns [for a complete list please go to website in the reference at the end of this article]:
1. Guaranteed Delivery - make messaging persistent so that they are not lost if the messaging system crashes. The messages are stored locally until it can be successfully delivered.
2. Datatype Channel - provide a separate channels for each type of message. The sender knowing what message to send can choose the appropriate channel to send it thru. The receiver knowing what channel the data was received from will know the type of message it receives.
3. Publish-Subscribe Channel - delivers a copy of the message to multiple subscribers.

Conclusion:

Messaging systems like MQTT will be an essential infrastructure to the internet of things. We should learn from the patterns that we learned from creating enterprise software that leverages messaging and apply the experience when rolling out devices for the internet of things.

Reference:
1[http://www.enterpriseintegrationpatterns.com/patterns/messaging/Introductio
n.html]

OAuth 1.0a Request Signing and Verification - HMAC-SHA1 - HMAC-SHA256

2015-12-15T19:20:00.000-08:00

This is a security project that I did this Fall semester. I was interested in why the OAuth1.0a specification did not require you to sign a request using RSA-SHA1 or any RSA algorithm. The first issue that I found is where would you store your private keys in a javascript client? - as a javascript encrypted file ? from the browser ? - that will be a security hole if you allow javascript to get private keys. These are some of the questions that got me asking and which I cannot find answers to for now.

OAuth 1.0a Signature Signing and Verification using JavaScript Implementation

Background

The OAuth specification tries to define an API authentication method that allows consumer websites or applications to access protected API Resources from a Service provider. The advantage that OAuth provides is that the user of the website or application is not required to disclose their credentials to their service provider. Another advantage of using OAuth to authenticate is that it does not require a specific user interface for how Service providers will authenticate a user [1].
Section 9 of the of the OAuth specification [1] specifies that all Token requests and Protected resources requests must be signed by the Consumer and verified by the Service Provider. As we have learned in class, Digital Signature provides authentication and some form of non-repudiation if it is signed by a private key. The signature in the OAuth specification is a hash of the request (a specific format is needed – the Signature Base String) using the users Consumer secret (sort of a private key). As mentioned in the specification, the purpose of signing requests is to prevent unauthorized parties from using the Consumer Key and Tokens when making Token request or Protected Resources requests. The signature is based on encoding the Consumer secret and Token secret (which is unique for each consumer) into a verifiable value which is included with the request.
OAuth does not mandate a particular signature method. The OAuth 1.0a protocol defines three signature methods: HMAC-SHA1, RSA-SHA1, and PLAINTEXT but Service providers are free to implement and document their own methods. This means that the protocol allows flexibility to use any signature method.
The signature is based on the Signature Base String (Section 9.1 [1]). The Signature Base String is the concatenation of following items into a single string, where each item is encoded (Section 5.1 [1]) and separated by ‘&’ (ampersand) character (ASCII code 38), even if the item is empty.
1. The HTTP Request method used to send the request. Value must be uppercase, for example: HEAD, GET, POST etc.
2. The request URL based on (Section 9.1.2 [1]).
(Please see section 9.1.2 how the URL is formed. Here is a link)
3. The normalized request parameters string (Section 9.1.1 [1]).
(Please see section 9.1.1 how to normalize request parameters. Here is a link)

Sample Signature base string: (based on the implementation for this research paper)
Blue = Request Method
Green = Request URL
Red = Normalized request parameters

POST&http%3A%2F%2Flocalhost%3A3000%2Fsample&oauth_consumer_key%3Dtestkey%26oauth_nonce%3DVNl2QAsefSVvZg0CxT9j6SJcPlnRDi01%26oauth_signature%3DA3jkT%252Bm4MZV6M2jxQxPGCRhJlcs%253D%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1446170859%26oauth_token%3Dtestkey%26oauth_version%3D1.0

Purpose of this research

The purpose of this research paper is to implement, learn the signing methods and verification methods for OAuth using JavaScript. Since the popularity of single page application using JavaScript and HTML is on the rise, this research will look into the following:
1. Cryptographic libraries in java script
a. Investigate what the current OAuth signing libraries are using to encode the signature
2. Implement OAuth signing in JavaScript using open source libraries
a. Add a signing method
3. Verify OAuth signature in JavaScript server side in Node using open source libraries
a. Add a verification for a signing method

Cryptographic libraries in JavaScript

While implementing the signing in JavaScript the following were used as the cryptographic libraries.
1. Crypto-js https://code.google.com/p/crypto-js/
This is the library that was used to do the HMAC-SHA1 and HMAC-SHA256 signing. This library has decent cryptographic capabilities like Hashing, Message Authentication Code and ciphers. It is good to know that most of the major algorithms for conventional cryptography are supported in this library (no RSA).

2. jsrsasign http://kjur.github.io/jsrsasign/
This is the library that was used to do the RSA-SHA1 signing. It also provides base64 encoding. This library was also used to generate the Private Key and Public Key pair. This JavaScript library is focused on private-public key cryptography (RSA). It also has support for Digital Certificates (X.509)

Maybe in the future a JavaScript library should be created to have conventional and public key cryptography in one library so that client and server side JavaScript will have less dependency.
Implement OAuth 1.0a signing in JavaScript using open source libraries

Since OAuth is being used by several popular services like Twitter, Flickr, Bitbucket, Linkedin, there are many JavaScript libraries that will help you create an OAuth signature. The one used for this research is oauth1.0a.js library which can be found here https://github.com/ddo/oauth-1.0a. The library implements the following signature signing on the browser out of the box HMAC-SHA1, HMAC-SHA256, and PLAINTEXT. The signature signing uses the Crypto-js library. The library has a dependency on Crypto-js. The implementation is missing the RSA-SHA1 implementation. I have added the RSA-SHA1 implementation for signature signing which uses the jsrsasign library.

Figure 1. Screen capture of the implementation of the RSA-SHA1 signing method. Lines 54-66 of the oauth1.0.a.js

Verify OAuth 1.0a signature in JavaScript server side in Node using open source libraries

After signing a request we need to be able to verify the signature on the server side. The server side implementation uses Node with the node module Express to serve up web static content and REST [2] services. To verify the OAuth signature a middleware function for Node Express application was used which can be found here https://github.com/keybuk/node-express-oauth. The library only supports PLAINTEXT and HMAC-SHA1 signature verification. I have added the HMAC-SHA256 and RSA-SHA1 implementation for signature verification. The server side library uses the same Crypto-js library to verify the signature. I have to add the jsrsasign dependency to do the RSA-SHA1 signature verification.

Figure 2. Screen capture of the implementation of HMAC-SHA256 and RSA-SHA1. Lines 181-195 of the oauth.js file
Implementation Details
The implementation used JavaScript on the browser and JavaScript on the server side in Node.

Figure 3. Browser and Server Communication Diagram

Implementation Folder Structure and Files
Here is the folder structure and files of the implementation. All of these files can be found in this repository https://github.com/petabyte/SEIS720. A short video to install and run can be found here https://youtu.be/0lXqHLTQmR0.

Figure 4. Implementation Folder Structure
Run the code

After install you should be able to hit http://localhost:3000 on your computer. The link will bring up a page like this.

Figure 5. Page after running the implementation
The page will contain each implemented signature method with the following details:

HMAC-SHA1

Response from server: This is the data that was requested via Oauth
Server response time (including signature verification): 1.513ms
Client Signature method: HMAC-SHA1
Client Signature Base String: POST&http%3A%2F%2Flocalhost%3A3000%2Fsample&oauth_consumer_key%3Dtestkey%26oauth_nonce%3DOcHPsdqNtRsTeonRhJsTE7eXTBPgYfdo%26oauth_signature%3DzYUBJA3RatAiV8eUGEkWvSXHAf8%253D%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1446416650%26oauth_token%3Dtestkey%26oauth_version%3D1.0
Client Signature: 4Prn5N6YRXHrriAwJtj97FWUJxM=
Client Side Timing: 5 in milliseconds

Timing for Each Signature Method

I tried to get the average time (5 request each) for each signature method.
Here are the timings

Signature Method	Client Side Signing Timing	Server Response Time (with Verification)	Average Total Time
HMAC-SHA1	2.41 ms	2.411 ms	~ 5 ms
HMAC-SHA256	2.2 ms	2.937 ms	~ 5.13 ms
PLAINTEXT	1 ms	1.301 ms	~ 2 ms
RSA-SHA1	31 ms	8.398 ms	~ 40 ms

Conclusion

Using JavaScript prove to be advantageous because the same library can be used in the client and server side. I have learned that there are some decent JavaScript libraries out there that supports conventional cryptography and public key cryptography. There is an opportunity to create a library that does both. Right now you have to use two or more JavaScript libraries to support both. The OAuth specification about signing request is flexible enough to accommodate any advancement in signature encoding as long as the methods of signing are conveyed to the Consumer by the Service provider. Key storage for key pairs can be an issue for using JavaScript. There is no protected key store management like the one provided in Java. The keys can be stored as ASCII (Base64) data. Even though this is supported by X.509 Certificate standard managing the file that contains the information can easily be copied.

References

[1] OAuth Core 1.0 Revision A. (n.d.). Retrieved October 30, 2015.
[2] Fielding, R. (2000). Architectural Styles and the Design of Network-based Software Architectures. Retrieved October 30, 2015.
[3] Fielding, R. (2000). CHAPTER 5. Retrieved October 30, 2015.

10 years of AJAX - Now much cleaner and standardized !

2015-07-15T11:38:00.001-07:00

A decade of Ajax

About 10 years ago the Ajax word was coined to describe the interactive web that Google created at that time Google Suggest and Google Maps (Ajax - Jesse James) . At the heart of that emerging technology is a Microsoft created object XHR(XMLHttpRequest - XMLHttpRequest) and is now adopted by Mozilla, Apple, and Google (IE is a different Web technology story). XHR is now being adopted as a standard in W3C (standardized XHR - W3C). But not only that there is XHR2 (HTML5 Rocks) - (I think this has been discontinued.)

XHR provides an easy way to retrieve data without doing a page refresh. Despite its name we are using XHR for any structured data that the server can provide us. (JSON, JSON-LD) - and binary data (SPDY HTTP 2.0)

We have used AJAX in a lot of creative

(SPA, Web API, MashUps)

ways and I am excited to see the next decade of highly interactive web designs.

The Future

What would the role of AJAX be in the internet of things ? Maybe we have an asynchronous object that consumes and receives data connecting to an MQTT (http://mqtt.org/)

As I feel nostalgic about Ajax here are some links that will help you revisit AJAX:

https://developer.mozilla.org/en-US/docs/AJAX/Getting_Started

http://jibbering.com/2002/4/httprequest.html