Idea Fountain

Agent AI Workflows, Deja Vu?

2024-12-08T00:04:00.002+05:30

LLMs have had a great run, up the hype cycle. After the initial euphoria, things seem to plateauing off in terms of potential usage, the burst of Generative AI not withstanding. Also the LLMs are not making much inroads into the super lucrative market of enterprise development. They have spent billions on evolving the models of these LLMs, the incremental evolution from here on can hardly pay the bills.

Refer my earlier post on the difficulties of LLM getting to learn about custom enterprise data models. In fact the only way to make inroads into the enterprise market and get a piece of the pie, is by somehow making enterprise business process development, dependent on the LLM.

As things stand right now, it would be very difficult and time consuming to enable LLMs to directly understand custom business processes, so what is the next best strategy now? Enter Agentic workflows ....

Every few years or so, this century, we have had technologies like EJBs, BPM, BPEL, ESB try to convince enterprises that they were the way to model business logic inside business processes. Well there is a new alternative which will now try to model business logic and processes, while being orchestrated by none other than the LLMs.

The #AgenticAI workflows with their nodes, edges, human-in-the-loop, sub-graphs for sub-processes, workflow designing studios, deployment servers will have the ability to loop with exit conditions contrasting them from DAG based BPM flows. The agentic workflow is a classical state machine implementation controlled to an extent by the LLM, but dictated primarily by business logic any which ways.

Meanwhile the inherent complexity of any business process stays just the same, irrespective of the technology used. Many of the lower intricacies of the business process may still need to be modelled as service implementations with persistent domain data models. The autonomous decision making part is left completely at the risk of designers who design the agents, agentic workflows, with LLM providing limited control or <ahem> intelligence.

The amount of sematic modelling involved in an advanced state machine, which will act as autonomous agents, is no trivial design exercise and maybe base agent reference implementations will evolve in due course to make business application development easier and less riskier.

Are there less expensive options of leveraging LLMs intelligence, in the enterprise, as pin-point integrations points within a custom business process, apart from RAG based approach? Comment below.

Meanwhile, I am rushing off to continue learning #LangGraph, #Autogen and #CrewAI, since we all need to be in the game, right?

Why I am starting to believe that time is an illusion

2020-11-29T11:26:00.002+05:30

We have refined our understanding of gravity from being a force of nature that attracts 2 masses, to gravity being merely the curvature of space-time to gravity being interpreted as time slowing down in space.

The other important ongoing discovery has to do with dark matter and dark energy which constitutes around 95% of our observable universe. Commensurate with evolving insights of dark matter and energy, dark energy at least seems to be the omnipresent energy in the vacuum of space, spread out over such large spaces that its detection is faint due to its density being very less.

I imagine, that dark matter may also on similar lines differ from ordinary matter, in that, it is much less dense and spread out over the vast expanse of inter galactic spaces. The current tug of war between our expanding universe v/s gravity, could then be seen as a war of equilibrium between normal matter/energy vs dark matter/energy, and clearly normal matter is getting rarefied, on the overall scale of the universe.

Next, lets for a moment assume that the nature of reality(matter and energy), truly is made up of E8 crystals or any other unit of reality, for that matter. So ordinary matter could be nothing but "densely packed E8 crystals" and dark matter could be "very rarefied" E8 crystals.

Time as we know it, is perceived by us, as beings made up of matter. Our perception of time primarily is from the point of view of how it appears in the "ordinary matter" portion of the universe. In the vacuum of space and in the dark matter part of the universe, is time just as relevant?

The perception of time is linked to causality, as experienced by us in the macro universe.

Time in the macro universe might seem ubiquitous but at sub atomic and quantum scales, where wave nature, duality, uncertainty and randomness are the norm, does the linear flow of time, as in the macro universe even hold relevance?

Time needs a frame of reference and an observer, without these, being clear to us, over the expanse of the universe and dark matter, is time a relevant variable for study, of particle physics itself?

Now can you blame me for tending to think, that time might be an illusion of our perception of the macro universe.

-GG

Throttling Kafka Consumption - Dispelling Myths

2020-08-21T18:54:00.000+05:30

Kafka as a messaging broker was designed for high throughput publish and subscribe. At times this very fact can prove to be an issue, when being used in more traditionally designed transactional systems. For example if an application is consuming messages from a kafka topic and inserting, updating in a relational database, the high rate of consumption of kafka messages, can put pressure on transactional persistence in the RDBMS and cause exceptions. Hence there arises a need to throttle message consumption from a kafka topic.

Starting out, lets assume, that we are only considering the message consumption inside a single instance of a consumer (which may be a part of a bigger consumer group). So we have a single consumer instance, picking up kafka messages from a single partition of a topic.

Most kafka client frameworks, will implement a decent kafka consumer poll-loop, for reading kafka messages from the partition as soon as possible and then hand these messages over to client application, as fast as possible, but without waiting/blocking for the client application to process each message.

For single thread technologies like node js, the above can be an issue,in the sense that, the result is that the application starts processing large number of messages in parallel. Many will resort to kafka configuration parameters like below, to control the rate of consumption of messages from kafka, with very little success, for the reasons I mention below:

fetch.min.bytes - This property allows a consumer to specify the minimum amount of data that it wants to receive from the broker when fetching records. If a broker receives a request for records from a consumer but the new records amount to fewer bytes than fetch.min.bytes, the broker will wait until more messages are available before sending the records back to the consumer.

fetch.max.wait.ms - By setting fetch.min.bytes, you tell Kafka to wait until it has enough data to send before responding to the consumer.If you set fetch.max.wait.ms to 100 ms and fetch.min.bytes to 1 MB, Kafka will receive a fetch request from the consumer and will respond with data either when it has 1 MB of data to return or after 100 ms, whichever happens first.

max.partition.fetch.bytes - This property controls the maximum number of bytes the server will return per partition.

session.timeout.ms - The amount of time a consumer can be out of contact with the brokers while still considered alive defaults to 10 seconds.

max.poll.records - This controls the maximum number of records that a single call to poll() will return.

receive.buffer.bytes and send.buffer.bytes - These are the sizes of the TCP send and receive buffers used by the sockets when writing and reading data.

If we manipulate above values, messages will be fetched more frequently OR less frequently. But if already 1000s of messages are lying in the topic, it wont matter a lot, given that fetch from topic to consumer is very quick(micro seconds). The consumer will still feel a steady supply of almost parallel messages.

Even playing around with commit offsets, will never block consumption of messages but only influence recovery, guaranteed delivery, once only delivery, etc. This will not be able to restrict super-fast, reading of messages by the consumer. (Its not like late acknowledgements, block, consumption of messages in traditional message brokers like JMS, MQ, etc)

The bottom line is that, the challenge of parallel consumption of messages in a kafka instance consumer, needs to be handled by the process/application and not by kafka configurations.

Consumer Side Technology

A much more important factor while dealing with throttling of kafka messages, is which tech/architecture you use, on the consumer side application.

Using Java / SpringBoot based consumer, will ensure that the consumer process runtime will be multi-thread-based and will be synchronous first or synchronous by default. The use of thread-pools in Java/Spring and given the fact that almost all calls in Java are synchronous by default, it is much easier to control the kafka message consumption in Java

Node JS based consumers are inherently single threaded an hence asynchronous first or asynchronous by default, using await, promises and callbacks in a way, that near-parallel consumption of kafka messages will be challenge, especially for less-experienced developers.

Process Mining - Practical Uses

2019-11-18T17:28:00.004+05:30

Process mining is a family of techniques that support the analysis of business processes based on event logs. During process mining, specialized data mining algorithms are applied to event log data in order to identify trends, patterns and details contained in event logs, to auto-discover and paint process definitions of processes. Process mining can be used to discover processes from events or improve process efficiency by identifying bottlenecks and exceptions

By itself process mining has several applications in large enterprises, to discover or validate business processes (in finance, manufacturing and retail) merely from events and timestamps, but the exciting thing is that the same technique can be used to get insights into user/customer behaviors on your website and also aid in customer segmentation.

Consider a simplistic apache web log (though much more detailed application logs can also be used)
127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] "GET /login.html HTTP/2" 200 1479
127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] "GET /catalog.html HTTP/2" 200 1479
127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] "GET /checkout.html HTTP/2" 200 1479
127.0.0.1 - peter [9/Feb/2017:10:34:12 -0700] "GET /checkstatus.html HTTP/2" 200 1479

The important pieces of information are the user id "peter", the date timestamp and the URL visited which can proxy up for the user's intended action. The above log can be easily transformed into

Sample CSV file

<user>,<action>,<action date>, <user attribute 1>, <user attribute 2>, <user attribute n>
UserId,Action,StartTimeStamp,EndTimeStamp,PremiumCustomer,UserLocation,UserOccupation
Peter,Logged In,12-20-2017 10:20:00 AM,12-20-2017 10:25:00 AM,premium_customer,US,architect
Bob,Logged In,12-20-2017 10:20:00 AM,12-21-2017 10:25:00 AM,premium_customer,US,architect
Peter,Browsed Catalog,12-20-2017 10:35:00 AM,12-20-2017 10:40:00 AM,premium_customer,US,architect
Alice,Browsed Catalog,12-21-2017 10:35:00 AM,12-21-2017 10:40:00 AM,premium_customer,US,architect
Bob,Browsed Catalog,12-21-2017 10:35:00 AM,12-21-2017 10:40:00 AM,premium_customer,US,architect

The above CSV data can now to be fed into a process mining tool to generate the "user/customer's" interaction process, as he navigates and uses the website. This can give many insights into how customers are using our websites and what we can do to make things more convenient for them.

Statistics like what time duration is spent by each user in any of the activities can also be readily obtained as a report from the tool. The major deviations in the main process flow can also be denoted. Animations of the process flow, over a period of months can be played out within minutes to get insight into the activities that are bottle-necking the overall process.

Instead of discovering processes for individual users we can generalize processes for say "platinum customers" by filtering the data within the process mining tool itself.

Process Mining deals with Petrinet and BPMN notations of process definitions and some popular algorithms for process mining are alpha miner, inductive visual miner and fuzzy miner.

Notes on Enterprise Application Development Strategy

2019-11-10T19:18:00.001+05:30

At the end of the day, technology is an enabler for extracting business value. Some sure fire ways to burn money needlessly, during enterprise application development are noted below. Use it as a checklist to guard against, in your own enterprise development. As always, use your own context, while weighing-in, any generic advice.

Don't care why and what the end user actually wants. Its more important to impress top management, with a good-looking demo
Using a particular tech because it is popular at google, netflix or facebook, irrespective of its applicability within your own context, especially NFR context
No clarity about the WHY, obsessed with only the HOW
Not able to contain scope creep and needless escalation of complexity
First thought that comes up becomes the definitive design, no choices, no alternatives considered
Resume driven or Ego driven, development by tech leads

Why use a suitable tech, when a tougher, riskier, newer tech is at hand, no matter how irrelevant

Underestimating integration and dependencies
Committing to technology choices without understanding the full scope of functionality and the non-functional requirements(NFR) in detail
Forgetting that no application is an island. Its important that an application can "fit into" the enterprise landscape in terms of security standards, IDAM (identity and access management), audit standards, deployment, reliability and business continuity standards
No thoughts given for migration path from existing application/s
Disproportionate emphasis on UI look and feel rather than UX and information models
Re-invent the wheel, under the guise of tech bravado
Dev Teaming mistakes

Thinking that employing 10 mediocre developers, instead of 3 crack devs, will be better
Believe the myth that application devs, still write framework level code
Never acknowledge, that app devs now, require to use proper frameworks and APIs, rather than belt out algorithms and complex programs

Blockchain - Usecases in supply chains

2019-10-21T13:31:00.003+05:30

Supply Chains, are the lifeline of businesses globally. Most likely, your business will find itself between upstream suppliers and downstream sales channels/distributors/customers. The figure below indicates this topology succinctly.

Generic Supply Chain connected via a Blockchain

Your organizations ability to fulfillment of its sales orders, totally depends on how efficient, agile your supply chain is. Some of the areas where blockchain technology can add great business value are:

Optimizing Inventory Costs
Though organizations have fairly automated internal processes, most still rely on antiquated ways of interacting with their suppliers and business partners, Orders can be placed over voice calls, email, logging into each other's portals, etc. Tracking of these orders status is never, in real time, mostly left to voice calls, email reminders, etc. In fact, it would not be uncommon for your organization to get notified of a supplier's delay in order fulfillment, close to your own, downstream, fulfillment commitments. By having, deep, blockchain-based integrations, with your supplier's ERP, you can ensure almost real-time notifications of any potential delays, in your order fulfillment. You can automate balancing of orders across your other suppliers as well.

If your distributors/sales partners are also on the blockchain, any weakness in demand, can also be notified to your organization in real-time, so as to re-balance supplier-side orders. This will lead to highly optimized inventories and substantial reduction in inventory costs. De-risking from supply side outages due to a specific supplier is another collateral benefit.

By having most of your suppliers on the chain, you will get unprecedented clarity about the big picture on the supply side, across all your suppliers. This can help you in coordinating receipt of goods from different suppliers, with a consolidated scheduling system based on blockchain data. This can further optimize the timing and logistics aspects of the inventory.

Minimizing Quality Costs using the blockchain
With the blockchain in place, your organization can have selective access to your supplier's ERP Data, including their quality control data. Having this level of accurate data, you can choose to minimize redundant quality checks, at your end. This is the good example of using a blockchain to enable trust and optimize costs.

Suppliers of a Supplier can also be brought onto the blockchain, to enable deeper and far reaching integration via the blockchain.

A blockchain can be used very effectively to "track the state" of a logical unit of delivery across supplier's parts. For example, if your organization was making refrigerators, every part, being supplied, tracked with the various suppliers, can be co-related with the logical entity "refrigerator 123" on the blockchain, hence, at any given point in time, you could not only know that 30 compressors, 26 coolant tubes are being supplied to me, but also that, I will be able to get out 26 complete refrigerators to my customers.

Bidding based supplier order management
The same trust based blockchain technology can be used to operate a bidding platform, to manage orders being issued out to the suppliers

Why traditional integration technologies like API, will just not work?
Point to point integrations with multiple suppliers not are just not scalable, in the supply chain ecosystem. imagine a supplychain with 5 nodes, each talking to other via point to API calls. Blockchains, along with providing a decentralized datastore, to track "shared state", also provide a standard stack for integration, at all peers, not to mention "consistent versions" of same distributed smart contracts for processing the data.

The suppliers that you bring onto the blockchain, may be direct competitors, inherently having a lack of trust about using processes, business logic, databases and codebases, belonging to each other, but the decentralized blockchain, will make it possible, to bring all such parties onto a common platform, sharing data/state, business processes and code.

Countering, Counterfeiting

2019-10-21T13:22:00.002+05:30

While researching into the practical applications of blockchain tech, in provenance, tracking and traceability, I came across claims that blockchain can help to tackle counterfeiting, which is a serious issue in the fashion, luxury and branded products industry.

The general opinion seemed to be that its tough to fight counterfeiting because establishing authenticity of an original product item, over a cleverly counterfeited one, was a tough nut to crack (at affordable technologies).

Lets get into the mind of counterfeiter then...Typically counterfeiters will reproduce a near-duplicate of the original, at a fraction of the cost, then package it exactly like an original and send it out to be sold. It would be next to impossible to catch a cleverly made and packaged counterfeit item.

The barcodes and qrcodes on the item package can be copied to the T, as well and then pasted onto the duplicates package. Standard barcodes (EAN-13) contain only the manufacturer code and product category information, which is popularly recognized as a GS1 standard GTIN, like is typically used to scan retail products as POS terminals. But the real problem like I said is that, the entire barcode can be copied and put onto a duplicate item, making it indistinguishable from the original. Authenticity is sadly not guaranteed even if additional attributes about the product item were embedded into extended barcode versions like the GS1 Databar or GS1-128.

For authenticity checking, the trick is to use 2 factor authentication. Imagine a qr code encoding a random number, printed on the outside packaging of a luxury product. Now another QR code with yet another encoded number present inside the product itself (say on the product lining or inside of the cap of the bottle). Now we have 2 numbers, one on the outside and one on the inside(can be accessed, only after breaking the seal of the product).

A simple web page/mobile app can be developed to validate the compatibility of the inside number and the outside number. As long as the compatibility checking algorithm is secret (maybe using a magic number/s), the end user can just scan the outside and inside QR codes/numbers, to let the website/mobile app, determine if the product is authentic or counterfeit.

Even if the counterfeiter manages to exactly copy, the outer packaging, he can never "quess" what the inside number should be, since he does not know the exact algorithm, that is used to derive/correlate to the inside number.

The last problem to solve, is the possibility that the counterfeiter may get hold of a few good samples of "valid" number pairs and use them with all the counterfeits repeatedly ! This can be easily tracked, since the website/app is centralized and can keep track how many times the query for checking authenticity is being received and from what sources, mobiles, geo-locations etc

Let know in the comments, any loopholes, I may have missed or any improvements you can think of.

Integrating existing enterprise apps with blockchain

2019-08-13T16:51:00.002+05:30

Lowering the barrier to entry, is key for increased adoption of blockchain based technologies into enterprises. Irrespective of the business value, that blockchains bring, there is a need to decrease the cost, pain and risk of integrating blockchains with existing applications in the enterprise.

Most large enterprises today, have established ERPs, CRMs, or cloud native custom applications, running 24 x 7. It makes sense to interface the new blockchain tech, harmoniously into the enterprise architecture landscape.

Its quite practical to assume that, most organizations, while they would want to participate in a blockchain and get all the benefits, will most likely be reluctant to do any major modifications to their existing enterprise apps. The need of the hour is, to ease ingestion of data onto the blockchain, without requiring expensive UI based / operational alternatives.

Presented below are a few ideas, for ingesting data onto blockchains, in an automated and batched manner, with minimal integration development and risk. Refer my earlier article about, wrapping up blockchain clients inside REST webservices, the below ideas can be used in conjunction with the REST adapters.

ERP Application Logs >> Logstash >> Http Post to Blockchain REST adapters >> Blockchain

Existing Application JDBC DB >> Database Polling >> Http Post to Blockchain REST adapters >> Blockchain

Existing Application publish message >> Topics (Message Provider/Kafka) >> Logstash >> Http Post to Blockchain REST adapters >> Blockchain

When the need for capturing data onto the blockchain is critical, more reliable and resilient, pipeline with Kafka based topics can be used.

Blockchain alternative to API integration across enterprises

2019-08-10T12:35:00.000+05:30

What is the go-to solution when 2 enterprises wish to communicate with each other. Right! Its via APIs, that too using REST APIs, with appropriate security in place.

Typically this involves one of the companies developing webservices and exposing those services, to be consumed by another enterprise. There are several constraining factors for this kind enterprise integration:

The webservices paradigm is uni-directional, that is, if org A, invokes a webservice hosted by org B, for org B, to confirm that desired state changes have occured, org B must call another webservice hosted by org A. This is essentially the issue, that the webservice calls are stateless. There is no state sharing at all between the organizations.

For true leverage and reuse of the webservices, there needs to be orchestration at the consumer side. This orchestration is not transparent to the provider of webservices.

The communication is primarily point to point. So the interactions, cannot be scaled easily if newer participants join in. Imagine 4 organizations, each one trying to call services into others and trying to keep track of some shared state, using and exposing status calls

All the above issues are elegantly solved using the "shared state" paradigm of blockchain. All participants talk to the blockchain almost as if it were a logical messaging bus, but more importantly, the bus is not just a physical connection, but maintains state in distributed ledger and also has smart contracts that can house, common and transparent pieces of code, that can be shared across the un-trusting organizations aka partners

Server to server API calls Security

2017-10-03T18:32:00.002+05:30

Calling Partner APIs from our application servers is a common use-case, a quick summary of options for securing such Server to server API calls

Cloud, Elasticity and Microservices

2017-08-13T22:38:00.002+05:30

Many enterprise customers are being seen talking about moving to the cloud and adopting microservices based architectures. This is especially seen in the Indian BFSI domain. Heres my two cents at how the things are evolving logically and trade offs that all BFSI Businesses should be aware of.

The incentive to move to the cloud
Conventional wisdom is slowly but surely tilting towards moving IT infrastructures to secure public clouds like AWS, Azure and IBM Bluemix. The reasons are many and varied and well established by now, I wont regurgitate the same here again. Irrespective of the particular reason for moving the newer enterprise applications to the cloud, it makes sense to ensure that the underlying applications can "leverage" the cloud based infrastructures to the fullest.

Elasticity is important
The cloud provides for dynamic provisioning of resources like processing power, storage, network bandwidth, etc. It then follows logically, that enterprise applications that are deployed on the cloud should be fully capable of "leveraging" this elasticity of the cloud to "scale out" over resources.
Most importantly since clouds are pay-as-you-go model, if the underlying architecture is able to "scale down" automatically, when customer demand is low, the cloud costs for running applications can be optimized. Such architectures which can optimize operating costs of the cloud are very desirable. Mind you, in the cloud, being able to "scale down" as per demand is not just a good-to-have but rather an imperative that has direct cost implications.

Enter Microservices
In a monolithic application based approach including app servers or ESBs or service orchestration containers, the size of the monolith restricts the extent to which the scale-down can occur. Hence the architectural approach of microservices which are much more granular lends itself to better scale down capabilities, along with various important technical advantages of microservices

Containerization and choices
OK so microservices lend themselves well to cloud deployments, due to their granularity! But managing microservices, monitoring them, ensuring fault tolerance, reliability, scalability, is now a challenging task. Using light weight and proven container technologies like docker becomes inevitable. Docker also provides the much needed portability.
But managing the containers can itself become a complex task. Using a container management tool like docker swarm is good for less complex deployments and if you need to a single out of the box solution for container management. For more complex enterprise wide deployments a open source and portable and pluggable docker management framework like kubernetes is much more practical.
For a quick compare between swarm and kubernetes

In addition, the docker container will need some deep integrations with your cloud provider, remember all this is running on the cloud ;-)
Here too, we have cloud provider based solutions like EC2's Container service which can be considered to be alternatives to kubernetes. But be aware that choosing a cloud provider based container management solution will mean no portability and vendor lock-in with your cloud provider

The architectural shift to microservices, is just the tip of the ice berg when considering your move to the cloud. In subsequent articles I will be talking about other aspects of your move to the cloud, which will affect the way your enterprise applications interact with the world. Some topics I intend to cover are Security and using API Gateways for integrations.

Spark Quick Review : Study Notes

2017-06-11T18:06:00.000+05:30

The main abstraction Spark provides is a resilient distributed dataset (RDD), which is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. RDDs are created by starting with a file in the Hadoop file system (or any other Hadoop-supported file system), or an existing Scala collection in the driver program, and transforming it. Users may also ask Spark to persist an RDD in memory, allowing it to be reused efficiently across parallel operations. Finally, RDDs automatically recover from node failures.

Driver program's in memory array can be converted to distributed data structure using parallelize

List<Integer> data = Arrays.asList(1, 2, 3, 4, 5);
JavaRDD<Integer> distData = sc.parallelize(data);

Spark can create distributed datasets from any storage source supported by Hadoop, including your local file system, HDFS, Cassandra, HBase, Amazon S3, etc.

JavaRDD<String> distFile = sc.textFile("data.txt");

RDDs support two types of operations: transformations, which create a new dataset from an existing one, and actions, which return a value to the driver program after running a computation on the dataset.
For example, map is a transformation that passes each dataset element through a function and returns a new RDD representing the results. On the other hand, reduce is an action that aggregates all the elements of the RDD using some function and returns the final result to the driver program (although there is also a parallel reduceByKey that returns a distributed dataset).

Since RDDs are distributed data stuctures, something special needs to be done to count elements inside RDD.

// Wrong: Don't do this!!
rdd.foreach(x -> counter += x);

rdd.collect().foreach(println) //can give out of memory since all data brought to drivernode

rdd.take(100).foreach(println)

Total Character Count Example

JavaRDD<String> lines = sc.textFile("data.txt");
JavaRDD<Integer> lineLengths = lines.map(s -> s.length());
int totalLength = lineLengths.reduce((a, b) -> a + b);

The following code uses the reduceByKey operation on key-value pairs to count how many times each line of text occurs in a file:

JavaRDD<String> lines = sc.textFile("data.txt");
JavaPairRDD<String, Integer> pairs = lines.mapToPair(s -> new Tuple2(s, 1));
JavaPairRDD<String, Integer> counts = pairs.reduceByKey((a, b) -> a + b);

Filter lines in a file for occurence of string "ganesh"

JavaRDD<String> logData = sc.textFile(logFile).cache();
JavaRDD<String> retData = logData.filter(s -> s.contains("ganesh"));

Different types of transformations and actions
https://spark.apache.org/docs/2.1.0/programming-guide.html

Word Count
JavaRDD<String> words = lines.flatMap(s->Arrays.asList(s.split(" ")).iterator()); JavaPairRDD<String, Integer> pairs = words.mapToPair(s -> new Tuple2(s, 1)); JavaPairRDD<String, Integer> counts = pairs.reduceByKey((a, b) -> a + b);

Shared Variables

Broadcast<int[]> broadcastVar = sc.broadcast(new int[] {1, 2, 3});

broadcastVar.value();
// returns [1, 2, 3]

LongAccumulator accum = jsc.sc().longAccumulator();

sc.parallelize(Arrays.asList(1, 2, 3, 4)).foreach(x -> accum.add(x));
// ...
// 10/09/29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 s

accum.value();
// returns 10

A gentle introduction to java based electronic trading using FIX

2015-11-19T09:45:00.001+05:30

I recently had the chance to work on a java based web application which provides electronic trading functionality using FIX(Financial Information eXchange). Wanted to share some of my experiences in this post :-)

Lets start out with the basics, though...

Electronic Trading the big picture, where does FIX fit in?
Trade life cycle can be divided into the following phases:

Pre-Trade - analytics and price discovery
Trade - order creation and execution
Post Trade - clearing, allocation and settlement
Post Settlement - risk mgmt, profit/loss account, position monitoring

FIX is mostly used in the Phases 1 and 2, whereas SWIFT is used in Phase 3 and beyond.

FIX protocol features like quote(RFQ), Indication of Interest (IOI) are used for price discovery, whereas features like DealAtQuote, Market Order and Limit Order are used for order placement.

How FIX is relevant | Who Uses FIX
FIX® has become the way the world trades. Virtually every major stock exchange and investment bank uses FIX for electronic trading, alongside the world's largest mutual funds, money managers and thousands of smaller investment firms. Leading futures exchanges offer FIX connections and major bond dealers either have or are implementing them. Identifying an exact number of users is impossible, as FIX is a free and open standard, but it is very clear that the world’s financial community now speaks FIX.

Enterprise apps - Using Open Source QuickFixJ, the java library for FIX
Most enterprise applications, act as the BUY side of FIX protocol ie these applications place trade requests, to the SELL side of FIX, which is provided by vendors like winterflood, which have connectivity to stock markets, traders, etc and enterprise applications require to, send FIX messages to vendor's FIX engine.

In order to be able to interface with a FIX engine, of a provider like winterflood, using java code, there is an excellent open source framework called QuickFixJ. This is a java library which can be used as a JVM embedded FIX engine. Detailed documentation can be found at their website.

In following article, the java code I will be show-casing code that uses the QuickFixJ APIs

Intro to common FIX flows
First, your application needs to communicate with a vendor who provides electronic trading services. Such vendors have a FIX server running with electronic trading capabilities and DMA(direct market access).

The whole idea is that our application will send a FIX message asking for a trade to be executed, to the vendors FIX engine. The vendor service will execute the trade and send back a response to our application. The responses are always asynchronous and can come in delayed or not at all.

Such trade instructions between our application and vendor FIX engine, using FIX messages format and protocol are called logically as FIX trade flows.

There are typically huge number of trade flow, sub-flows, combinations, varied responses, but the trick is to narrow down the number of trade flows by pre-deciding with vendor, which trade flows we will consider and also clamp down on the possible requests and responses.

This logical ring fencing of the possible trade flows will ensure, you can provide robust, deterministic functionality related to electronic trading, in your own application.

Quote with Deal At Quote Order
The most common trade interaction which can be done during market hours is the Quote and Deal functionality.

As per this trade flow, user sends out a quote request, for a ticker/symbol in the form of a FIX Message of type R. Asynchronously, but within a few milli seconds, a Quote response is received by the application. This quote response contains the market price of the ticker/symbol as a quote. This price in quote is applicable only for a a few seconds say for 30 seconds

The user now can send another FIX message within 30 seconds, asking to book order for above mentioned stock, with certain number of shares at the price agreed upon in the quote above

A more interactive and verbose narrative is mentioned below, along with the sample FIX messages, good luck deciphering :-)

Quote and Deal
Quote (R) - give me quote for stock with symbol VOD
<20150923-07:47:20, FIX.4.2:APP->VEND, outgoing> (8=FIX.4.2 9=194 35=R 34=2 49=APP 50=D2C 52=20150923-07:47:20.027 56=VEND 115=12345 131=1442994439993 146=1 55=VOD.L 48=GB00BH4HKS39 22=4 207=XLON 54=2 38=10 64=20150925 60=20150923-07:47:20.026 15=GBP 120=GBP 10=009 )

Quote Response (S) - Unit market price for VOD is 2.445 pounds, valid for next 30 seconds
<20150923-07:47:20, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=245 35=S 49=VEND 56=APP 34=2 52=20150923-07:47:20 131=1442994439993 117=X90FZI23H4719001 55=VOD.L 22=4 48=GB00BH4HKS39 167=CS 207=XLON 132=2.1325 134=3750 62=20150923-07:47:50 64=20150925 15=GBP 120=GBP 152=21.325 76=VENDOR 60=20150923-07:47:20 10=088 )

Deal At Quote (D) - book an order for VOD @ 2.445 pounds for 100 shares
<20150923-07:47:20, FIX.4.2:APP->VEND, outgoing> (8=FIX.4.2 9=238 35=D 34=3 49=APP 50=D2C 52=20150923-07:47:20.575 56=VEND 115=12345 11=1442994440565 15=GBP 21=1 22=4 38=10 40=D 44=2.1325 48=GB00BH4HKS39 54=2 55=VOD.L 59=4 60=20150923-07:47:20.575 63=3 64=20150925 117=X90FZI23H4719001 120=GBP 207=XLON 10=153 )

Exceution Report (8) for VOD @ 2.445 pounds for 100 shares with order status=FILLED
<20150923-07:47:20, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=359 35=8 49=VEND 56=APP 34=3 52=20150923-07:47:20 57=D2C 128=12345 1=WINTERG0 6=2.1325 11=1442994440565 14=10 15=GBP 17=b90FDI23H4720002 20=0 22=4 29=4 30=XLON 31=2.1325 32=10 37=b90FDI23H4720002 38=10 39=2 40=D 44=2.1325 48=GB00BH4HKS39 54=2 55=VOD.L 59=4 60=20150923-07:47:20 63=6 64=20150925 76=VENDOR 110=0 119=21.33 120=GBP 150=2 151=0 167=CS 207=XLON 10=244 )

other possible order status are: CANCELLED, REJECTED, PARTIALLY_FILLED, etc

A quote can also be rejected, in which case a FIX message of type b is received by the application

Market Order
A market order is an order request which asks to buy/sell a certain number of shares of a symbol at the best price, at the time of request.

Market Order Request - 35=D order, 40=1 market, 54=2 sell, 55=VOD symbol, 38=110 num shares
<20150923-08:21:39, FIX.4.2:APP->VEND, outgoing> (8=FIX.4.2 9=234 35=D 34=2 49=APP 50=D2C 52=20150923-08:21:39.383 56=VEND 115=12345 11=1442996499366 15=GBP 21=3 22=4 38=110 40=1 48=GB00BH4HKS39 54=2 55=VOD.L 59=6 60=20150923-08:21:39.380 63=3 64=20150925 120=GBP 126=20150925-08:21:39.381 207=XLON 10=110 )

Execeution Report (8) with order status=FILLED
<20150923-08:21:39, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=357 35=8 49=VEND 56=APP 34=2 52=20150923-08:21:39 57=D2C 128=12345 1=WINTERG0 6=2.1325 11=1442996499366 14=110 15=GBP 17=0D0FDI23I2139010 20=0 22=4 29=4 30=XLON 31=2.1325 32=110 37=0D0FDI23I2139010 38=110 39=2 40=1 44=2.1325 48=GB00BH4HKS39 54=2 55=VOD.L 59=3 60=20150923-08:21:39 63=6 64=20150925 76=VENDOR 119=234.58 120=GBP 150=2 151=0 167=CS 207=XLON 10=120 )

Limit Order
A limit order is an order request which asks to buy/sell a certain number of shares of a symbol, when the stock price of the symbol, reaches a certain specified limit price.

Limit Order Request - 35=D order, 40=2 market, 54=2 sell, 55=VOD symbol, 38=110 num shares
<20151009-09:05:22, FIX.4.2:APP->VEND, outgoing> (8=FIX.4.2 9=206 35=D 34=229 49=APP 50=D2C 52=20151009-09:05:22.933 56=VEND 115=12345 11=1444381522933 15=GBP 21=2 22=4 38=110 40=2 44=7.885 48=GB0004082847 54=1 55=STAN 59=0 60=20151009-09:05:22.933 63=3 120=GBP 207=XLON 10=112 )

Exceution Report (8) with order status=NEW (Early ACK for limit order)
Limit orders will be filled only when limit is reached
<20151009-09:05:23, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=296 35=8 49=VEND 56=APP 34=229 52=20151009-09:05:23 57=D2C 128=12345 6=0 11=1444381522933 14=0 15=GBP 17=y6u2743950 20=0 21=2 22=4 29=1 31=0 32=0 37=y6u2743950 38=110 39=0 40=2 44=7.885 48=GB0004082847 54=1 55=STAN 59=0 60=20151009-09:05:23 63=3 120=GBP 150=0 151=110 167=CS 198=y6u2743950 207=XLON 10=065 )

when limit price is reached, Exceution Report (8) with order status=FILLED
<20151009-09:22:41, FIX.4.2:APP->VEND, incoming> (8=FIX.4.2 9=342 35=8 49=VEND 56=APP 34=268 52=20151009-09:22:41 57=D2C 128=12345 6=7.4 11=1444381522933 14=110 15=GBP 17=1040601735 20=0 21=2 22=4 29=4 30=XLON 31=7.4 32=110 37=y6u2743950 38=110 39=2 40=2 44=7.4 48=GB0004082847 54=1 55=STAN 59=0 60=20151009-09:22:41 63=3 64=20151013 76=VENDOR 119=814 120=GBP 150=2 151=0 152=814 155=0 167=CS 207=XLON 10=208 )

The Java Code (finally!)

Quote request and deal-at-quote
Writing java code to submit a quote is as easy seen below

quickfix.fix42.QuoteRequest quote = new quickfix.fix42.QuoteRequest(new QuoteReqID(quoteReqId));

//optionally set header feilds that your Trade Provider may require
quote.getHeader().setField(new OnBehalfOfCompID(ONBEHALF_COMP_ID));
quote.getHeader().setField(new SenderSubID(SENDER_SUB_ID));

//create a new 'repeating group' as per FIX spec
quickfix.fix42.QuoteRequest.NoRelatedSym group = new quickfix.fix42.QuoteRequest.NoRelatedSym();

group.set(new Symbol("VOD"));
group.set(new SecurityID(GB00BH4HKS39 )); //value of ISIN: GB00BH4HKS39
group.set(new IDSource("4")); //4=ISIN, 2=SEDOL
group.set(new SecurityExchange("XLON")); //stock exchange ID
group.set(Side.BUY); //this could just as well be Side.SELL

group.set(new OrderQty(10)); //u can buy 10 shares of ticker VOD
//group.setField(new CashOrderQty(4000.00)); //u can buy shares worth 4000 instead of 10 shares

group.setField(new SettlCurrency("GBP"));
group.set(new Currency("GBP"));
group.set(new TransactTime(new Date()));

quote.addGroup(group);

System.out.println("getQuoteForVEND FIX42 message "+quote);
Session.sendToTarget(quote, sessionId);

Deal@Quote
Writing java code to submit a Deal@Quote is as easy seen below
NewOrderSingle order = new NewOrderSingle(new ClOrdID(orderId),
new HandlInst(HandlInst.AUTOMATED_EXECUTION_ORDER_PUBLIC),
new Symbol("VOD"),
new Side(Side.SELL), new TransactTime(new Date()), new OrdType(OrdType.PREVIOUSLY_QUOTED));

//optionally set header feilds that your Trade Provider requires
order.getHeader().setField(new OnBehalfOfCompID(ONBEHALF_COMP_ID));
order.getHeader().setField(new SenderSubID(SENDER_SUB_ID));

order.set(new SecurityID("GB00BH4HKS39")); //ISIN for ticker VOD
order.set(new IDSource("4")); //4=ISIN, 2=SEDOL
order.set(new SettlmntTyp(SettlmntTyp.T_PLUS_2));

order.set(new SecurityExchange("XLON")); //stock exchange london

order.set(new QuoteID(origQuoteId)); //original quote ID
order.set(new Price(request.getPrice())); //price of previous quote

//order.setField(new OrderQty(10)); //u can buy 10 shares of ticker VOD
order.set(new CashOrderQty(4000.00)); //u can buy shares worth 4000 instead of 10

order.set(new TimeInForce(TimeInForce.DAY)); //can be other values like
order.set(new SettlCurrency("GBP"));
order.set(new Currency("GBP"));

System.out.println("sending out DealAtQuotye FIX42 message "+order);

Session.sendToTarget(order, sessionId);

Market Order
Writing java code to send a market order is as easy as seen below

NewOrderSingle order = new NewOrderSingle(new ClOrdID(orderId),
new HandlInst(HandlInst.AUTOMATED_EXECUTION_ORDER_PUBLIC),
new Symbol("VOD"),
new Side(Side.SELL), new TransactTime(new Date()), new OrdType(OrdType.MARKET));

//optionally set header feilds that your Trade Provider requires
order.getHeader().setField(new OnBehalfOfCompID(ONBEHALF_COMP_ID));
order.getHeader().setField(new SenderSubID(SENDER_SUB_ID));

order.set(new SecurityID("GB00BH4HKS39")); //ISIN for ticker VOD
order.set(new IDSource("4")); //4=ISIN, 2=SEDOL
order.set(new SettlmntTyp(SettlmntTyp.T_PLUS_2));

order.set(new SecurityExchange("XLON")); //stock exchange london

order.setField(new OrderQty(10)); //u can buy 10 shares of ticker VOD
//order.set(new CashOrderQty(4000.00)); //u can buy shares worth 4000 instead of 10

order.set(new TimeInForce(TimeInForce.DAY)); //can be other values like
order.set(new SettlCurrency("GBP"));
order.set(new Currency("GBP"));

System.out.println("sending out MarketOrderForVEND FIX42 message "+order);
Session.sendToTarget(order, sessionId);

Limit Order
Writing java code to send a Limit order is as easy as seen below.
Only change from above code is using the OrdType.LIMIT

NewOrderSingle order = new NewOrderSingle(new ClOrdID(orderId),
new HandlInst(HandlInst.AUTOMATED_EXECUTION_ORDER_PUBLIC),
new Symbol("VOD"),
new Side(Side.SELL), new TransactTime(new Date()), new OrdType(OrdType.LIMIT));

Using the above concepts, frameworks and codes, provides some idea of the electronic trading functionalities, that can be integrated into your java based enterprise application.

Cheers!

REST APIs for Simpler Payment Gateway Integration into your web and mobile apps

2015-08-28T12:00:00.002+05:30

I recently had the chance to integrate the current web application I am working on with a popular UK based payment gateway for credit and debit cards, named Worldpay Online . Carrying out the above integration was surprisingly easy and below are the details.

Our requirement was that our end users should be able to pay for our application services using their debit / credit cards. I embarked on the payment gateway integration journey initially thinking of traditional ways of integration. We did not wish to incur the additional overhead of any PCI DSS compliance that occurs, if you choose to store any card details on your site. Hence, I started considering more traditional approaches like our site will redirect payment requests to the payment gateway hosted "payment pages".

These typically gives the end user the ability to choose their payment method, input their card details and ask for their card to be charged for relevant amounts.

Even worldpay as our preferred payment gateway provider had these options under the integration type of "HTML Redirect". However the main drawbacks of this kind of "payment page" based integration were as follows:

No control and limited customisation of the payment pages
Lack of a strong application context while the user was on the payment pages
The mechanism of gateway provider letting us know that the card was charged successfully made the whole process asynchronous, as it was a post back to our application servers

Well, it turns out there is a much simpler approach available in most modern payment gateways. Using the payment gateways REST APIs. I used the worldpay online REST APIs, to quickly integrate the payment gateway into our application. Here I will talk a bit about my experience using the worldpay online payment APIs.

Though we use the payment REST APIs, we still do not need to have an PCI-DSS compliance. The reason is that we never capture or store the card details on our application. The entire process is broken down into 2 steps:

Step 1: Our application displays inline or as pop-up a worldpay online screen. This worldpay screen captures the card details and gives back a temporary token to the application.

Step 2: The application uses the above temporary token and actually calls the REST payment API. The token acts as a proxy for the card details like card number, expiration date, CVV etc. Application never knows about card details, they are known only to worldpay online!

The best part is that:

Except for a small portion of page which captures card details, rest of webpage is under applications control and hence customizable
The actual REST API call is synchronous so error handling effort is reduced by a huge amount

The actual payment REST call is as simple as shown below

Example Request

curl https://api.worldpay.com/v1/orders
    -H "Authorization:your-service-key"
    -H "Content-type: application/json"
    -X POST
    -d
'{
    "token" : "your-token",
    "orderDescription" : "order-description",
    "amount" : 500,
    "currencyCode" : "GBP"
}'

Example Response

{
    "orderCode": "worldpay-order-code",
    "token" : "your-token",
    "orderDescription" : "your-order-description",
    "amount" : 500,
    "currencyCode" : "GBP",
    "customerOrderCode":"my-customer-order-code",
    "paymentStatus": "SUCCESS",
    "paymentResponse": {
        "type": "ObfuscatedCard",
        "name": "name",
        "expiryMonth": 2,
        "expiryYear": 2015,
        "cardType": "VISA_CREDIT",
        "maskedCardNumber": "**** **** **** 1111",
        "billingAddress": {
            "address1": "18 Linver Road", 
            "postalCode": "SW6 3RB", 
            "city": "London", 
            "countryCode": "GB"
        },
        "cardSchemeType": "consumer",
        "cardSchemeName": "VISA CREDIT",
        "cardIssuer": "LLOYDS BANK PLC",
        "countryCode": "GB",
        "cardClass": "credit",
        "cardProductTypeDescNonContactless": "unknown",
        "cardProductTypeDescContactless": "unknown",
        "prepaid": "false"
    },
    "shopperEmailAddress": "name@domain.co.uk",
    "environment": "TEST",
    "riskScore": : {
        "value": "1" 
    }
}

For more information please visit Worldpay online documentation

Worldpay as a gateway provides various features, (many of which I implemented successfully) like

Recurring Payments
Card-on-File (for re-using previously entered card information)
3d-Secure integration
Refunds
Dispute Defence
Telephone orders
.. and so on

Happy integration, people :-)

Databases for HA and DR

2015-04-08T09:42:00.000+05:30

During a recent participation in a bid, responding to an RFP, I had to wear my ops hat and it was a refreshing change, from my dev endeavours in recent past.

One of the things, that I was required to do was to come with a solution that includes HA and DR. When considering the database options for HA and DR, I had to brush up my basics related to database replications, options and also discuss and validate with other senior tech architects who specialize in ops. Following a brief of the learnings that came out of the experience.

To know the basics of replication, the wikidpedia link for replication is very good. But first some basics.

Background: why database synchronization is required?
Scalability in the middle tier (for example from app servers) is easy to achieve through horizontal scaling, all you would need is mutiple app servers, all symmetrical and fronted by a load balancer.But in the database tier achieving scalability is lot tougher. Especially for data centric applications like OLTP, the database is the single most stressed tier. Though scale-up is an option, it does not provide for a good high availability option. Hence we have to think of 2 or more copies of database, to provide scalability as well as high availability.

Multiple databases can be a tricky option. We can have a cluster of databases all symmetric masters, fronted by a load balancer, but synchronizing data across the databases, is a huge overhead and has potential for failures such as deadlocks, during synchronization.

But to think up other options such a single master, multiple read-only slaves, one must appreciate that, not all data access is equal. If in our application, we can visualize data access as being divided into readonly and readwrite, we may find that only small percentage of application's data access is read write, say 30%. In such cases having a single master database which caters to read-write and multiple slave databases which cater to read-only can be resorted to.

In any case, having mutiple copies of the database and trying to synchronize data across those databases is something you will have to inevitably deal with, either for database load balancing, high availability(HA) or on a more remote note for disaster recovery (DR).

So what are the options for database synchronization?
There are 3 broad options for replication:

Storage based replication
File based replication
Database replication: this is usually supported natively by the specific database

Storage based replication
Active (real-time) storage replication is usually implemented by distributing updates of a block device to several physical hard disks. This way, any file system supported by the operating system can be replicated without modification, as the file system code works on a level above the block device driver layer. It is implemented either in hardware (in a disk array controller) or in software (in a device driver).

When storage replication is done across locally connected disks it is known as disk mirroring. A replication is extendable across a computer network, so the disks can be located in physically distant locations. For replication, latency is the key factor because it determines either how far apart the sites can be or the type of replication that can be employed.

Basic options here are synchronous replication, which guarantees no data loss at the expense of reduced performance and aysnchronous replication where-in remote storage is updated asynchronously hence it cannot guarantee zero data loss.

File based replication
File-based replication is replicating files at a logical level rather than replicating at the storage block level. There are many different ways of performing this. Unlike with storage-level replication, the solutions almost exclusively rely on software.

File level replication solution yield a few benefits. Firstly because data is captured at a file level it can make an informed decision on whether to replicate based on the location of the file and the type of file. Hence unlike block-level storage replication where a whole volume needs to be replicated, file replication products have the ability to exclude temporary files or parts of a filesystem that hold no business value. This can substantially reduce the amount of data sent from the source machine as well as decrease the storage burden on the destination machine.

On a negative side, as this is a software only solution, it requires implementation and maintenance on the operating system level, and uses some of machine's processing power (CPU).

File based replication can be done by a kernel driver that intercepts calls to the filesystem functions, filesystem journal replication or batch replication, where-in source and destination file systems are monitored for changes.

Database replication
Database replication can be used on many database management systems, usually with a master/slave relationship between the original and the copies. The master logs the updates, which then ripple through to the slaves. The slave acknoledges the updates stating that it has received the update successfully, thus allowing the master, sending (and potentially re-sending until successfully applied) of subsequent updates.

Also database replication becomes difficult when it scales up to support either larger number of databases or increasing distances and latencies between remote slaves.

Some common aspects to consider while choosing database replication:

Do you wish to cater to HA only or you want load balancing as well?
What is latency for replicaton? Are participant servers in replication, local or remote(site)?
Are you ready to contend with multiple masters or is single master good enough?
Can you split your database access into read-write and read-only queries?

The actual process of database replication usually involves shipping database transaction logs, from master to slave. This shipping can be done as file based transfer of logs or logs can be streamed from master to slaves. Databases can support "specialized transaction logs" which are solely for purposes of replication and are hence optimized.

If you wish only HA or redundancy, than you can have slave/s in warm standby mode. In this case, master sends all updates to slave but the slave is not used even for read-only access, it is only an up-to-date standby.For HA, using shared storage for active master and inactive standby can be resorted to. But the shared storage becomes the SPOF(single point of failure).

If you intend to have single master and mutiple read-only slaves, either your application must be smart enough to balance, database access as readwrite connections and read-only connections. Some databases like postgresql provide third party middle-ware libraries which bifurcate client's database access and send read-write request to master and read-only request balanced across multiple slaves.

If your application has need to load balance even the read-write requests, then you might think of going in for multiple masters. Many databases refer to this as database clustering or mutli-master support. Here, read-write commands are sent out to one of the masters and masters are synchronized through mechanisms like 2 phase commits. Multi master replication has challenges such as reduced performance and increased probability of transactional conflicts and related issues like deadlocks.

In addiiton, databases can support features like ability to load balance read-only queries, by allowing parallel execution of single query across multiple read-only slaves.

Synchronous replication implies, the synchronizations across slaves is guaranteed at the cost of commit performance against the master

For DR, remoteness introduces high latency, hence log shipping or async streaming of logs, to a remote warm standby is acceptable.

Agile - Tips From Live Projects

2015-03-18T09:57:00.000+05:30

I recently had a chance to learn about a live agile project as part of agile training programme.
Below are some of the tips I picked up, while interacting with the key members on the team.

The Big Picture
The overall release is divided into several phases as mentinoned below.
Notice the SIT and UAT are run independently, towards end of release.
This is time consumimg but does reduce risk of any bugs in release.
This is more relevant for softwares which needs extremely thorough testing.

Pre foundation Phase (1 week)
Foundation Phase (2 weeks)
Sprint 1 (2 weeks)
Sprint 2
Sprint 3
Sprint 4
Sprint 5 Final / Hardening Sprint
SIT Test (1 month)
UAT Test (1 month)
Release

During foundation phase high level estimation done using technique called T-Shirt sizing
(S, M, L, XL, XXL, XXXL). This helps in deciding scope of sprints.

Plan for sprints to progessively gain velocity

Balancing RnD and delivery
Instead of running 2 week Spikes like Sprints, only Spike Tasks are run to balance RnD efforts with delivering usable functionality.

1 Story = Spike Task1 + Task2 + Task3....

Sprint includes a sensible mix of techincal stories and functional stories.
Stubs are used for technical components planned for future technical stories.

Estimation
Stories estimated as story points using estimation poker
Task are always estimated / tracked at hours spent.
There is provision in tool, to estimate task hours and also put in actual hours.
Over several successive sprints, a good average estimate of 1 Story point = xyz hours crystalizes

Sample Story Status:
New
Analyzed
Approved
Dev WIP
Dev Blocked
Dev Done
Test WIP
Test Blocked
Test Done

Dev Done includes code review
Test Done includes show-n-tell, given by tester to convince the BA

Very succint definition of done

Sample DOD for dev complete:

Impact Analysis document done
Code complete
Code reviewed
NFR, performance, security tests pass
Regression tests pass
All known bugs raised by test team resolved

About Code and CI

Release-wise branch in SVN (version control)
No branches per sprint
No parallel sprints
SVN checkin format:
<Release>_<Module>_<Story>_<Task/Defect>: Description
Automated regression testing in place

Other Takeaways:
1.
Very detailed acceptance criteria, Yes No questions, fill blanks, unambigious answers
Is xxx panel visible as seen in logical diagram? Yes/No
Does the table/grid contain 13 rows Yes/No
Quality of story determined by how detailed is acceptance criteria

2.
Story Status like "Test Blocked", if tester cannot test a story, calls up dev immediately or writes email.
All Blockers get mentioned in standup

3.
Always testing team is lightly loaded at start of sprint and overloaded towards end of sprint, to reduce this pain point ....
Have stories at very fine-grained level, eg each story should add up to only few hours of dev task.
This way, dev always has something testable for tester throughout the sprint Idle time for testers (waiting for testable functionality) is reduced.

Rapid development of REST webservices with persistence

2015-02-06T11:08:00.002+05:30

Development of modern business applications, has to become faster and easier, if we are to meet business delivery demands, spanning over ever shorter cycles. Also we are now required to deliver softwares continuosly and incrementally, in an agile manner.

Jump-starting the development process with the right tools and technolgies is the need of the hour.

Rapid development, using currently popular technology, has always been in vogue. Right now exposing middle tier services as REST APIs is all the rage. So how do we develop REST services with persistence, in a rapid manner, with least effort?

I had written an earlier post on grails 2_3 providing out of box support for REST CRUD , writing only a simple java bean (DTO), using the grails framework. But that would mean using an entirely new framework like grails only to exploit its REST CRUD functionality. The engineer in me, thought that was an unoptimized use of grails. Why shoudlnt we have this, out of box REST CRUD using spring, hibernate the technologies that we know best already :-)

Well more recently, advancements made in the Spring Data JPA framework, makes it very very easy to develop CRUD REST services out of the box, writing only the entity classes with JPA annotations.
Well here is how...

Effectively, we will see, how to write only a simple java bean, annotated with JPA annotations for persistence and using spring data jpa and spring boot, we can automatically generate REST CRUD services for our bean / entity. All this using very little code :-)

Create a typical maven java project (not a web project)
eg. using maven-archetype-quickstart

We will use spring boot based spring data jpa and spring data rest

Use a pom.xml similar to below

https://github.com/ganeshghag/RestSpringDataJpa/blob/master/pom.xml

As per spring data jpa, create persistent entity class say Person as a POJO and annotate it with JPA annotations

package com.ghag.rnd.rest.domain;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import javax.persistence.Id;

@Entity
public class Person {

@Id
@GeneratedValue(strategy = GenerationType.AUTO)
private long id;

private String firstName;
private String lastName;
private String email;
private String address;
private String mobile;
private Integer employeeId;

//getters and setters go here
}

Again as per spring data jpa framework, create an interface which will tell the spring data jpa framework, to create hibernate jpa based CRUD functionality with respect to the entity class Person we defined above

package com.ghag.rnd.rest.repository;

import java.util.List;
import org.springframework.data.repository.PagingAndSortingRepository;
import org.springframework.data.repository.query.Param;
import org.springframework.data.rest.core.annotation.RepositoryRestResource;
import com.ghag.rnd.rest.domain.Person;

@RepositoryRestResource(collectionResourceRel = "people", path = "people")
public interface PersonRepository extends PagingAndSortingRepository<Person, Long> {

List<Person> findByLastName(@Param("name") String name);

}

The methods in the interface like findByLastName, impart filtering capability on the the entity queries by entity attributes like lastName.

The annotation below
@RepositoryRestResource(collectionResourceRel = "people", path = "people")
ensures that, the CRUD rest services for the Person entity are auto-magically generated for us. This is the main fetaure, I wanted to publicize in this article.

Thats it, now using Spring Boot and its default sensible configurations, you can just right a class similar to below and you should be able to start up the applicatoin with an embedded tomcat server.

package com.ghag.rnd.rest;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Import;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.data.rest.webmvc.config.RepositoryRestMvcConfiguration;

@Configuration
@EnableJpaRepositories
@Import(RepositoryRestMvcConfiguration.class)
@EnableAutoConfiguration
@ComponentScan //important and required
public class Application {

public static void main(String[] args) {
SpringApplication.run(Application.class, args);
}
}

The application will expose the CRUD REST services for persistent entity Person at
http://server:port/people

you can easily test the services using curl commands like following:
curl -i -X POST -H "Content-Type:application/json" -d @insert.cmd http://localhost:8080/people
where file insert.cmd contains json for the Person object like thus:
{
"firstName" : "Ganesh",
"lastName" : "Ghag",
"email":"some@some.com",
"address":"flower valley, thane",
"mobile":"3662626262",
"employeeId":8373
}

In order to customize the application, writing custom REST controllers is as easy as writing below POJO

package com.ghag.rnd.rest.controllers;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;

import com.ghag.rnd.rest.repository.PersonRepository;

@RestController
@RequestMapping("/custom")
public class MyCustomController {

@Autowired
PersonRepository personRepository;

@RequestMapping(value="/sayHello/{input}", method=RequestMethod.GET)
public String sayHello(@PathVariable String input){

System.out.println("findall="+personRepository.findAll());
personRepository.deleteAll();
return "Hello!"+input;
}
}

To further enhance the CRUD application, we can easily

use above application as a microservice, using spring boot as is
export above application as a war and deploy it on traditional app server
use various logging for debugging
use spring based validations
use spring security
switch CRUD database from current default H2 to any of our choice using spring boot config file application.properties

Please refer spring boot documentation for accomplishing the above mentioned enhancments.
http://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/
http://docs.spring.io/spring-data/jpa/docs/current/reference/html/

By using hibernate tools reverse engineering, given an existing database, auto-generating JPA annotated entities is a no-brainer and with the above knowledge of generating REST CRUD services, it means, you can expose a database, as REST services, with less than a days work.

Database multi-tenancy using Hibernate 4

2014-09-08T14:17:00.001+05:30

With many business applications being written, with intent to be deployed on cloud, there is increasingly a need to design multi-tenancy into your application.

For persistence layer multi-tenant design offers a few options:
http://msdn.microsoft.com/en-us/library/aa479086.aspx

separate database per tenant
separate schema, same database per tenant
same schema, same database for all tenant with discriminator column for tenant rows in same table

In this article, I am going to talk more about implementing the options 1 and 2 using the support in hibernate 4, for multi-tenancy.

Also for the database connections, we have choice of having a common database connection pool, from which connections are allocated to each tenant request, or separate connection pools for each tenant.
For the common connection pool approach, each connection before being used for data access, needs to be "primed" for usage against a tenant database or schema, using statements like "use <tenant_schema>" or "use <tenant_database>"

Multi-tenancy in Hibernate

For the "separate connection pool per tenant approach"

First we need to implement a custom connection provider as follows

import java.sql.Connection;
import java.sql.SQLException;

import org.apache.commons.dbcp2.BasicDataSource;
import org.hibernate.engine.jdbc.connections.spi.ConnectionProvider;

public class ConnectionProviderImpl implements ConnectionProvider {
 
 private final BasicDataSource basicDataSource = new BasicDataSource();
 
 public ConnectionProviderImpl(String database){

                //this should be read from properties file
  basicDataSource.setDriverClassName("com.mysql.jdbc.Driver");
  basicDataSource.setUrl("jdbc:mysql://localhost:3306/"+database);
  basicDataSource.setUsername("myuser");
  basicDataSource.setPassword("mypassword");
  basicDataSource.setInitialSize(2);
  basicDataSource.setMaxTotal(10);
 }

 @Override
 public boolean isUnwrappableAs(Class arg0) {
  return false;
 }

 @Override
 public  T unwrap(Class arg0) {
  return null;
 }

 @Override
 public void closeConnection(Connection arg0) throws SQLException {
  arg0.close();
 }

 @Override
 public Connection getConnection() throws SQLException {
  return basicDataSource.getConnection();
 }

 @Override
 public boolean supportsAggressiveRelease() {
  return false;
 }

}

Next we need to implement AbstractMultiTenantConnectionProvider of hibernate, as follows
Here I am maintaining a map of database identifiers against, connection providers.
When hibernate invokes the selectConnectionProvider( ) method with the tenant identifier, we use the tenant identifier to return the "appropriate" connection provider from the map.
It is necessary to implement a method like getAnyConnectionProvider( ), which should return, a sensible - default connection provider.

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;

import org.hibernate.engine.jdbc.connections.spi.AbstractMultiTenantConnectionProvider;
import org.hibernate.engine.jdbc.connections.spi.ConnectionProvider;

public class MultiTenantConnectionProvider extends AbstractMultiTenantConnectionProvider {
 
 private HashMap connProviderMap = new HashMap();
 
 public MultiTenantConnectionProvider(){
  

  List providerNames = new ArrayList();
  providerNames.add("default_db");
  providerNames.add("db1");
  providerNames.add("db2");
  //need to get above from properties file
  
    for (String providerName : providerNames) {
       connProviderMap.put(providerName, new ConnectionProviderImpl(providerName));
    }
  
 }
 
 @Override
 protected ConnectionProvider getAnyConnectionProvider() {
  System.out.println("inside MultiTenantConnectionProvider::getAnyConnectionProvider");
  return connProviderMap.get("default_db");
 }

 @Override
 protected ConnectionProvider selectConnectionProvider(String tenantId) {
  ConnectionProvider connectionProvider = connProviderMap.get(tenantId);
  if(connectionProvider == null)
   connectionProvider =  new ConnectionProviderImpl("default_db");
  
  return connectionProvider;
 } 

}

For common connection pool for all tenants.

We need to implement MultiTenantConnectionProvider of hibernate, as follows
Since we are using the same connection pool, in getConnection(), we need to prime the connection using a SQL statement like 'use <database>', so that the connection's further use will be in database as per tenant_id.
Also in releaseConnection( ), we have a 'use <default_db>' as a fallback.

import java.sql.Connection;
import java.sql.SQLException;

import org.hibernate.HibernateException;
import org.hibernate.engine.jdbc.connections.spi.ConnectionProvider;
import org.hibernate.engine.jdbc.connections.spi.MultiTenantConnectionProvider;

public class MultiTenantConnectionProviderWithSingleDBPool implements
  MultiTenantConnectionProvider {
 
 private final ConnectionProvider connectionProvider = new ConnectionProviderImpl(CurrentTenantIdentifierResolver.DEFAULT_TENANT_ID); 
 

 @Override
 public boolean isUnwrappableAs(Class arg0) {
  return false;
 }

 @Override
 public  T unwrap(Class arg0) {
  return null;
 }

 @Override
 public Connection getAnyConnection() throws SQLException {
  System.out.println("inside MultiTenantConnectionProvider::getAnyConnection");
  return connectionProvider.getConnection();
 }

 @Override
 public void releaseAnyConnection(Connection connection) throws SQLException {
  connectionProvider.closeConnection( connection );
 }

 @Override
 public Connection getConnection(String tenantIdentifier) throws SQLException {
  final Connection connection = getAnyConnection();
  try {
     connection.createStatement().execute( "USE " + tenantIdentifier );
  }
  catch ( SQLException e ) {
     throw new HibernateException(
       "MultiTenantConnectionProvider::Could not alter JDBC connection to specified schema [" +tenantIdentifier + "]",e);
  }
  return connection;
 }

 @Override
 public void releaseConnection(String tenantIdentifier, Connection connection) throws SQLException {
  try {
   connection.createStatement().execute( "USE default_db" );
  }
  catch ( SQLException e ) {
     throw new HibernateException(
     "Could not alter JDBC connection to specified schema [" +
   tenantIdentifier + "]",e);
  }
  connectionProvider.closeConnection( connection );
 }

 @Override
 public boolean supportsAggressiveRelease() {
  return false;
 }

}

refer hibernate docs
http://docs.jboss.org/hibernate/orm/4.1/devguide/en-US/html/ch16.html

Now, to persuade hibernate to use tenant identifier before any database access, we can

manually set the tenant identifier on hibernate session factory
we can implement hibernate tenant identifier resolver

Manually set the tenant identifier on hibernate session factory
This can be done, in a web application in any suitable interceptor where hibernate session is available

Session session = sessionFactory.withOptions()
        .tenantIdentifier( yourTenantIdentifier )
        ...
        .openSession();

Hibernate tenant identifier resolver can be implemented as follows.
Here I am using a ThreadLocal to get the tenant identifier. The threadlocal itself can be set early on in the http request thread in say, a servlet filter.

import org.hibernate.context.spi.CurrentTenantIdentifierResolver;

public class CurrentTenantIdentifierResolver implements
  CurrentTenantIdentifierResolver {
 
 public static ThreadLocal _tenantIdentifier = new ThreadLocal();
 public static String DEFAULT_TENANT_ID = "default_db";

 @Override
 public String resolveCurrentTenantIdentifier() {
  System.out.println("from inside resolveCurrentTenantIdentifier....");
  String tenantId = _tenantIdentifier.get();
  if(tenantId == null)
   tenantId = DEFAULT_TENANT_ID;
  
  System.out.println("threadlocal tenant id ="+tenantId);
  return tenantId;
 }

 @Override
 public boolean validateExistingCurrentSessions() {
  return true;
 }

}

Finally, the hibernate configuration for specifying that we are using hibernate mutli-tenancy, in hibernate.cfg.xml

That's it people,
Happy multi-tenanting :-)

Cheers!

Using free openshift java PAAS, to run a web application on an online version of tomcat

2014-04-17T12:28:00.002+05:30

Many times you have a war file, that you would like to deploy such that the application becomes available online.A classical use case is using a REST service as stub for your mobile app or thin client web app.

Openshift online PAAS is an easy solution for such free hotsitng of a java web application

Create a free account at open shift online
Download their rhc command line tool
Fire following command to create 1 of 3 free apps, where name of new hosted web app is for example say, restgrails

rhc create-app restgrails tomcat-7

Go to the open shift web console and note down the ssh url of the git repository
setup ssh keys for the open shift git repo using command

rhc setup

If required, place the keys generated in <user-dir>\.ssh into <git-install-dir>\.ssh

fire following command on your local machine

git clone <git_url> restgrails

on your local machine the above git command will create a new directory called restgrails and checkout default web app from openshift git repo to your local machine

cd restgrails

Fire following commands to get rid of the default web app's src and pom.xml, since we will deploy a ready made war file onto the openshift PAAS

git rm -rf src/ pom.xml

git commit -am "deleted default source code"

Now prepare the ready made war file for deploying onto open shift

copy war file to restgrails/webapps/
rename war file to ROOT.war --this is important if you want to map your webapp to the openshift online app URL

git add .
git commit -am "added new root.war to webapps"
git push --force

Above git push command, will trigger a deploy of the war file into the remote tomcat on open shift and soon the web app will become available online as say

http://restgrails-ideafountain.rhcloud.com/
ie
http://<web-app-name>-<domain-name-reg-with-openshift-online>.rhcloud.com/

you can tail the tomcat server logs with the following command
rhc tail restgrails

For databases, openshift online does support databases like mongodb, mysql and postgresql, so if your app uses hibernate, it can easily be configured to use online database like say mysql or postgresql, while running it online

to check logs

Go to openshift web console, and click on Remote access, want to login to your application

Fire ssh command, in textbox on web console, from your dos prompt

Example: ssh acvahdvahdvah@plateletdonation-ideafountain.rhcloud.com

You will be get logged-in to the VM.

Here give following commands:

[plateletdonation-ideafountain.rhcloud.com ]\> find . -name *.log

find: `./.tmp': Permission denied

find: `./.sandbox': Permission denied

find: `./.ssh': Permission denied

./app-root/logs/jbossews.log

./jbossews/tmp/stacktrace.log

[plateletdonation-ideafountain.rhcloud.com ]\> tail -f ./app-root/logs/jbossews.log

Why is enterprise application development different from just programming?

2014-03-26T15:58:00.001+05:30

Any monkey can write code that works!
Writing code that fulfills a functionality is a very important first step in the process of developing an enterprise application, but it is only the beginning, the enterprise application is far from done...

So what else is involved in developing an enterprise application?
As it turns out there are several other aspects of development that have to be considered:

Is the code unit tested with sufficient test coverage?
Is the code thread safe, in the face of multiple users accessing it?
Is the code transactional?
Is the code secure?
Is the code easy to debug? Is it instrumented with logging at appropriate levels?
Does the code handle exceptions and errors?
Is the code optimized for performance and for memory consumption?
Is the code resilient? Can it recover, from a failure?
Is the code easily extensible and maintainable?
Is the code portable across deployment environments?
Are all the configurable properties externalized and easily changed during deployment?
Is the code able to inter-operate with legacy systems to be integrated?

Answering all the above questions to the satisfaction of the non-functional requirements, is equally important. Many developers are aware of all the above aspects of development but completely ignore them, when asked about estimates when they think they can complete a particular functionality or feature of the enterprise application under development :-(

On a side note, even when we as developers decide to write our own reusable component or framework, please take into consideration that all the above aspects need to be built into the component/framework and hence many times it makes better sense to use an opensource component which is popular used or has withstood the test of time.Don't re-invent the wheel :-)

Taking an application from developers machine to production
Taking a functionally complete application from a developers machine upto production is a long walk. Lets see some of the common things that need to be done:

automated integration tests especially for regression testing
scripts for automated checkout, packaging and deployment
automated configuration of application as per environment eg. integration, qc, uat and prod
scripts for application life cycle management - start, restart, stop, application
scripts/tools for application database cleanup and archival
scripts/tools for application log files cleanup and archival
scripts/tools for monitoring application health and sending alerts
scripts/tools for scrapping application logs for exceptions, etc and sending out alerts
scripts/tools for performance and memory monitoring of application
migrating existing database to new application database + testing app against migrated data
scripts for scheduling house keeping work around the application

As can be easily seen, there is a very substantial and important component of "operations" which needs to be undertaken, for successful implementation of any application. All these activities can hardly be classified as typical coding. These need to be factored into time estimates as well as workforce planning

When and why to use NODE.JS

2014-03-25T14:40:00.002+05:30

Node.js !

The new kid on the block, has definitely been turning main stream. So, you must have heard buzzwords like async io, server side javascript, etc getting associated with the node hype. Some remarks from the sceptics as well - you cant handle dynamic javascript with your immature development workforce or node.js is ok for hello-world apps but not necessarily for primetime on the enterprise applications landscape.

Well heres my attempt clearing up the mist, with regards to node.js and related frameworks...

When to use Node.js

As very rightly pointed out by the authors of node.js, it is an excellent fit for DIRT-y applications. DIRT stands for Data Intensive, Real Time.

How do Java EE web apps work

Lets start out with a traditional Java EE web applications hosted on say a tomcat container, when a 1000 users concurrently access the web application, for each user request a thread is spawned by the web container. Each thread, does processing, passing control through, various application layers like view, controller, service, data access to finally hit a database or backend, get the relevant data and bring it back to presentation tier from where it may be formatted like an html response and sent back to the client browser as an Http response. The hundreds/thousands of threads running in the web container, ensure scalability of the application. But each thread contains synchronous method calls. Each thread is held up, while the sync call completes.The best to emphasize this is through a database call such as
ResultData data = personDao.updatePersonDetails(person);
( above call blocks till all DB rows are fetched )

it is not uncommon for many threads in such an application to be waiting for IO either disk io or network io (as in a database call). As a result one can clearly observe that hardly 10%-20% of cpu is utilized on the web tier, most latency is due to threads waiting on IO.

Can the above latency be reduced?

Enter Node.js with its aysnc io

Node.js at its simplest and most fundamental level prefers non-blocking function calls, through incessant use of callbacks. For example the above blocking database call can be simplified logically to the following async call:
personDao.updatePersonDetails(person, function(err, numRowsAffected ){....} );
Above call issues request to database driver for update but does not wait for results, it continues to next line of execution, and when the database update has been made, the callback i.e. the anonymous functiona gets executed, returning back the error if any and the number of rows affected by the updated.

Though the above difference between using sync blocking functions v/s using async functions with callbacks may seem trivial, if we adopt the async non-blocking function calling as the default way of programming soon things start adding up.

Instead of having 1 thread dedicated for an http request and having thousands of such threads, in node.js we have async functions, which do processing in bits and dont hold up a single thread. The node.js model starts exhibiting more scalability, due to non-blocking nature of calls in the processing stack.

Where are the benchmarks?

Well, consider this, apache webserver uses the multi threaded model whereas nginx uses non-blocking IO, it has been proven several benchmarks that nginx with its non-blocking io shows roughly 20% better scalability than apache webserver. So even for serving static content, we know, async io is always going to be more scalable.

Can Node.js leverage multiple cores? Is it fault tolerant?

This was another valid criticism raised against node,js platform in early days of the framework. We need to understand that node.js server has only a single thread available for the application' usage (there may be others for doing the server's internal housekeeping). Due to the above limitation, node.js processing runs on a single core and is also susceptible to outage since one thread crashing can bring processing down for all clients. Thankfully multiple node.js servers can be run on the same machine with a load balancer upfront. Or node.js modules like "cluster", "upstart" and "forever" ensure, node.js servers can be horizontally scaled and will have features like guaranteed uptime and automatic restarts, on crash.

Node.js for realtime apps

Out of the acronym DIRT, we already saw how node.js is a good fit for "data intensive" applications, well its time to talk about one more category of applications, namely "realtime applications" for which node.js would be suitable.
Again lets consider, traditional web apps, which require to show data that changes frequently at the backend. The traditional approach using Http is for browser based javascript clients to "poll" the server using "auto-refresh" tags. This approach apart from loading up the server is also very unoptimized, instead newer full-duplex protocols like websocket, can be used by the servers to push content, messages, notifications to the client browsers, "whenever" the server events occur.

This event driven approach, to pushing from server to client is very compatible with node.js async and non-blocking APIs, hence there is excellent support in node.js based modules like socket.io, for server push. This makes node.js very suitable for web applications that require high interactivity, server push and notifications.

What is node.js and how useful is it to build real life web apps from scratch?

Well node.js core is really a http server and a platform for hosting web apps. The core node.js hence does not include any features which directly are used in the making of a web application. To create a real life web application, one needs a framework with many features like ability to host static content from the server's native file system, user authentication, session and cookie management, templatable views, nagivation and routing framework, binding model and views, error handling, request body parsing etc.
For providing all above functionalities the core node.js needs to have relevant modules added on top of it
Node.js provides a robust module system called npm stands for Node package manager, which helps in installing such node.js modules.
The most critical module which provides pluggable middle-ware is a module called connect.
On top of connect is a module called express.js which provides a full fledged web application framework.
More custom requirements for the web app can be fulfilled by installing appropriate modules freely available via the npm (node package manager).
Similarly, templating engines like jade and ejs can be plugged into frameworks like express.js, thereby enriching the node.js ecosystem for your web application development.

In subsequent articles, I will be walking through developing a basic node.js web application for doing CRUD (create, read, update, delete) into a mongo database. You will be pleasantly surprised how easy and elegant it is to develop web applications in node.js based frameworks.

Cheers!

How to build a cross platform mobile app using only a browser

2014-03-10T09:45:00.001+05:30

Securing your REST API

2014-02-07T11:08:00.002+05:30

With developers exposing application functionality as REST services, willy nilly all over the enterprise, the onus of securing these REST services is on them as well. Here I will discuss the options for means of authenticating your REST services. For authenticating REST services you have the following broad options:

Use HTTP Basic Auth, Digest, Form based or Client Certificates
Use custom token based authentication mechanisms
Use OAuth 2.0 - Two legged or Three Legged Auth

Using HTTP Basic Auth, Digest, Form based or Client Certificates

Since HTTP is the most popular implementation of REST, Java EE based authentication mechanisms hold true for REST service calls as well. Here we have 3 options, as mentioned well in Java EE Security documentation.

To secure REST API exposed at http://<hostname>:<portnumber>/web-app-context/webapi/rest-resource we will use the following configuration in web application's web.xml

 
<security-role>
  <role-name>admin</role-name>
</security-role>

<security-constraint>
  <display-name>SecurityConstraint1</display-name>
  <web-resource-collection>
   <web-resource-name>WRCollection</web-resource-name>
   <url-pattern>/webapi/*</url-pattern>
  </web-resource-collection>
  <auth-constraint>
   <role-name>admin</role-name>
  </auth-constraint>
</security-constraint>
<login-config>
  <auth-method>BASIC</auth-method>
</login-config>

Above, configuration uses the "Basic Auth" scheme, similarly REST based (logical) resources, can be secured using DIGEST, Form Based and client-certificates schemes as well. Please refer Java EE security documentation for further details.

Also, suitable realm can be used to refer to credential stores. If you want to make the actual authentication to be portable across application servers you can use JAAS and write-up your own custom LoginModule and CallbackHandler.

So far so good, securing REST services is like any conventional Java EE web application. But....

Many times, the REST services are consumed by not conventional browser based UIs, but rather by mobile applications based on say android or iOS. In such cases, having a mobile app which needs a re-login after HttpSession timeout of say 30 minutes(for tomcat), can be very tedious and undesirable. Instead most mobile apps allow user to enter the username and password, once and then store it on the mobile device as an app preference.

In order to provide a feature such as above, with mobile clients in mind, we can use "custom tokens with expirations". The idea itself is quite simple.

The first time a mobile user uses the app, he is asked for a username and password for authentication.
This server backend provides a REST API such as /login which takes a Http POST and username and password as part of Http request body.
The password itself can be sent across using encoding such as base-64 (over SSL obviously!)
The server validates the username and password going against a credential store such as LDAP based or ActiveDirectory based. Once the user is authenticated, a unique token is generated by server, the server saves this token against the username, along with say a timestamp and expiry date time and also the token is sent back to the client.
The client stores this token say in HTML5 localstorage and uses it in subsequent REST requests, in a custom http header
For subsequent requests, the server looks at the custom header retrieves the token, validates that it is valid by checking against the username and also ascertaining that it has not expired as per expiry policy.
If token is valid, the request goes through, if token is invalid or expired, the server throws back a 401 error, which makes the client go back to login page for re-login

The above mechanism is depicted in representative code below, to clarify the explanation.

The server side authentication and token check can be done using, a servlet filter interceptor as depicted below:

package com.mastek.opensource.security.filters;

import java.io.IOException;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class SecurityTokenFilter implements Filter {
 
 @Override
 public void doFilter(ServletRequest req, ServletResponse resp, 
   FilterChain chain) throws IOException, ServletException {
  
  HttpServletRequest request = (HttpServletRequest)req;
  String token = request.getHeader("MyTokenField");
  
  //additionally check if the token is valid
  //valid can also check if token is expired as per policy
  if(token == null || token.length() == 0)  
     //send HTTP resp code 401: Unauthorized
     ((HttpServletResponse)resp).setStatus(HttpServletResponse.SC_UNAUTHORIZED); 
  else
     chain.doFilter(req, resp);
  
 }


 @Override
 public void destroy() {

 }
 
 @Override
 public void init(FilterConfig arg0) throws ServletException {

 }
 
}

The servlet filter entry in web.xml is shown below:

 
<filter>
  <filter-name>SecurityTokenFilter</filter-name>
  <filter-class>your.package.SecurityTokenFilter</filter-class>
</filter>
<filter-mapping>
  <filter-name>SecurityTokenFilter</filter-name>
  <url-pattern>/webapi/*</url-pattern>
</filter-mapping>

The ajax request which sends out the token in custom Http header and redirects to login page if server throws a 401 is depicted below:

$(document).ready(function() {
$.ajax({
    type: 'GET',
    url: "my-url",
    dataType: 'json',
    data: "{ 'name': 'somedata'}",
    beforeSend: function (request){
        request.setRequestHeader("MyTokenField", localStorage.userToken);
    },
    success: function(mydata) {
        console.log('data ret='+JSON.stringify(mydata)) 
    },
    error: function(mydata){
        if(mydata.status === 401){
            alert('Redirecting you back to login page....');
            window.location = "login.html";
        }
    }
});
 
});

I have'nt put up the /login API, but its easy to imagine that, as a service which takes in username and password, and returns back a token.

The above mechanism is still suspectible to man-in-the-middle attacks, for which the transport needs to be secured with SSL, any which ways. Libraries for generating hash as the unique token are available in javascript as well as java.

The flexible thing about custom tokens, is that you can put in code on your server side to implement any particular policy for the token expiration. Example tokens can be to expire in a day or in a week. Also, keeping track of active users in your system is easier.

I will leave the OAuth 2.0 two-legged mechanism for a subsequent article, since this one is getting too lengthy as it is :-)

Hope you enjoyed the discussion above, related to basics of securing REST services.

Cheers!

Backbone.js - Getting Started Tutorial

2014-02-04T16:56:00.001+05:30

When I started learning backbone.js, though there were several getting started tutorials, all of them seemed a little aggressive, since they introduced all the backbone features in a single, supposedly simple application. In this tutorial I am going to introduce each of backbone basic concepts incrementally and kind of provide a less steeper introduction to backbone.js for beginners.

Lets jump right in....

We will start out with a bare bones html file, which can be thought off, as the anchor or home page for the backbone application. Lets name it bb-intro.html.

Initial listing of bb-intro.html

Next we will create a "router" in backbone, it is like a navigator component, which maps URLs to views/templates.

Lets have 3 pages(views) in our application, to be able to display these on demand we will define the following routes.

 var AppRouter = Backbone.Router.extend({
   routes : {
    "" : "showPage1",
    "page1" : "showPage1",
    "page2" : "showPage2",
    "page3" : "showPage3"
   },
   showPage1 : function() {
    new AppViewPage1().render();
   },
   showPage2 : function() {
    new AppViewPage2().render();
   },
   showPage3 : function() {
    new AppViewPage3().render();
   }
});

var appRouter = new AppRouter();
Backbone.history.start();

The route entry "" implies the default html page and refers to the following URL
http://<host-name>:<port number>/<web-context>/<path-to-html>/bb-intro.html
or
http://<host-name>:<port number>/<web-context>/<path-to-html>/bb-intro.html#

The entry "page1" refers to the following URL:
http://<host-name>:<port number>/<web-context>/<path-to-html>/bb-intro.html#page1

The entry "page2" refers to the following URL:
http://<host-name>:<port number>/<web-context>/<path-to-html>/bb-intro.html#page2

and so on....I think you get the picture here :)

Next up, is the views like AppViewPage1, AppViewPage2, AppViewPage3, these are as you might have guessed javascript objects defined as follows:

 var AppViewPage1 = Backbone.View.extend({
  template : _.template($('#page1').html()),
  render : function() {
   $('#container').html(this.template());
  }
 });
  
  
 var AppViewPage2 = Backbone.View.extend({
  template : _.template($('#page2').html()),
  render : function() {
   $('#container').html(this.template());
  }
 });
  
 var AppViewPage3 = Backbone.View.extend({
  template : _.template($('#page3').html()),
  render : function() {
          $('#container').html(this.template());
  }
 });

The code for each AppViewPage is quite simple stereotypical, for illustration purposes.
We use the underscore library's api _.template( ) to load a template string
And in the view's render method, we simply assign html content of the div with id "container" using familiar jQuery syntax.

Next up is the definitions of the template themselves. These are html templates containing valid html elements.

<script id="page1" type="text/template">
      <div class="view">
        <h2>This is Page 1</h2>
        <a href="#page2">Goto Page 2</a>
      </div>
</script>

 <script id="page2" type="text/template">
      <div class="view">
        <h2>This is Page 2</h2>
        <a href="#page3">Goto Page 3</a>
      </div>
</script>
 
<script id="page3" type="text/template">
      <div class="view">
        <h2>This is Page 3</h2>
        <a href="#page1">Back to Page 1</a>
      </div>
</script>

Thats it! When you access the above html in browser, it should show the following screens with proper navigation when clicked on the hyperlinks.

Notice that the hrefs in the templates, refer to relative URLs, with values like "#page1", "#page2" and "#page3". This is same as the values we specified in the mapping in our router definition.

Now that we are comfortable with navigating between views, using backbone. Lets turn our attention to features called model and collection provided by backbone.

Lets jump right in by defining a model and collection and try to use the same in our view.

 MyModel = Backbone.Model.extend({
  defaults : {
   title : 'Default Title',
   status : 'Initiated at '+new Date(),
   attrib1 : 'Value1',
   attrib2 : 'Value2',
   attrib3 : 'Value3'
  }
});

 MyCollection = Backbone.Collection.extend({
  model : MyModel,
 });
 
 var myCollection = new MyCollection();
 myCollection.add(new MyModel({title:'Page1'}));
 myCollection.add(new MyModel({title:'Page2'}));
 myCollection.add(new MyModel({title:'Page3'}));
 myCollection.add(new MyModel());

The above code defines a MyModel and MyCollection. We then instantiate a MyCollection object and add 4 MyModel objects with custom titles.

When the template is attached to the view, we need to additionally pass the collection as a template parameter. This can be done as shown in the code below:

  
var AppViewPage1 = Backbone.View.extend({
template : _.template($('#page1').html()),
   render : function() {
 $('#container').html(this.template({mydata:myCollection.toJSON()}));
   }
});

The template itself needs to be tweaked to iterate over the collection and start displaying the passed data parameters. This can be easily done as seen in the snippet below:

 

<script type="text/template" id="page1">
      <div class="view">
        <h2>This is Page 1</h2>
        <h2>Data on page:</h2>
        <% _.each(mydata, function(row) { %>
        <%=row.title%> <br>
        <%  });   %>

       <br><br>
       <a href="#page2">Goto Page 2</a>
      </div>
</script>

Apart from above mentioned features, the backbone model does provide the following important features:

Ability to listen to changes in model
In the model's initialize function, we can listen for changes to specific model attributes as follows:

initialize: function(){
            this.on("change:myattrib1", function(model){
                var newvalue = model.get("myattrib1");
                alert("Changed myattrib to " + newvalue );
            });
}

Interact transparently with a REST service back end
On the model we can invoke stock framework methods like save( ), destroy( ) and fetch( ), which result in HTTP methods PUT, POST, DELETE and GET, getting fired under the hood by the framework against a URL specified in the model's attribute called 'urlRoot'.

Thus there is no need to explicitly use jquery or ajax to make server side requests, the backbone.js model does it for you transparently.

    var UserModel = Backbone.Model.extend({
        urlRoot: '/user',
        defaults: {
            name: '',
            email: ''
        }
    });

    var user = new Usermodel();
    // Notice that we haven't set an `id`
    var userDetails = {
        name: 'Ganesh Ghag',
        email: 'ganeshghag@gmail.com'
    };

    // Because we have not set a `id` the server will call
    // issue a post  /user with a payload of userDetails data above
    // The server should save the data and return a response containing the new 'id'
    user.save(userDetails, {
        success: function (user) {
            alert(user.toJSON());
        }
   })

Support for Model data validation
There is a provision in the backbone model to provide standard callback to validate model data before submission to server and also to raise error( ) callback in case validation fails. This is depicted below:

// If you return a string from the validate function,
// Backbone will throw an error
validate: function( attributes ){
            if( attributes.age < 0 ){
                return "Negative values are not allowed";
            }
},
initialize: function(){
            this.bind("error", function(model, error){
                alert( error );
            });
}

So far we have been able to use backbone to display views, navigate between views and bind and show model data and collections, into the views.

Lastly to attach events to elements in our backbone views is as easy as specifying events attribute of Backbone.View as follows:

SearchView = Backbone.View.extend({

        events: {
            "click input[type=button]": "doBtnClick"
        },
        doBtnClick: function( event ){
            alert( "from inside search button click" );
        }
}

});

With this we have covered most basics about the the backbone framework. You should now be heading towards more advanced tutorials on backbone.

Cheers!

Easy Bootstrap 3 based responsive web design using drag drop

2014-01-16T16:05:00.000+05:30

Designing responsive web sites and user interfaces, which adjust their layouts and sizes depending on the device on which the website is being used, is all the rage now.

Twitter bootstrap is the most popular framework for designing responsive websites. Bootstrap primarily uses CSS3, javascript, jquery and most importantly media queries to make a website responsive.

But having to add all the botostrap css classes by-hand is quite painful and makes developers who are not designers cringe. Just when HTML/CSS builder tools were becoming easy to use, comes this little twist of using bootstrap responsive css classes all over html markup.

Well, just found this little website, http://www.layoutit.com, which allows developers to quickly build basic skeletons of bootstrap 3 based components, using drag and drop, instead of hand coding.

I use it to quickly build a basic responsive page markup, then paste it in my IDE and fine-tune it from there on.

This has definitely taken the pain out of remembering and putting in bootstrap class names by hand.

Do check it out.

This is a good convenience till more sophisticated and free tools for building responsive UIs come up in the market.