RicePlate

noreply@blogger.com (Partha S Sengupta) — Wed, 13 May 2026 08:08:52 -0700

The Rarest Skill in Software Engineering

On the morning of August 1, 2012, Knight Capital Group — then one of Wall Street's most dominant trading firms, responsible for a significant share of all US equity trading — deployed a software update to eight servers. One server was missed. A dormant piece of legacy code from 2003, called Power Peg, was accidentally reactivated on that unpatched machine. No one caught it. When the market opened at 9:30 a.m., that one server began firing millions of erroneous orders into the market. Within 45 minutes, Knight Capital had lost over $440 million. The firm nearly ceased to exist by close of business. It was acquired months later.

Read the post-mortems carefully and a pattern emerges that has nothing to do with bad intentions and everything to do with a missing discipline. The SEC investigation found that Knight had no documented deployment procedures, no kill switch, no circuit breaker logic in the order router, and dead code sitting undisturbed in a production system for nearly a decade — code nobody had ever formally designed out, because nobody had maintained a living picture of what that system actually contained at the class and method level.

The architecture of Knight Capital's system was not the problem. Nobody had ever asked the question that low-level design answers: "When a new flag is set in this router, what code paths does it activate, what objects are instantiated, and what happens if one of eight instances receives a different binary than the others?"

Knight Capital's architecture was sound. It had layers, services, and clear separation of concerns on every diagram. What was missing was a living design of what actually lived inside those boxes — and what happened when one of them misbehaved.

Architecture tells you where things live. Design tells you how things work. The industry has become obsessed with the former and almost completely forgotten the latter.

Fig 1 · The Architecture Illusion vs. The Design Reality

Architecture diagrams show you the map. Low-level design shows you the terrain.

The Mythology of the Three-Tier Diagram

Open LinkedIn right now. Search "system design." You'll drown in diagrams. Load balancers. CDNs. Message queues. Kubernetes clusters. API Gateways. Microservice meshes. They are visually impressive. They communicate scale. They look like engineering.

But ask the architects who drew them: "Inside that 'Order Service' box — what classes exist? What does the constructor of OrderService look like? How is the repository injected? What happens to a partially validated order when the payment service times out mid-transaction?"

The room goes quiet.

📖 Evidence · HealthCare.gov · 2013

The 2013 launch of HealthCare.gov — the US federal insurance marketplace — is one of the most documented failures of low-level integration design in enterprise software history. The architecture diagram was enormous and impressive: dozens of contractors, a data hub, identity management, federal data services. But when engineers from Google were brought in after the catastrophic launch, what they found, according to contemporary reporting, was that the software simply did not "talk" to other software. The front end, built by one contractor, was not wired to the back end built by another. A design decision — requiring users to create an account before browsing plans — created a bottleneck that collapsed the system under even modest load. Stress tests one day before launch revealed the site struggled with just over a thousand simultaneous users against an expected tens of thousands. The architecture meetings had happened. The system design reviews had happened. What had never happened was a careful, end-to-end design of what a single user action — "create account" — actually triggered across every service, every data contract, every handshake between components owned by different teams. The eventual cost of the project exceeded $1.7 billion.

The Three Invisible Layers of Real Design

When I talk about low-level full stack design, I mean three distinct disciplines that the industry has nearly abandoned in favour of framework tutorials and cloud certifications:

1. Frontend Logic Design — Not UI/UX

The internet is awash with Figma screens, wireframes, and UX flows. That is not what I mean. I mean: how is your frontend component tree structured as a runtime object model? How does state flow? Where does business logic bleed into the view layer and why is that catastrophic? How are your API calls orchestrated — sequentially, in parallel, with cancellation tokens? What happens to your component when the network request is in-flight and the user navigates away?

2. API and Service Class Decomposition

A box labelled "Payment Service" tells me nothing. I want to see the interfaces. The dependency graph. Whether you understand the Single Responsibility Principle well enough to know that PaymentProcessor, FraudEvaluator, and ReceiptEmitter are three different design responsibilities that merely cooperate. I want to see where your DTOs are defined and why they are different from your domain entities. I want to see your transaction boundaries marked on a sequence diagram, not just "Service calls Database."

3. Data Structure and Schema Design

Not "we use PostgreSQL" or "we have a NoSQL database." I want to know: why is this a table and not a document? Why is this relationship modelled with a join table and not an embedded array? Where are your indexes and what queries motivated them? A schema is a design artifact of the highest order. It encodes every assumption your team has ever made about the real world. And most teams treat it as an afterthought.

Fig 2 · Runtime Journey: "Place Order" — What Actually Happens

A runtime sequence revealing design decisions — not just that services talk, but HOW they handle every failure mode.

The Skills Gap, In Practice

In the course of my career I have sat across the table from many engineers — fresh graduates and principal engineers with fifteen years of experience alike. I ask a consistent set of questions:

Draw me the class diagram for a service you built recently. Not the architecture — the classes.
Walk me through what happens in memory when a user submits this form, from first keystroke to database write.
Show me how your API layer knows when to return a 200 vs a 422 vs a 500 — and where that logic lives.
Explain the index strategy on your most-queried table and why you chose it.

The pattern is remarkably consistent, regardless of seniority title: most candidates can answer one or two of these questions with real precision. Very few can answer all four. The rest reach for buzzwords: "We used microservices," "We followed SOLID," "We had a repository pattern." Fine — but those are categories, not designs. And this is not a knock on individual engineers. It is a structural indictment of how the industry teaches, promotes, and rewards engineering work.

📖 The Pattern I See Repeatedly

A team builds a well-structured microservices platform. The architecture review passes. The cloud diagram gets the CTO's sign-off. Then production happens. An "Add to Cart" action starts failing silently for a fraction of users during peak traffic. Nobody can trace it — not because the team is incompetent, but because nobody had ever designed what CartService.addItem(userId, productId, qty) actually does at a method level: what it validates, what events it publishes, what it rolls back on failure, and how the frontend's optimistic UI update reconciles with the eventual consistency model. The architecture exists. The design does not. I have seen this pattern — the invisible gap between the diagram and the code — more times than I can count, across banking, retail, healthcare, and logistics.

The Paradox of Modern Engineering: We have more tools, frameworks, cloud services, and AI assistants than at any point in history — and yet the ability to design what happens inside a single service, across a single request lifecycle, has never been rarer. We've outsourced the thinking to the framework and called it productivity.

Fig 3 · What "Order Service" Actually Contains — Class-Level Design

This is what the "Order Service" box actually contains. Every decision here has a consequence at runtime.

Why This Matters More in the Age of Generative AI

Here is where I need to be direct, because this is the conversation the industry is not having.

Generative AI and agentic AI systems are being bolted onto enterprise software at extraordinary speed. But here is the uncomfortable truth, backed by research: AI generates code at the architectural level it is prompted at — and if your prompts are architectural, your code will be architectural.

The security research firm Veracode, analysing AI-generated code across major language models, found that roughly 45% of AI-generated code introduces security vulnerabilities — and crucially, security performance has remained largely unchanged even as models have dramatically improved at generating syntactically correct code. The Cloud Security Alliance notes that AI assistants generate code without inherent awareness of your application's risk model, internal standards, or threat landscape — producing endpoints that accept user input without validation, services that skip authorisation checks, and transaction logic with no rollback design, simply because the prompt never mentioned these concerns.

A Georgetown CSET study further identified that AI code generation models produce code that frequently aligns with the MITRE CWE Top 25 most dangerous software weaknesses — not because the models are poorly built, but because they were trained on the vast body of open-source code that humanity has written, inheriting its insecure patterns alongside its brilliant ones.

📖 The Knight Capital Pattern, Replayed at AI Speed

Consider what the Knight Capital post-mortem teaches us in the age of AI: a dormant piece of code, improperly scoped, reactivated by a flag that wasn't understood — all because there was no living low-level design of what that system actually contained. Now imagine that same scenario, but the code was generated by an AI agent responding to the prompt "add support for the new RLP routing flag." The AI produces working code. Tests pass. The flag exists. The dead code path, invisible without a thorough class-level design review, is never questioned. The architectural description — "router handles RLP orders" — is correct. The design, invisible inside the box, is catastrophic. This is not hypothetical. Researchers have documented AI models hard-coding secrets, bypassing input validation, and confidently claiming their insecure implementation "follows best practices."

The engineers who can catch, prevent, and fix these problems are the engineers who have low-level full stack design fluency. They are the ones who look at AI-generated code and say: "This method is doing six things. That's a design smell. This query has no index hint. This object is mutable when it should be immutable. This boundary is wrong."

The engineers who cannot do this will accept AI-generated code at face value — because they have no framework to evaluate it at the level that matters.

Fig 4 · Agentic AI Systems Need Design Fluency More, Not Less

Every arrow an AI agent follows is a design decision. Without design fluency, you cannot govern what the agent does.

The Disappearing Craft

I started my career in the era of two-tier applications. Client. Server. That was it. And within those two tiers, you had to understand everything. How the form data was serialized. How the stored procedure was structured. How the cursor iterated over the result set. How memory was managed in the client application. You had no choice but to design — because the framework would not save you.

Then came three-tier. Then SOA. Then microservices. Each abstraction layer was genuinely valuable. But each one also created a new category of engineer who understood the layer they worked in and nothing above or below it. The frontend engineer who has never read a SQL execution plan. The backend engineer who has never traced an event through a message queue with a debugger. The data engineer who has never opened a network tab in a browser.

And now, with generative AI, we are adding one more layer of abstraction on top of all of these — a layer that will write the code that crosses all the other layers. The question is not whether AI can generate that code. It can. The question is whether any human on your team can evaluate it.

A generation of engineers is being trained to prompt AI systems they cannot fully read, reviewing code they were never taught to design. This is not a productivity gain. It is accumulated technical debt at machine speed.

What "Low-Level Full Stack Design" Actually Looks Like

Let me be concrete. When I say a team has low-level design fluency, I mean they can produce and reason from:

Component interaction diagrams showing how frontend components pass props, emit events, share state, and call services — with lifecycle hooks and error states documented.
Class-level diagrams for every non-trivial service showing constructor dependencies, method signatures, return types, and the design patterns chosen (Strategy, Factory, Repository, etc.) and why.
Sequence diagrams tied to use cases — not generic "Service calls DB" arrows, but specific: what method is called, what parameters, what is returned, what exception is thrown and caught where, what compensation logic runs on failure.
Data model diagrams with column types, nullability decisions, index definitions, foreign key constraints, and annotations explaining the business rule encoded by each constraint.
Transaction boundary maps showing exactly where commits and rollbacks occur across the call stack, including distributed transactions and saga compensations.

These are not academic exercises. Every one of these diagrams is a conversation — with your team, your future self, and increasingly, with the AI agent you are about to ask to extend the system.

The Opportunity in the Gap

I am not writing this to be pessimistic. I am writing this because this gap is your greatest competitive advantage if you choose to fill it.

The market is producing engineers who can deploy to Kubernetes, configure CI/CD pipelines, and prompt AI tools with impressive facility. It is producing almost no engineers who can sit down with a blank diagram and say: "Here is the class structure of this feature, here is what happens in memory and on the wire when a user triggers this use case, here is why the database is structured this way, and here is every failure mode and how the system handles it."

That person — the one who combines the breadth to see the full stack with the depth to design within each layer — is the person who will govern AI-augmented systems safely. Who will make the right architectural choices because they understand what those choices mean at the code level. Who will look at a generative AI output and know, immediately, what it got wrong.

📖 Story · What It Looks Like When It's Right

There were a few engineers who spoke infrequently in architecture meetings. But when they did speak, it was always the same kind of sentence: "That's fine at the architecture level, but in the service, the validator will be called after the repository open, which means we're holding a connection during validation. Move the validator upstream or we'll leak connections under load." They were right every time. They didn't just see the boxes. They saw what happened inside them. They are worth many architects who can only draw diagrams.

✦

A Call to the Industry

Write more about class design. Write more about sequence diagrams that actually show method calls. Write more about schema decisions and why you made them. Write more about what happens at runtime — in memory, on the wire, in the database — when a user takes a single action.

Stop treating "system design" as synonymous with "high-level architecture." That is the floor, not the ceiling. The ceiling is understanding what happens inside every box you draw, at every level of the stack, under every condition — success, failure, and the messy middle.

In a world where AI writes the code inside the boxes, the engineers who understand what should be in those boxes are not less relevant. They are the last line of defence.

That is the rarest skill in software engineering. It always was. We just used to call it engineering.

If this resonated with you — if you are a developer who wants to build this skill, or an architect who misses seeing it in teams, or a CTO wondering why your AI-assisted development keeps producing systems that are hard to debug and maintain — this conversation is worth having. Share this with your team. Then ask them: "When I click this button in our app, can you walk me through every class, every query, every design decision involved?" Their answer will tell you everything.

Sources & Evidence Referenced

SEC Enforcement Action: Knight Capital Americas LLC, 2013. SEC charged Knight with market access rule violations following the August 2012 trading incident. sec.gov/newsroom/press-releases/2013-222
Dolfing, H. "The $440 Million Software Error at Knight Capital." Case study documenting the root cause: dead code, reused flags, absent deployment controls, and no kill switch. henricodolfing.com
HHS Office of Inspector General Report (2014). HealthCare.gov total project cost exceeded $1.7 billion. System collapsed under minimal load during launch; front-end and back-end components built by separate contractors failed to integrate. oig.hhs.gov
Washington Post (2016). Federal officials received 18 written warnings that HealthCare.gov was mismanaged and off course before launch, including capacity planning failures and deviations from IT standards. washingtonpost.com
Veracode (2025). AI introduces security vulnerabilities in 45% of code generation cases. Security performance of AI models has not improved alongside syntactic accuracy improvements. veracode.com
Cloud Security Alliance (2025). A recent study found 62% of AI-generated code contains design flaws or known security vulnerabilities. AI assistants are unaware of application risk models and omit validation and access controls by default. cloudsecurityalliance.org
Ji et al., Center for Security and Emerging Technology, Georgetown (2024). "Cybersecurity Risks of AI-Generated Code." AI models frequently output code aligned with MITRE CWE Top 25 dangerous weaknesses, inherited from insecure patterns in training data. cset.georgetown.edu

10 Ways to Screw up an IT Project with Technology Madness

noreply@blogger.com (Partha S Sengupta) — Thu, 21 Apr 2016 12:41:00 -0700

Whether an IT project is of development, integration or transformation type, Technology Selection has always been a critical activity for it. It is critical because the technologies selected are used during the entire life cycle of the project and remain as the main drivers for successful completion of the project with desired Quality of Service (QoS) of the product. This activity requires an intense study of the business, functional, non-functional requirements, expected QoS, user requirements and the existing system if any.

Usually this process falls in the area of the technical leads or architects. Selection of right kind of software, technologies, frameworks and tools becomes a boon to a project, provided you have right hands to operate these instruments efficiently. However the main objective is to deliver high quality (QoS) software with desired outcome which can never be compromised during this process.

There are well-known project management issues to screw up projects which are the subjects of project management experts however technology selection is entirely a subject of the techies. Gil Edelman provided a nice and brief overview of technology selection process in his article “How to Choose Your Tech Stack” (http://svsg.co/how-to-choose-your-tech-stack/). Primary factors that drive the technology selection process are

· Scope of Project

· Functional / Non-functional requirements

· Budget for Licenses

· Availability of Skills / Experience

· Political influences

· Strategic vision & Management Buy in

There are very few technical pros-cons that really matter. For example, PHP, C#, Python and Java – all of them work perfectly. The choice is usually driven by budget, strategic vision, political reason or available experience more than anything else.

Quality standards e.g. ISO or CMMi ensure that things are done properly e.g. Right Design, Right Requirements, Right Budget, Right Testing and Right Attitude etc. These standards can ensure whether the process of software manufacturing, testing and delivery are done properly or not. The quality parameters of the actual software system (e.g. Quality of Service) and technology are not the primary focus for them, while Quality of Service is the primary focus area for technology selection process.

However there are situations where wrong technology selection can be disastrous. Having all cutting edge technologies together doesn’t always help projects to fly especially when the technology is under research and experimentation.

1. Poor Statement of Work

When no one is sure about what needs to be accomplished, things start to get fragile from the beginning itself. One of the key ways to screw up a project is to avoid creating a roadmap, definition of project requirements and expectations of all stakeholders in a documented format at the beginning of project and get approval from all stakeholders. This is a common approach as mentioned by Jennifer Lonoff Schiff in her article “15 Ways to Screw Up an IT Project” (http://www.cio.com/article/2384088/project-management/15-ways-to-screw-up-an-it-project.html).

2. Lack of Clarity on Requirement

When functional requirements of a system under development are not understood well, we can’t decide whether a web server or portal server (for example) is required. This is a typical decision point on whether an application is going to be offered over the web or not. There are several basic questions and checklists which may drive us to different kinds of software types e.g. Infrastructure services, Network Services, Messaging Services and Security Services etc.

An application connecting with external systems raises more questions on which transport techniques, protocols and security measures to be used, which might drive us toward an ESB or set of connectors or adapters.

If the business logic is expected to be configurable such as a rule, a business rule engine is helpful.

Certain questions usually get their answers from the Application Architecture of a system. TOGAF has rightly depicted the dependency of Technology Architecture on Application Architecture (http://pubs.opengroup.org/architecture/togaf9-doc/arch/chap12.html). Without diving deep into the functional requirements, arriving at decisions in these areas is next to impossible.

Apart from functional requirements, identification of Quality of Services (QoS) and non-functional requirements play a critical role in technology selection.

The top most priorities of systems like Google search, Twitter, Facebook have been speed, parallel processing, security and processing of gigantic volume of data. If we have similar requirement, technologies like big data and faster real time processing can be considered, otherwise it is an overkill of effort such as putting a Jet Engine in a bicycle.

Finally the end user requirement means a lot. I have seen systems with full functionality being rejected by users due to the lack of user friendliness. A study of user communities, their requirements, ease of use and comfort helps greatly in deciding whether to go for computer, tablet, mobile or other devices. Sometimes experienced users with mainframe backgrounds might not like the idea of playing with their mouse on a beautiful web responsive GUIs. Popular user interfaces can be a way to screw up the success here. Managing comfort and expectation is the key to success.

3. Too Many Stakeholders

Projects with multiple stakeholders (including customers, management, end-users and technical architects or leads) really need the best of luck to survive. In this situation applying technical governance is a big challenge. Starting projects of this type without a technical governance framework is the surest way to screw up.

Presence of multiple stakeholders usually requires the technology stack to be decided and reviewed by many.

Stakeholders with their various backgrounds and experiences possess different perspectives and try to help out during technology selection process by contributing their thoughts. Managing and convincing them efficiently can obviously save the project from being doomed. This is a sensitive task which should be handled with care by considering the sentiments, interests, importance and political angles. TOGAF nicely describes Stakeholders Management as one of the key areas (http://pubs.opengroup.org/architecture/togaf9-doc/arch/chap24.html).

4. Personal Aspirations & Technology Madness

Developers always enjoy learning new technologies (I am confessing). Most of us are or were developers. Using young developers’ mindset and perspective during technology selection is another way to screw the project. Learning new technologies for fun and applying them in real project deliveries have different risks involved. Not all cutting edge technologies are required to achieve desired outcome from the system under construction. Sometimes they might appear in your nightmares when you are suffering from lack of support and newly reported bugs.

In 2010, Twitter’s search engine was re-developed using Apache Lucene by replacing MySQL. Again in 2011, they shifted from Ruby-on-Rails to JVM for better performance and scalability. So analyzing the pros-cons, finding out the availability of support and forums are mandatory activities before jumping for any new technology. After all, the customer will not wait till you find a new solution or fix a bug in your tech stack. Therefore technology madness is a strong driver behind immature decisions.

5. Poor Communication

Mr. Architect has defined all technology layers with latest technology components, discussed in meetings and prepared slide decks to share his thoughts. But designers and developers are not sure how to use them in code.

Well I might love to use Spring Framework annotations for dependency injection (DI) to avoid the pain of defining the beans in appContext.xml but another developer might not have that knowledge, hence we end up with some beans defined in xml while some are auto wired with annotations in the code. Eventually the team which gets to manage the code later will be in a soup.

Similar situations can arise while using a message oriented middleware (MoM). I am using my SOAP objects over HTTP transport while my friend likes JSON over HTTP. Therefore, defining a descriptive document with all areas covered is the primary task of an architect. Apart from that, the architect should hang around with the developers to ensure the architecture standards, patterns, technology best practices are being followed and enforced properly.

6. Lack of Skills & Experience

A very crucial point, lack of skills and required experiences to define and use technologies can work as a knife through the butter, when comes to screwing up a project. This applies not only for developers but also for the architects. An architect without knowledge of latest technologies and their impact on the target system is really helpless in technology selection process. Upgrading skills and knowledge can be a savior in situations like this.

The architect must provide enough rationale and logically proof the suitability, capability and applicability of the selected stack with respect to the requirement of the system under construction. She should also list out all architectural goal and constraints (aligning with the technical vision and strategy) and show how the selected stack is supporting the goals and constraints.

Lack of developers with required skills is the most critical issue that can easily screw a project. Proper screening, recruitment of technical members and training them can help on this to a large extent, however to touch the seventh heaven, experienced professionals are your only instruments. This is where the technology stars and heroes are born to save the project.

7. Compromising on Quality of Service

Quality of Service (QoS) is a significant factor which drives the technology architecture of a system, especially at the age of component or service based architecture. The parameters of QoS e.g. Performance, Availability, Scalability, Security, Serviceability, Reliability, Maintainability, Evolvability, Survivability, Adaptability and Reusability etc. are considered as the measure of a highly efficient and robust system.

Decisions to apply architectural styles such as SOA or Microservices, architectural and design patterns can help in reusability, serviceability and adaptability.

Cassandra database is failsafe, highly scalable, specialized in fast insertions, however query executions with joins (such as in SQL) is not a sweet spot for Cassandra. For new queries, you might need to copy data to a matching data structure on a SQL database. Shifting design paradigm from a relational background to the world of name/value or column families is not an easy job.

To avoid bumps ahead, sometimes the QoS of the system is compromised with the technologies or tools in our comfort zone. Usage of legacy technologies sometimes restricts the system to leverage improved features from the cutting edge technologies, speedy development life cycle.

On the other hand, using all latest technologies with lack of experienced developers and lack of support can surely screw up the maintainability of a project.

8. Technology Life Cycle

The architect can easily ensure the birth of a system with a bunch of outdated technologies by selecting the tech stack with technologies which are out of life (such as the legacy ones from history which can only find few old hands e.g. Visual Basic, Power Builder, COBOL, Sybase etc.).

On the other hand, selecting the alpha, beta, early release or latest versions of software tools can put the system in great risk due to lack of support and unknown bugs.

Using the previous version of software than the latest can be a safer option. You can check out the defect lists from the release notes before doing so.

9. Lack of Compatibility with Strategic Vision

Business or enterprise strategies are usually established as a result of political influences, business vision, governance policies and achieve specific business targets such as profitability, cost cutting, promotions etc. Technology strategies are driven by the enterprise strategies. In fact, most of the software development, transformation, integration and implementation projects are kicked off as per business strategies and plan. Most of the large scale transformation, migration, rationalization and modernization projects are aids to cutting cost on maintenance.

One will surely not migrate to an IPv4 network when the strategic vision demands for a rapid growth preferably to an IPv6 network.

Selection of right kind of technology stacks is a sensitive issue. It requires study of entire enterprise landscape, business processes, features, interfaces and standards to be able to define the target technology stack so that redundant applications are decommissioned or consolidated to the strategic stack. Poor decision making on tech stacks takes its toll hugely on these projects.

10. Budget and Cost

The budget and cost of software licenses and cost of human resources are two major factors that influence the technology selection process. For example, Open Source software is a popular instrument for cost cutting however maintaining them can be a pain due to lack of managed services. We need to arrive at an optimum balanced figure between cost and maintainability here.

Finally, it is suggested considering the Technology Selection process with required care by realizing its importance and effectiveness at the beginning of IT projects. It should be initiated by identifying significant factors, drivers and stakeholders followed by a proper study and evaluation of technologies. Preferably performing proof-of-concepts of technologies (once decided) can give us better peace of mind.

WaveMaker: Rapid Web App Generator

noreply@blogger.com (Partha S Sengupta) — Mon, 5 May 2014 02:10:00 -0700

Now-a-days web applications are demanded like customers buying goods from the retailers' shops. In fact companies like Google, Apple, Microsoft and others have already introduced concepts like application market or store on mobile and other areas as well. Software applications are no more expected to follow a baby-birth process these days which demands for a high speed rapid application development or generation. During the last few years MDA and other paradigms have attracted us toward meta-programming and different code generators. But they were not matured enough to generate a full-fledged web application so far.

Looking at the future demand and technology trend, I was in search of such a web application generator which can generate simple web applications for database CRUD (Create, Read, Update, Delete) operations. I was excited with Naked Objects (NO) framework
(http://www.nakedobjects.org/) and anticipating a vital outcome on this direction. Eventually Apache Isis (http://isis.apache.org/index.html) attracted me since this is a Java based implementation of NO. It uses But the project is in incubator and has no definite road map for release and the product has some areas still grey. There is another implementation of NO on .NET known as NuGET. Hope readers will pardon my Java inclination here.

Apache Isis is a framework for rapidly developing domain-driven apps in Java. Write your business logic in entities, domain services and repositories, and the framework dynamically (at runtime) generates a representation of that domain model as a webapp or as a RESTful API. Use for prototyping or production. Apache Isis follows the hexagonal architecture pattern (also known as the onion architecture or ports-and-adapters architecture). It places the domain model in the middle, with presentation, persistence and other services dependent upon the domain model. Though Apache Isis uses a very promising architecture with latest Open Source frameworks like Wicket, JPA, Hibernate and RESTFul communication but the amount of change to be made to enable it for professional usage is little de-motivating.

On continuation of my search someone referred WaveMaker ( http://www.wavemaker.com/) to me. While exploring WaveMaker (WM), ultimately I felt like I found something which falls in line with my hunt. WM is easy to deploy and learn. They have nice videos and tutorials to jump start development. The nice and browser based integrated development environment (IDE) for development and deployment. Most of the well known databases can be integrated with the IDE with export and import features. The IDE offers options to develop entity objects that can be engineered forward onto database tables. Also when the existing tables can be imported to WM, it generates entity objects automatically. WM uses DOJO for frontend code generation and hibernate for object relational mapping (ORM).
It can also link with LDAP or database for user authentication and authorization. It uses a panel with DOJO and HTML elements to be use on the GUI by simple drag-drop feature. Each element has its own property panel that reminds me of IDEs like JBuilder, Visual Age, Power Builder and Visual Studio. It uses Tomcat as an embedded server to run the IDE and test run applications. WM provided limited numbers of GUI templates with pre-generated style-sheets which can be further modified in the IDE.

Integrated Software Development Factory: An Idea

noreply@blogger.com (Partha S Sengupta) — Wed, 9 Oct 2013 06:48:00 -0700

Software Development: The Process

Software development and delivery process is the backbone of business for a professional organization in software industry. Software development is a team activity involving more than one professional with defined set of roles and responsibilities. The involvement of multiple people in the process requires a well defined work flow to be maintained. Apart from human resources the software development process (SDP) typically requires development software tools, platforms and hardware infrastructure and most importantly a strong collaboration mechanism within the team involving the artifacts of the delivery. Ideally a software development and delivery process (as per the water falls model) involves certain technical steps or phases as briefly described below. Phases including financial, business and management activities such as scoping, estimation, planning are not included here.

Requirement Study & Analysis: Where a person (Requirement Analyst) interacts with the user or customer to understand the process and requirement that the software (to be developed) should implement. The information captured in this phase is generally captured in Software Requirement Specification (SRS). Test cases to validate the system (after development) are also defined following this phase.

Architecture & Design: Specifying and documenting the architecture and design for the software system to visualize the basic structure or model which should be followed to construct the software system. This phase can have two primary steps such as -

· Defining the business architecture or functional design by covering the business processes and logic

· Defining the technical architecture or system design explaining the structure of system components required to implement the business processes.

Design and Architecture specifications are documented in this phase.

Development & Coding: In this phase, different software components are built or developed according to the design specified in the previous phase and tested. The outcome of this phase is source code and executable code.

Deployment & System Testing: All components developed are put together and tested to evaluate the ability of the system to provide the functionalities specified during the requirement capturing phase. Test cases defined during the requirement capturing phase are executed in this phase and test results are captured and defects found during testing are fixed.

User Acceptance: Once system testing is completed the software is submitted to the customer for final acceptance. This phase ideally have customer testing involvement. Defect fixing activities (if any) are also involved in this phase.

Operation & Maintenance: Once the software is accepted by the customer, it is finally delivered and installed for final use. In case any change in requirement takes place or any problems are identified by the customer after the delivery is done, necessary changes to implement new requirement or to fix problems are implemented in the system. In this case a new waterfall (starting with Requirement Analysis phase) can be followed for each new requirement or defect. Please refer to the waterfall model at http://en.wikipedia.org/wiki/Waterfall_model.

Monitoring & Tracking: Overall monitoring and tracking activities for the entire project are performed by a project manager (PM) throughout the phases of a software development lifecycle (SDLC).

The steps mentioned above are depicted in figure 1 below.

Figure 1: Software development lifecycle phases

Pain Areas in the Process

Æ Majority of employees in a software development organization (SDO) are involved in the process of software development for customers. The development activities (projects) are performed by forming project teams consisting of different people. Typically a lead or manager is involved to monitor and track a project. Apart from the project team members (PTM) other stakeholders such as reviewers, users and customer also get involved throughout the lifecycle of a project. Nowadays most of the stakeholders come from different geographic locations even with different time zones. This is the reason why a project requires a strong collaboration mechanism to be in place.

Æ Usage of standard practices, software tools and frameworks increases the efficiency of the software under development by reducing defects and improving performance, flexibility, extensibility etc. Enforcing standards and best practices over a team of people can be a challenge without a centralized control and strong collaboration.

Æ The process of requirement study & analysis does not only involve human interactions, it can also involve several documents and pre-existing software systems as well. Sometimes existing source code from the previously existing system can also be taken as the input for requirement. Recovering the design and logic from an existing system or source code might need a reverse engineering mechanism to be in place.

Æ Proper event tracking technique always helps in getting everyone in a team notified about the latest updates and help them to react promptly.

Æ The next problem commonly faced is increasing gap in communication which often results in a big difference between the requirement and the development software system. Figure 2 illustrates the same with a popular cartoon. Usually this problem leads to wastage of investment and following rework.

Figure 2: Gap between requirement and implemented system

Æ Misunderstandings in certain areas of requirement between a person with and without software background create major problems in communication as illustrated by the famous Dilbert Cartoon in figure3. Ideally a predefined flow of communication and precisely designated requirement capturing interface with relevant questions and clarifications can help in reducing such gaps. Educating the stakeholder (without software background) a little also helps here.

Figure 3: Misunderstanding in requirement capturing

Æ Communication Gaps can exist between the requirement analyst (who understands the requirement from the customer or user) and the architect or designer which can be minimized by recording requirement with all necessary information which can help the architect or designer. Similar gaps can exist between designer and developer which lead to defects introduced during development phase.

Æ Sometimes manual effort spent in build, deployment and installation procedures leads to human dependency. This process can be made automatic and time-driven with the help of a Continuous Integration (http://www.thoughtworks.com/continuous-integration) technique.

Æ Rather than putting the test cases in document only and executing manually during the testing phase, Test Driven Development (TDD) Tools can be used to record the use cases and execute them automatically by integrating the same with Continuous Integration tools. Martin Fowler’s article (http://www.martinfowler.com/articles/continuousIntegration.html) on Continuous Integration explains this more clearly.

Æ Overall a central repository with version control mechanism is a very important component to keep all artifacts which are referred (e.g. customer supplied documents) and produced (e.g. SRS, Design Document, Source Code etc.) during a project secured.

Goals to Achieve

The primary goal of software development process is to produce deliverables according to requirement with expected quality. The artifacts produced by a project which can be treated as deliverables are usually the SRS, architecture document, design document or source code etc. The final goal of a software development project should ideally be a working piece of software meeting customer’s requirement.

During SDLC, injection of defects is inevitable. Defects are introduced in different phases of a software development project such as Requirement Analysis, Design or Development. The impact of defects on delivery process and cost to fix them is directly proportional with time. The bugs discovered at the end of the SDLC would cost more to get fixed. So it is better, to find problems in the initial state of SDLC and fix them at the earliest. Based on a study (https://buildsecurityin.us-cert.gov/articles/best-practices/code-analysis/business-case), it has been found that, the savings potential from early defect detection is huge; nearly 60 percent of all defects usually exist by design phase. Figure 4 shows how the cost to fix a defect increases with time.

There are certain techniques such as Pair Programming (PP), Continuous Integration (CI), and Test Driven Development (TDD) etc. which are found to be highly effective in catching the bugs early. While we think of an integrated software development environment (ISDE) as a solution, these techniques need to be considered as well. So we are assuming that, our solution will be equipped to facilitate such techniques. For this article, let’s assume a name for the solution as Integrated Software Development Factory (iSDF).

Figure 4: Increasing Cost to Fix a Defect with time

Integrated Software Development Factory (iSDF): In Search of a Pain Killer

A discussion on so many pain areas together is enough to get a strong headache for anyone. So if we quickly look for a pain killer, a solution to address all primary pain areas in developing a software solution or system, we need to map the resolutions of the pain areas as use cases to the pain killer. Let’s name this pain killer as Integrated Software Development Factory (iSDF). Typical use cases for iSDF deduced from the pain areas described above are the following.

· Submit Requirement: The software end-user or customer should be able to enter his / her requirement in our iSDF using a graphical user interface. The customer should also be able to submit associated documents supporting the requirement. The requirement will be stored in system and later referred by other involved in the project team. Once a requirement is submitted, all relevant stakeholders such as requirement analyst, PM and architect should be notified to take necessary actions to process the same.

· Study & Analyze Requirement: The requirement analyst should have an interface to view the requirement statements and supporting documents entered by the customer for analysis. The analyst should also be able to interact with the end-user / customer for clarifications on the requirement. The analyst should be able to produce and submit an SRS document according to pre-defined format or template. Once the SRS is submitted, required stakeholders such as PM and architect should be notified to take necessary actions to process the same. iSDF should capture requirement in predefined format such as use case or user story so that other PTMs can refer to the same following a specific standard and paradigm (e.g. Object-Oriented Paradigm). Specially, the designer should be able to refer to the use cases or user stories by using a design tool integrated with iSDF. This will help in reducing the interpretation gap between requirement and design.

· Define Architecture: iSDF should provide an interface for the Architects to view SRS and define the architecture for the software application based on the SRS submitted by the analyst. Relevant project members such as PM and architecture reviewer should be notified while architecture is defined and submitted in iSDF.

· Review Architecture: Once the architecture is produced, a technical reviewer should be able to review the architecture and provide comments on the same. The architect will incorporate changes suggested by the reviewer (if any) and update the architecture document. Proper notification mechanism (to PM and Architect) should be in place on review comments submission.

· Define Test Cases: iSDF should provide an interface for the Testers to view requirement and define required test cases for software application based on the SRS submitted by the analyst. Relevant project members such as PM, analyst and customer (if required) should be notified while test cases are identified and submitted in iSDF.

· Review Test Cases: Once test cases are produced, a reviewer (ideally analyst or business users) should be able to review the test cases and provide comments on the same. The test case creator will incorporate changes suggested by the reviewer (if any) and update the test cases. Proper notification mechanism (to PM, analyst and other concerned members) should be in place on review comments submission.

· Define Design: Once the architecture is finalized after review, the designer should be notified to start producing the software design. iSDF should provide an interface for the designer to view the requirement (in form of use case or user story), architecture and supporting documents such as SRS and architecture document. The designer should be able produce the software application design by using some design tools integration with iSDF and submit. Once submitted iSDF will store the design and relevant documents and notify the PM and design reviewer.

· Review Design: Once the design is created, a technical reviewer should be able to review the design and provide comments on the same. The designer will incorporate changes suggested by the reviewer (if any) and update the design document. Proper notification mechanism (to PM and designer) should be in place on review comments submission.

· Develop & Build: The next phase to design is development and build, which primary includes the generation of source code in desired programming languages, customization and configuration. The developers should be provided with proper interface (in iSDF) to be able to build or develop software applications to work in this phase.

Integrated development environments (IDE) like Eclipse and NetBeans are few examples of development tools which can be integrated with iSDF so that, design models (e.g. UML) can be viewed and referred using the IDE. Code generation options can also be integrated to generate code from the design models whenever required. The source code, configuration files and other associated artifacts produced during this phase should be stored in a centralized repository secured by a version control system such as Git, SVN and CVS etc. so that everyone in a development team should be able to share and update the source code as required.

An automatic build tool can be integrated with iSDF so that, the build process can be scheduled in a periodic manner or based on an event (such as on any update in source code repository). Usually this feature is available in typical Continuous Integration (CI) tools like Hudson, Apache Gump and CruiseControl etc. Once triggered by a time-driven or a event-driven approach, the scope of activities for a CI tool should be to check out the source code and configuration files from version controlled repository, compile and build the same, deploy the application on an execution platform e.g. an application server for a test tool to start testing.

Technical reviewers should be notified on any code change or repository update so that, the new code change can be reviewed.

Figure 5: iSDF use cases

· Review Code: Once a new code or modified piece of code or other artifacts are submitted in repository, a notification should be send out to assigned reviewer to review the new or modified artifact and provide comments or approve the change. Once the comments or approval is posted the concerned developer or changer should be notified by iSDF notification mechanism. For any clarification, an integrated collaboration mechanism can be used by the reviewer and changer if required. To support required activities during the review, a code review interface, a notification, a collaboration mechanism and workflow should be in place.

· Test: This is an important phase in a SDLC. iSDF should have an interface for the testers to produce test cases associated with the requirement of a project during the requirement analysis phase. Once the development is completed, testers should be able to configure the test cases in automated testing tools integrated with iSDF. For example, Fitness is a very useful wiki based tool to implement Test Driven Development (TDD). Such tools can be integrated with the CI layer (within iSDF) so that, build, deployment and testing can be automated removing human dependencies. Once test results are produced, a notification mechanism can circulate the results to concerned parties (e.g. developer, project manager and tester etc.) for further actions.

· Defect Tracking: Defects can be recorded and monitored using a Defect Tracking tool such as Bugzilla, Jira, Trac, Traxis and Mantis etc. In order to do this, iSDF should be integrated with a Defect Tracking Tool. While defects are introduced concerned parties can be notified by the system for further actions.

· Deploy & Install: Different versions of application execution platforms or environments for development, testing, staging and production can be set up and integration with iSDF so that, once the application is developed and tested is can be easily deployed or installed on a designated execution environment. Developers can use a development environment to deploy and perform unit testing on their code. On completion of development and code review, individual code changes can be integrated and deployed on a testing environment for the testers to start testing. Once the testing is over, the tested application can be deployed on staging environment for customer’s acceptance testing or review.

· Acceptance by Customer: Our iSDF solution should also provide an interface for the customers or business users to test and review the completed application and provide final acceptance or sign off. Typically, a staging environment and an interface to report defects are provided for customer acceptance testing. The defect reporting interface should have notification capabilities to keep the concerned parties informed.

· Monitor & Track SDLC: Apart from an event based notification mechanism, iSDF should also provide a facility so that the entire SDLC followed in a project should be monitored and tracked by a project manager and other concerned stakeholders on a regular basis. Usually the progress of a project, issues and defects are the primary items of interest for monitoring.

· Administration: Capabilities like User management, user authentication, authorization, project creation, project member allocation can be implemented as administrative activities required for iSDF.

Recipe of a Pain Killer: The Solution

In order to implement the abovementioned use cases certain technical capabilities will be necessary for our iSDF. These capabilities (listed below) together can be considered as a good recipe for iSDF.

iSDF should have a Web based Graphical User Interface or portal to be able to provide all user interfaces required for requirement submission, requirement analysis, collaboration and document management.

A Workflow Management Framework will be required to manage workflows and human interactions in terms of requirement analysis and reviews etc.

A Software Design Tool or interface should be integrated with iSDF so that, the requirement documents or SRS can be referred by the designer and design the software accordingly. Ideally a use case model prepared by the requirement analyst should be captured and shared among relevant users within iSDF in a predefined format so that the design tool can interpret the same as the starting point of design. Apart from preparing design models (e.g. class diagram, sequence diagrams and activity diagrams etc.), the designer can also prepare a design document and store the same in centralized repository of iSDF. The centralized repository can be controlled by version control systems to secure the documents.

A Code Generation feature can be added to iSDF to generate initial code structure from design. For example, once the designer completes class diagrams for an application and executes the code generator on the same, the code generator should be able to read the class diagrams and generate initial source codes in desired programming languages (e.g. Java, C++, C# etc.). Once the initial source code structure is generated, they can be stored (checked) in a version controlled repository for reference later by the developers.

Development Tools or environments can be integrated with iSDF, where developers can check out the existing source codes from version controlled repository, modify the code, compile and check in the modified code back in the repository. Once there is an update in source code repository, an automatic event or notification should be generated in iSDF that should trigger the build process (through a CI tool).

A Reverse Engineering facility of iSDF can help in generating design models from code (in case existing source codes are referred as requirement artifacts). For example, class diagrams can be generated from Java source code.

A Reusable Component Repository (containing standards, frameworks, reusable code etc.) should be available with iSDF for the designer, architects and developers to refer to and reuse during architecture definition, design and development.

An Email Notification mechanism to notify members involved in a project or administrator whenever there is a change in repository. For example, the PM and requirement analyst should be notified whenever the customer or business user submits a requirement. Similarly architects, designers should be notified whenever the SRS is finalized and code reviewers should be notified once there is an update on source code.

A Collaboration Framework for the geographically distributed members to collaborate within each other during a review, analysis and general discussions. The mode of communications can be implemented through chat, email, desktop sharing and video chatting etc.

A Continuous Integration platform or tool to automate build, testing and deployment processes should be implemented in iSDF. This is make the process of compiling, building source codes and deploying application packages on desired execution platforms e.g. application or web server etc. Once an update is made on source codes by the developers and automatic event will be generated which will trigger the CI tool to start the build and deployment process. Figure 6 depicts a typical continuous integration process.

A Build Tool should be provided as an integrated feature of iSDF. This tool should ideally be integrated with the CI component so that, whenever there is an update in code repository this tool can be automatically invoked based on an event. On invocation, the tool should be able to check out the latest source code changes from the code repository, compile, build and deploy the same on an execution platform (e.g. application server) for execution. Some popular build tools used nowadays are Apache Ant, Apache Buildr, Apache Maven, Cabal, Gradle, Bamboo etc.

Integrated Testing Tools for software application testing should be provided as an important feature of iSDF. The testing tools which can implement a test driven development (TDD) process such as Fitnesse. It should provide the feature to configure the test cases and test data in it. The testing tool should be integrated with the continuous integration tool or platform so that, on each update in source code or application configuration, testing can be executed automatically and the results can be shared (using email) with appropriate stakeholders. Ideally when the application is built and deployed on the execution platform (e.g. Application Server), the testing tool should start executing the test cases which were already pre-configured in it.

Figure 6: Continuous Integration Process

A Defect or Bug Tracker tool can be integrated with iSDF to manage and track defects. These tools can be integrated with iSDF to register and track bugs or defects found during testing. Monitoring and tracking of defect life cycle and maintaining a defect repository for further analysis are important aspects of this.

A User Authentication & Authorization mechanism to authenticate and authorize users (preferably directory service integration will be a modern technique for this) should be integrated with iSDF. This will enable security and restrict user access based on authority.

Reporting Tool is a significant component for any application software. For iSDF, the customer, project manager and other designated stakeholders can monitor and track the software development process by using desired web based reporting interfaces or dashboards.

Figure 7 shows the entire iSDF solution as conceptualized in this article.

Figure 7: iSDF Solution

References

1. Waterfall model for project management and execution at http://en.wikipedia.org/wiki/Waterfall_model

2. Continuous Integration technique at http://www.thoughtworks.com/continuous-integration

3. Martin Fowler’s article on Continuous Integration at http://www.martinfowler.com/articles/continuousIntegration.html

4. FitNesse, the acceptance testing framework at http://fitnesse.org/

The BAM Book

noreply@blogger.com (Partha S Sengupta) — Fri, 11 May 2012 11:23:00 -0700

On Packt publishers' request I reviewed 2 chapters of their BAM book. It has a good coverage and its a concise reading. http://t.co/ZXa3NCGi -- Partha S Sengupta (@psengupta1973)

Dynamic Proxy: A simple example

noreply@blogger.com (Partha S Sengupta) — Sat, 30 Apr 2011 11:23:00 -0700

The proxy pattern defined by the Gang of Four (GoF) refers to a structure where a method call of an object (proxy) is actually redirected to a method of another object which is a sibling (inherited from the same parent) of the proxy object. The figure below explains a simple proxy.

A simple proxy

A simple proxy typically implements a single method. There are situations when multiple interfaces need to be implemented by a single class to behave in different ways. Dynamic proxy is a mechanism by which we can achieve the same, provided you follow certain guidelines and restrictions. While I was going thru the JavaWorld artcle (http://www.javaworld.com/javaworld/jw-11-2000/jw-1110-proxy.html) on DProxy, I noticed a case where the author is explaining the dynamic proxy with the example of automobiles.

If a vehicle is considered as a system primarily three categories of users can be thought of for the same – a passenger who rides, a mechanic who repairs and a driver who drives. Each one is concerned with the different perspective of the vehicle. On the other hand the vehicle should also be able to present three different interfaces for these users, such as the driver will be interested in the interface of driving which is presented by steering, gear, break, speedometer etc. Similarly the passenger will be presented with the passenger’s seat, door, window, music, television etc, while the mechanic should be able to inspect and repair the engine, chassis, tyres etc.

I tried implementing the case with Java. I will try to explain the same in this article.

To explain the situation I considered mainly six classes.

1. Drivable interface – To represent the interface of a vehicle designated for drivers.

2. Ridable interface - To represent the interface of a vehicle designated for passengers.

3. Repairable interface - To represent the interface of a vehicle designated for mechanics.

4. Vehicle class - To represent a vehicle that implements the three interfaces mentioned above.

5. VehicleInvocationHandler class - To represent the class responsible for creating the dynamic proxy class and map the methods defined in the above interfaces and the respective vehicle. This class in implementing java.lang.reflect. InvocationHandler class add the feature which will enable it to load the interfaces and the vehicle class dynamically at runtime and map them with each other.

6. DProxyTester class – Representing the client which uses the system as a client.
The figure below explains the structural aspect of the system.

The figure below explains the structural aspect of the system.

Dynamic Proxy: The Automobile Case

From the above structure it’s obvious that the client i.e. DProxyTester will remain unaffected from any changes in Vehicle class.

The Java code for the classes and interfaces are given below.

Drivable.java

public interface Drivable {
   public void drive(String source, String dest, int speed);
}

Ridable.java

public interface Ridable {
    public void ride(String[] passengers);
}

Repairable.java

public interface Repairable {
    public void repair(String[] parts);
}

Vehicle.java

public class Vehicle {

String name = null;
public Vehicle(String name){
    this.name = name;
}

public void move(String start, String end, int speed) {
     System.out.println(name +" is going from "+start+" to "+end+" at a speed of "+speed+"mph");
}

public void carry(String[] passengers) {
     for (String str : passengers)
         System.out.println(name+" is Carrying "+str);
}

public void getRepaired(String[] parts) {
     for (String str : parts){
         System.out.print(str+", ");
    }
    if(parts.length > 1){
         System.out.println("are getting Repaired on "+name);
    }
    else{
         System.out.println("is getting Repaired on "+name);
    }
}
}

VehicleInvocationHandler.java

import java.lang.reflect.*;

class VehicleInvocationHandler implements InvocationHandler
{
    Vehicle vehicle = null;
    private VehicleInvocationHandler(Vehicle vehicle){
        this.vehicle = vehicle;
    }

    public static Object getProxy(String name){
       Vehicle vehicle = new Vehicle(name);
    // the proxy can be provided as drivable, ridable or repairable.
       Class[] interfaces = new Class[] { Drivable.class, Ridable.class, Repairable.class};
       Object proxy = Proxy.newProxyInstance(vehicle.getClass().getClassLoader(), interfaces, new VehicleInvocationHandler(vehicle));
       return proxy;
    }

    public Object invoke(Object proxy, Method method, Object[] args){
         if(method.getName().equals("drive")){
               vehicle.move((String)args[0], (String)args[1], ((Integer)args[2]).intValue());
         }
         if(method.getName().equals("ride")){
               String[] passengers = (String[]) args[0];
               vehicle.carry(passengers);
         }
         if(method.getName().equals("repair")){
               String[] parts = (String[]) args[0];
               vehicle.getRepaired(parts);
         }
         return null;
    }
}

DProxyTester.java
public class DProxyTester
{
    public static void main(String[] args){
        Object proxy = VehicleInvocationHandler.getProxy("Bus");
        Drivable d = (Drivable) proxy;
        Ridable r = (Ridable) proxy;
        Repairable p = (Repairable) proxy;
        System.out.println("Dynamic Proxy created... ");
        d.drive("Lombard", "Chicago", 60);
        r.ride(new String[]{"Tom", "John", "Harry"});
        p.repair(new String[]{"Wipper", "Head lights", "Gear Box"});
    }
}

BAM: Capabilities & Applicability (Part 5)

noreply@blogger.com (Partha S Sengupta) — Fri, 4 Mar 2011 04:59:00 -0800

CEP: A Brainier BAM
Complex Event Processing (CEP) is a rising rule-based technology that combines real-time information from distributed message-based systems, applications and databases. CEP dynamically searches for patterns and trends that would otherwise go unnoticed. CEP allows enterprises to identify and predict opportunities represented by outwardly distinct events across complex, heterogeneous IT environments. Figure 5.1 shows how CEP extracts these insights. Businesses capture hundreds of thousands of business events to form an event cloud. These events include, updates to customer records in a CRM application, sales in a point-of-sale application, orders scanned upon shipping or delivery, GPS data for delivery trucks and even weather information. Key features of CEP implementation is the ability to model business activities, specifically the dependencies among servers, network infrastructure, applications, people, and processes. CEP uses these models to identify constraints or patterns that map formal or informal processes to expected outcomes. These models provide a framework for designing, automating, and managing intelligent decisions across the enterprise.

Figure 5.1: CEP delivers insight from enterprise events

Implementing CEP

The steps below are required to implement a CEP solution.

1. Enable the enterprise IT infrastructure to capture events.
2. Correlate/match these events to business objects for context and to determine causality.
3. Create the capability to recognize event patterns.
4. Once patterns are recognized, aggregate pertinent event patterns into higher-level event structures also known as “complex” events.
5. Provide business process models and state-based timing expectations (e.g., timeouts/lack of event support) to represent meaningful process transitions.
6. Create the capability to act/react to drive the business object state to mitigate problems or take advantage of opportunities.

Figure 5.2: CEP Systems are noted for their ability to provide insight into operations in real time. They are iterative and can create new events as patterns of activity from employees, vendors, customers, and other stakeholders

Oracle Complex Event Processing (OCEP)

Among the real-time process monitoring and event processing tools Oracle has offered Business Activity Monitor and Complex Event Processing. One can perform real-time detection of specific patterns, across multiple data streams and time windows, through Oracle Complex Event Processing (CEP). One can also gain real-time visibility into operation and performance of business processes, along with the ability to respond to specific situations, through Oracle Business Activity Monitoring (BAM).

At its core, Oracle Complex Event Processing is a Java container implemented with a lightweight, modular architecture based on the Open Services Gateway initiative (OSGi™). It provides a complex event processing (CEP) engine and Continuous Query Language (CQL), as well as a rich development platform without sacrificing performance.

Architecture Overview of OCEP

Oracle Complex Event Processing is architected as a set of relatively fine-grained software modules. Each module contains a set of Java classes that implement some aspect of the server's functionality. A module may act as a library for other modules by exporting classes for client modules to use. A module may also register services in a service registry for use by other modules. The figure 5.3 shows the high level software architecture of an Oracle Complex Event Processing instance. At the lowest level there is a Java virtual machine with deterministic garbage collection (DGC). The JVM provides the foundation for support of applications that demand deterministic latency by limiting the length of garbage collection pauses. The next layer is composed of the modularity framework which allows modules to control the import and export of Java classes. The modularity layer also handles class loading and provides versioning support for classes. The service framework is responsible for instantiating the classes that implement a service and for resolving dependencies between services.

Figure 5.3: OCEP Architecture Overview

BAM: Capabilities & Applicability (Part 4)

noreply@blogger.com (Partha S Sengupta) — Wed, 2 Mar 2011 21:22:00 -0800

On The Way to an Agile Enterprise

In many ways BAM can be considered as a key enabler for achieving agility in the business processes across the enterprise. Agility is all about the sustenance to change. It’s the ability of a business to accommodate and respond to a change faster. BAM can address the key pre-requisites to achieve agility in the enterprise:
• Visibility into daily operations and faster response to variations from the normal trend
• Awareness of change both from external and internal sources
• End-to-end view of the processes with important signs, corrective and proactive measures to be taken by the stakeholders.
Figure 4.1 shows how the BAM solutions and the agility pre-requisites are mapped to the example stated in this article. It’s important to identify the key factors that can increase the agility of the business. For example, the business should focus on external sources that are applicable to the domain and identify the stakeholders who need to get notified on the change and also the people who can act effectively. The more quickly the respond state is reached, the more agility the business will gain.

Figure 4.1: BAM for Agility

Optimizing Business Processes

In a conventional technical way BAM does not have capabilities for performance modeling and application tuning which are the key features of performance engineering. But if BAM is considered from the business process point of view, it provides an approach on how to monitor, tune and track the business processes for performance improvement. This feature in turn adds up in achieving agility, responsiveness and resilience to the business. Eventually these are the vital ingredients needed to optimize the performance of a business process.

Solutions and Implementations

There are several distinct Business Activity Monitoring Solutions present today. Some of them are offered by vendors which are focused exclusively on the Business Activity Monitoring sector such as Systar. Most of the realistic Business Activity Monitoring applications in the market come from the primary enterprise software vendors such as Microsoft, IBM, TIBCO and Oracle. These companies were able to sense the future need of BAM in the marketplace and stapled Business Activity Monitoring capabilities with some of their existing software products. For example Microsoft introduced Business Activity Monitoring features to its BizTalk Server suite, IBM added them to WebSphere; and Oracle enhanced its application server offering with Business Activity Monitoring features. Some popular BAM solutions available today are briefly described below.

TIBCO BusinessFactor: TIBCO BusinessFactor™ software enables the development and delivery of performance visibility and business activity monitoring (BAM) solutions that let users quickly understand current operational performance in context of business objectives, past performance and related activities. It collects and correlates data from across the organization, presents information via customizable interactive dashboards, and alerts users when thresholds are exceeded. By performing these functions, the software helps users more quickly and accurately identify and address risks and opportunities.

It Visual representations of data – including maps, blueprints, technical drawings, charts, or graphs – give users the ability to analyze current conditions and performance for any level of location specificity or timeframe to quickly identify trends and predict likely outcomes. BusinessFactor's intuitive user interface shows critical performance indicators and supporting contextual information so users can perform quick historical analysis, drill into data, and take action. BusinessFactor also gives users the ability to collaborate with other users and interact with systems and processes to take appropriate action.

TIBCO SpotFire: By the way, nowadays TIBCO is promoting a more dynamic product in this area called SpotFire. SpotFire Operations Analytics is designed not only to monitor real-time business events but also to create and deliver root-cause analysis applications in response to those events automatically. Unlike reporting or dashboard based tools, those can only provide alerts on operational problems, SpotFire helps in solving problems, by automatically merging real-time monitored data with contextual information from multiple sources into applications designed to find outliers, trends and relationships.

Oracle Business Activity Monitoring: Oracle BAM is a key technology component of Oracle Fusion Middleware and SOA stack of products. It satisfies a rising need to enable Business Executives and particularly operations managers, to make decisions by making real-time actionable information available 24x7. Visibility into key business metrics empowers managers to improve the efficiency of their business processes in a step-by-step approach to process optimization.

Features of Oracle BAM

The proposed Business Activity Monitoring is Oracle BAM. By deploying the SOA framework and using BAM for monitoring the services, business users can derive actionable timely intelligence on their business operations by monitoring the flow of data, service requests, and process invocations. All the components of Oracle SOA Suite including Oracle BEPL Process Manager (Oracle BPEL PM) and/or the integration adaptors can be configured to sends Oracle BAM information emanating from the various integrated application processes using the JMS protocol. With Oracle BPEL PM this can be achieved by leveraging the built in “sensor” technology wherein a part or the entire state of an instance transaction at a process node can be sent to Oracle BAM as a JMS message for further analysis. Oracle BAM can then correlate disparate events that it received, as messages, within the context of pre-defined monitoring models and rules that can relate to business process KPIs to alert business users of process problems and opportunities.

Architecture Overview of Oracle BAM
The Oracle BAM architecture utilizes messaging, data integration, advanced data caching, analytics monitoring, alerting, and reporting technology to deliver requested critical information within seconds of an event or change in status. Because the primary source of data is messages, Oracle BAM is able to update reports and generate alerts at speeds that traditional architectures simply can’t match. Oracle BAM can accept tens of thousands of updates per second into a memory-based persistent cache that is at the center of the Oracle BAM architecture. Any application can send events using Web services or over any JMSbased messaging protocols. Legacy application can integrate with Oracle BPEL PM using custom created adaptors and can in turn integrate with Oracle BAM via the Oracle BPEL PM native sensor architecture. Oracle BAM can additionally also send alerts to external web services when specified threshold conditions within the analytics engine are met. The figure 4.2 shows the Oracle BAM architecture.

Figure 4.2: Oracle BAM architecture

Business process data can be monitored and tracked using by Oracle Business Activity Monitor. Oracle Business Activity Monitoring (Oracle BAM) is a complete solution for building interactive, real-time dashboards and proactive alerts for monitoring business processes and services. Oracle BAM gives business executives and operation managers the information they need to make better business decisions and take corrective action if the business environment changes.

Figure 4.3: Oracle BAM and surrounding components

IBM WebSphere Business Monitor: IBM WebSphere Business Monitor (WBM) offers real-time insight into business processes.

WBM extends the extensive distributed platform for monitoring of events native to the mainframe platform. It provides business users with a real-time, end-to-end view of business process performance on user-friendly and customizable dashboards, corporate portals and mobile devices, as well as on the desktop. It can improve business results by enabling business users to preempt problems with predictive KPIs and to detect and manage business situations. It boosts up productivity by empowering business users to create new dashboards, KPIs and alerts with minimal IT involvement. It accelerates continuous process improvement and business innovation through a tight integration with other IBM BPM products.
WebSphere Business Monitor V6.1 provides two types of dashboards: Web-based dashboards and portlet-based dashboards. Web-based dashboards are implemented as Web pages. Portlet-based dashboards are a component of WebSphere Business Monitor V6.1 that operates within the WebSphere Portal V6.0.1.1 environment. Figure 4.4 on and Figure 4.5 illustrate the runtime architecture respectively. As indicated by the flow in both figures, you follow this sequence:

1. After you finish the development of the monitor model, export the resulting monitor application that is deployed into the runtime environment (WebSphere Application Server, WebSphere Enterprise Service Bus, or WebSphere Process Server) by using the WebSphere Administrative Console. Ensure that you configure the WebSphere Business Monitor server application to link appropriately to CEI to be registered to consume events from the emitting application.
2. On another server (for example, WebSphere Process Server or WebSphere MQ Workflow), deploy the application that emits events to a CEI server.
3. Upon receipt of the registered event types, the CEI server sends the appropriate Common Base Event to the server registered in step 1.
4. The monitor model executes. It polls for events waiting on the queue, processes them according to the monitor model instructions, and stores business measures to the Monitor database.
Optionally, the monitor model application can obtain real-time event-related data from the event or data source through code that you can provide and invoke as a user-defined XPath function.
1. Optionally, in addition to receiving events as in step 3, the monitor model application can detect business situations (for example, thresholds) and emit events that can be consumed by the WebSphere Business Monitor action services.
2. The CEI server routes events received through the event or data source or the monitor model application to the action services.
3. The action services component takes action such as sending notifications, calling Web services, and invoking Service Component Architecture (SCA) components to perform actions.
Independently, the business user uses the dashboards, which invoke Representational State Transfer (REST) services to return data from the Monitor database and show monitoring results through different views.

Figure 4.4

Figure 4.4 shows usage of the lightweight Web dashboard, which is implemented as Web pages. DB2 Alphablox package installation is optional and is only the prerequisite for use of the dashboard’s dimensional and report views.

Figure 4.5

Figure 4.5 shows usage of the Portal dashboard, which uses WebSphere Portal Server to invoke a REST service. DB2 Alphablox package installation is optional and is only the prerequisite for use of the dashboard’s dimensional and report views.

Systar Business Activity Monitoring for Telecom: Systar Business Activity Monitoring (BAM) enables telecom firms to go beyond simply monitoring network and IT infrastructure processes, instead delivering a clear, 360-degree view of how incidents affect the health of critical business services that must seamlessly flow together.

Systar BAM enables organizations to improve operational efficiency and service quality through unprecedented real-time insight into core telecom business processes, including: Order fulfillment, Service Activation, Revenue Assurance and Supplier and Partner Service Assurance.

Microsoft Business Activity Monitoring: Pro Business Activity Monitoring in BizTalk 2009 focuses on Microsoft's BAM tools, which provide a flexible infrastructure that captures data from Windows Communication Foundation, Windows Workflow Foundation, .NET applications, and BizTalk Server.
Microsoft BAM allows the user to manage aggregations, alerts, and profiles to monitor relevant business metrics (called Key Performance Indicators or KPIs). It provides an end-to-end visibility into the business processes, providing accurate information about the status and results of various operations, processes and transactions so the users can address problem areas and resolve issues within the business.

BAM: Capabilities & Applicability (Part 3)

noreply@blogger.com (Partha S Sengupta) — Tue, 1 Mar 2011 20:53:00 -0800

BAM Matrix

The primary input to the BAM engine can be gathered in a BAM process matrix (table below) that interconnects different systems from business process point of view and also outlines the significant signs to monitor along with the corrective and proactive actions to take in case of contingency. As shown in the third row of the table below, if the estimated delivery date is not generated by the inventory system within three hours of receiving the order, the BAM engine will first fire an API call to check if any detailed status message can be obtained. This will be followed up by sending an email alert to its system owner along with the order and status details. Proactively, the BAM engine can also update the billing engine to delay the billing start date of this order.

BAM process matrix

Integrating Intelligence

A typical business process is influenced by both internal and external sources of information. The BAM model described so far only takes the internal sources of influence into account. The same model can be extended to include external sources of information like weather forecasts, selective news feeds and customized dashboard for manually feeding in intelligence information collected offline. Figure below shows the manner in which this system can be extended.
For instance, when the model depicted above receives some critical weather information about an incoming cyclone, it can update the inventory system to stock appropriate items in the store. Considering another example, the administrator can pull up the list of best selling items during a festival season from the BI system and manually feed it in using the Admin dashboard. At times critical news items can also influence the business process. For instance, in case a popular news source publishes the reviews of a movie with a rating of 5 stars, the systems can send out an alert to increase its inventory. Alternatively, if a stocked movie has really got a bad review, then the stock reorder requests can be put on hold without any manual intervention.

BAM integrated with external systems and feeds for intelligence

Considerations for Implementation

Rolling out a BAM process takes a considerable amount of effort and commitment from the business side. Here are few things for the stakeholders to consider before deciding to go ahead with BAM implementation.

BAM is a Business Solution, Not a Technology Solution: A systematic study of the business processes in context is required as the first step to BAM roll out. Once the study is over a process matrix (table above) has to be created to outline the key stages in the lifecycle of a business entity as it goes through the business process. After the matrix is defined, it will be used to configure the BAM solution to monitor the appropriate significant signs and take the right corrective action. This is purely a business activity and does not have much of technology dependency. It has to be ensured that the required time and effort is committed by the business personnel to make this activity successful.

The idea might need to be sold: A number of existing important business applications might require to get modified for a BAM rollout to be effective. It is important to get the accurate data out of the applications and analyze the same to get precise meaning out of them so that they can be used properly during the BAM implementation. The applications those are architected to be extensible or that follow SOA paradigm will not have many issue in incorporating changes. For the rest, willingness and preparedness to accommodate the changes has to be explained and understood. Most of the time this would require selling idea to all the parties involved and explaining them how the investment will bring in greater benefits to the business.

Compatibility Aspects: Though it sounds very exciting, there is a problem at the last lap of BAM rollout. Since BAM works as a information aggregator, it needs to interact with multiple systems based on the diverse platform and technologies. To avoid the complicacies caused by the heterogeneous platforms and technologies it’s recommended to use web services and SOA paradigm wherever possible during the integration. This would bring in compatibility among the applications running on a heterogeneous environment. While planning for a custom built BAM solution it is very important to consider the needs of the existing applications and the time required to integrate the BAM into the existing environment.

To Start on a Large Scale or Not: For an effective BAM roll out it’s very critical to integrate it with all the systems that participate in the business processes. Before planning for the roll out the complexity of all the candidate systems should be evaluated. Typically all enterprises will have a combination of small and large scale business processes. Small ones will have lesser issues for integration than the large ones. Time and effort required for each of the business processes should be estimated properly. Once the estimation is done the business processes should prioritize based on their sizes otherwise this can put a hold on the business operations for a long period. For example, if the Shipping and CRM systems are found to be the most active ones among the other systems participating in a business process, they should be targeted as the initial candidates for BAM roll out. Once they are integrated with BAM and start to produce results on the BAM reporting interfaces the rest of the systems can be integrated gradually. Another option of the roll out is to break up the business processes in small segments and start the integration in an incremental manner.

BAM: Capabilities & Applicability (Part 2)

noreply@blogger.com (Partha S Sengupta) — Sun, 27 Feb 2011 21:39:00 -0800

BAM Architecture in General

Architectural diagram on figure below shows general architecture of BAM solution and explain functions of each of them.

Modeling tool

Usually GUI based toolkit is used to define monitor model. The model contains definition of events that should be “caught”, metrics, triggers and KPIs. Finished model is deployed and monitoring can start.

BAM core

This is the main component of BAM solution. It generates event patterns according to the monitor model and deploys them on CEP (Complex Event Processing) engine. It later processes the received event data and transforms them in the desired form. BAM core can also communicate with the Process engine, start and stop the processes, generate events etc. according to triggers and alarms defined in the monitor model.

CEP engine

This is the lowest level component responsible for event filtering. The CEP engine receives query patterns from the BAM core. It queries the event flow both from the Process engine and ESB and sends the “wanted” events back to BAM core. Usually only process engine events are used for BAM, but CEP engine allows also filtering of ESB flow events. That can be used in more technical oriented monitoring, or combined with the business data, to monitor performance, resources usage etc.

Front-end

Front-end receives data already processed, with computed KPIs etc. and takes care of their visualization. Every BAM solution should deliver graphical front-end witch rich visualization possibilities, such as charts, dashboards or graphs. An advanced OLAP-like visualization tool can be also useful. One of the advanced features some front-ends also provide is a real-time view of processes, visualized on process graph. Here you can view process diagram as it was modeled and track the state of a particular process instance. It is also common that at least one front-end should be web based. In this case front-end should provide a possibility to assemble your dashboards “on fly”. This is done usually by AJAX based scripts. AJAX becomes a standard for web-based BI tools today and it is also supported by Jasper Dashboards, one of the hot candidates for open source BAM front-end.

Long term data mining BI tools

An optional component that can analyze data from long-term perspective, realize data mining and knowledge discovery processes. Discovered information is delivered back to BAM core. Mostly external tools are used for such cases nowadays but hopefully in future some of those tools will become a part of BAM solutions. That will allow the feedback obtained by those tools to be sent back to the bam core or the process server.

General architecture for BAM

Where to Apply BAM

A business process typically spreads across multiple large systems which in turn made up of subsystems and components. For example, an order (i.e. a business entity) for a broadband connection flows from one system to another. It goes through some level of transformation while getting processed. The figure 1 shows how an order flows among multiple systems such as CRM, Billing and Inventory etc. to fulfill an order in a typical business process.
Once the order for the broadband connection is placed by the customer online, it flows through different systems on the way to get fulfilled. The basic steps followed in the process above are:

• The order must be validated online for availability and geographical access.
• A broadband router must be shipped from the warehouse to the customer’s address.
• Customer’s phone line and broadband connection must be linked to the same billing account.
• CRM should be updated with the order fulfillment information for the call center to access when providing support to the customer.
The fault with this setup is that each of the systems is autonomous. They have their own alerting and reporting mechanism. This might lead to a scenario where the customer experiences problems with - the delivery of the broadband router in time, unable to receive a discount (if associated) on the combination of a broadband and a land line. Due to the absence of a proper BAM solution the service provider won’t be able to sense these faults before the customers do, causing an unwanted blot in the customer relationship.

In an opposite scenario with a BAM implemented on the service provider’s side:

• The data from not only the systems involved in the business process but also the related subsystems and components can be captured proactively.
• The collected data can be tracked and analyzed against the defined measures.
• Alerts can be raised to the applications and the people when something is not behaving as desired.

How to Employ BAM

To be able to employ a BAM solution effectively the preferred option is to put the entire business process under the supervision of a BPM engine as shown in the figure below.

Business processes executed by BPM

This setup can be extended to incorporate a BAM solution as shown in figure below. Under this situation, the BAM adapter is integrated with the BPM engine that in turn is integrated with all the business process systems. The BAM adapter traces the business entity through the business process and keeps comparing it against the standard or defined values. Let us set a KPI for efficient delivery in the abovementioned example, such that an order should not rest within the inventory system more than seven hours. To handle this situation the BAM adapter should track all the orders that go to the inventory management system. If it does not receive a status change notification within seven hours, an email or an SMS alert is sent to the system owner. In case an API (application programming interface) is available to interact with the inventory system, the specific order can be identified and the owner can be notified with an alert on the same. If there is no status change noticed within twenty four hours, an alert (email or SMS) can be sent to the customer with an updated date of delivery.

BAM integrated with BPM infrastructure

This way, BAM brings in agility and resilience in the business, not only that, it can also cut down the burden on the support services by proactively notifying the customers about their order status.

Enterprises that use middleware instead of a BPM infrastructure for business process orchestration can also incorporate the BAM support in their existing setup using an Enterprise Service Bus (ESB) and a logging framework.

In this case, the applications that cater to the part of the business process will require using a BAM logging framework for updating the status of the business entity (e.g. order) through its lifecycle. Multiple instances of this logging framework can be deployed throughout the enterprise and all of them can forward the logging calls to the BAM engine through the use of an ESB or an asynchronous messaging service.

BAM incorporated in a setup with middleware but no BPM

The BAM engine can revive the lifecycle trail of the business entity from these logged messages, trace its progress and raise alerts, alarms and communications as appropriate.

BAM: Capabilities & Applicability (Part 1)

noreply@blogger.com (Partha S Sengupta) — Tue, 22 Feb 2011 22:10:00 -0800

Nowadays BAM (Business activity monitoring) is catching up like a mandate for the administration and management for an enterprise. There are few topics that discussed in this article explaining the capability and applicability of a BAM system. The references to the telecom IT sector has been used due to my current involvement in telecom industry.

Proactive data mining

Like other industry verticals the telecom sector depends on the execution of complex business processes. Usually business processes include interactions among multiple heterogeneous IT systems and human resources. The efficiency and success of these business processes in an operational environment can be monitored based on multiple Key Performance Indicators (KPIs) of the business such as customer satisfaction, overall business performance and adaptability to changes. As a reason it is becoming important to monitor the performance of the business processes and optimize them based on the results. Presently businesses come to know about a problem when a failure occurs in the business processes. Monitoring is one of the best ways to sense an imminent problem and avoid it with corrective actions in advance so that, the impact on business and the users can be minimized.
This article will try to present a solution to monitor and optimize the business processes based on the currently available technologies and their maturity.

Purpose of Business Activity Monitoring

The term business activity monitoring was introduced by Gartner, Inc. in 2001 as their notion of aggregation, analysis, and real-time presentation functionality of business relevant information. The main objective of business activity monitoring is to predict or sense the upcoming problem based on the operational transactions and report the same for a preventive measure.

It is becoming increasingly significant in today’s enterprises to consider informed decisions in timely manner by consolidating information present in various silo data sources. These decisions eventually have major impact on the business parameters such as profitability, revenue, customer satisfaction and agility. IT systems like Enterprise Resource Planning (ERP), Customer Relationship Management (CRM), Order Management System (OMS), Business Process Management (BPM), etc. have served with considerable efficiencies in their own areas of the business. However mostly the business processes span across multiple systems.
If we imagine of an enterprise with the capability of foreseeing the next steps, predicting impact of events, sensing an unseen fault by putting two and two together from different systems and eventually spot and plug any hole in the plumbing before the damage is done, a tremendous agility, resilience and efficiency in the business processes can be introduced.
A layer that is able to monitor business processes in the above mentioned manner should be able to understand and perform real time monitoring of vital signs in the context of business process. This monitoring should be made proactive as much as possible. Apart from the failures, the deviations in the normal trends based on the historical data need to be flagged and alerted to appropriate stakeholders with minimum time lag. Among the on-demand features, it should have ad hoc reports, dashboards for the decision making community within the enterprise. Such a layer would bring in manifold benefits.
For example, let’s consider an e-commerce scenario with the key business process of fulfillment of orders placed online. The customer places an order with a set of items and gets notified with a date of delivery. Under a happy path scenario, the customer receives the items on the promised date. Considering a variation when the logistics of delivery such as trucks, delivery boys etc. are operating at the full capacity due to a sudden increase in the number orders placed in, an individual order might get delayed in the delivery. The activity monitoring layer could potentially correlate the high volumes being handled by the system to a potential delay in delivery times. This can trigger an alert or communication to customers informing them about the delay before it actually takes place.
Imagine a more interesting scenario of inventory management for the e-commerce site. Assuming that there is a forecast for thunderstorms for couple of days in some of the areas, the monitoring layer can send an alert to the inventory management group to stock up on rainy weather essentials such as umbrellas and raincoats to enable the business to meet the un-foreseen increased demand.

Components of a Business Activity Monitoring

BAM consists of the following generic components:

Business Event

A business event points to the occurrence of an event that is significant to an operation of a business activity. It is separate from an IT event. An IT event usually shows the occurrence of an event in the context of an IT resource.

Business activity

A business activity offers services or functionalities required to serve a specific business purpose.

Metric

A metric is a measurement of a specific property belonging to business activity that can be used to monitor business operations.

Key performance indicator

KPIs are the metrics used to measure the progress of a business goal. KPIs have to be measurable as well as achievable and provide the possibility to aggregate measures over a set of instances. In order to define KPIs, companies should have clear performance requirements, and they must also have a clear definition of their strategic target which can be compared to runtime values.

Techniques of Today

Though BAM is relatively a new term are some techniques that have been around for a while. The stalwarts amongst these are

• IT system monitors
• Business Intelligence (BI)
• Reporting and Analytics
• Business Process Management (BPM)

Each of the above has its own pros and cons when the BAM features are evaluated. Let us dig a little deeper in these three techniques.

IT System Monitors: To be able to monitor failures and success events at times many enterprises have implemented IT system monitoring measures in effective ways. The main objective of such systems is to alert the support services in a timely manner to meet the service level of agreement and availability levels of the application. This technique targets individual applications for monitoring and helps them to react under a error condition at the application level. While it does not consider the entire business process as a monitoring target as a result it is unable to track any fault at the business process level. IT system monitors are mostly reactive than proactive. They keep watching the applications, logs, network, server, hardware in isolation for faults rather than the end-to-end business process. IT system monitors will be unable to detect an error at the business process level while the applications are running smoothly. IT system monitors watch for the malfunctions and failures only, while an ideal BAM should look for success and failures both to yield meaningful activity metrics. They are also not aware of the business process level KPIs (Key Performance Indicators).
For example, if a customer is promised with the certain duration for an order fulfillment process, a KPI needs to be set at the business process level for order processing. IT system monitors are unable to detect a degradation of this kind of KPIs. The scope of alerts is limited to the application and does not correlate to failures in other participating application in the same process. It is also unable to detect any variation in the business process when the applications are functioning properly.
In spite of all shortcomings these tools have been serving as monitoring instruments for many organizations. In fact there have been conscious design efforts introduced to log success events (along with the failures) in individual applications’ logs against a unique correlation id and also a listener mechanism to poll the logs in a timely manner and correlate the events using the unique id for each business process transactions. However since the number of I/O operations increase (due to write in log and timely polling of listeners), the performance of the applications is compromised seriously under this situation.

Business Intelligence (BI): Business Intelligence has been occupying the place of a backbone for decision making process of the organizations for quite some time now. This is due to its capabilities of analytics and extensive reporting. BI is capable of processing enormous amount of data stored in huge data marts at predefined frequencies by using ETL (Extraction Transformation and Loading) tools and put the same into the data stores which gets reported eventually. This way BI enables businesses to make informed decisions to a great extent. However when it comes to BAM some constraints apply.

• Since BI and data warehousing are based on batch processes, proactive and real time decisions can not be made easily using BI.
• The statistical analysis is performed manually in BI. Since the monitoring framework is not updated with the historical trends, it’s not able to flag events which are deviating from the normal trend.
• Since BI is not a real time system, it can not track real time errors or deviations and generate alerts for that.

Business Process Management (BPM): Enterprises use business process management to manage, model and automate business processes. Externalization of all business processes those are entrenched deeply in applications to a central location is one of the main tenets of using BPM. BPM accomplishes quite a lot from that angle in terms of improved process agility and centralized governance of vital processes. But BPM also has some shortcomings as a BAM solution:

• BPM has no built in monitoring capabilities. Therefore, alerts based on the success or failure of a process can not be generated by the BPM infrastructure. Furthermore the success and failures can not be evaluated against the historical trend to compose intelligent reports necessary for the right changes or decisions to the activities.
• BPM has the knowledge of a business process while it is lacking the knowledge of relationship between the participating applications, historical data and statistical analysis which are required for anticipating the unknown errors in a business process.

The table below presents a comparative analysis of the BAM techniques described above. It also shows the rating of the three techniques as an ideal BAM solution.

Table 1: Comparative analysis of three BAM techniques

MBT

noreply@blogger.com (Partha S Sengupta) — Tue, 31 Aug 2010 11:15:00 -0700

Another trend is catching up along with WEB2.0, SOA and EA (Enterprise Architecture) which is Business transformation or IT transformation to be specific. I believe this is going to be the next big thing in IT industry. Most of the enterprises who piled up multiple big and small applications to support their daily business transactions are realizing a need to consolidate the pool of software applications to have a better control monitoring and governance. There is a burning need to control the growth of IT landscape. As a result the enterprises are going for transformation of their IT stack. I foIn search for a standard methodology of business transformation found most of the consulting farms have come up with proprietary methodologies. There is one called MBT (Methodology for Business Transformation - http://www.doi.gov/archive/ocio/architecture/mbt/guidance.htm) presented by the US Department of the Interior. This is in fact in line with some methodologies that I have been using for the last couple of months. This method considers the vital ingredients of EA and IT governance along with the technical aspect of process re-engineering and transformation roadmap. As TOGAF describes the ADM (Architecture Development Method), MBT plays an important role in listing out the important activities to be done on the way to transformation.

BAM: Now and Beyond

noreply@blogger.com (Partha S Sengupta) — Mon, 26 Jul 2010 01:10:00 -0700

BAM is still not matured enough to be used smoothly in an enterprise. It requires the implementer to take more responsibilities to fit with the business in context. But the promises that BAM brings in is really very beneficial to all the industries and businesses. If implemented in a proper way that fits with the business strategies it can bridge the gap between the business and the technology. The enterprises those roll out BAM will have an obvious advantage over their competitors with respect to the performance of the business systems.
As far as the agility is concerned, the best option for an enterprise to get the maximum benefit out of a BAM solution is to have centralized control over the business processes using a BPM platform and complement the same by a BAM framework. BPM will help the enterprise to have an end-to-end view on the business processes and their control while BAM will bring in the capabilities to monitor the changes, foresee a variation from the normal trend and act accordingly.
The changes that can be monitored by a BAM could be anything that happens in the market but the real intelligence that can earn a greater profitability for a business. This can be done with a continuous mining of news feeds and analyzing them to find out which are relevant to the business and can potentially change the landscape of a business. This is how technology can be employed to watch for the business opportunities like a hawk and enrich the business to proceed toward a profitable growth, operational efficiency, reducing operational costs and adaptability. CEP is a brainier and improved BAM that tracks the real time business events and figure out a business pattern based on the same. In case of a deviation from the normal pattern the CEP notifies and enables systems to resolve issues automatically.

Way to effective SOA Governance: A case study

noreply@blogger.com (Partha S Sengupta) — Tue, 29 Jun 2010 23:00:00 -0700

There are multiple issues faced by conventional IT companies on the way to implement effective SOA governance.

Lack of decision on what services to be built and how to make them reusable and sharable leads to interoperability issues.
No standard mechanism to reuse existing services.
No mechanism to impose standards and policies on the services
No control on service provisioning.
Lack of management and visibility on dependencies

Traditional IT governance believes on ownership of IT assets. This model does not help in SOA paradigm. SOA leads to reusable shared packages of IT assets.
As an example, we can consider a fictitious telecom service provider company as SDFTELECOM. They have various departments looking after customer management, inventory management, billing, service provisioning and activation. Each department has their own stack of systems to serve their respective areas of operation. Let us consider a set of systems participate in an end-to-end order fulfillment process. The systems were moderately integrated using SOA infrastructure. As per the ownership model followed, the participating systems belong to different departments of SDFTELECOM. With significant growth in business and increasing customer demand the systems need to undergo changes quite often. But due to the individual ownership of systems they are facing challenges in managing their SOA landscape.
In most of the cases the changes are made on adhoc basis. Services introduced during these changes are often developed in silos. Overall, they are not reaching the intended business agility goals due to the following issues:

There are significant issues with service reuse (or lack reuse).
The company is struggling to identify new service candidates and to prioritize them.
Governance of changes and versions for services is crude and unrefined.
Lack of mechanism to enforce operational policies consistently across the SOA runtime environment.
Adoption of standards is unsystematic.

SDFTELECOM has identified three key requirements to be addressed to overcome the abovementioned issues.
Govern how new services are created.
Maximize the reuse existing services.
Enforce standards and best practices across SDFTELECOM SOA landscape.

To implement effective SOA governance SDFTELECOM needs to accomplish the following activities.

Setting up SOA vision and strategy in line with corporate and IT strategy.
Business service portfolio modeling
Services and implementation lifecycle
Setting service standards, policies and prioritization
Setting up a service funding strategy
Controlling changes
Aligning SOA strategy with Enterprise Architecture.
Managing Service inventory and registries

The organization structure of SDFTELECOM IT is shown as below.

IT organization structure of SDFTELECOM

The organization is headed by a Top level IT Steering Committee (TLITSC) led by the CIO. The domain owners within the TLITSC are accountable for the business functionality within their appropriate business domain. These domain owners direct reporting responsibilities within their business domain as well as to the CIO. The TLITSC has a panel of key technical members as well. The technical members along with the domain owners attempt to have a convergence between business and IT.
The organizational model has a layer of Business Domain Directors and SOA Direction team next to TLITSC. Domain owners in TLITSC receive insight on priorities in their respective domains from the business domain directors. The business domain directors also work with the SOA Direction team to give insight on required business functionality. The SOA Direction team is in command of defining and refining the strategy and direction for the SOA initiatives, high-level direction of the architecture, and proper guidance of the service portfolio in association with the Business Domain Directors.
Core SOA Team helps out the implementation of required services with the guidance of SOA Direction team. The role of Core SOA Team is to help in defining various processes throughout the service life cycle as well as standards and best practices to ensure that the processes are properly communicated, enforced, and evolved. Similar to other layers of the Center ofCompetency (SOACC), the SOA Core Team also has representation from both the business and IT perspectives.

Distribution of responsibilities among different teams of SDFTELECOM is shown in the table below.

Distribution of responsibilities among SDFTELECM IT teams

Supposing a new requirement for a new service to check customer’s credit rating is placed by the billing department. The decision flow on whether the service to be created or reused is depicted below. The activities are numbered with 1,1A, 2 etc.

Decision flow for service creation

The distribution of responsibilities (shown in the flow above) among the teams are given below.

Similarly to service creation, more decision flows can be created for improving the service reusability across the organization and enforcing of standards and best practices for the services.

MyApp on WebSphere Application Server v6.1 (ND): A case study (PART 2)

noreply@blogger.com (Partha S Sengupta) — Sun, 4 Apr 2010 18:43:00 -0700

This is a continuation of Part 1 of the same series posted on 31st March 2010.
The application I wanted to install on WebSphere Application Server (WAS) is packaged in MyApp.ear.

Installing WAS and Enterprise

Pick up the pieces

In my case I collected MyApp.ear, Linux.c88STML.tar.gz (installer for WebSphere Application Server 6.1 network deployment on Linux), downloaded.updii.61013.linux.ia32.zip (UpdateInstaller), 6.1.0-WS-PLG-LinuxX32-FP0000013.pak (Fix Pack 13), jconn2.jar on the box under installation.

Installing WAS6.1nd

1. Created a dir as /opt/was61nd_install and execute
# tar -zxvf Linux.c88STML.tar.gz -C ./was61nd_install
2. Executed /opt/was61nd_install/launchpad.sh
3. Accepted License Agreement.
4. GUI came up with a warning regarding the OS prerequisite check failed. I ignored and clicked on "next" button.
5. Accepted Sample Application Installation.
6. Selected Installation directory as /opt/IBM/WebSphere/AppServer61nd. Click "next".
7. In "WebSphere Application Server Environment" selected "Cell" environment.
8. In "Enable Administration Security" screen, clicked on the check box "Enable Administration Security".
9. Entered user/password (I tried wasadmin/password) for Admin and entered password for Sample Application (mine is same as Admin).
10. It will come up with "Installation Summary". Clicked next to start installation.

Installing FixPack13

1. The FixPack13 needs the UpdateInstaller to be installed first. The UpdateInstaller for WAS610 can be found at http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg24012718.
2. The instructions for installing the UpdateInstaller are at
http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/tins_updi_install.html
3. The FixPack13 for WAS61 on Linux is available at http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg24017311.
4. The instructions for installing the FixPack13 is available at http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg27010985#steps.
5. Unzipped "unzip download.updii.61013.linux.ia32.zip -d /opt/installers/was610nd/untar/UpdateInstaller/" under /opt/installers/was610nd/UpdateInstaller.
6. Used the installRegistryUtils command with the required attributes to examine installation locations for the installed Update Installer products. The command file is named as installRegistryUtils.sh in the app_server_root/bin directory.
7. Went to /opt/installers/was610nd/untar/UpdateInstaller/UpdateInstaller and opened "responsefile.updiinstaller.txt" in vi.
8. Set the option '-OPT silentInstallLicenseAcceptance="true"' and "-OPT installLocation=/opt/IBM/WebSphere/UpdateInstaller"(where the UpdateInstaller should be installed).
9. Executed "./install -silent -options ./responsefile.updiinstaller.txt".
10. Executed cp /opt/installers/was610nd/FixPack13/6.1.0-WS-WAS-LinuxX32-FP0000013.pak /opt/IBM/WebSphere/UpdateInstaller/maintenance/.
11. Opened /opt/IBM/WebSphere/UpdateInstaller/responsefiles/install.txt in vi.
12. Uncommented and set '-W maintenance.package="/opt/IBM/WebSphere/UpdateInstaller/maintenance"' (the location where the FixPack13 is residing).
13. Set '-W product.location="/opt/IBM/WebSphere/AppServer61nd"' (where WAS is installed), saved install.txt.
14. Stopped all WebSphere related processes as below.
a) Went to Admin Console -> System administration -> Nodes. (Figure 2)
The Admin Console URL is https://10.15.2.19:9043/ibm/console ,the port might vary to 9044.
b) Checked localhostNode01 and clicked Stop.
c) Went back to Putty session and moved into AppServer61nd/profiles/Dmgr01/bin.
d) Executed "./stopManager.sh -username wasadmin -password password".

Figure 2

15. Executed /opt/IBM/WebSphere/UpdateInstaller/update.sh -options ./responsefiles/install.txt -silent.
16. Started up AppServer61nd/profiles/Dmgr01/bin/startManager.sh -username wasadmin -password password.
17. Started up AppServer61nd/profiles/AppSrv01/bin/startNode.sh -username wasadmin -password password.
18. Opened Admin Console on the browser (https://10.15.2.19:9043/ibm/console).
19. Went to Servers->Application Servers and started server1. (Figure 3)
20. Went to Applications -> Enterprise Applications and checked if SampleGallery has been started already, if not, it has to be started.
21. Went to Servers->Application Servers and created 1 more server as server2 (in addition to the existing one - server1) under the same node. Selected server type as "default". Opted for WebSphere to generate the ports for all servers.
22. Once server2 was created, clicked on Save link on the following page and synchronized the Node under System Administration -> Nodes.
23. Tried starting Servers -> Application Servers -> server2 to see if it's ok.
24. Stopped all the servers from the Admin Console for further configuration changes.

Figure 3

Installing MyApp

1. Went to Applications -> Enterprise Applications. Clicked Install button on the right panel. (Figure 4)
2. Browsed to the path to MyApp.ear and select the path to it and clicked next.
3. Clicked Finish on the Summary page.
4. After seeing "Application MyApp installed successfully.” Clicked on "Save" link.

Figure 4

Configuring MyApp

1. Placed jconn2.jar (I used Sysbase database for MyApp and jconn2.jar was containing the JDBC drivers for Sybase) under AppServer61nd/optionalLibraries/sybase.
2. Went to Security -> Secure administration, applications, and infrastructure. (Figure 5)

Figure 5

3. Expanded the link "Java Authentication and Authorization Service" and clicked on "J2C authentication data". (Figure 6)

Figure 6

4. Clicked on New button to add.
5. Entered Alias as "MyDBAdmin", entered user and password, clicked apply.
6. On next page clicked "Save" link.
7. Went to Resources -> JDBC -> JDBC Providers.
8. On "JDBC providers" screen on the right selected the scope as localhostNode01.server2 from the dropdown list. (Figure 7)

Figure 7

9. Clicked New to add a JDBC provider.
10. On next page selected "sybase" as the database type, "Sybase JDBC 2 driver" as the provider type and "connection pool data source" as the implementation type. Accepted provider name same as the provider type, i.e. "Sybase JDBC 2 driver". Clicked next. (Figure 8)

Figure 8

11. Entered AppServer61nd/optionalLibraries/sybase as the path to the jconn2.jar. (Figure 9)

Figure 9

12. Clicked finish button on the summary page. Clicked on "Save" directly to the master configuration.
13. Went to Resources -> JDBC -> JDBC Provider.
14. Clicked on "Sybase JDBC 2 driver" link on the right.
15. Clicked on "Data Sources" (under Additional Properties) link on the next page. (Figure 10)

Figure 10

16. Clicked on New button to create a new data source under this provider. (Figure 11)

Figure 11

17. Entered "MyDS" as the data source name, "jdbc/MyDS" as the JNDI name and selected “DBAdmin" from the J2C authentication alias list. Clicked next. (Figure 12)

Figure 12

18. On next page (entered database specific properties for the data source), entered database name as MYDB, server name and port, clicked next.
19. On the next page (i.e. summary) clicked finish. Clicked Save on the following page.
20. Selected "MyDS" from the list and clicked on "Test Connection". In case a warning is displayed on top of page click on the "Synchronize" link in the warning message. Don't forget to click the checkbox for across node synchronization.
21. With this I went to Servers->Application Servers and restarted server2 and my application is ready to go :)

MyApp on WebSphere Application Server v6.1 (ND): A case study (PART 1)

noreply@blogger.com (Partha S Sengupta) — Wed, 31 Mar 2010 23:56:00 -0700

It’s better late than never :) As I have promised to present a case study of installing an application in WebSphere Application Server (WAS) environment in my previous article “Multihomed J2EE Clustering: A Case Study”, here is one on WAS network deployment version.
In this post I will try put together a brief on WAS concepts and overview required to go ahead with the deployment plan.

WebSphere Application Server: An overview

WAS is the implementation by IBM of the Java 2 Enterprise Edition (J2EE) platform. It conforms to the J2EE 1.4 specification.
WebSphere Application Server is available in three unique packages that are designed to meet a wide range of customer requirements.
_ IBM WebSphere Application Server - Express V6
_ IBM WebSphere Application Server V6 - Base
_ IBM WebSphere Application Server Network Deployment V6

Network deployment platform

We used Network deployment (ND) version for the installation described below.
With ND, a distributed server configuration can be set up, which enables

# central administration,
# workload management and
# failover.

In this environment, one or more application servers are integrated into a cell that is managed by a deployment manager. The application servers can reside on the same machine as the deployment manager or on multiple separate machines. Administration and management is handled centrally from the administration interfaces via the deployment manager. With this configuration, multiple application servers can be created to run unique sets of applications and then manage those applications from a central location. However, more importantly, one can cluster application servers to allow for workload management and failover capabilities. Applications that are installed in the cluster are replicated across the application servers. When one server fails, another server in the cluster continues processing. Workload is distributed among Web containers and Enterprise JavaBeans containers in a cluster using a weighted round-robin scheme. Figure 1 illustrates the basic components of an application server in a distributed server environment.

Application servers, nodes, and cells

Regardless of the configuration, the WebSphere Application Server is organized based on the concept of cells, nodes, and servers. While all of these elements are present in each configuration, cells and nodes do not play an important role until you take advantage of the features provided with Network Deployment.

Application servers

The application server is the primary runtime component in all configurations and is where an application actually executes. All WebSphere Application Server configurations can have one or more application servers. In the Express and Base configurations, each application server works as a separate entity. There is no workload distribution or common administration among application servers. With Network Deployment, a distributed server environment can be built consisting of multiple application servers maintained from a central administration point. In a distributed server environment, one can cluster application servers for workload distribution.

Nodes, node groups, and node agents

A node is a logical grouping of server processes that are managed by WebSphere and that share common configuration and operational control. A node is associated with one physical installation of WebSphere Application Server. In a stand-alone application server configuration, there is only one node. With Network Deployment, multiple nodes can be configured that can be managed from one common administration server. In these centralized management configurations, each node has a node agent that works with a deployment manager to manage administration processes.
A node group is a new concept introduced with WebSphere Application Server V6. A node group is a grouping of nodes within a cell that have similar capabilities. A node group validates that the node is capable of performing certain functions before allowing those functions. For example, a cluster cannot contain both z/OS nodes and nodes that are not z/OS. In this case, multiple node groups (one for the z/OS nodes and one for nodes other than z/OS) can be defined. A DefaultNodeGroup is automatically created based on the deployment manager platform. This node group contains the deployment manager and any new nodes with the same platform type.

Cells

A cell is a grouping of nodes into a single administrative domain. In the Base and Express configurations, a cell contains one node. That node may have multiple servers, but the configuration files for each server are stored and maintained individually.In a distributed server configuration, a cell can consist of multiple nodes which are all administered from a single point. The configuration and application files for all nodes in the cell are centralized into a cell master configuration repository.

Figure 1

Servers

WebSphere Application Server supplies application servers which provide the functions that are required to host applications. It also provides the ability to define external servers to the administration process.
In the next post (PART 2) of this series I will describe the installation process and strategy of MyApp on WAS.

Killer Servlet: a performance tuning and garbage collection solution

noreply@blogger.com (Partha S Sengupta) — Wed, 24 Mar 2010 01:13:00 -0700

An applet is a Java application executed by a web browser to show dynamic graphics on the front-end. After the advent of Rich Internet Application (RIA) technologies, applets are disappearing from the tech landscape but once upon a time it was the only way to show rich contents on the front-end. A servlet in brief is a piece of Java code executed on the server side (typically a web server with an embedded servlet engine) to process request, collect data from data source or rendering a well formatted HTML page on the browser.
An applet-servlet communication is typically done where one can not avoid having a thick client such as an applet on the front-end posting requesting and / or receiving feed from the server side. For example, suppose an applet is continuously drawing and updating a graph based on the feeds received from the servers on the stock prices.
Whenever a user is opening an applet on his/her browser, the browser is downloading the applet and starts executing it. The applet draws a nice graph on the front-end (browser) and on the background connects to a server to fetch stock prices. When we had such a situation back in 2001/2002 (original situation has been modified a little to keep the privacy of the project intact) the architecture followed to implement this is as follows.
Among the technical components there were

• A Stock Price Applet with
o a nice graph panel
o a thread to render the graph on the front-end
o a thread to communicate with a servlet at the back-end

• A Stock Price Servlet (HTTPServlet) on the server with a “while” loop in the doPost() method and having some sub-components as below.
o a JMS Listener attached to the Stock price JMS topic
o a shared message cache where the subscriber stored the messages and the servlet used to read from.

• A Stock price JMS Topic on the JMS server

• Multiple Stock price publishers connected with different external Stock servers on the outer end and with the Stock price JMS topis on the inner end (in the server). Figure 1 explains the architecture.

Figure 1

The applet used to run on the browser and communicate (by opening a http stream) with the StockPriceServlet at the back-end. The servlet used to instantiate the JMS subscriber on receiving the applet’s request. The JMS subscriber was connected with the Stock price JMS topic to listen to Stock price update messages. On the other hand the JMS publishers kept adding messages as and when received from the external servers. To keep the http session alive between the applet and the servlet an infinite “while” loop is placed in the doPost() method of the servlet. The communication was a server push within an http session continuation.

Everything was nice and smooth but after serving a certain number of clients (i.e. applets) the server was getting slower and slower. It was a real-time application and delays to a certain extent was permissible but not beyond that.
So we started investigating the reason behind it. We discovered that the StockPriceServlets those were serving the applets on the front-end were kept alive even after the applet window was closed. The reason was the infinite loop in the doPost() method. The JMS subscriber was still alive listening to the JMS topic and throwing the messages to the servlet. The StockPriceServlet was writing the messages on the http response stream without any errors/exceptions because it was unable to sense that the receiver on the other side of the stream (i.e. the applet) has been stopped.

There was an option to put a timeout on the servlet so after a certain period pf time it dies out but that was not optimal. With an increased frequency of updates received by the stock servers the system might choke anytime.
Another option was to send a completion request from the applet to the servlet so that it can come out of the loop. But once the servlet receives the first request from the applet it starts the JMS subscriber and goes inside the loop. Http communications are stateless therefore a new request will invoke a new servlet.

Looking for more options we figured out a way to stop the StockPriceServlet and free up the resources with a programmatic technique. It was quite clear that the servlet needed to be killed on the completion of the applet’s activities. So an event or update needs to be thrown back to the servlet notifying that the applet has been stopped. Based on this update the servlet had to be killed. So we felt the need of an agent on the server side who will receive the notification from the applet’s end. The agent needs to accomplish the following activities.

1. Receives the completion notification from the applet.
2. Identifies the StockPriceServlet instance that was serving the applet.
3. Notify the servlet by updating a flag that makes the servlet to break out from the infinite loop and stop the subscriber attached to it.

So the agent took birth as the KillerServlet (please don’t laugh :) it’s the birth of the terminator ;). Activity 1 can be easily achieved by sending a http request from the applet to the KillerServlet. The most critical was the step 2 i.e. to identify the servlet instance. All servlet instances were maintained by the servlet container and hence getting a hold with them is tough. Well, getting the reference of a servlet from the ServletContext method was easy and official during 2001/2002 , but the issue was to identify the exact subject for kill i.e. a correlation. To establish a correlation the participants in the communication channel i.e. the applet and the servlet should be marked / tagged with a unique id (a correlation id) that can be used later to get a reference of the participants.
Figure 2 explains the modified architecture to include the solution. The solution technique was conceptualized as below.

Figure 2

1. The applet connects with the StockPriceServlet.
2. The StockPriceServlet generates a unique id (correlation id) and sends back the same to the applet with the first response before entering the loop. The StockPriceServlet should have a flag to terminate the while loop e.g. “while(!completed){}” instead of “while(true){}”. The StockPriceServlet should also have a unique id generator (preferably a singleton object).
3. The StockPriceServlet sets an attribute to the ServletContext (ServletContext.setAttribute()) with its own reference against the unique id.
4. Once the applet receives the unique id it should store the same.
5. Once the applet is stopped or terminated it should be able to sense the same (may be by using a JavaScript or a close button) and invoke the KillerServlet with the unique id.
6. The KillerServlet receives the unique id and looks up for the same in the ServletContext. (ServletContext.getAttribute()).
7. On receiving the appropriate reference to the StockPriceServlet instance the KillerServlet should update the completion flag to “true” in the StockPriceServlet instance.
8. Once the while loop is terminated, the StockPriceServlet stops the JMS subscriber and cleans up the message cache.

We did not want to invoke the destroy method to avoid touching the lifecycle controls and left the rest of things on the servlet container.

TOGAF and SOA

noreply@blogger.com (Partha S Sengupta) — Mon, 8 Mar 2010 02:09:00 -0800

I am with a mission to define the technology architecture i.e. the phase D in TOGAF ADM (http://www.opengroup.org/architecture/togaf8-doc/arch/chap03.html). Days are changing, the current trend (try with google) is SOA :)
So everything should be done in SOA way. TOGAF is an architecture framework while SOA is an architectural style. TOGAF does not mention any specific architectural style to be followed, so I can use SOA with out any issue. Technology architecture as defined by TOGAF needs input from the business and application architecture as well.
During the last couple of years people used object oriented paradigm more conveniently with most of the IT system architecture, analysis, design and development. SOA can be used as an architectural style during the analysis (not sure if the business can always be analised in terms of services) design and modeling but once it comes to the implementation arena it's no more SOA. Services are implemented as web services or something else similar to that. Finally it boils down to files, dlls, java classes, exe, xml file etc. It's hard to distinguish the bricks collapsed out of a charch or a library or a skyscraper after an earthquake :)
When there's no abstraction, there's no SOA or object orientation or any other style, they are all the same. Finally all the services and objects are going to be converted to the binary language for the machine to execute.
To me the technology architecture that talks about the concrete deployment environment for a system or application has got very less to do with SOA.
Digging the lower level details of the BPEL process engines or ESBs will return the application server underlying. ESB and BPEL process engines are mostly deployed as applications on the top of an application server for most of the vendors (be it Webspehere or Weblogic).

Multimedia search engine

noreply@blogger.com (Partha S Sengupta) — Wed, 24 Feb 2010 00:39:00 -0800

One more abstract useless thought.
Till date the search engines that we use on the internet are also text-based search engines. These search engines will typically have a box to put your text followed by a neat button to click on to begin with the search. On submission of your search request you will get a well formatted page containing all the search results. Usually a bunch of links following different websites are presented. In case it's an image or video search links to the image/video resources are listed.
Computers are intended to become close to human brain with the ability to interact using all the sense organs like human beings. Will you recognize my face from a group photo taken while I was in school? tough :) If you can I must say you have a good eye-sight and power of pattern recognition, that's human brain. I hope someday we will have search engines to upload the image of my face and find out a group photo posted somewhere on the net. Exciting, aint it!! we can do this with many media patterns such as

# image
# video
# sound/music

I could have added more to the list if we had smell, taste and touch :) but we don't have input/output devices for them yet. ooops!! sorry it's not my concept (well every computer user might have thought this way while seeing a lovely photo of a food S/he likes) but has been patented recently (http://www.patentstorm.us/patents/7562816/description.html). I would hope we will see it commercially available in the market soon. Even there are touch printers in the market too (http://www.gizmag.com/sense-taste-smell-touch-printer/13689/).

Image pattern recognition will help greatly in finding out photos from the internet for the purpose of research, investigation.
Sound patterns would track down the copied or pirated music ;) and videos will help in tracking down piracy.

I don't see my words in your design

noreply@blogger.com (Partha S Sengupta) — Fri, 12 Feb 2010 01:42:00 -0800

It has always been a long and tough session to make the business people understand the thought process behind attaining a particular design model. In my career as a software designer I faced several categories of stakeholders for my design such as the IT-managers from the client's side, technical IT people, business people, end-users etc.
Usually we go and meet these users to grab the requirement of the system/application they are looking for and then try to draw a mapping between the requirement and the software logical constructs such as classes, objects, services, components etc. During the process some new terms pop up based on the abstraction of the designer which might not surface the exact word specified in the requirement. Designers try to map the features/requirement of the application with the logical software constructs which is sometimes beyond the understanding of some of the stakeholders.
Based on the technical knowledge on software design of the stakeholder it becomes harder or easier to bridge the gap and make them happy. I am sure all the designers have experienced these kinds of sessions where the audience is shouting at you syaing "I don't see my words in your design, make me understand how are you arriving at this model of yours..". It will be followed by a long discussion of why I did it.
For example, a person who drives a car might not be very familier with the internal structure or mechanism of a chasis or gear box, but you have to make him/her understand how is it related to the process of driving else S/he will not buy the car... I just had this kind of session today :)

Websphere Datapower

noreply@blogger.com (Partha S Sengupta) — Mon, 25 Aug 2008 11:15:00 -0700

IBM came up with a hardware to work as an Enterprise Service Bus (ESB). Well not entirely. It's called Datapower, a part of Websphere SOA solution stack. In our project there's a need to convert data from a mainframe fixed field format to XML and vice versa. So we were looking for a solution that can support a huge volume of transactions in a fraction of second. Using an ESB could be a possible approach but this one is easy to setup and since the conversion is done through hardware, it's really fast. They have three models for the appliance - XS35, XA40 and XI50. We were interested about the XI50 which includes the features of XS35 and XA40. It's a big (approx. 2.5ft / 2ft) blue box weighing nearly 20lbs. (I was about to break my back while trying to carry it). The cooling fans make loud noises. It has got an USB interface to the computer. You can set up the initial configurations using that and then it will let you access a web-interface (embedded as a firmware - like a wireless router) to handle configuration. Among the other main features

Transforms disparate message formats - binary, legacy, and XML. It can load XSL templates, so XML to XML translations can be carried out using XSLT.

Provides message routing (basic routing using round-robin) and security supporting standards such as WS-Security and WS-SecurityPolicy.

Able to connect with MQ, HTTP or FTP.

Helps heterogeneous applications to connect to registries and repositories, as well as directly to database.

Able to translation modules generated by WTX (Websphere Transformation Extender). We tried this while converting XML from/to fixed field format. So it supports non-XML such as COBOL Copybooks.
A piece of routing logic can be injected using the routing policies.

Here is a link for more information http://www-01.ibm.com/software/integration/datapower/

It doesn't support adapters like ESBs. It's a hardware so it can give you a power of processing equivalent to 10 servers running ESBs. Looking at the current prices of servers and the limitations on the customization making a selection will be a little tough. Unless it's not an XML to XML conversion WTX should be kept handy.

Pluggable knowledge base for human brain

noreply@blogger.com (Partha S Sengupta) — Mon, 14 Jul 2008 11:22:00 -0700

I bought my first 1GB USB flash drive in 3300 Indian Rupees ($80 approx.) in 2004. Nowadays I see 4GB flash drives are available in $14.99. It's so cheap these days and I hope someday we are going to have 1TB USB drive in $50 or less. USB has become universal and you can find it in computers, cameras, media players bluetooth devices, external drives; all most any device that's pluggable with a computer.
I remember the movie Terminator to be good instance of man-machine fusion. Imagine someday we all will have an USB port attached to our heads. All our knowledge base will be made available in market in USB drives so that we can plug that into our brain and use them. An idea called Brain-Computer Interface close to this is already out in the market (http://en.wikipedia.org/wiki/Brain-computer_interface).

The OS vendors can see a nice business case in developing a firmware to install in the brain so that it understand and interact with the USB port.
An interpretation layer to translate the data from the external media in a format that the brain can understand (may be an electric pulse language :))
A lucrative business opportunity for Google to come up with a search utility to look up the knowledge base and find out the required material and supply it to the brain :)
We will have knowledge bases available for a doctor, an engineer, a lawyer, software engineer and whatever else you can think of under this sky.
If I need to be a doctor in the morning a pilot in the evening and a cook in the night, I can do every thing very skillfully with all these equipments :)
Well you will still need to practice if you are going for a swimming competition.

Yes you are correct I have some free time to relax my brain and talk rubbish :)

Code Generator Part VIII: The generated code

noreply@blogger.com (Partha S Sengupta) — Tue, 17 Jun 2008 10:45:00 -0700

As the concluding part of the Code Generator series here is an example of how the generator works to generate the code. In this post I am going to present a sample object definition as the input (order.xml) to generate couple of Java source files using the home made code generator.

A piece of code generated using your home made Code Generator

Here is an example of an XML object specification for Customer, Order, OrderItem and Item objects to be supplied as the input for the code generation.

Order.xml

<?xml version="1.0" encoding="UTF-8"?>

<OOSpec name="Order">
<Package name="">
<Class name="Customer" stereotype="class" visibility="public" baseInterfaces="Transformer">
<Import reference="java.util.Vector"/>
<Attribute name="code"/>
<Attribute name="names" multiplicity="3" defaultValue="Partha,Sarathi,Sengupta"/>
<Attribute name="age" type="integer"/>
<Attribute name="contacts" multiplicity="*"/>
<Operation constructor="yes" documentation="This is default constructor"></Operation>
<Operation directive="" documentation="This is transform operation" name="transform" returnType="string" bodyClassName="com. codegenerator.ToXMLStringBody">
</Operation>
</Class>
<Class name="Order" baseInterfaces="Transformer">
<Attribute name="number"/>
<Attribute name="date" type="date"/>
<Operation constructor="yes"></Operation>
<Operation name="transform" returnType="string" bodyClassName="com. codegenerator.ToXMLStringBody" synchronized="yes">
</Operation>
</Class>
<Class name="OrderItem" baseInterfaces="Transformer">
<Attribute name="number" type="integer"/>
<Attribute name="quantity" type="integer"/>
<Operation constructor="yes"></Operation>
<Operation name="transform" returnType="string" bodyClassName="com. codegenerator.ToXMLStringBody">
</Operation>
</Class>
<Class name="Item" documentation="This is a sample documentation">
<Attribute name="number"/>
<Attribute name="description" defaultValue="This is a description of the item"/>
<Attribute name="price" type="double" defaultValue="2.12"/>
<Operation constructor="yes"></Operation>
<Operation name="transform" returnType="string" visibility="public" bodyClassName="com.codegenerator.ToXMLStringBody">
</Operation>
</Class>
<Class name="Transformer" stereotype="interface" baseInterfaces="java.io.Serializable">
<Operation name="transform" returnType="string"></Operation>
</Class>
</Package>
<Association name="">
<Role class="Customer" multiplicity="1" name="" start="yes" navigable="no"/>
<Role class="Order" multiplicity="2" name="myOrders" navigable="yes"/>
</Association>
<Association name="">
<Role class="Order" multiplicity="1" name="order" start="yes" navigable="no"/>
<Role class="OrderItem" multiplicity="4" name="orderItems" navigable="yes"/>
</Association>
<Association name="">
<Role class="OrderItem" multiplicity="1" name="" start="yes" navigable="no"/>
<Role class="Item" multiplicity="1" name="item" navigable="yes"/>
</Association>
</OOSpec>

The Java source codes generated from the above specification are presented below. Notice how the transform method has been generated differently in each class.

Customer.java

/* This file has been generated by the CodeGenerator */
package com.order;
import java.util.Vector;

public class Customer implements Transformer {
private String code;
private String[] names;
private int age;
private java.util.ArrayList contacts;
private Order[] myOrders;

public String getCode() { return this.code; }
public void setCode(String code) { this.code = code; }
public String[] getNames() { return this.names; }
public void setNames(String[] names) { this.names = names; }
public int getAge() { return this.age; }
public void setAge(int age) {this.age = age;}
public java.util.ArrayList getContacts() {return this.contacts;}
public String getContacts(int index) {return (String)this.contacts.get(index);}
public void setContacts(java.util.ArrayList contacts) {this.contacts = contacts;}
public void addContacts(String contacts) {this.contacts.add(contacts);}
public Order[] getMyOrders() {return this.myOrders;}
public void setMyOrders(Order[] myOrders) { this.myOrders = myOrders;}
public Customer() {
code = new String();
names = new String[3];
names[0] = new String("Partha");
names[1] = new String("Sarathi");
names[2] = new String("Sengupta");
age = 0;
contacts = new java.util.ArrayList();
myOrders = new Order[2];
for(int i=0; i<myOrders.length; i++) { myOrders[i] = new Order(); }
}

/*This is transform operation*/
public String transform() {
StringBuffer sb = new StringBuffer();
if(code != null && !code.toString().equals("")) {
sb.append("<code>"+code+"</code>");
}
else {
sb.append("<code/>");
}
for(int i=0; i<names.length; i++) {
if(names[i] != null && !names[i].toString().equals("")) {
sb.append("<names>"+names[i]+"</names>");
}
Else {
sb.append("<names/>");
}
}
sb.append("<age>"+age+"</age>");

for(int i=0; i<contacts.size(); i++) {
if(contacts.get(i) != null && !contacts.get(i).toString().equals("")) {
sb.append("<contacts>"+((String)contacts.get(i)).toString()+"</contacts>");
}
Else {
sb.append("<contacts/>");
}
}
for(int i=0; i<myOrders.length; i++) {
if(myOrders[i] != null){
sb.append("<myOrders>");
sb.append(myOrders[i].transform());
sb.append("</myOrders>");
}
Else {
sb.append("<myOrders/>");
}
}
return sb.toString();
}
}

Order.java

/* This file has been generated by the CodeGenerator */

package com.order;
import java.util.*;

public class Order implements Transformer{
private String number;
private java.util.Date date;
private OrderItem[] orderItems;
public String getNumber() {return this.number;}
public void setNumber(String number) { this.number = number; }
public java.util.Date getDate() { return this.date; }
public void setDate(java.util.Date date) { this.date = date; }
public OrderItem[] getOrderItems() { return this.orderItems; }
public void setOrderItems(OrderItem[] orderItems) { this.orderItems = orderItems; }
public Order(){
number = new String();
date = new java.util.Date();
orderItems = new OrderItem[4];
for(int i=0; i<orderItems.length; i++) {
orderItems[i] = new OrderItem();
}
}
public synchronized String transform() {
StringBuffer sb = new StringBuffer();
if(number != null && !number.toString().equals("")){
sb.append("<number>"+number+"</number>");
}
else {
sb.append("<number/>");
}
if(date != null && !date.toString().equals("")) {
sb.append("<date>"+date+"</date>");
}
else {
sb.append("<date/>");
}
for(int i=0; i<orderItems.length; i++) {
if(orderItems[i] != null){
sb.append("<orderItems>");
sb.append(orderItems[i].transform());
sb.append("</orderItems>");
}
else {
sb.append("<orderItems/>");
}
}
return sb.toString();
}
}

Item.java

/* This file has been generated by the CodeGenerator */

package com.order;
import java.util.*;

/*This is a sample documentation*/
public class Item implements java.io.Serializable{
private String number;
private String description;
private double price;
public String getNumber() { return this.number; }
public void setNumber(String number) { this.number = number; }
public String getDescription() { return this.description; }
public void setDescription(String description) { this.description = description; }
public double getPrice() { return this.price; }
public void setPrice(double price) { this.price = price; }
public Item() {
number = new String();
description = new String("This is a description of the item");
price = 2.12;
}
public String transform(){
StringBuffer sb = new StringBuffer();
if(number != null && !number.toString().equals("")){
sb.append("<number>"+number+"</number>");
}
else{
sb.append("<number/>");
}
if(description != null && !description.toString().equals("")) {
sb.append("<description>"+description+"</description>");
}
else{
sb.append("<description/>");
}
sb.append("<price>"+price+"</price>");
return sb.toString();
}
}

OrderItem.java

/* This file has been generated by the CodeGenerator */

package com.order;
import java.util.*;

public class OrderItem implements Transformer{
private int number;
private int quantity;
private Item item;
public int getNumber(){ return this.number; }
public void setNumber(int number) { this.number = number; }
public int getQuantity(){ return this.quantity; }
public void setQuantity(int quantity){ this.quantity = quantity; }
public Item getItem(){ return this.item; }
public void setItem(Item item){ this.item = item; }
public OrderItem(){
number = 0;
quantity = 0;
item = new Item();
}
public String transform(){
StringBuffer sb = new StringBuffer();
sb.append("<number>"+number+"</number>");
sb.append("<quantity>"+quantity+"</quantity>");
if(item != null && !item.toString().equals("")) {
sb.append("<item>");
sb.append(item.transform());
sb.append("</item>");
}
else{
sb.append("<item/>");
}
return sb.toString();
}
}

Hope this article will help those who are looking for preliminary information on the concepts and technologies used in code generation. Please feel free to leave your comments.

Code Generator Part VII: Controller Layer

noreply@blogger.com (Partha S Sengupta) — Wed, 4 Jun 2008 13:48:00 -0700

Finally here comes the control room for the code generation process. This is the place where the Importer, Exporter and the IOM get together to generate the code for me. This layer controls the orchestration of the entire process and is designed to be unaffected by any change in the Importer, Exporter or IOM layers. In my next post I will try to provide an example to explain how the code generator actually works in reality and generates code.

The Control Layer
It’s layer from where the Code Generator gets executed by the user. It has mainly two classes Generator and GeneratorUtility as shown in Fig 1.

Fig 1

This shows that the Generator class instantiates the Importer and Exporter type of classes. In case of Java source code generation, the Generator instantiates the XMLOOImporter (Fig 6) which implements the Importer interface for importing the XML input model. Once the XML model is imported and stored in the memory in the form of IOM model, the Generator instantiates the JavaExporter (Fig 7) which implements the Exporter interface for generating the Java source files as per the IOM model (Fig 3).

Generator is the common entry point for both code generation and template generation processes. This class has mainly two methods called “generate” and “main”.
The “generate” method takes 4 parameters as input – input model type (i.e. XML), output source code language (i.e. Java), name of the input model file (i.e. XML model file name) and the output destination directory. To generate the source codes, this class invokes the importer and the exporter simultaneously to read the class specifications from the XML input model file and to generate Java source code according to the class specifications.
Since this class also has a “main” method, it can be invoked from the command-line by using the 4 parameters described above. The Generator class is shown in figure 1.

GeneratorUtility is a common utility class that gets used by most of the classes involved in the code generator system. It contains some static values and utility methods used for intermediate processing or formatting. The GeneratorUtility class is shown in figure 1.
Below is a high-level sequence diagram of the Code Generator.

Fig 2

Actor invokes the Code Generator by calling the generate() method of the main class “Generator”. The actor specifies the input type (i.e. XML), output type (i.e. Java), input model file (XML file) name and the output destination (directory or path) while calling this method (Fig 2).
In case of Java source code generation, the generate() method instantiates XMLOOImporter class (which implements Importer) and invokes its start() method to read the class specifications from the XML model file provided as input (Fig 2). Once the classes are identified from the input model (XML), Generator instantiates the JavaExporter class (which implements Exporter interface) to generate java source files from those classes as specified in the XML model (Fig 2). At the end of generating the source files the Code Generator system will display a list of source files that have been generated and a message with the number of source files. Incase an invalid (not an XML) input document is provided to the system it will throw an exception and display the stack-trace for the exception.
Figure 3 shows detailed sequence diagram of the code generation process.

Fig 3

This section provides the relationship or mapping between the XML-tags used in the XML model supplied as the input to the Code Generator and the corresponding IOM class that represents the tag in the memory object model (IOM).
Figure 4 shows the relationship between different components defined in XML and realized in IOM for the source code generation process. Blue boxes marked with "XML Tag" in the diagram represents the tags used in the input XML model and other boxes are representing the IOM classes already shown in Part IV of this series. The attributes shown in the XML-tags (blue boxes) are the XML-attributes. The names of the XML-attributes are quite identical to those of the corresponding IOM class e.g. the attributes – name, stereotype, visibility, documentation, isStatic, isFinal, isAbstract etc. of IOMClass are having identical names with the respective XML-attributes (i.e. name, stereotype, visibility, documentation, static, final, abstract) in the Class tag. The attributes that are not in the XML-tags but present in corresponding IOM class are used for internal processing of Code Generator.
Since XML does not allow data types other than character strings, the data type of XML-attributes is given as string. After the XML input model file is parsed, individual XML tags are parsed and translated to the respective IOM objects and stored as the internal object model in the memory. During the translation of input XML the string data types are also converted to appropriate Java data types e.g. value of isStatic is given as “yes/no” in the XML, it’s converted to Java boolean (true/false) type. While generating the output the exporter reads the internal object model from the memory and generates appropriate documents accordingly.

Fig 4

Code Generator Part VI: Exporter Layer

noreply@blogger.com (Partha S Sengupta) — Wed, 21 May 2008 14:17:00 -0700

It's been a long time I managed to keep myself busy :) , back again to stretch a little bit on the blog. It's the turn of the Exporter part of my code generator. I would try to put together a rough idea about the design model of the Exporter. This layer is responsible for exporting the object model (IOM) that has been created in the memory by the Importer to a desired code (e.g. Java, C++ etc.). The input for the exporter should be the IOM (stored in the memory) and the output should be the generated code in a desired programming language or any other format (e.g. CSV, RTF), provided the exporter has to be made knowledgeable enogh to transform the IOM into the desired format. I will try put the pieces (Importer, Exporter, IOM) together in a a controlling framework in my next post.

The Exporter Layer

The exporter has the responsibility of translating the IOM created by the importer and generating the desired output document (i.e. Source Codes or Documents in a specific format).
Incase of code generation the exporter translates the IOM created by the importer and generates the desired output source codes from the IOM. The code generator can have different exporters depending on different output types (i.e. output languages like Java, C++ etc.). Currently we are generating the Java source codes by the exporter. The exporter has been designed using a generic interface (Exporter) as the parent. Any class that generates source codes in a specific object-oriented language from the IOM should have the Exporter interface as its parent. In our case we have an exporter class (JavaExporter) that exports Java source codes according to the object-oriented class specification as defined in the input XML model. This class, called JavaExporter implements another interface called OOCodeExporter which already implements the Exporter interface (Fig 7). The reason behind introducing the OOSpecExporter interface is to encapsulate the activities involved in generating source code in all object-oriented languages in a single interface. This way the Exporter interface can be further implemented by other exporters for generating source codes in different object-oriented languages other than Java (e.g. C++, C# etc.). Below are the descriptions for the classes that belong to the exporter layer.
Please refer to Fig 1 for the classes involved in the Exporter layer.

Fig 1

Exporter is designed as a generic interface. The interface contains the start(), initialize() and finalize() methods. The main entry point of the code generator i.e. the Generator class invokes the exporter using the start() method of this interface. To use the code generator for generating source code in a specific object-oriented language (e.g. Java, C++, C# etc.), new exporter classes can be introduced implementing the same Exporter interface.

OOCodeExporter is an interface implementing the Exporter interface. The purpose of this interface is to encapsulate the operations required to generate source codes in object-oriented languages. Currently the JavaExporter implements this interface to generate Java source codes according to class specifications defined in the input XML model. In future new exporters can be introduced as children to this interface for generating source codes in different OO languages other than Java, e.g. C++, C#, VB etc. The implementation of the Exporter interface depends on the type of OO languages (Java in our case) in which the source code will be generated, so different languages can have very different implementations.

JavaExporter implements the OOCodeExporter interface. The architecture of this class is closely bound to the SAX parser philosophy. This class works on the IOM objects (i.e. IOMClass, IOMAttribute, IOMOperation etc.) stored in the IOMOOController by XMLOOImplorter. The start() method begins to navigate through each object in the internal object model (IOM) and calls the startClass() method each time a IOMClass object is processed, and in general the startXxx() method each time an instance of IOMXxx is processed. The exporter then invokes the endXxx() methods when an instance of IOMXxx is finished to be processed. It creates an output file using the name of the class and adding the JAVA extension. Then it writes, by using the information coming from the IOMClass object, Java code for declaring a class. By implementing all the methods, the JavaExporter class generates the Java classes corresponding to XML model in terms of attributes, operations, and associations. The JavaExporter works based on certain assumptions as described below.
The generated Java source files involved in the DTOs will by default
- Do not need to implement a base class.
- Implement the "Serializable" interface.
- Import "java.util.*" and “java.io.Serializable”.
- Have the stereotype as "class".
- Have visibility "private" for attributes.
- Uses public getter/setter operations.
- Has non-final and non-static attributes or operations.
- Use attributes with type "String".
- Use attributes with multiplicity as "1".