Shayan's Weblog

Batch Processing System - EAI

2010-03-22T12:28:00.000-07:00

Batch Processing

Batch Processing is execution of a series of programs ("jobs") on a computer without manual intervention.

Batch jobs are set up so they can be run to completion without manual intervention, so all input data is preselected through scripts or command-line parameters. This is in contrast to "online" or interactive programs which prompt the user for such input. A program takes a set of data files as input, process the data, and produces a set of output data files. This operating environment is termed as "batch processing" because the input data are collected into batches on files and are processed in batches by the program.

Benefits

Batch processing has these benefits:

• It allows sharing of computer resources among many users and programs,
• It shifts the time of job processing to when the computing resources are less busy,
• It avoids idling the computing resources with minute-by-minute manual intervention and supervision,
• By keeping high overall rate of utilization, it better amortizes the cost of a computer, especially an expensive one.

Modern Systems

Despite their long history, batch applications are still critical in most organizations. While online systems are now used when manual intervention is not desired, they are not well suited to the high-volume, repetitive tasks. Therefore, even new systems usually contain a batch application for cases such as updating information at the end of the day, generating reports, and printing documents.
Common batch processing usage
Databases
Batch processing is used for efficient bulk database updates and automated transaction processing, as contrasted to interactive online transaction processing (OLTP) applications.
Converting
Batch processing is also used for converting a number of computer files from one format to another. This is to make files portable and versatile especially for proprietary and legacy files where viewers are not easy to come by.

Job scheduling

A job scheduler is an enterprise software application that is in charge of unattended background executions, commonly known for historical reasons as batch processing.
Synonyms are batch system, Distributed Resource Management System (DRMS), and Distributed Resource Manager (DRM). Today's job schedulers typically provide a graphical user interface and a single point of control for definition and monitoring of background executions in a distributed network of computers. Increasingly job schedulers are required to orchestrate the integration of real-time business activities with traditional background IT processing, across different operating system platforms and business application environment
Features
Basic features expected of job scheduler software are:
• Interfaces which helps to define workflows and/or job dependencies
• Automatic submission of executions
• Interfaces to monitor the executions
• Priorities and/or queues to control the execution order of unrelated jobs
Organizations needing to automate highly complex related and un-related IT workload will also be expecting more advanced features from a job scheduler, such as:
• Real-time scheduling based on external, un-predictable events
• Automatic restart and recovery in event of failures
• Alerting and notification to operations personnel
• Generation of incident reports
• Audit trails for regulatory compliance purposes

Main Concepts

There are many concepts that are central to almost every job scheduler implementation and that are widely recognized with minimal variations:
• Jobs
• Dependencies
• Job Streams
• Users

Workload Automation

The concept of Workload Automation is an evolution of traditional job schedulers which needed to react to the dynamic demands of IT. Workload automation needs to be able to coordinate—in real-time—a varied set of workload types with complex dependencies across a broad spectrum of operating systems and application platforms.
A scheduling model based only upon calendar events is not sufficient to meet the demands of today’s automated data centers. The requirement for batch submission of workloads driven by date/time dependencies still exists, but on-demand IT processing requirements have expanded job submission triggers far beyond the time dimension.
Workload Automation solutions should provide: a service-oriented architecture (SOA); integration capabilities at the application services level for web services and J2EE-based applications; a service-level orientation for managing workloads and finally, a critical path analysis and reporting capabilities.

Cybermation ESP

Cybermation ESP is a good example of an event-driven, cross platform Enterprise Job Scheduling solution that provides the opportunity to use a single point of control across all environments, operating systems, applications and databases. By consolidating multiple scheduling tools and providing real-time automation, Data Center Managers benefit from these capabilities to mask the complexity of their heterogeneous environment, gain from increased agility, and also limit the number of resources needed to manage their systems.
History

Job Scheduling has a long history. Job Schedulers are one of the major components of the IT infrastructure since the early mainframe systems. At first, stacks of punch cards were processed one after the other, hence the term “batch processing.”
From a historical point of view, we can distinguish two main eras about Job Schedulers:

• The mainframe era - Job Control Language (JCL) on IBM mainframes: Initially based on JCL functionality to handle dependencies this era is typified by the development of sophisticated scheduling solutions forming part of the systems management and automation toolset on the mainframe.

• The open systems era - Modern schedulers on a variety of architectures and operating systems: With standard scheduling tools limited to such as Cron, the need for mainframe standard job schedulers has grown with the increased adoption of distributed computing environments.

In terms of the type of scheduling there are also distinct eras:

• Batch processing - the traditional date and time based execution of background tasks based on a defined period during which resources were available for batch processing (the batch window). In effect the original mainframe approach transposed onto the open systems environment.

• Event-driven process automation - where background processes cannot be simply run at a defined time, either because the nature of the business demands that workload is based on the occurrence of external events (such as the arrival of an order from a customer or a stock update from a store branch) or because there is no / insufficient batch window.

• Service Oriented job scheduling - recent developments in Service Oriented Architecture (SOA) have seen a move towards deploying job scheduling.

Scheduling

Various schemes are used to decide which particular job to run. Parameters that might be considered include:
• Job priority
• Compute resource availability
• License key if job is using licensed software
• Execution time allocated to user
• Number of simultaneous jobs allowed for a user
• Estimated execution time
• Elapsed execution time
• Availability of peripheral devices
• Occurrence of prescribed events

Sample implemented Patch Processing System

Following diagram illustrates different pieces of an EAI system implemented in a large scale enterprise application (CMS project at WorkSafeBC). The purpose of this system is to provide a reliable asynchronous data transformation between External systems: MSP/Emergis and newly implemented system:CMS which includes following components:
• IBM z/OS Mainframe
• FTP Location (for incoming/outgoing files)
• Mainframe Jobs (written in JCL)
• ESP Job Schedular
• MS Windows BAT files
• Java Script files
• Curam Batch Launcher
• CMS interfaces (Batch Processes)

IBM z/OS Mainframe

z/OS is a 64-bit operating system for mainframe computers, created by IBM. It is the successor to OS/390, which in turn followed MVS and combined a number of formerly separate, related products. z/OS offers the attributes of modern operating systems but also retains much of the functionality originating in the 1960s and each subsequent decade that is still found in daily use. (Extreme backward compatibility is one of z/OS's central design philosophies.) It is derived from OS/390 and was introduced in October of 2000.
z/OS supports mainframe staple technologies such as CICS, IMS, DB2, RACF, SNA, WebSphere MQ, record-oriented data access methods, REXX, CLIST, SMP/E, JCL, TSO/E, and ISPF. However, z/OS also supports 64-bit Java, C/C++, and UNIX (Single UNIX Specification) APIs and applications, with UNIX/Linux-style hierarchical HFS and zFS file systems. As a result, z/OS now can host Oracle database (RDBMS software) and SAP(ERP software). z/OS can communicate directly via TCP/IP, including IPv6, and includes a standard HTTP server along with other common services such as FTP, NFS, and CIFS/SMB. Another central design philosophy is support for extremely high quality of service (QoS), even with a single operating system instance, although z/OS has built-in support for Parallel Sysplex clustering.

FTP Location

This is the location of the incoming/outgoing files as well as intermediate encoded files produced as the result of intermediate main frame jobs in different stages.

MSP and Emergis Systems would FTP their data files such as transactions, reference data, providers information and etc to this location. These data would arrive on daily/weekly/Bi-weekly/monthly/on-demand basis to be processed by CMS.

On the other hand, where WorkSafeBC needs to send the result of processes to External Systems, it would FTP the outgoing files from this location. These data includes: pharmacy drug profile confirmation file, invoice decision file and etc.

Mainframe Jobs

These jobs are written in JCL and basically are similar to legacy code with some changes to be compatible with
CMS system. The main purpose of these jobs are to encoding the file from EBCDIC to ASCII, Sorting and FTP the out put file to the location where CMS jobs would pick it up for further processing.

ESP Job Scheduler

Cybermation ESP is used as a facility to definition and monitoring the jobs based on the information given in design time through worksheets which mainly includes the description of the job for ESP users, timing of the jobs: daily/weekly/bi-weekly/monthly, events that would result in submission of the job, dependencies to other jobs and manual/automatic actions to be taken based on return codes from the jobs.

MS Windows BAT files

BAT file are called by scheduler which subsequently would call java programs (CMS interfaces) through Curam Batch launcher. The main purpose of these BAT files are to located the input file or submit the output file, execute the batch launcher, eventually returning the result of batch process to ESP scheduler.

Java Script files

In some circumstances, there is a need to pull the input file from the external system by java scripts and eventually making it ready for other jobs to further processing.
Curam Batch Launcher
Curam Batch Launcher is the starting point to run the CMS interfaces (batch processes). This utility is called by BAT files.

Interfaces (Batch Processes)

These are the java programs to load the input files to CMS database after validation of the input file for inbound interfaces or generate output file in case of outbound interfaces such as invoice decision files. To avoid locking the Data Base tables for long period of time, checkpoint restart mechanism is used. I.e., after processing of certain number of records, transaction is committed to Data Base and check point is stored. Besides, in case job failure in middle of the process, the ESP Scheduler would restart the job from last check-point providing the job runtime id.

Enterprise Application Integration

2010-02-18T10:15:00.000-08:00

EAS as System integration

In engineering, system integration is bringing together of the component subsystems into one system and ensuring that the subsystems function together as a system. In information technology, systems integration is the process of linking together different computing systems and software applications physically or functionally.

The system integrator brings together discrete systems utilizing a variety of techniques such as
A. computer networking
B. enterprise application integration
C. business process management
D. Manual programming.
In this document we are covering enterprise application integration.

EAI Concept

Enterprise Application Integration is an integration framework composed of a collection of technologies and services which form a middleware to enable integration of systems and applications across the enterprise. Enterprise Application Integration (EAI) is defined as the use of software and computer systems architectural principles to integrate a set of enterprise computer applications. One of the Purposes of EAI is Data (information) Integration, ensuring that information in multiple systems is kept consistent. This is also known as EII (Enterprise Information Integration).

Enterprise Application Integration is related to middleware technologies such as message-oriented middleware (MOM), and data representation technologies such as XML. Other EAI technologies involve using web services as part of service-oriented architecture as a means of integration. Enterprise Application Integration tends to be data centric. In the near future, it will come to include content integration and business processes.

EAI Mediation pattern

Here, the EAI system acts as the go-between or broker between (interface or communicating) multiple applications. Whenever an interesting event occurs in an application (e. g., new information created, new transaction completed, etc.) an integration module in the EAI system is notified. The module then propagates the changes to other relevant applications.

EAI Implementation Pitfalls

In 2003 it was reported that 70% of all EAI projects fail. Most of these failures are not due to the software itself or technical difficulties, but due to management issues. Integration Consortium European Chairman Steve Craggs has outlined the seven main pitfalls undertaken by companies using EAI systems and explains solutions to these problems.
• Constant change
The very nature of EAI is dynamic and requires dynamic project managers to manage their implementation.
• Shortage of EAI experts
EAI requires knowledge of many issues and technical aspects.
• Competing standards
Within the EAI field, the paradox is that EAI standards themselves are not universal.
• EAI is a tool paradigm
EAI is not a tool, but rather a system and should be implemented as such.
• Building interfaces is an art
Engineering the solution is not sufficient. Solutions need to be negotiated with user departments to reach a common consensus on the final outcome. A lack of consensus on interface designs leads to excessive effort to map between various systems data requirements.
• Loss of detail
Information that seemed unimportant at an earlier stage may become crucial later.
• Accountability
Since so many departments have many conflicting requirements, there should be clear accountability for the system's final structure.
Other potential problems may arise in these areas:
• Emerging Requirements
EAI implementations should be extensible and modular to allow for future changes.
• Protectionism
The applications whose data is being integrated often belong to different departments that have technical, cultural, and political reasons for not wanting to share their data with other departments

Advantages and Disadvantages

Advantages:

o Real time information access among systems
o Streamlines business processes and helps raise organizational efficiency
o Maintains information integrity across multiple systems
o Ease of development and maintenance

Disadvantages:

o High initial development costs, especially for small and mid-sized businesses (SMBs)
o Require a fair amount of up front business design, which many managers are not able to envision or not willing to invest in. Most EAI projects usually start off as point-to-point efforts, very soon becoming unmanageable as the number of applications increase

Middleware as EAI solution

Middleware is computer software that connects software components or applications. The software consists of a set of services that allows multiple processes running on one or more machines to interact. This technology evolved to provide for interoperability in support of the move to coherent distributed architectures, which are used most often to support and simplify complex, distributed applications. It includes web servers, application servers, and similar tools that support application development and delivery. Middleware is especially essential to modern information technology based on XML, SOAP, Web services, and Service-Oriented Architecture (SOA).

Message-oriented middleware (MOM)

Message-oriented middleware (MOM) is infrastructure focused on sending and receiving messages that increases the interoperability, portability, and flexibility of an application by allowing the application to be distributed over heterogeneous platforms. It reduces the complexity of developing applications that span multiple operating systems and network protocols by insulating the application developer from the details of the various operating system and network interfaces. API's that extend across diverse platforms and networks are typically provided by MOM.

MOM is software that resides in both portions of client/server architecture and typically supports asynchronous calls between the client and server applications. Message queues provide temporary storage when the destination program is busy or not connected. MOM reduces the involvement of application developers with the complexity of the master-slave nature of the client/server mechanism.
MOM comprises a category of inter-application communication software that generally relies on asynchronous message-passing, as opposed to a request-response metaphor.
Most message-oriented middleware depend on a message queue system, but there are some implementations that rely on broadcast or multicast messaging systems. Below is two different technologies to implement MOM.

Java Message Service (JMS)

The Java Message Service (JMS) API is a Java Message Oriented Middleware (MOM) API for sending messages between two or more clients. JMS is a part of the Java Platform, Enterprise Edition. It is a messaging standard that allows application components based on the Java 2 Platform, Enterprise Edition (J2EE) to create, send, receive, and read messages. It enables distributed communication that is loosely coupled, reliable, and asynchronous.

Microsoft Message Queuing

MSMQ is essentially a messaging protocol that allows applications running on separate servers/processes to communicate in a failsafe manner. A queue is a temporary storage location from which messages can be sent when conditions permit. This enables communication across heterogeneous networks and between computers which may not always be connected. By contrast, sockets and other network protocols assume that direct connections always exist.

Sample implemented EAI System

Following diagram illustrates different pieces of an EAI system implemented in a large scale enterprise application. The purpose of this system is to provide a reliable asynchronous data transformation between existing systems and newly implemented system. EAI Components are as below:

o Data Base Replication
o Triggers & Stored Procedures
o Messaging Controller
o Messaging DB (XML)
o Messaging Service System which consists of
o Message Producer
o Message Consumer
o Messaging Service Provider
o Message Queue

Database Replication

Database replication is necessary to avoid lowering the performance of existing system due to overhead resulting from middleware. As the result of this replication, existing systems can operate without any system dependency to middleware which ensures reliable asynchronous messaging service.

Triggers & Stored Procedures

When a transaction occurring in the existing system would need to be reflected in the new system then Triggers and Stored Procedures written on Replicated DB come to play. These integration modules are notified as the result of an event in existing system and would create XML messages to be stored in Message DB. E.g., the event of completing the registration of an Employer in Existing System would occur when all necessary information are present in the existing system. This event would cause a trigger on corresponding table on Replicated DB to invoke relevant stored procedure. The Stored Procedure subsequently would collect all required information such as head office address, legal and trade names, etc and format it in XML layout and eventually store in Message DB.

Messaging Service System

This component could be implemented by either Java Message Service (JMS) or Microsoft Message Queuing solutions to send the XML messages between Existing and New applications. When a new message appears in Messaging DB, Message producer extracts the XML data and send it to Message Queue using Messaging Service Provider through established connection.

Web Services and Interfaces

After Message Consumer successfully received the XML message, would create relevant SOAP message and calls the web services for further processing. Web services on the other hand would call interfaces to new System to populate relevant tables. Interfaces code basically would be parsing and validating the XML message before populating relevant tables to be used by new application. In case updates to pervious transferred data is required, mapping tables between Excising and New Systems are used to find the relevant data before any update happens.

Message Controller

If any error such as XML validation error or DB integrity happened in interfaces, Message Consumer
Would receive the log and flag corresponding message as suspended in Message DB. At that time Messaging Controller would take necessary actions to resolve the issue to restart the message processing for awaiting messages in the queue.

Shayan Manoochehri
System Integration Architect

Reference:
http://worldofasp.net/aspnet/Tutorials/Basic_Introduction_about_MSMQ_in_.NET_Framework_98.aspx
http://en.wikipedia.org/wiki/Systems_integration
http://en.wikipedia.org/wiki/Enterprise_application_integration
http://en.wikipedia.org/wiki/Middleware
http://en.wikipedia.org/wiki/Message-oriented_middleware
http://en.wikipedia.org/wiki/Java_Message_Service
http://en.wikipedia.org/wiki/Microsoft_Message_Queuing
http://en.wikipedia.org/wiki/Message_queue