Artificial intelligence (AI) is evolving rapidly, and so are organizations’ approach to its use. In April 2026, BMC released a statement of direction on the integration of AI into the BMC AMI suite of mainframe solutions. In The Next Evolution in Enterprise AI is Purpose Built, BMC mainframe Senior Vice President and General Manager John McKenny discusses the shift from generative AI (GenAI) to agentic AI and our intent to embed agentic AI across the BMC AMI portfolio.

While the statement of direction explains how we’re moving into the future with agentic AI workflows, over the past two years we have developed purpose-built AI for the mainframe, offering in-context expertise through BMC AMI Assistant. With Knowledge Expert Chat, practitioners can harness the power of AI to find the answers they need when and where they need them, increasing the quality and efficiency of their work. Each quarter, we have added to what mainframe professionals can accomplish while increasing integration of BMC AMI Assistant across the BMC AMI portfolio.

The April 2026 release of enhancements to the BMC AMI portfolio focuses on this advancement of mainframe GenAI as well as the open integration of the platform with the broader enterprise IT ecosystem.

GenAI powered by organizational intelligence

Even as we increase the use of agent-driven automation on the mainframe, the efforts to improve GenAI assistance continue, including this quarter’s General Availability of Knowledge Hub, which surfaces institutional knowledge from across the organization to offer contextual information and answers at the moment of decision through BMC AMI Assistant Knowledge Expert Chat.

Knowledge Hub works behind the scenes, ingesting provided institutional knowledge from past issue resolutions, operational insights, and information from runbooks, tickets, and shared files, then combining it with BMC mainframe expertise to create Knowledge Expert Chat responses that are contextually aware and tailored to user’s mainframe environment. These responses are available directly in workflows using existing BMC tools, enabling quicker decisions made with expert-level confidence, regardless of the user’s experience level.

AI-generated application analysis reports

Just as Knowledge Hub captures institutional knowledge to inform assistance provided by Knowledge Expert Chat, a new feature in BMC AMI zAdviser Enterprise adds hard-won institutional knowledge to zAdviser’s collection of development tool and toolchain data to help development managers and their developers understand the applications they are modifying.

BMC AMI zAdviser Enterprise gathers BMC AMI tool usage data and DevOps metrics to give development managers a clear picture of development effectiveness. With new application analysis reports, they now get a single AI-generated view of how their applications work, where the risk is, and where their team’s time is going, accelerating modernization decisions and cutting weeks off of developer onboarding.

BMC AMI zAdviser Enterprise’s Application Analysis turns tribal knowledge into organizational knowledge before it walks out the door. The narrative assessments capture the knowledge of experienced developers, sharing that expertise with development managers and developers, creating more efficient application review, planning, and optimization.

Beyond a greater understanding of individual programs, these application analysis reports provide a clear picture of which programs are attracting a disproportionate share of developer attention because of failures and modifications. By correlating failure history with code complexity and maintenance patterns, they enable development teams to target problem applications with proactive remediation, improving system resilience and development efficiency.

Standardizing enterprise security

Manual management of security certificates decreases efficiency and weakens system security. For some time, security teams have been able to automate certificate management on distributed systems using third-party tools.BMC revolutionized mainframe security management with BMC AMI Enterprise Connector for Venafi. This April, we introduce BMC AMI Certificate Manager, a new product within BMC AMI Security that gives customers more choice and flexibility in integrating enterprise security solutions with the mainframe.

Designed to integrate with IBM Z^® enterprise security management (ESM) environments, including RACF^®, ACF2^™, and Top Secret^® for z/OS to automate the full certificate lifecycle without the need for infrastructure changes, BMC AMI Certificate Manager extends leading enterprise certificate management platforms to the mainframe. This gives CISOs and security teams the ability to use the same vendor solutions for distributed and mainframe certificate management, further integrating the mainframe with—and simplifying—enterprise security efforts.

BMC AMI Certificate Manager currently integrates with Venafi^® and Keyfactor^®, with further integrations to follow in future releases. With this unified integration layer between enterprise certificate management tools and IBM Z^® ESMs, organizations can automate certificate issuance, renewal, and enforcement with one BMC solution while supporting multiple vendors.

Breaking down silos with new technology

This quarter’s enhancements further BMC’s commitment to optimizing what is possible on the mainframe through open integration of the platform and continuous improvements to AI capabilities. With the addition of organization-specific knowledge to AI engines, Knowledge Hub empowers mainframe professionals to make the right decisions as they do their jobs, regardless of their experience and skill levels. Application analysis reports in BMC AMI zAdviser Enterprise combine development and application performance data with system telemetry to provide a clear picture of how applications are performing and where attention and efforts should be focused. And the new BMC AMI Certificate Manager enables security teams to employ certificate management policies across platforms without the need for separate tooling.

Each of these enhancements improves the productivity and efficiency of mainframe teams, a goal BMC is committed to pursuing with each of our quarterly releases. Make your mainframe the engine of faster, better, and smarter answers. When you BMC First.

These are just a few of the innovations included in the April 2026 release of BMC AMI features. To learn more about everything included in the release, visit the What’s New in Mainframe Solutions webpage.

Mainframe organizations continue to run some of the most critical systems in the global economy while managing increasing complexity and rising expectations for innovation. Artificial intelligence (AI) provides a practical way to scale expertise, reduce operational friction, and help teams move work forward with greater confidence.

BMC believes organizations will increasingly leverage AI to broaden access to mainframe capabilities and integrate these systems more fully into their broader enterprise IT ecosystems. In many cases, AI will also act as a catalyst for modernization in place—helping organizations evolve applications, workflows, and operational practices while continuing to run critical workloads on the platform that already powers their business.

The Direction: From AI assistance to coordinated intelligence

Enterprise AI is moving beyond generating insights and explanations. Organizations now expect intelligence that can safely participate in execution across critical systems. Our direction is clear: BMC AMI solutions are evolving into intelligent participants in a governed ecosystem of AI agents that collaborate autonomously across development, operations, data, and security workflows to accelerate innovation on the mainframe.

Supporting this direction requires enterprise AI on the mainframe to operate across three coordinated layers:

Intelligence layer — Domain knowledge and reasoning grounded in mainframe expertise, operational telemetry, and institutional knowledge.

Coordination layer — Orchestrated AI agents collaborating across workflows to connect understanding with action.

Governance layer — Policy-aware controls ensuring AI-driven actions remain secure, transparent, auditable, and subject to human oversight.

Together, these layers enable AI to extend beyond analysis and participate responsibly in execution. This foundation now enables the next phase of our direction: extending AI beyond knowledge and insight toward coordinated, governed execution across the BMC AMI platform.

AI in production: Trust earned, not claimed

In July 2024, we published a Statement of Direction focused on infusing intelligence directly into BMC AMI product experiences. Our objective was straightforward: reduce complexity, close knowledge gaps, and make it easier for teams to work confidently on the mainframe.

That direction was grounded in three core principles:

• Consistent and contextual intelligence, with a common AI foundation infused across the AMI portfolio
• LLM freedom, allowing organizations to choose the models that align with their enterprise AI strategies
• In-the-moment intelligence, embedding AI directly into workflows to surface answers and insights where work happens

Since then, we have delivered.

BMC AMI Assistant was introduced beginning with GenAI-powered code explanation and expanded across consecutive quarterly releases into a cohesive intelligence layer embedded throughout the BMC AMI portfolio. Using AI as an advisor, developers gained faster understanding of unfamiliar applications, while operators gained explainable insight into system and operational issues. Investigation that once required manual analysis increasingly includes guided insight and next steps directly inside the workflow.

The addition of Knowledge Expert Chat and Knowledge Hub further expanded this intelligence layer. Together, these capabilities combine curated product intelligence, decades of BMC mainframe expertise, and each organization’s own institutional knowledge, making trusted guidance available at the moment of decision. This approach helps less-experienced staff work with greater confidence while allowing senior experts to focus on higher-value work, turning institutional knowledge into a shared operational asset. Watch this short video for a demo of how Knowledge Expert Chat provides answers at the moment of need.

Instead of searching across documentation, runbooks, and tickets, teams can surface answers through natural-language questions using a knowledge expert chat inside their workflow. Early usage shows that this approach not only accelerates troubleshooting and decision-making but also helps users discover capabilities and documentation they did not previously know existed.

Trust in AI is earned through real outcomes. This intelligence foundation now supports the next phase of our direction.

Customers are redefining what AI must do

Through advisory boards, design programs, beta participation, and enterprise AI readiness discussions, a clear message has emerged: customers want AI that does more than explain what happened. They want orchestrated intelligence that helps move work forward.

They want to reduce repetitive analysis, move faster from insight to action, and make it easier for teams to build expertise and focus on higher-value priorities. They want to realize the full promise of BMC AMI as automated mainframe intelligence.

Generative AI laid the foundation by helping teams understand, explain, and accelerate work. Agentic AI builds on that foundation by enabling intelligence to plan, decide, and safely carry work forward.

To meet these expectations at enterprise scale, our new Statement of Direction is grounded in five core principles:

• Establish the core first: Build an orchestrated agentic intelligence foundation.
• Specialized agents: Reflect domain expertise with focused, autonomous actions.
• Human-in-the-loop: Governed, transparent, and observable autonomy.
• Open ecosystem: Open standards-based, composable, and extensible.
• Outcome-driven adoption: Deliver measurable value early and expand over time.

The next phase extends AI beyond insight toward trusted participation in execution, within enterprise guardrails.

And that shift defines our next statement of direction.

From AI assistance to orchestrated intelligence

Building on the foundation of the AI-driven knowledge and guidance we have delivered with BMC AMI Assistant, our next statement of direction is clear: extending that foundation toward proactive, coordinated intelligence that can safely execute actions within defined governance policies across the BMC AMI portfolio.

This evolution introduces coordinated AI agents with specialized, policy-aware capabilities operating across development, operations, data, and security. Each agent is domain-specific, working together within enterprise guardrails to deliver governed, collaborative outcomes.

In this next phase, AI agents for operations, development, data, and security will operate as orchestrated participants across domains—supporting actions such as system and performance diagnostics, development workflows, security validation, and operational recovery. They will learn across past incidents and participate in execution with transparency, verifiability, enterprise governance, and human oversight built in by design.

We will expand beyond explanation and recommendation to enable validated action inside workflows. Agentic workflows will move from detection to investigation to explanation to resolution within a single, governed AI experience.

AI will not replace expertise. Human validation remains essential. Principal engineers and system programmers will continue to define policy, validate outcomes, and shape how intelligence operates. AI scales expertise rather than removing it.

The result is faster problem identification, more contextual decision-making, and reduced operational friction—while ensuring the mainframe participates fully in broader enterprise AI strategies. The mainframe will operate as a connected, policy-aligned participant in the enterprise AI ecosystem across hybrid environments, maintaining the security, reliability, and trust on which organizations depend.

Delivering agentic workflows across the BMC AMI portfolio

Over the coming releases, this direction will take shape through agentic workflows that coordinate knowledge, reasoning, and action across BMC AMI solutions. Initial workflows could include agentic AIOps incident resolution, agentic diagnostics, AI-assisted application insights, development troubleshooting, and test-case generation.

These workflows build on the intelligence layer already established across the BMC AMI portfolio, combining development and operations telemetry, mainframe domain expertise, and organizational knowledge to provide the context required for responsible execution.

From there, coordinated AI agents operate across solutions, aligning development, operations, data, and security activities that were previously siloed. Instead of isolated AI features, intelligence becomes part of the workflow itself, helping teams connect system understanding with the next operational step.

Every action remains governed and transparent. Human validation, enterprise policy, and operational guardrails remain central, ensuring AI participation strengthens reliability rather than introducing risk.

The objective is clear: translate insight into trusted action while preserving the control and discipline enterprise systems require.

Establishing the foundation for governed AI execution

As AI becomes more operational, governance and trust become increasingly essential. Organizations expect AI not only to inform decisions, but to operate safely, predictably, and transparently within enterprise policy boundaries.

Across the industry, fragmented approaches are already emerging: isolated AI integrations, independently managed execution layers, and disconnected tool endpoints. While intended to accelerate innovation, these models often introduce new complexity, inconsistent policy enforcement, and operational sprawl.

Enterprise mainframe environments cannot afford that fragmentation, especially as AI Agent orchestration becomes a core capability for coordinating multi-step agentic workflows. This can result in agents giving conflicting recommendations and teams lose confidence and revert to manual work. Without a unified and governed approach, orchestration itself becomes fragmented, leading to unpredictable behavior, loss of insight, and reduced trust in AI-driven operations.

As part of our next phase, our direction includes establishing an MCP Gateway as a shared, governed access layer across the BMC AMI portfolio. Rather than creating multiple independently governed AI entry points, the MCP Gateway will provide a centralized, secure, and policy-aware interface through which AI agents interact with BMC AMI solutions. Every AI-driven action will be visible, governed, and aligned to enterprise policy with consistent controls across the platform. This enables AI agents not only to access BMC AMI systems, but to safely execute actions across them within defined policy boundaries.

Supporting this architecture, we will introduce an Agent Gateway to facilitate how AI agents communicate and collaborate. Instead of agents interacting independently, interactions will flow through the Agent Gateway—where the agent interactions are visible—ensuring governance, auditability, logging, and policy enforcement across agentic workflows.

Together, the MCP Gateway and Agent Gateway extend the BMC AMI Platform into a governed AI foundation that enables coordinated intelligence and trusted execution across the portfolio (see Figure 1). The BMC AMI Platform serves as the enterprise foundation for this direction—providing a unified layer of core capabilities and services that connects intelligence, orchestration, and governance across the BMC AMI portfolio. It brings together innovations such as BMC AMI Assistant, the Agent Gateway, and the MCP Gateway to simplify mainframe transformation, accelerate innovation, and enable AI to operate consistently and securely at scale.

Industry standards define how AI systems connect to tools and agents. Our direction focuses on how those connections are governed, secured, and operationalized at enterprise scale—without introducing fragmentation or operational risk.

This is how we intend to evolve agentic AI into an enterprise-grade foundation across the BMC AMI portfolio (see Figure 1).

Figure 1: High-level overview of BMC AMI Platform’s agentic architecture supporting enterprise-grade AI.

What you can expect — and how to shape what comes next

Over the coming releases, you will experience a meaningful shift in how AI participates in the BMC AMI environment. Intelligence will extend beyond explanation and guidance to support the safe execution of operational tasks. Capabilities that once appeared as isolated features will increasingly operate as coordinated agentic workflows across the BMC AMI portfolio, connecting development, operations, data, and security activities.

AI will participate responsibly within enterprise guardrails, operating under the policies, validation models, and controls defined by the teams who run these systems. This direction represents more than a single capability release, it establishes the foundation for a new way of working with BMC AMI solutions.

We invite customers to help us shape this next phase. You can engage with us in several ways: join our Customer Design Partner program to help refine and validate the most impactful agentic workflows, participate in early access initiatives focused on execution-oriented capabilities, and work with us to shape the governance and policy models that will guide enterprise-scale AI execution.

Moving forward with confidence

We delivered on our commitments. We established a foundation of trust. And we are now stepping into the next phase with clarity.

The future of AI on the mainframe centers on purpose-built enterprise intelligence, where your choice of AI models, your institutional knowledge and operational practices, your people, and your chosen platforms work together to drive intelligent execution across the enterprise. In this model, the mainframe is not an isolated environment. It is a fully governed participant in enterprise AI strategies, capable of supporting intelligent execution across hybrid systems.

Organizations that embrace this direction will not simply modernize their mainframe environments. They will unlock new ways to operate complex workloads with greater visibility, control, and intelligence.

And this is only the beginning. Before you run AI on your most essential platform, BMC First.

Are you truly ready to move from AI insight to AI-driven execution on the mainframe?
Assess your organization’s AI readiness and maturity level—register for an AI Readiness Discovery Workshop to take the next step.

Most organizations don’t start by thinking about service orchestration. They start with symptoms: a morning dashboard showing incomplete data, an SLA missed because an upstream process ran late, or a cross-team Slack thread at 7 AM trying to figure out what broke and why. 

By the time the word “orchestration” enters the conversation, the workarounds have usually been in place for months. And that’s exactly the problem—those workarounds are invisible because they look like “process.” 

The real distinction isn’t between working and broken systems. It’s between environments that are quietly compensating for a lack of coordination and those built on a foundation that can actually support end-to-end execution. That’s where the difference between scheduling and orchestration begins—and why it matters.

Where Job Scheduling Hits Its Ceiling

Technically there’s nothing wrong with the built-in automation tools that come with modern applications. ERP systems, CRMs, data warehouses, and large database platforms all ship with native scheduling capability. For environments that aren’t particularly complex, those tools do exactly what they’re supposed to do: automate jobs within the boundaries of the application that owns them. 

The problem starts at the boundary. 

When a business outcome requires work that transcends multiple applications – supply chain operations, financial close, ML pipeline execution – there is no common coordination layer to manage it. Each application scheduler knows its own world and nothing else. The result is a set of individually automated workflows that have no shared understanding of each other’s state, progress, or failure. 

This is the core architectural gap that service orchestration addresses. A control plane above existing schedulers manages end-to-end execution with business objectives as the organizing principle. 

What Cross-System Coordination Actually Looks Like in Practice

A customer submits an application through a web portal. From there, the process fans out across a stack of systems that have very little in common: data validation runs through an API service in Kubernetes containers; identity verification calls out to a third-party SaaS provider; risk scoring runs on a model deployed in the cloud; the actual account gets created in a core banking system that lives on-premises; the CRM updates the customer profile; compliance documentation gets generated and archived. 

That’s six distinct environments – cloud, on-premises, containerized infrastructure, SaaS – all participating in a single business transaction. And that’s the simplified version. 

Now add the business constraint that makes this genuinely difficult to manage: the customer expects the whole process to complete within one hour of submitting their application. To be clear, that’s a business commitment – and it applies to the entire chain, not to any individual step. 

No application scheduler was built to coordinate that topology under that deadline. Each one was designed to manage its own job queue, not to understand where it sits in a larger business process or what happens downstream if it falls behind.

How the Problem Reveals Itself Operationally

Organizations don’t usually discover this gap through architectural reviews. They discover it through patterns of operational friction that gradually become normalized. 

The most telling early signal is what might be called the time buffer contract. When data teams and application teams operate without a common coordination layer, they start negotiating informal timing agreements. The data from the application layer usually lands by 4 PM, so the ETL pipelines are configured to kick off at 5, with an hour of buffer built in as insurance. 

That buffer looks like responsible planning, but it masks missing dependency management. It fails in both directions – if the upstream data is late, the downstream pipeline starts anyway and processes whatever is there; if the data arrives early, the pipeline sits idle waiting for a clock to tick. 

The second pattern is more disruptive. A data pipeline ingests records from a CRM and an ERP expecting a complete dataset – a million records, in a typical case. An upstream workflow encountered a problem, so now the data arrives incomplete. The pipeline has no way to know this, so it runs to completion and updates business dashboards with partial data. No one finds out until a business user notices something looks wrong. 

What follows is a cross-team reconstruction effort: Slack messages, conference calls, manual tracing of execution steps across multiple tools to figure out where in the chain the problem originated. This kind of incident response – teams scrambling across disconnected systems to piece together what happened – is one of the clearest indicators that an organization has outgrown its current approach to workflow coordination. 

The absence of process lineage across the full stack – from the business application layer down through the data layer – is what makes these incidents so difficult to resolve. As Basil Faruqui, Sr. Director of Product Marketing at BMC Software, puts it: “You cannot manage what you can’t see.” And you can’t see what no single tool was designed to show you. 

What Orchestration Requires in the Modern Era 

Listing orchestration capabilities as a feature set understates what’s actually required. The architecture needs to be a direct response to the failure modes that siloed scheduling creates. 

The first requirement is broad environmental support. A control plane that only works in cloud environments, or only connects to certain application types, is just another silo with a better UI.

Orchestrating a business outcome end-to-end is complex and needs to span SaaS applications, multi-cloud infrastructure, on-premises data centers, and even mainframe systems in some cases. For many industries like banking, financial services and insurance mainframe support isn’t optional. 

The second requirement is end-to-end visibility with process lineage. Knowing that a job completed isn’t the same as knowing whether the business service it belongs to is on track. Effective visibility means being able to trace the state of a workflow across every system it touches, in real time. 

The third requirement is SLA management that carries business context. A notification that says a process is running late is less useful than a notification that says a supply chain workflow is running late and will affect five downstream business services if it doesn’t recover within the next 20 minutes. The former tells you something is wrong. The latter tells you what to do about it. 

Self-healing is the fourth requirement, and it’s where organizational knowledge becomes operational policy. When a workflow step fails because of a transient network issue, someone investigates and decides to rerun the step. That resolution should be capturable as a policy – and increasingly, AI-enabled capabilities can identify that the same failure pattern has occurred before, surface the historical remediation, and automate the response with human approval where appropriate. The goal is to build institutional knowledge into the platform rather than relying on it to live only in the heads of the people who were on shift when the incident happened. 

Finally, CI/CD integration matters more than it’s often given credit for. Orchestration is, at its core, a team sport. Business users, application teams, and operations teams all have a stake in how workflows are defined, tested, and deployed. Treating orchestration workflows as code – with version control, automated testing, and deployment pipelines consistent with modern DevOps practice – isn’t a nice-to-have. It’s what makes the platform manageable at all scales.

AI Workloads Break the Pipeline Model 

One of the more common misconceptions in this space is that AI and ML workloads can be handled by extending existing data pipeline tools. The assumption is that model training is just a compute-intensive batch job, and inference is just another API call. Both assumptions fall short. 

Consider a customer churn detection model. The training data has to be sourced from systems that aren’t data systems at all – ERP, CRM, custom applications, potentially social media feeds. Getting that data into a staging environment already requires coordination across the application layer. Then the model training pipelines need to run. So far, that’s two layers of coordination before any inference happens. 

Once the model is in production and identifies customers at risk, the business response – sending a promotional offer, for example – doesn’t happen inside the model. It happens in the CRM and ERP layer where customer profiles and promotional workflows live. The model is one step in a business process that starts with applications and ends with a business action. 

Confining AI workloads to a data pipeline tool means the platform can manage the middle of that chain but not the ends. The data sourcing and the downstream business response are both outside its scope. Siloing the AI workload in a single tool leaves you with a system that can’t map or control the process from source to business action. 

This matters even more as agent-based architectures become more common. Organizations are already deploying agents that take on discrete tasks within larger processes. A workflow like order-to-cash will not be handed over to agents in its entirety anytime soon – but agents are increasingly handling specific components of it. That means an orchestration platform needs to be able to invoke agents, pass them the right context, enforce guardrails, and manage their execution according to the same SLAs that govern the rest of the workflow. For organizations with active agent deployments, agent orchestration is already a present consideration. 

Rethinking the Question

When organizations run into workflow coordination problems, the instinct is to ask whether they need better automation. More often, what they actually need is orchestration. 

Better automation improves individual jobs. Orchestration manages the relationships between them. For workflows that stay within the boundaries of a single application, better automation is probably the right answer. For workflows that span systems, teams, and environments to deliver a business outcome with a defined deadline, automation alone can’t provide what’s needed. 

The symptoms are identifiable. Time buffers masquerading as process discipline. Partial data incidents that require multi-team investigation to trace. SLA breaches that can’t be attributed to a specific failure point because no single tool has visibility across the full chain. ​​None of these are signs that automation needs tuning. They point to a missing control plane. 

The distinction matters because building more automation on top of a coordination gap doesn’t close the gap. It adds more things to coordinate.

A data pipeline is a series of automated steps for moving data from one or more sources to a designated destination, often transforming it along the way. Raw, disparate pieces of data enter one end, undergo processes like cleaning, restructuring, and enrichment, and emerge at the other end as usable insights. Data pipelines are the foundation of every analytics dashboard, machine learning model, and data-driven operational decision. 

You could think of a data pipeline like an airport baggage system: bags (data) enter the conveyor system, get scanned (validation), sorted (transformation), and routed to the correct flight (destination database). If one belt jams, everything backs up—just like a bottleneck in a data pipeline.

What’s the difference between a data pipeline and ETL?

A data pipeline and an ETL pipeline are not the same thing, though the terms are often used interchangeably.

A data pipeline is an umbrella term for any set of processes that move data from one system to another. This includes simple data ingestion, real-time streaming, batch processing, and complex multi-step workflows. 

An ETL pipeline (Extract, Transform, Load) is a specific type of data pipeline. Its purpose is to extract data from sources, transform it into the right format, and load it into a destination system like a data warehouse or database. All ETL pipelines are data pipelines, but not all data pipelines are ETL pipelines. A pipeline that moves raw data without transforming it, or streams data in real time, is still a data pipeline—but not an ETL pipeline. 

Is SQL a data pipeline?

No. SQL (Structured Query Language) is a language used to query, manage, and manipulate data in relational databases—it’s a tool, not a workflow. 

A data pipeline is the full automated process that moves data from one place to another. SQL can be used inside a data pipeline to filter, join, or transform data, but SQL doesn’t constitute the pipeline itself. If you’re building a house, SQL is the hammer and saw—essential tools, but not the entire construction project. 

Why do data pipelines matter?

Data pipelines are essential because organizations collect data from too many sources—customer interactions, social media, sales transactions, website logs, IoT devices, and internal applications—to manage manually. Without a systematic way to collect, process, and deliver this data, it quickly becomes unmanageable rather than useful. 

Data pipelines power the data-driven decisions we encounter every day: personalized recommendations, real-time fraud alerts, and analytics dashboards. For DataOps teams specifically, pipelines help ensure the reliability, scalability, and governance that organizations need to use data confidently instead of being overwhelmed by it. 

What are the benefits of a well-executed data pipeline?

Data pipelines don’t just move data—they make it fit for purpose, delivering it where and when it’s needed. Five benefits go beyond the basic transport function: 

Enabling analytics and business intelligence: Pipelines feed cleaned, structured data into data warehouses and analytical platforms, allowing analysts to discover trends, identify opportunities, and monitor performance. 

Fueling machine learning and AI: AI models require large volumes of high-quality, pretreated data. Data pipelines help ensure models get the data they need to learn and make accurate predictions. 

Ensuring data quality and governance: As data gets cleaned, validated, and standardized, data pipelines support greater confidence in data-driven decisions. They also enforce governance rules for compliance and security. 

Improving operational efficiency: By integrating data from various systems, pipelines provide a holistic view of operations, automating workflows and flagging issues in real time. 

Facilitating data democratization: Pipelines make data accessible and understandable to more people within an organization, empowering more teams to make informed decisions by connecting data sources to decision-makers. 

Without strong data pipelines, organizations can fly blind—making decisions based on intuition rather than evidence. 

What are the core components of a data pipeline?

Every data pipeline is made up of five essential components. Understanding these elements reveals how data flows and transforms from its original source to its final destination. 

1. Source: Where your data lives

Source is the origin point of your data—the starting line of the pipeline. The type of source determines how the data will be extracted. Common data sources include: 

• Databases: relational (e.g., MySQL) and NoSQL (e.g., MongoDB) 
• Applications: CRM systems (e.g., Salesforce), ERPs (e.g., SAP), marketing automation platforms 
• APIs: third-party services, social media platforms, public data feeds 
• Files: CSVs, JSON, XML, Parquet, Avro—often stored in cloud storage (e.g., Azure Blob) 
• Streaming data: real-time event streams from IoT devices, website clicks, financial transactions 
• Logs: system logs, web server logs, application logs 

2. Extraction: Getting your data out

Extraction is the step where data is pulled from its original source—often involving different file types, formats, and sometimes unstable or slow source connections. The goal of extraction is to get a raw copy of the data without altering the source system.

Three common extraction methods: 

• Batch extraction: Data is pulled in chunks at scheduled intervals (e.g., nightly, hourly). Used for data that doesn’t change frequently or where immediate updates aren’t critical. 
• Incremental extraction: Only new or changed data since the last pipeline run is extracted. Faster than full extraction, but requires change-detection techniques like timestamps, version numbers, or Change Data Capture (CDC). 
• Streaming extraction: Data is continuously pulled from sources as events occur, typically using message queues or event streaming platforms like Kafka or Kinesis. 

3. Transformation: Cleaning and shaping your data 

Transformation is usually the most complex part of a data pipeline. This is where the messiness of raw data gets cleaned and turned into actionable information. The goal is to ensure data quality, consistency, and suitability for the intended destination. Common transformation steps include:

• Cleaning: Removing duplicates, handling missing values, correcting errors 
• Filtering: Selecting only relevant rows or columns 
• Aggregating: Summarizing or categorizing data (e.g., total sales per day) 
• Joining or merging: Combining data from multiple sources using common keys (e.g., joining customer data with order data) 
• Standardizing or normalizing: Ensuring consistent data types, formats, and units (e.g., standardizing currency codes) 
• Enriching: Adding new data points by looking up external information or deriving new features (e.g., adding geographical data based on an IP address) 
• Structuring: Converting unstructured or semi-structured data into a structured format

4. Loading: Delivering your data

Once extracted and transformed, data is loaded into the system where it will be used—a database, data warehouse, or analytics platform—so applications, reports, and analytics tools can access it. Three common loading strategies:

• Full load: The entire destination table or dataset is overwritten with new, transformed data. Simpler to implement, but resource-intensive for large datasets. 
• Incremental load: Only new or changed records are appended to the destination. More efficient, but requires diligent management of data updates and deletions. 
• Streaming load: Data is continuously loaded as it arrives, often into specialized real-time databases or analytical engines. 

5. Destination: Where your data rests

Destination is the final storage location where processed data is available for consumption by analysts, data scientists, and applications. Common destinations include: 

• Data warehouses: Optimized for complex queries and reporting on large volumes of historical data (e.g., Snowflake) 
• Data lakes: Hold raw or semi-structured data at scale for advanced analytics and machine learning (e.g., Azure Data Lake Storage) 
• Databases: Operational systems for everyday applications like websites or apps (e.g., MongoDB) 
• Business intelligence (BI) tools: Software that turns data into dashboards and reports (e.g., Tableau) 
• File storage: Simple storage for archiving or later processing 

Note: loading and destination are conceptually distinct but closely related in practice. Loading is the action of writing processed data into a system; destination is the place where that data ends up and stays until it’s needed. Loading is like putting groceries into the fridge. Destination is the actual fridge. 

Bringing it all together—an e-commerce scenario: Source (CRM database) → Extract (SQL query for new customer orders) → Transform (clean addresses, calculate total order value, join with product details) → Load (insert into data warehouse) → Destination (data warehouse for reporting). The entire process runs automatically on a schedule, ensuring a continuous flow of refined information. 

What are the three types of data pipelines?

Just as there are different ways to transport goods, there are different types of data pipelines—each optimized for specific needs around speed, volume, and complexity. 

Batch processing: the daily shuttle

Batch processing works like a commuter train on a defined schedule. It picks up a large group of passengers (data) at scheduled times and delivers them to their destination. Data is collected over a period, then processed as a single, large batch.

• Characteristics: High latency (data may be hours or days old); processes large volumes efficiently; often scheduled during off-peak hours 
• Use cases: Nightly reports, monthly financial summaries, loading historical data into a data warehouse, running complex analytical jobs that don’t require immediate results 

Real-time streaming: the instant delivery service

Real-time streaming works like an instant delivery service. As soon as a package (data event) is created, it’s picked up, processed almost immediately, and delivered to its destination with minimal delay.

• Characteristics: Low latency (data is typically seconds or milliseconds old); handles continuous streams of individual events; requires infrastructure optimized for speed 
• Use cases: Fraud detection, real-time personalized recommendations, IoT sensor data analysis, monitoring system health, live dashboards

Hybrid approaches: the best of both worlds

Many organizations combine batch and streaming pipelines. Two common hybrid patterns:

• Lambda architecture: Uses separate batch and streaming layers. The streaming layer provides real-time views; the batch layer processes historical data for accuracy and completeness. Results from both are then merged. 
• Kappa architecture: A simpler approach that handles both real-time and historical processing using a single streaming engine, often by replaying streams.

Choosing the right pipeline type depends entirely on your business requirements for data freshness, volume, and complexity. 

What are the main challenges in data pipeline management?

Building a data pipeline is one thing; keeping it running smoothly and reliably is another. Here are the most common data pipeline management challenges—and the best practices for each. 

Ensuring data quality: Data can be inconsistent, incomplete, or incorrect at the source, leading to garbage in, garbage out. Best practices: implement data validation rules at every stage; use data profiling tools to understand data characteristics; create data quality checks within transformation steps; use data observability platforms to detect anomalies early. 

Scalability and performance: As data volumes grow or requirements shift to real-time, pipelines can become slow or break entirely. Best practices: design for scalability from the outset; use distributed processing frameworks (e.g., Apache Spark); use cloud-native services that scale automatically; implement incremental loading strategies; optimize queries and transformation logic. 

Security and compliance: Data pipelines handle sensitive information, requiring stringent security and compliance measures. Best practices: encrypt data at rest and in transit; implement strong access controls (least privilege); audit data access and changes; redact sensitive data during transformation where necessary. 

Monitoring and alerting: Without proper monitoring, pipeline failures or data quality issues can go undetected and impact downstream applications. Best practices: implement comprehensive monitoring for pipeline health, performance metrics, and data quality metrics; set up automated alerts for critical failures, latency breaches, or data anomalies; use dashboards for operational visibility.

What are the top data pipeline use cases?

Data pipelines are versatile, powering applications across industries. Seven key examples of how pipelines transform raw data into actionable insights:

• Business intelligence and reporting: Aggregating sales data, customer demographics, and marketing spend into a data warehouse for daily, weekly, or monthly reports and dashboards that guide strategic decisions 
• Customer 360-degree view: Combining data from CRM, sales, support, and marketing platforms to create a holistic profile of each customer, enabling personalized experiences and targeted campaigns 
• Fraud detection: Ingesting real-time financial transactions, social media activity, and user behavior to identify suspicious patterns and instantly flag potential fraud 
• IoT analytics: Collecting streams of data from sensors (e.g., factory machines, smart city devices) to monitor performance, predict maintenance needs, and optimize operations 
• Personalized recommendations: Processing user browsing history, purchase data, and demographic information to power content recommendations on streaming platforms 
• Log analytics: Consolidating logs from applications and servers to monitor system health, troubleshoot issues, and detect security threats in real time 
• ML model training: Preparing, cleaning, and feeding large datasets to machine learning models for tasks like image recognition, natural language processing, or predictive analytics

What makes a data pipeline modern?

Modern data pipelines share several key characteristics that distinguish them from traditional approaches:

• Cloud-native and serverless: Modern pipelines use cloud services (e.g., AWS, Azure) that scale automatically and reduce operational overhead 
• ELT-first approach: Instead of transforming data before loading, modern pipelines often load raw data into a cloud data warehouse (e.g., Snowflake, BigQuery) and transform it within the warehouse using SQL—leveraging the destination’s compute power and enabling greater flexibility 
• Data lake integration: Modern pipelines frequently integrate with data lakes to store vast amounts of raw, multi-structured data for future use, advanced analytics, and machine learning 
• Real-time capabilities: Streaming technologies like Apache Kafka and Amazon Kinesis increasingly power modern pipelines for immediate insights 
• Orchestration and automation: Tools like Apache Airflow or cloud-native orchestrators automate scheduling, manage dependencies, and monitor pipeline health 
• Data observability: Modern pipelines go beyond basic monitoring to actively track the health, quality, and lineage of data—detecting anomalies and ensuring data trustworthiness 
• Data governance and security by design: Security, privacy, and compliance are built into the pipeline architecture from the start, not added as an afterthought 
• Flexibility and agility: Modular components make modern pipelines easier to adapt to new data sources and changing business requirements 

How do traditional and modern data pipelines compare? 

Modern data pipelines are designed to be more agile, scalable, cost-effective, and resilient than traditional approaches. Here’s a side-by-side comparison:

What are the most common data pipeline tools?

The data pipeline tool landscape is broad and evolving. Four categories to consider: 

Data integration platforms: Comprehensive ETL/ELT tools, often with visual interfaces and pre-built connectors. Examples: Talend, Informatica. A good fit for teams that want an end-to-end solution without heavy coding; businesses with multiple data sources needing pre-built connectors; and organizations prioritizing ease of use and quick deployment. 

Cloud-native services: Major cloud providers offer services specifically designed for scalable data pipelines. Examples: Amazon Kinesis, Google BigQuery. A good fit for companies already invested in a specific cloud ecosystem; teams needing scalable, cost-effective solutions for batch and streaming; and use cases requiring tight integration with other cloud services. 

Open-source frameworks: Flexible, developer-friendly options for orchestration and processing. Examples: Apache Airflow, Apache Kafka. A good fit for engineering teams with strong technical skills; organizations wanting maximum flexibility and control; and scenarios with custom requirements or large-scale data processing needs. 

Enterprise workflow orchestration platforms like Control-M are ideal for teams that need to orchestrate BigQuery data pipelines alongside mainframe, cloud, and on-premises workloads — providing SLA management and audit capabilities that native GCP scheduling does not offer. A good fit for large enterprises with complex, mission-critical workflows; businesses needing robust scheduling, compliance, and audit capabilities; and teams managing cross-platform jobs spanning mainframe, cloud, and on-premises systems with high reliability requirements. 

The right tool depends on your budget, team skill set, data volumes, and real-time requirements. 

5 key data pipeline takeaways

Whether you’re learning about data pipelines for the first time or refreshing on the fundamentals, here are the most important points to carry forward: 

1. Understand the pipeline lifecycle

A data pipeline isn’t just about moving data—it involves extraction, transformation, loading, orchestration, monitoring, and governance.

2. Orchestration is key

Orchestration ensures repeatability, scalability, and observability across the entire data pipeline. 

3. Embrace automation and CI/CD

Integrating data pipelines into CI/CD workflows enables faster, safer changes. DataOps applies DevOps principles to data: automated testing, deployment, and version control for pipelines.

4. Prioritize data quality and monitoring

Pipelines can fail silently if data quality isn’t checked. Implement validation, anomaly detection, and alerts to catch issues early. Observability is critical for trust and compliance.

5. Design for scalability and flexibility

Modern data pipelines must handle batch and streaming, adapt to schema changes, and scale with data growth. Cloud-native and modular architectures are essential for agility. 

Bonus tip: Learn the full ecosystem—ETL tools, orchestration frameworks, cloud services—and how they fit together to build pipelines that serve your organization at scale. 

Frequently asked questions about data pipelines

How is a data pipeline different from a data warehouse? 

A data pipeline is the process that moves and transforms data; a data warehouse is a destination where processed data is stored for analysis. Data pipelines feed data warehouses—the pipeline handles transport and preparation, while the warehouse provides the storage and query environment. The two are complementary, not interchangeable. 

What skills do you need to build a data pipeline? 

Building a data pipeline typically requires proficiency in SQL and at least one programming language (Python and Scala are common), familiarity with data transformation concepts (ETL/ELT), and working knowledge of at least one orchestration or workflow tool. Cloud platform experience (AWS, Azure, or GCP) is increasingly essential for modern data pipeline work. 

How do you test a data pipeline? 

Data pipeline testing involves validating data at multiple stages: confirming that extraction produces complete, unaltered source data; verifying that transformation logic produces expected outputs; and checking that loaded data matches expected row counts, data types, and values at the destination. Automated testing frameworks integrated into CI/CD pipelines can run these checks continuously and catch regressions early. 

What is data pipeline latency? 

Data pipeline latency is the time between a data event occurring at the source and that data becoming available at the destination. Batch pipelines have high latency (hours or days); real-time streaming pipelines have low latency (seconds or milliseconds). Acceptable latency depends on the use case—fraud detection requires near-zero latency, while monthly financial reporting can tolerate daily batch processing. 

How does BMC Control-M help manage data pipelines? 

BMC Control-M is an enterprise workflow orchestration platform that automates, schedules, and monitors complex data pipelines across environments—including mainframe, cloud, and on-premises systems. Control-M provides centralized visibility, dependency management, and audit capabilities, making it well-suited for mission-critical pipelines that require high reliability, compliance, and cross-platform coordination. 

The views and opinions expressed in this post are those of the author and do not necessarily reflect the official position of BMC. 

Control-M from BMC enables teams to orchestrate data pipelines on Google Cloud by defining, scheduling, and monitoring workflows across services like Cloud Storage, Dataflow, and BigQuery—all from a single interface. This article walks through the five key orchestration challenges, the Google Cloud services involved, and a real-world credit-card fraud detection example that puts Control-M into action. 

The Google Cloud Platform is designed specifically to accommodate organizations in a variety of positions along their cloud services journey, from large-scale machine learning (ML) and data analysis to services tailored to SMBs to hybrid-cloud solutions for customers that want to use services from more than one cloud provider. When BMC was migrating our Control-M application to this cloud ecosystem, we had to be very thoughtful about how we managed this change. The SADA engineering team worked alongside the BMC team to ensure that we had a seamless integration for our customers. 

SADA supported this project by providing an inventory of the Google Cloud configuration options, decisions, and recommendations to enable the data platform foundation deployment, collaborated with BMC on the implementation planning, provided automation templates, and designed the Google Cloud architecture for the relevant managed services on the Google Cloud Platform.

In this article, we will discuss the end result of this work, and look at an example using a credit-card fraud detection process to show how you can use Control-M to orchestrate a data pipeline seamlessly in Google Cloud.

What are the five challenges of orchestrating an ML data pipeline?

Orchestrating a machine learning data pipeline on Google Cloud involves five primary challenges that teams must address before workflows can run reliably at scale.

Understand the workflow

Examine all dependencies and any decision trees. For example, if data ingestion is successful, proceed down this path; if it is not successful, proceed down that path.

Understand the teams

If multiple teams are involved in the workflow, each needs to have a way to define their workflow using a standard interface, and to be able to merge their workflows to make up the pipeline.

Follow standards

Teams should use repeatable standards and conventions when building workflows. This avoids having multiple jobs with identical names. Each step should also have a meaningful description to help clarify its purpose in the event of a failure.

Minimize the number of tools required

Use a single tool for visualization and interaction with the pipeline (and dependencies). Visualization is important during the definition stage since it’s hard to manage something that you can’t see. This is even more important when the pipeline is running.

Include built-in error handling capabilities

It’s important to understand how errors can impact downstream jobs in the workflow or the business service level agreement (SLA). Failure of a job should not halt the pipeline altogether and require human interaction. Criteria can be used to determine if a failed job can be restarted automatically or whether a human must be contacted to evaluate the failure—if, for instance, there are a certain number of failures involving the same error.

How did BMC and SADA meet these orchestration challenges?

Addressing these challenges required a solid foundation and presented opportunities for collaboration. BMC and SADA aligned using the SADA POWER line of services to establish the data platform foundation. Some notable elements in this technical alignment included work by SADA to:

• Apply industry expertise to expedite BMC’s development efforts. 
• Establish a best practices baseline around data pipelines and the tools to orchestrate them. 
• Conduct collaborative sessions to understand BMC’s technical needs and provide solutions that the BMC team could integrate and then expand upon. 

SADA’s Data Platform Foundation provided opportunities to leverage Google Cloud services to accomplish the complex analytics required of an application like Control-M. The BMC and SADA teams worked together to establish a strong foundation through:

• Selecting data and storage locations in Google Cloud Storage. 
• Utilizing the advantages provided by Pub/Sub to streamline the analytics and data integration pipelines. 
• Having thorough discussions around the extract, transform, and load (ETL) processes to truly understand the end state of the data. 
• Using BigQuery and writing analytic queries. 
• Understanding the importance of automation, replicability of processes, and monitoring performance in establishing a system that is scalable and flexible. 
• Using Data Studio to create a visualization dashboard to provide the necessary business insights. 

How does fraud detection illustrate data pipeline orchestration on Google Cloud?

Credit-card fraud detection is a practical, real-world example of how Control-M can orchestrate a complex ML data pipeline on Google Cloud—combining real-time and batch processes across multiple services.

Digital transactions have been increasing steadily for years, and the accelerating adoption of digital payments by businesses and consumers has brought with it increased fraud and operational risks. With fraudsters improving their techniques, companies are relying on ML to build resilient and efficient fraud detection systems. Since fraud constantly evolves, detection systems must be able to identify new types of fraud by detecting anomalies that are seen for the first time—making fraud detection a perpetual task that requires constant diligence and innovation.

Common types of financial fraud that customers work to prevent with this application include:

• Stolen/fake credit card fraud: Transactions made using fake cards, or cards belonging to someone else.
• ATM fraud: Cash withdrawals using someone else’s card.

Fraud detection is composed of both real-time and batch processes. The real-time process is responsible for denying a transaction and possibly placing a hold on an account or credit card, thus preventing the fraud from occurring. It must respond quickly, sometimes at the cost of reduced accuracy.

To minimize false positives, which may upset or inconvenience customers, a batch phase is used to continuously fine-tune the detection model. After transactions are confirmed as valid or fraudulent, all recent events are input to the batch process on a regular cadence. This batch process then updates the training and scoring of the real-time model to keep real-time detection operating at peak accuracy. This batch process is the focus of this article.

How can you try the demo system?

SADA and BMC created a demonstration version of this solution so you can experiment with it on Google Cloud. You can find all of the code, plus sample data, in GitHub.

Resources included are:

• Kaggle datasets of transaction data, fraud status, and demographics
• Queries
• Schema
• User-defined functions (UDFs)

How does the pipeline work?

For each region in which the organization operates, transaction data is collected daily. Details collected include (but are not limited to):

• Transaction details: Describes each transaction, including the amount, item code, location, method of payment, and so on.
• Personal details: Describes the name, address, age, and other details about the purchaser.

This information is pulled from corporate data based on credit card information and real-time fraud detection that identifies which transactions were flagged as fraudulent.

New data arrives either as batch feeds or is dropped into Cloud Storage by Pub/Sub. This new data is then loaded into BigQuery by Dataflow jobs. Normalization and some data enrichment is performed by UDFs during the load process.

Once all the data preparation is complete, analytics are run against the combined new and historical data to test and rank fraud detection performance. The results are displayed in Data Studio dashboards.

Figure 1: Control-M orchestration

Which Google Cloud services power the pipeline?

Control-M orchestrates a coordinated set of Google Cloud services—Cloud Storage, Dataflow, BigQuery, and Data Studio—each handling a distinct stage of the data pipeline.

Cloud Storage provides a common landing zone for all incoming data and a consistent input for downstream processing. Dataflow is Google Cloud’s primary data integration tool.

SADA and BMC selected BigQuery for data processing. Earlier versions of this application used Hadoop, but while working with the team at SADA, we converted to BigQuery as this is the recommended strategy from Google for sophisticated data warehouse or data lake applications. This choice also simplified setup by providing out-of-the-box integration with Cloud Dataflow. UDFs provide a simple mechanism for manipulating data during the load process.

What are the two ways to define pipeline workflows in Control-M?

Control-M supports two approaches for defining pipeline workflows, giving teams flexibility to work visually or programmatically.

Using a graphical editor

This provides the option of dragging and dropping the workflow steps into a workspace and connecting them.

Use RESTful APIs

Define the workflows using a jobs-as-code method, then use JSON to integrate into a continuous integration/continuous delivery (CI/CD) toolchain. This method improves workflow management by flowing jobs through a pipeline of automated building, testing, and release. Google Cloud provides a number of developer tools for CI/CD, including Cloud Build and Cloud Deploy.

How are jobs defined in Control-M?

The basic Control-M execution unit is referred to as a job. There are a number of attributes for every job, defined in JSON:

Job type

Options include script, command, file transfer, Dataflow, or BigQuery.

Run location

For instance, which host is running the job?

Identity

For example, is the job being “run as…” or run using a connection profile?

Schedule

Determines when to run the job and identifies relevant scheduling criteria.

Dependencies

This could be things like whether the job must finish by a certain time or output must arrive by a certain time or date.

Jobs are stored in folders and the attributes discussed above, along with any other instructions, are applied to all jobs in that folder.

Below is an example of the JSON code that describes the workflow used in the fraud detection model ranking application. You can find the full JSON code in the Control-M Automation API Community Solutions GitHub repo. While there, you can also find solutions, the Control-M Automation API guide, and other code samples.

{
"Defaults" : {
},
"jog-mc-gcp-fraud-detection": {"Type": "Folder",
"Comment" : "Update fraud history, run, train and score models",
"jog-gcs-download" : {"Type" : "Job:FileTransfer",…},
"jog-dflow-gcs-to-bq-fraud": {"Type": "Job:Google DataFlow",…},
"jog-dflow-gcs-to-bq-transactions": {"Type": “Job:Google DataFlow",…},
"jog-dflow-gcs-to-bq-personal": {"Type": "Job:Google DataFlow",…},
"jog-mc-bq-query": {"Type": "Job:Database:EmbeddedQuery", …},
"jog-mc-fm-service": {"Type": "Job:SLAManagement",…},
},
"flow00": {"Type":"Flow", "Sequence":[
"jog-gcs-download",
"jog-dflow-gcs-to-bq-fraud",
"jog-mc-bq-query",
"jog-mc-fm-service"]},
"flow01": {"Type":"Flow", "Sequence":[
"jog-gcs-download",
"jog-dflow-gcs-to-bq-transactions",
"jog-mc-bq-query", "jog-mc-fm-service"]},
"flow02": {"Type":"Flow", "Sequence":[
"jog-gcs-download",
"jog-dflow-gcs-to-bq-personal",
"jog-mc-bq-query",
"jog-mc-fm-service"]}

}
}

The jobs shown in this workflow correspond directly with the steps illustrated previously in Figure 1.

The workflow contains three fundamental sections:

Defaults. These are the functions that apply to the workflow. This could include details such as who to contact for job failures or standards for job naming or structure.

{  "Defaults" : {"RunAs" : "ctmagent", "OrderMethod": "Manual", "Application" : 
       "multicloud", "SubApplication" : "jog-mc-fraud-modeling", 
      "Job" : {"SemQR": { "Type": "Resource:Semaphore", Quantity": "1"},
      "actionIfError" : {"Type": "If", "CompletionStatus":"NOTOK", "mailTeam": 
          {"Type": "Mail", "Message": "Job %%JOBNAME failed", "Subject": 
                 "Error occurred", "To": deng_support@bmc.com}}}
    }, 

Job definitions. This is where individual jobs are specified and listed. See below for descriptions of each job in the flow.

Flow statements. These define the relationships of the job, both upstream and downstream.

"flow00": {"Type":"Flow", "Sequence":["jog-gcs-download", 
           "jog-dflow-gcs-to-bq-fraud", "jog-mc-bq-query", 
           "jog-mc-fm-service"]},
"flow01": {"Type":"Flow", "Sequence":["jog-gcs-download", 
           "jog-dflow-gcs-to-bq-transactions", 
           "jog-mc-bq-query", "jog-mc-fm-service"]},
"flow02": {"Type":"Flow", "Sequence":["jog-gcs-download", 
           "jog-dflow-gcs-to-bq-personal", "jog-mc-bq-query", 
           "jog-mc-fm-service"]} 

How does Control-M schedule pipeline workflows?

Control-M uses a server-and-agent model for scheduling. The server is the central engine that manages workflow scheduling and submission to agents, which are lightweight workers. In the demo described in this article, the Control-M server and agent are both running on Google Compute Engine VM instances.

Workflows are most commonly launched in response to various events such as data arrival but may also be executed automatically based on a predefined schedule. Schedules are very flexible and can refer to business calendars; specify different days of the week, month, or quarter; define cyclic execution, which runs workflows intermittently or every “n” hours or minutes; and so on.

How does Control-M process the data?

Control-M processes data through a sequence of job types—File Transfer, Dataflow, and SLA Management—each mapped to a distinct stage of the pipeline.

File Transfer job type

The first job, called jog-gcs-download, is of type Job:FileTransfer. This job transfers files from a conventional file system described by ConnectionProfileSrc to Google Cloud Storage described by ConnectionProfileDest.

The File Transfer job type can watch for data-related events (file watching) as a prerequisite for data transfer, as well as perform pre/post actions such as deletion of the source after a successful transfer, renaming, source and destination comparison, and restart from the point of failure in the event of an interruption. In the example, this job moves several files from a Linux® host and drops them into Google Cloud Storage buckets.

"jog-gcs-download" : {"Type" : "Job:FileTransfer",
        "Host" : "ftpagents",
        "ConnectionProfileSrc" : "smprodMFT",
        "ConnectionProfileDest" : "joggcp",
        "S3BucketName" : "prj1968-bmc-data-platform-foundation",
        "Description" : "First data ingest that triggers downstream applications",
        "FileTransfers" : [
          {
            "TransferType" : "Binary",
            "TransferOption" : "SrcToDestFileWatcher",
            "Src" : "/bmc_personal_details.csv",
            "Dest" : "/bmc_personal_details.csv"
          },
          {
            "TransferType" : "Binary",
            "TransferOption" : "SrcToDestFileWatcher",
            "Src" : "/bmc_fraud_details.csv",
            "Dest" : "/bmc_fraud_details.csv"
          },
          {
            "TransferType" : "Binary",
            "TransferOption" : "SrcToDestFileWatcher",
            "Src" : "/bmc_transaction_details.csv",
            "Dest" : "/bmc_transaction_details.csv"
          } 
        ]
      }, 

Dataflow

Dataflow jobs are executed to push the newly arrived data into BigQuery. The jobs appear complex, but Google Cloud provides an easy-to-use process to make the definitions simple.

Go to the Dataflow Jobs page (Figure 2). If you have an existing job, choose to Clone it or Create Job from Template. Once you’ve provided the desired parameters, click on Equivalent REST at the bottom to get this information (Figure 3), which you can cut and paste directly into the job’s Parameters section.

Figure 2: Dataflow Jobs page

Figure 3: Cut and paste into job Parameters section

"jog-dflow-gcs-to-bq-fraud": {"Type": "Job:ApplicationIntegrator:AI Google DataFlow",
        "AI-Location": "us-central1",
        "AI-Parameters (JSON Format)": "{"jobName": "jog-dflow-gcs-to-bq-fraud",
        "environment": {        "bypassTempDirValidation": false,
        "tempLocation": "gs://prj1968-bmc-data-platform-foundation/bmc_fraud_details/temp",
        "ipConfiguration": "WORKER_IP_UNSPECIFIED",
        "additionalExperiments": []    },    
        "parameters": {
        "javascriptTextTransformGcsPath": "gs://prj1968-bmc-data-platform-foundation/bmc_fraud_details/bmc_fraud_details_transform.js", 
        "JSONPath": "gs://prj1968-bmc-data-platform-foundation/bmc_fraud_details/bmc_fraud_details_schema.json",
        "javascriptTextTransformFunctionName": "transform",
        "outputTable": "sso-gcp-dba-ctm4-pub-cc10274:bmc_dataplatform_foundation.bmc_fraud_details_V2",
        "inputFilePattern": "gs://prj1968-bmc-data-platform-foundation/bmc_fraud_details/bmc_fraud_details.csv", 
        "bigQueryLoadingTemporaryDirectory": "gs://prj1968-bmc-data-platform-foundation/bmc_fraud_details/tmpbq"    }}",
        "AI-Log Level": "INFO",
        "AI-Template Location (gs://)": "gs://dataflow-templates-us-central1/latest/GCS_Text_to_BigQuery",
        "AI-Project ID": "sso-gcp-dba-ctm4-pub-cc10274",
        "AI-Template Type": "Classic Template",
        "ConnectionProfile": "JOG-DFLOW-MIDENTITY",
        "Host": "gcpagents"
      }, 

SLA management

This job defines the SLA completion criteria and instructs Control-M to monitor the entire workflow as a single business entity.

"jog-mc-fm-service": {"Type": "Job:SLAManagement",
	 "ServiceName": "Model testing and scoring for fraud detection",
	 "ServicePriority": "3",
	 "JobRunsDeviationsTolerance": "3",
	 "CompleteIn": {
	    "Time": "20:00"
	  }
	},

The ServiceName specifies a business-relevant name that will appear in notifications or service incidents, as well as in displays for non-technical users, to make it clear which business service may be impacted. Importantly, Control-M uses statistics collected from previous executions to automatically compute the expected completion so that any deviation can be detected and reported at the earliest possible moment. This gives monitoring teams the maximum opportunity to course-correct before any impact to business services is detected.

How do you monitor the pipeline state in Control-M?

Control-M provides a real-time monitoring interface that shows the status of every job in the pipeline, making it straightforward to identify failures and take action without switching between tools.

Control-M provides a user interface for monitoring workflows (Figure 4). In the screenshot below, the first job completed successfully and is green, the next three jobs are executing and depicted in yellow. Jobs that are waiting to run are shown in gray.

Figure 4: Control-M Monitoring Domain

You can access the output and logs of every job from the pane on the right-hand side. This capability is vital during daily operations. To monitor those operations more easily, Control-M provides a single pane to view the output of jobs running on disparate systems without having to connect to each application’s console.

Control-M also allows you to perform several actions on the jobs in the pipeline, such as hold, rerun, and kill. You sometimes need to perform these actions when troubleshooting a failure or skipping a job, for example.

All of the functions discussed here are also available from a REST-based API or a CLI.

Conclusion

Coordinating and monitoring workflows across an ML pipeline remains a complex task, even with the rich set of ML tools that Google Cloud provides. Anytime you need to orchestrate a data pipeline on Google Cloud that combines file transfers, applications, data sources, or infrastructure, Control-M can simplify your workflow orchestration. Control-M integrates, automates, and orchestrates application workflows whether on-premises, on Google Cloud, or in a hybrid environment.

Frequently asked questions 

What is the best tool to orchestrate a data pipeline on Google Cloud?

Control-M from BMC is purpose-built for orchestrating data pipelines across cloud and hybrid environments. On Google Cloud, Control-M integrates natively with Cloud Storage, Dataflow, BigQuery, and Pub/Sub, enabling teams to define, schedule, and monitor workflows from a single interface using either a graphical editor or RESTful APIs. 

What is the difference between Dataflow and Control-M?

Google Cloud Dataflow is a managed data integration service that moves and transforms data between sources and destinations such as BigQuery. Control-M is a workflow orchestration engine that coordinates and monitors the execution of multiple tools—including Dataflow jobs—as part of a broader end-to-end pipeline. Dataflow handles the data movement; Control-M manages the sequencing, scheduling, and error handling of the entire workflow. 

Can Control-M run jobs as code on Google Cloud?

Yes. Control-M supports a jobs-as-code approach using JSON and RESTful APIs, which can be integrated into a CI/CD toolchain. Google Cloud developer tools including Cloud Build and Cloud Deploy are compatible with this method. 

What Google Cloud services are used in a Control-M data pipeline?

A typical Control-M-orchestrated pipeline on Google Cloud uses Cloud Storage as the data landing zone, Pub/Sub for streaming data ingestion, Dataflow for ETL processing, BigQuery for analytics and querying, and Data Studio for visualization. Control-M coordinates the sequencing and monitoring of all these services. 

How does Control-M handle pipeline failures on Google Cloud? 

Control-M includes built-in error handling and SLA management capabilities. When a job fails, Control-M can automatically restart it or escalate to a human based on predefined criteria. The SLA Management job type monitors the entire workflow as a single business entity and uses historical execution data to predict completion times, alerting monitoring teams to deviations before business SLAs are breached. 

The views and opinions expressed in this post are those of the author and do not necessarily reflect the official position of BMC.

DataOps applies agile engineering and DevOps best practices to data management, helping organizations rapidly turn raw data into fully operationalized production deliverables that unlock real business value. For companies struggling to extract results from their data investments, DataOps provides the framework—and the right workflow orchestration platform provides the engine—to run data pipelines reliably at enterprise scale. 

Across every industry, companies continue to put increased focus on gathering data and finding innovative ways to garner actionable insights. Organizations are willing to invest significant time and money to make that happen. 

However, despite high levels of investment, data projects can often yield lackluster results. A recent survey of advanced major analytics programs by McKinsey found that companies spend 80 percent of their time doing repetitive tasks such as preparing data, where limited value-added work occurs. Additionally, they found that only 10 percent of companies feel they have this issue under control. 

So why are data project failure rates so high despite increased investment and focus? 

Many variables can impact project success. Often cited factors include project complexity and limited talent pools. Data scientists, cloud architects, and data engineers are in short supply globally. Companies are also recognizing that many of their data projects are failing because they struggle to operationalize the data initiatives at scale in production.

How does DataOps help unlock data initiatives?

DataOps is the application of agile engineering and DevOps best practices to the field of data management—helping organizations rapidly turn new insights into fully operationalized production deliverables that unlock business value from data. By treating data pipelines with the same discipline applied to software delivery, DataOps reduces the time between a raw insight and a production-ready, business-usable output.

The number of organizations adopting DataOps practices to help them unlock their data is increasing exponentially, so much so that analyst firms have started tracking DataOps tools as a market.

In 2022, industry analyst Gartner® published the Market Guide for DataOps Tools, in which it provided this market definition: 

“DataOps tools provide greater automation and agility over the full life cycle management of data pipelines in order to streamline data operations. The core capabilities of a DataOps tool include:

• Orchestration: Connectivity, workflow automation, lineage, scheduling, logging, troubleshooting, and alerting
• Observability: Monitoring live/historic workflows, insights into workflow performance and cost metrics, impact analysis
• Environment Management: Infrastructure as code, resource provisioning, environment repository templates, credentials management
• Deployment Automation: Version control, release pipelines, approvals, rollback, and recovery
• Test Automation: Business rules validation, test scripts management, test data management”

As the Gartner market definition indicates, orchestration of data pipelines is a key element of DataOps capabilities. However, data workflow orchestration comes with its own set of challenges.

What are the data orchestration challenges?

Most data pipeline workflows are immensely complex and run across many disparate applications, data sources, and infrastructure technologies that need to work together. While the goal is to automate these processes in production, the reality is that without a powerful workflow orchestration platform, delivering these projects at enterprise scale can be expensive and often requires significant time spent doing manual work.

Data workflow orchestration projects have four key stages: ingestion, storage, processing, and delivering insights to make faster and smarter decisions.

Figure 1. Data projects have four stages with many moving parts across multiple technologies.

Ingestion involves collecting data from traditional sources like enterprise resource planning (ERP) and customer resource management (CRM) solutions, financial systems, and many other systems of record in addition to data from modern sources like devices, Internet of Things (IoT) sensors, and social media.

Storage increases the complexity with numerous different tools and technologies that are part of the data pipeline. Where and how you store data depends a lot on persistence, the relative value of the data sets, the refresh rate of your analytics models, and the speed at which you can move the data to processing.

Processing has many of the same challenges. How much pure processing is needed? Is it constant or variable? Is it scheduled, event-driven, or ad hoc? How do you minimize costs? The list goes on and on.

Delivering insights requires moving the data output to analytics systems. This layer is also complex, with a growing number of tools representing the last mile in the data pipeline.

With new data and cloud technologies being frequently introduced, companies are constantly reevaluating their tech stacks. This evolving innovation creates pressure and churn that can be challenging because companies need to easily adopt new technologies and scale them in production. Ultimately, if a new data analytics service is not in production at scale, companies are not getting actionable insights or achieving value.

What capabilities should a workflow orchestration platform have?

Successfully running business-critical workflows at scale in production doesn’t happen by accident. The right workflow orchestration platform can help you streamline your data pipelines and get the actionable insights you need.

Here are eight essential capabilities to look for in your workflow orchestration platform:

1. Support heterogeneous workflows: Companies are rapidly moving to the cloud, and for the foreseeable future will have workflows across a highly complex mix of hybrid environments. For many, this will include supporting the mainframe and distributed systems across the data center and multiple private and/or public clouds. If your orchestration platform cannot handle the diversity of applications and underlying infrastructure, you will have a highly fragmented automation strategy with many silos of automation that require cumbersome custom integrations to handle cross-platform workflow dependencies.
2. Service level agreement (SLA) management: Business workflows—ranging from ML models predicting risk to financial close and payment settlements—all have completion SLAs that are sometimes governed by guidelines set by regulatory agencies. Your orchestration platform must be able to understand and notify you of task failures and delays in complex workflows, and it needs to be able to map issues to broader business impacts.
3. Error handling and notifications: When running in production, even the best-designed workflows will have failures and delays. It is vital that the right teams are notified so that lengthy war room discussions just to figure out who needs to work on a problem can be avoided. Your orchestration platform must automatically send notifications to the right teams at the right time.
4. Self-healing and remediation: When teams respond to job failures within business workflows, they take corrective action, such as restarting a job, deleting a file, or flushing a cache or temp table. Your orchestration platform should enable automation engineers to configure such actions to happen automatically the next time the same problem occurs.
5. End-to-end visibility: Workflows execute interconnected business processes across hybrid tech stacks. Your orchestration platform should be able to clearly show the lineage of your workflows. This is integral to helping you understand the relationships between applications and the business processes they support. This is also important for change management—when making changes, it is vital to see what happens upstream and downstream from a process. 
6. Self-service user experience (UX) for multiple personas: Workflow orchestration is a team sport with many stakeholders such as data teams, developers, operations, business process owners, and more. Each team has different use cases and preferences for how they want to interact with the orchestration tools. This means your orchestration platform must offer the right user interface (UI) and UX for each team so they can benefit from the technology. 
7. Production standards: Running workflows in production requires adherence to standards, which means using correct naming conventions, error-handling patterns, and so on. Your orchestration platform should have a mechanism that provides a very simple way to define such standards and guide users to the appropriate standards when they are building workflows.
8. Support DevOps practices: As companies adopt DevOps practices such as continuous integration and continuous deployment (CI/CD) pipelines, the workflow development, modification, and even infrastructure deployment of workflows, your orchestration platform should be able to fit into modern release practices. 

How do Control-M and Control-M SaaS support DataOps? 

DataOps tools and methodologies can help you make the best use of your data investment. But if you want to succeed in your DataOps journey, you must be able to operationalize the data. Control-M (self-hosted) and Control-M SaaS provide a layer of abstraction to simplify the orchestration of complex data pipelines. These application and data workflow orchestration platforms enable end-to-end visibility and predictive SLAs across any data technology or infrastructure.

Figure 2. Control-M is a layer of abstraction to simplify complex data pipelines.

Control-M and Control-M SaaS can help you orchestrate your data pipelines, put your data to effective use, and improve your data-driven business outcomes. Both platforms are used by thousands of companies globally and are proven to help companies run data pipeline workflows in production at scale.

Here are some examples of the robust capabilities Control-M and Control-M SaaS have and how they can help you streamline your data pipeline workflow orchestration:

Robust integrations

The tools required to run a modern business vary widely. Often, each department utilizes its own technologies, requiring manual scripting to connect workflows across the business. Control-M and Control-M SaaS feature a vast library of out-of-the-box integrations that allow businesses to orchestrate the latest technologies.

SLA management and impact analysis

With Control-M and Control-M SaaS, you can track the status of business service levels along with corresponding workflows, so you know exactly how business services are performing at any given time. The two platforms can predict that a service will be late if a job is delayed or has failed upstream because they are using historical data to calculate how long a downstream job usually takes to run. Using this data, they can notify stakeholders not only that a particular job is late, but which business services are at risk of being delayed.

Python client

Many teams within an organization need to interact with your workflow orchestration platform for various reasons. Developers are a particularly important stakeholder in the orchestration process. They develop the applications that will run in production and be orchestrated by Control-M and Control-M SaaS. The Python client allows developers to natively invoke their functions from their Python code.

Visibility for business users

Business users are an important stakeholder, as well. They are ultimately responsible for the timely delivery of the services they own. With the Control-M mobile app and web interface, they can track the status of their workflows anytime, from anywhere, without having to contact the application teams or operations for status updates.

The need for data is on the rise and shows no signs of abating, which means that having the ability to store, process, and operationalize that data will remain crucial to the success of any organization. DataOps practices backed by the powerful data orchestration capabilities of Control-M and Control-M SaaS can help you orchestrate data pipelines, streamline the data delivery process, and improve business outcomes.

To learn more about how Control-M/Control-M SaaS can help you deliver data-driven outcomes faster, visit our website. 

1. *Market Guide for DataOps Tools; December 5, 2022; Robert Thanaraj, Sharat Menon, Ankush Jain

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.

Frequently asked questions 

What is DataOps and why are organizations adopting it?

DataOps is the application of agile engineering and DevOps best practices to data management—helping organizations rapidly operationalize data insights into production-ready deliverables. Organizations adopt DataOps because the complexity of modern data environments makes it nearly impossible to deliver reliable, scalable data pipelines without a structured framework for automation, orchestration, and governance. 

What are the four stages of data workflow orchestration?

Data workflow orchestration projects move through four stages: ingestion (collecting data from traditional and modern sources), storage (managing persistence, value, and refresh rates), processing (handling compute requirements and scheduling), and delivering insights (routing data output to analytics systems for decision-making). 

What are the most important capabilities in a workflow orchestration platform?

The eight essential capabilities are: support for heterogeneous workflows across hybrid environments, SLA management and business impact mapping, error handling and automatic notifications, self-healing and remediation, end-to-end workflow visibility and lineage, self-service UX for multiple personas, production standards enforcement, and support for DevOps and CI/CD practices. 

How do Control-M and Control-M SaaS simplify data pipeline orchestration?

Control-M and Control-M SaaS act as an abstraction layer that simplifies complex data pipeline orchestration by providing a vast library of out-of-the-box integrations, predictive SLA management, dependency-aware scheduling, and end-to-end visibility across any data technology or infrastructure. Both platforms are used by thousands of companies globally to run data pipeline workflows reliably at enterprise scale. 

What is the difference between Control-M and Control-M SaaS?

Control-M is a self-hosted workflow orchestration platform, while Control-M SaaS is a fully managed, cloud-delivered version of the same platform. Both provide core capabilities—including SLA management, predictive impact analysis, robust integrations, and end-to-end visibility—but Control-M SaaS eliminates infrastructure management overhead, making it well-suited for organizations standardizing on cloud-first operations. 

The views and opinions expressed in this post are those of the author and do not necessarily reflect the official position of BMC.

The multi-agent AI era isn’t coming—it’s already here. According to Deloitte, 75% of organizations are investing in AI agents, driving a surge in enterprise adoption. And according to IDC, this isn’t incremental—it’s a structural shift, with agentic AI–driven investment expected to reach $1.3 trillion by 2029.

On the surface, more agents should mean more value.

It doesn’t.

The very force accelerating AI adoption—agent proliferation—may ultimately constrain its impact. As Deloitte notes, once organizations move toward multi‑agent systems, orchestration becomes essential to unlocking their full potential. Yet many enterprises still frame the problem as one of intelligence: bigger models, smarter agents, more autonomy.

That framing is incomplete. The real challenge facing enterprises today isn’t intelligence—it’s execution.

This article explores why orchestration, not additional agents, is the critical missing layer in enterprise AI. It explains how agent sprawl creates complexity, why agent‑only orchestration falls short, and why enterprises must treat orchestration as a control plane—coordinating agents, workflows, data pipelines, and legacy systems—to reliably translate AI into real business outcomes.

Five realities enterprises must confront

The challenge isn’t whether agents will be adopted. That’s already happening. The real question is whether enterprises are prepared for what comes next.

1. Agent sprawl is inevitable

As agents deliver value, organizations will deploy more of them—quickly, on a multitude of platforms. What starts as a targeted approach becomes a distributed ecosystem of autonomous components. Left unchecked, this fragmentation creates a coordination problem—multiple agents making decisions across disconnected environments with no shared understanding of timing, dependencies, or outcomes. It’s a bit like the “Not Hot Dog” app from the series Silicon Valley—a model that could perfectly identify a hot dog and confidently label everything else as “not hot dog.” Technically impressive. Practically useless beyond a very narrow context

2. Orchestrating agents isn’t enough

The instinctive response is to orchestrate the agents themselves. But that only solves part of the problem.

Agents don’t operate in isolation—they plug into larger business processes. Financial close, trade reconciliation, inventory replenishment, even data pipelines that power inference, RAG, and BI all remain multi-step workflows. Agents may automate decisions within them, but they don’t run them end-to-end.

Which means orchestration can’t be designed around agents alone. It has to coordinate agents alongside scripts, APIs, batch jobs, and serverless functions that make up the rest of the process.

Otherwise, you’re not eliminating complexity—you’re creating another orchestration silo that still has to be connected to everything else.

3. AI-ready data doesn’t solve itself

Another emerging lesson is that agents are only as good as the data they consume. As enterprises invest heavily in models and agents, many discover that the real bottleneck is data readiness. Fragmented, outdated, or poorly governed data leads to unreliable outputs. What is new, however, is orchestration’s role in resolving that bottleneck. Preparing AI-ready data requires coordinating data pipelines, application workflows, and event triggers across the enterprise. The intelligence layer depends on that foundation.

4. The enterprise is more hybrid than ever

Despite the hype around new technologies, most enterprises operate in deeply hybrid environments. Mainframes remain the lifeblood of many of the world’s largest companies—powering core transactions and systems of record—while cloud-native platforms and microservices drive new digital experiences and AI innovation. Modern data tools interact with both, and critical processes now span generations of infrastructure. These systems aren’t disappearing anytime soon. The challenge isn’t replacing them—it’s ensuring that new AI-driven capabilities work alongside them. That’s where orchestration across the entire stack becomes essential.

5. Reliability still defines success

In the race to deploy the newest AI tools, it’s easy to overlook something fundamental: reliability. Enterprise workflow orchestration has long been judged by a simple standard—it just works. Think of it like a Swiss watch: precise, dependable, and trusted to run critical operations. AI systems must meet that same bar. Autonomy is powerful, but enterprises won’t accept fragile automation in mission-critical environments. The orchestration layer must ensure workflows remain predictable, auditable, and resilient—even as intelligence becomes more distributed.

The path from complexity to simplicity

Most enterprise problems aren’t glamorous. It’s easy to get excited about frontier models, GPUs, and agents that can reason and act—that’s where the headlines are. But the problems haven’t changed: How do we accelerate financial close? How do we detect and prevent fraud before it impacts customers? How do we execute trades reliably at market speed? How do we keep shelves stocked and orders fulfilled? How do we ensure critical healthcare data is available when it’s needed most?

Simple in nature. Relentless in execution.

Delivering those outcomes means coordinating workflows across agents and traditional systems like ERPs, CRMs, data lakes that span everything from multiple clouds to mainframe systems—all while meeting SLAs, audit, traceability, and explainability requirements. There’s nothing flashy about that. But without it, AI stays in the lab and never graduates to production environments, which is where systems deliver business value.

This is why orchestration isn’t just a tool—it’s a strategy. A control plane for execution.

The goal isn’t more agents—it’s better outcomes. Achieving that requires something often overlooked: simplicity. Orchestration is what turns complexity into simplicity. As Leonardo da Vinci put it, “Simplicity is the ultimate sophistication.”

ETL (extract, transform, and load) is the wrong approach for mainframe data migration because it was built for structured, database-to-database transfers—not the flexible, cloud-ready data movement that modern mainframe environments require. ETL’s complexity, labor demands, processing costs, and inability to handle unstructured data make it a poor fit for organizations pursuing mainframe data migration today. ELT (extract, load, and transform), which moves raw data to its destination first and transforms it there, is better aligned with how mainframe modernization actually works.

Change is good—a familiar mantra, but one not always easy to practice. When it comes to moving toward a new way of handling data, mainframe organizations, which have earned their keep by delivering the IT equivalent of corporate-wide insurance policies (rugged, reliable, and risk-averse), naturally look with caution on new concepts like extract, load, and transform (ELT). 

Positioned as a lighter and faster alternative to more traditional data handling procedures such as extract, transform, and load (ETL), ELT definitely invites scrutiny. And that scrutiny can be worthwhile. 

Definitions provided by SearchDataManagement.com say that ELT is “a data integration process for transferring raw data from a source server to a data system (such as a data warehouse or data lake) on a target server and then preparing the information for downstream uses.” In contrast, another source defines ETL as “three database functions that are combined into one tool to pull data out of one database and place it into another database.” 

The crucial functional difference is that ETL focuses exclusively on database-to-database transfer, while ELT is open-ended and flexible. In the mainframe world, ETL is a tool with a more limited focus—ELT is focused on jump-starting the future.

Why does ETL fail for mainframe data migration?

ETL falls short across five key dimensions: complexity, labor intensity, processing bottlenecks, structural rigidity, and high processing costs. Here is a closer look at each.

1. ETL is too complex

ETL was not originally designed to handle all the tasks it is now being asked to do. In the early days, ETL was often applied to pull data from one relational structure and fit it into a different relational structure—including cleansing the data along the way. 

For example, a traditional relational database management system (RDBMS) can get befuddled by numeric data where it is expecting alpha data, or by the presence of obsolete address abbreviations. ETL is optimized for that kind of painstaking, field-by-field data checking, “cleaning,” and movement—but not for feeding a Hadoop database or modern data lake. ETL was not invented to take advantage of all the ways data originates and all the ways it can be used today. 

2.ETL is labor-intensive

Because the ETL process is built around transformation, everything depends on the timely completion of that transformation step. With larger amounts of data in play—think Big Data—the required transformation times can become inconvenient or impractical, turning ETL into a functional and computational bottleneck. 

3.ETL demands structure

Because the ETL process is built around transformation, everything depends on the timely completion of that transformation step. With larger amounts of data in play—think Big Data—the required transformation times can become inconvenient or impractical, turning ETL into a functional and computational bottleneck. 

4.ETL demands structure

ETL is not designed for unstructured data and can add complexity rather than value when asked to handle it. ETL works best for traditional databases but does not help much with the massive waves of unstructured data that organizations need to process today.

5.ETL has high processing costs

ETL is especially challenging for mainframes because mainframe workloads generally incur MSU (million service unit) processing charges—burdening systems that need to be handling real-time workloads at the same time. ELT, by contrast, can be accomplished using mostly the capabilities of built-in zIIP engines, which cuts MSU costs, with additional processing handled at a chosen cloud destination. In response to ETL’s high processing costs, many organizations have already moved the data transformations stage to the cloud to support analytics workloads and data lake creation. 

Why is ELT the better path forward for mainframe organizations?

It would be wrong to oversimplify a decision between ETL and ELT—there are too many moving parts and decision points to weigh. But the key insight is this: ELT speaks to the evolving IT paradigms that ETL was never built to address.

ELT is ideal for moving massive amounts of data to the cloud—typically to a data lake built to ingest any and all available data so that modern analytics can get to work. That is why ELT is growing and making inroads specifically in the mainframe environment. ELT represents perhaps the best path to accelerating mainframe data movement to the cloud at scale, making ELT a key tool for IT organizations aiming at modernization and at maximizing the value of their existing investments.

Frequently asked questions

What is ETL and why has it been used in mainframe environments?

ETL (extract, transform, and load) is a data integration method that pulls data from a source, transforms it to match a target schema, and loads it into a destination database. Mainframes have historically relied on ETL because it was well-suited to the structured, RDBMS-to-RDBMS data movement that dominated before cloud and big data workloads became standard. 

What is the difference between ETL and ELT?

ETL transforms data before loading it into the destination system. ELT loads raw data to the destination first and transforms it there, leveraging the destination system’s processing power. ELT is more flexible and better suited to modern cloud environments and data lakes than ETL. 

Why does ETL become a bottleneck for mainframe data migration?

ETL requires all transformation to complete before data can move to its destination. At the scale of modern mainframe data volumes, this dependency on sequential transformation creates delays that make ETL impractical. ELT avoids the bottleneck by separating the load and transform steps entirely. 

What is a zIIP engine and how does it reduce mainframe processing costs?

A zIIP (IBM Z Integrated Information Processor) is a specialty engine on IBM mainframes designed to offload eligible workloads from general-purpose processors, reducing MSU charges. ELT workloads are often eligible for zIIP processing, making ELT significantly more cost-efficient than ETL for mainframe data migration projects. 

When does ETL still make sense over ELT?

ETL remains appropriate for structured, database-to-database migrations where schema alignment and data quality transformation must happen before the data reaches its destination—particularly when data volumes are manageable and the target system requires pre-transformed data. For large-scale, cloud-bound mainframe modernization, ELT is generally the better choice. 

The views and opinions expressed in this post are those of the author and do not necessarily reflect the official position of BMC.

BMC has been named the overall highest performer and a Value Leader in the 2025 EMA Radar for Workload Automation and Orchestration—for the eighth consecutive time. Control-M also earned EMA’s recognition for Excellence in Mission-Critical Orchestration, reinforcing its standing as the market leader for enterprise-class orchestration. For organizations evaluating workload automation platforms, this recognition reflects Control-M’s maturity, breadth, and forward-looking platform strategy.

Control-M: Leading Enterprise Orchestration

EMA’s report highlights Control-M’s unmatched maturity, scalability, and innovation cadence as the defining factors behind BMC’s Value Leader designation. With a modern, API-first architecture and feature parity across SaaS and on-premises deployments, Control-M empowers organizations to orchestrate data pipelines, applications, and infrastructure with confidence and control.

Key Differentiators Called Out by EMA

EMA identified five areas where Control-M outperforms market averages across the EMA Radar workload automation evaluation criteria.

Mission-critical orchestration 

Control-M leads in managing complex, infrastructure-intensive business processes with precision and reliability—the capability EMA specifically recognized with its Excellence in Mission-Critical Orchestration award. 

Hybrid and multi-cloud reach 

Native integrations with AWS, Azure, GCP, Oracle Cloud, and Kubernetes enable seamless orchestration across hybrid and multi-cloud environments, without requiring separate tooling per platform. 

Data pipeline and DataOps leadership 

Deep integrations with Snowflake, Databricks, Apache Airflow, and other modern data platforms make Control-M a cornerstone for enterprise data operations and DataOps workflows. 

DevOps and Jobs-as-Code 

Developers can define, test, and promote workflows using JSON or Python, embedding workload orchestration directly into CI/CD pipelines and developer-native toolchains. 

Observability and AI 

Embedded SLA clocks, anomaly detection, and Jett—Control-M’s fully integrated GenAI copilot—bring closed-loop intelligence to orchestration. Jett supports workflow optimization, SLA prediction, and real-time guidance, making AI a foundational part of the platform experience. 

A Vision for the Future: Orchestrator of Orchestrators

EMA recognized BMC’s forward-looking strategy to position Control-M as the “orchestrator of orchestrators”—a unifying layer that spans ERP systems, DevOps pipelines, service management tools, and AI platforms. This vision is already taking shape through expanded integrations, enhanced Workflow Insights, and GenAI-powered Advisors. 

Control-M SaaS continues to gain momentum, offering global reach, hybrid visibility, and enterprise-grade resilience. With a single console view across SaaS and on-premises environments, organizations can modernize at their own pace without compromising governance or control.

Strategic value drivers

BMC’s strategy in the workload automation and orchestration market rests on three core principles:

• End-to-end orchestration: Delivering orchestration of AI, data, and application workflows across hybrid environments—from multi-cloud to mainframe. 
• Agentic orchestration: Building on the GenAI-powered advisor Jett toward an agentic model, enabling a fleet of specialized AI agents to dynamically build, execute, and manage end-to-end workflows. 
• Flexible deployment: Providing SaaS or self-hosted options with a unified view that ensures consistency, governance, and control across environments. 

These principles define how Control-M helps enterprises turn operational complexity into competitive advantage—operating with resilience and scaling innovation with confidence. 

To find out more about Control-M’s recognition and continued leadership in the EMA Radar workload automation space, download a copy of the report here.

Frequently asked questions

What is the EMA Radar for Workload Automation and Orchestration?

The EMA Radar for Workload Automation and Orchestration is an independent analyst evaluation by Enterprise Management Associates (EMA) that assesses vendors across criteria including functionality, architecture, deployment, cost advantage, and vendor strength. Value Leader status indicates the highest combined score for performance and value among evaluated vendors. 

What does it mean that BMC was named a Value Leader in the EMA Radar?

Being named a Value Leader means BMC’s Control-M achieved the highest overall performance score in EMA’s 2025 evaluation while also delivering strong cost-to-value positioning. BMC has held this designation for eight consecutive years, reflecting sustained leadership rather than a single-year result. 

What is Excellence in Mission-Critical Orchestration?

Excellence in Mission-Critical Orchestration is a specific EMA recognition awarded to vendors that demonstrate top-tier capability in managing complex, high-stakes, infrastructure-intensive workflows. Control-M earned this designation in the 2025 EMA Radar evaluation alongside its Value Leader status. 

How does Control-M support hybrid and multi-cloud workload automation?

Control-M provides native integrations with major cloud platforms—AWS, Azure, GCP, and Oracle Cloud—as well as Kubernetes, enabling organizations to orchestrate workloads consistently across on-premises and cloud environments from a single unified console. 

The views and opinions expressed in this post are those of the author and do not necessarily reflect the official position of BMC.

To operationalize data projects, organizations need automated, orchestrated, end-to-end visibility across every stage of the data pipeline. Without a structured operationalization framework, even the most sophisticated data initiatives stall before reaching production. A unified workflow orchestration platform that abstracts the complexity of disparate data tools is the critical bridge between a working prototype and enterprise-scale deployment.

Data is abundant—and growing exponentially. But simply having the data isn’t enough. Businesses consistently struggle to move data projects from development into production, and the cost of that gap is significant. In 2018, Gartner^® predicted in their report entitled “Predicts 2019: Artificial Intelligence Core Technologies” that only 15 percent of cutting-edge data projects would make it into production by 2022—meaning 85 percent would fail to produce results. Separately, in its Top Trends in Data and Analytics, 2022 report, Gartner warned that organizations without a sustainable data and analytics operationalization framework risk seeing their initiatives set back by up to two years.

Why do data projects struggle to reach production?

The core challenge is scale. A project can work well in prototype in one location, but if it can’t be scaled nationally or globally, it has essentially failed.

As companies recognize the need to build operationalization into their plans, the industry has refocused on IT operations (ITOps)—generating a range of discipline-specific Ops frameworks: DataOps for data, MLOps for machine learning, AIOps for artificial intelligence, and ModelOps for analytics modeling. This proliferation has even produced the catch-all term XOps—a placeholder, as some in the industry put it, for “we don’t know what’s coming next but it will involve Ops somehow.” The problem isn’t ambition. It’s complexity.

What are the four stages of a data pipeline?

Every data project shares the same four foundational stages, which form the building blocks of data pipelines: 

1.Data ingestion

Orchestrating data from traditional sources such as enterprise resource planning (ERP) and customer relationship management (CRM) systems, financial platforms, and other systems of record—combined with data from devices, sensors, social media, weblogs, and IoT sensors and devices.

2. Data storage

Where and how data is stored depends significantly on persistence, the relative value of data sets, the rate of refresh for analytics models, and the speed at which data can move to processing. 

3. Data processing

Processing requirements vary widely: How much compute is needed? Is it constant or variable? Is workload scheduled, event-driven, or ad hoc? How are costs minimized? 

4. Insight delivery

The last mile—moving data output to analytics systems. The insights layer is complex and shifts constantly as markets adopt new technologies. A new data analytics service that isn’t in production at scale delivers no actionable insights and no business value, whether measured in revenue generation or operational efficiency.

Why is end-to-end orchestration essential for data pipelines?

The operational goal is to run data pipelines in a highly automated fashion with minimal human intervention and full visibility across every component. But almost every technology in the data pipeline comes with its own built-in automation utilities and tools—tools that are often not designed to work with each other. Stitching these together for end-to-end automation and orchestration is where teams hit the wall. 

This challenge has driven the rise of application and data workflow orchestration platforms: solutions that operate with speed and scale in production and abstract the underlying automation utilities so teams can focus on outcomes rather than infrastructure. 

How does Control-M help operationalize data projects? 

Figure 1. Gartner Data and Analytics Essentials: DataOps by Robert Thanaraj

Control-M from BMC is an application and data workflow orchestration and automation platform that serves as the abstraction layer to simplify the complex data pipeline. Control-M enables end-to-end visibility and predictive service level agreements (SLAs) across any data technology or infrastructure, delivers data-driven insights in production at scale, and integrates new technology innovations into even the most complex data pipelines.

The Control-M platform offers a range of capabilities to automate and orchestrate application and data workflows:

• The Control-M Automation API, which promotes collaboration between Dev and Ops by allowing developers to embed production-ready workflow automation while applications are being developed.
• Out-of-the-box support for cloud resources including Amazon Web Services (AWS) Lambda and Azure Logic Apps, Functions, and Batch to help you leverage the flexibility and scalability of your cloud ecosystems.
• Integrated file transfers with all your applications that allow you to move internal and external file transfers to a central interface to improve visibility and control.
• Self-Service features that allows employees across the business to access the jobs data relevant to them.
• Application Integrator, which supports the creation of custom job types and deploys them in your Control-M environment quickly and easily.
• Conversion tools that simplify conversion from third-party schedulers.

Data projects will continue to grow in strategic importance. Successfully operationalizing data workflows as a core part of project planning and execution is essential to business outcomes. An application and data workflow orchestration platform should be a foundational step in every DataOps journey.

To learn more about how Control-M can help you find DataOps success, visit our website.

Frequently asked questions

What does it mean to operationalize a data project?

To operationalize a data project means to move it from prototype or development into reliable, repeatable production at scale. Operationalization requires automating the core pipeline stages—ingestion, storage, processing, and insight delivery—so that data projects consistently produce actionable results without manual intervention. 

Why do most data projects fail to reach production?

Most data projects fail to reach production because of complexity at scale. Each pipeline stage relies on different tools and technologies that don’t natively integrate, making end-to-end automation difficult to achieve. Without an orchestration layer to unify these components, teams cannot maintain the visibility and control needed to run pipelines reliably in production. 

What is the difference between data automation and data orchestration?

Data automation refers to executing individual tasks or processes without human intervention. Data orchestration coordinates multiple automated tasks across tools, systems, and environments to execute a complete data pipeline as a single managed workflow. Orchestration is what enables automation to work at enterprise scale. 

What role does DataOps play in operationalizing data projects?

DataOps applies DevOps principles—continuous integration, automation, and collaboration—to data pipeline development and operations. A DataOps framework helps organizations build operationalization into data projects from the start, reducing the time from prototype to production and improving the reliability of data-driven outcomes. 

What should I look for in a workflow orchestration platform?

A workflow orchestration platform for data projects should provide end-to-end pipeline visibility, support for heterogeneous data technologies and cloud environments, event-driven and scheduled workload capabilities, integrated file transfer management, self-service access for business users, and APIs that enable Dev and Ops collaboration during development. 

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.

The views and opinions expressed in this post are those of the author and do not necessarily reflect the official position of BMC.