As digital certificate lifetimes drop to 47 days, automation becomes essential to maintain availability, security, and compliance. BMC AMI Digital Certificate Manager extends automated certificate lifecycle management to the mainframe, enabling standardization across the enterprise using your current CLM vendor.

Digital certificates are the connective tissue of the enterprise environment, enabling systems, workloads, applications, and APIs to verify each other and communicate securely. When a certificate fails due to expiration or error, the impact on service availability and security can be immediate and severe. And with regulators driving certificate lifetimes down to as little as 47 days by 2029, organizations face a sharp increase in both operational and compliance risk. That makes digital certificate management a top priority for BMC customers.

By enabling organizations to discover, track, and renew certificates across the infrastructure, digital certificate management prevents the outages and security gaps that can result from expired, misconfigured, or otherwise compromised certificates. This task has grown more difficult in recent years, and even greater challenges are on the horizon. But BMC has a solution.

Why digital certificate management is becoming an urgent challenge

Traditionally, many organizations have managed mainframe certificates through manual processes centered on spreadsheets and tribal knowledge. In recent years, the growing number of system identities relying on these certificates has pushed these methods to the breaking point. Now, regulatory changes have made them completely unsustainable.

To reduce the exposure that can result from a compromised certificate, the CA/Browser Forum has announced aggressive reductions in TLS certificate lifetimes. Until this month, companies were allowed a relatively manageable 398-day renewal cycle. Now that window has been nearly cut in half to 200 days. Next March, it will shrink once again to 100 days, and by March 2029, TLS certificates will be good for only 47 days. Each of these reductions effectively multiplies the certificate management workload for mainframe security teams, and with it, the chance of manual errors, expirations, and system outages.

This isn’t a future problem. The regulatory deadlines are fixed, the timelines are non-negotiable, and their impact is inevitable. That’s why I’m excited to announce BMC AMI Digital Certificate Manager (DCM)—a new solution that fundamentally changes certificate management on the mainframe.

How to extend enterprise digital certificate management to the mainframe

Most enterprises already invest in Certificate Lifecycle Management (CLM) platforms for their distributed and cloud environments. These platforms haven’t been able reach the mainframe, however, leaving z/OS as a manual island in an otherwise automated estate. Now BMC is filling that gap with the only solution enabling digital certificate management platforms to extend automation to the mainframe as part of a consistent enterprise strategy.

Proven in operational environments for over five years, DCM provides a unified integration layer to connect Venafi and Keyfactor digital certificate management tools to mainframe ESMs including RACF, ACF2, and Top Secret. With DCM, you can standardize on one BMC solution for your mainframe while supporting whichever certificate vendors your organization already uses, no rip-and-replace required.

End-to-end automated certificate operations

DCM extends your organization’s CLM to automate the entire mainframe certificate lifecycle, from issuance and renewal to replacement and rollback. The impact is immediate and measurable:

• Dramatic effort reduction: Mainframe certificate implementations that previously took up to three hours of manual work are now fully automated.
• Eliminated outage risk: Expired or mismanaged certificates are a major cause of preventable mainframe outages. DCM’s scheduled renewals and built-in rollback ensure continuous availability without late-night firefighting.
• Reduced dependency on scarce skills: Mainframe security expertise is increasingly hard to find. DCM removes the need for skilled personnel to manually execute certificate commands across RACF, ACF2, or Top Secret, freeing them for higher-value work.
• Complete audit visibility: Every action is logged with full detail, including which commands were issued, which ESM responses were received, who authorized the change, and when.

Real-world impact at a major financial institution

One of the world’s largest financial institutions evaluated DCM against its current, pre-automation state. With certificate volumes growing over 30 percent year over year, a small core team currently handles digital certificate management manually across many application owners and faces an increasing risk of outages, audit failures, and security gaps.

The firm projected the five-year value of deploying DCM as $8.6 million, driven by manual effort reduction and avoided headcount ($3.6M), eliminated application outages ($2.2M), compliance and audit risk reduction ($1.4M), operational efficiency gains ($0.8M), and future-proofing against accelerating certificate volumes ($0.6M). Beyond these measurable financial gains, the solution supports the institution’s broader strategic priorities around operational resilience and responsible growth.

Strengthening Zero Trust across the enterprise

Machine identity is foundational to Zero Trust: Every workload, process, and system must be authenticated. Working alongside BMC AMI Security, DCM becomes part of a comprehensive Zero Trust strategy for the mainframe, enabling continuous threat detection, automated response, and end-to-end protection across your most critical environment. Security teams gain the observability, policy enforcement, and confidence they need to report to the chief information security officer (CISO) and the board that the mainframe is truly protected.

Looking ahead

The certificate landscape continues to move toward shorter lifetimes, more frequent renewals, higher volumes, and tighter regulatory scrutiny. All of these trends will drive an exponential growth in manual digital certificate management workloads. By acting now, organizations can stay ahead of increasingly urgent certificate deadlines while preparing their infrastructure for continuous, automated certificate renewals.

BMC AMI Digital Certificate Manager is generally available. We invite you to learn more about how DCM can modernize certificate management across your mainframe environment—preserving your existing tools, eliminating manual effort, and building the operational resilience your business demands.

BMC is aware of recent industry discussion about the use of advanced AI techniques to identify software vulnerabilities, including initiatives such as Project Glasswing.

As part of normal security operations, BMC continuously monitors emerging research, threat intelligence, and industry developments related to software and supply chain security. These developments reflect an acceleration in how vulnerabilities may be identified across the industry.

BMC maintains a defense-in-depth security program and regularly evaluates opportunities to enhance controls and processes as technologies evolve. We engage with customers, partners, and the broader security community to remain aligned with industry best practices.

As new information becomes available, BMC will use its proactive notification process to help keep customers up to date.

To register for proactive notifications, please see the following BMC Support Central article:

https://docs.bmc.com/xwiki/bin/view/Standalone/BMC-Support-Central-User-Guide/supportcentraluserguide/Manage-Your-Support-Account/Favorite-Products-and-Alerts/

The Orchestration Imperative for the AI Era

Something interesting is happening in enterprise technology right now. In conversations with clients, I’m hearing that as companies modernize, invest heavily in AI, and build increasingly complex digital ecosystems, many are running into the same reality.

Innovation is moving faster than operations.

And the gap is now showing up in execution. Customers report that as they move quickly to launch more AI pilots and adopt new platforms, the operational foundations beneath these innovations often remain fragmented. Automation silos, disconnected pipelines, and governance gaps create that fragmentation—and introduce risk just when scale matters most.

What’s interesting is where this leads: automation itself is no longer the goal. The question is no longer how do we automate more? It’s how do we orchestrate everything reliably at enterprise scale?

Based on numerous conversations with customers and prospects in recent months, here are five operational shifts I believe every enterprise will face if they aren’t already there.

1. From AI experimentation to AI execution

Many enterprises have already piloted GenAI. They are now expecting measurable outcomes. The focus is shifting from isolated experiments to production-grade AI embedded directly in operations. Leaders are asking practical questions:

Does AI …

• Reduce incidents?
• Accelerate root cause analysis?
• Improve SLA adherence and resource utilization?

AI will increasingly power predictive workflow orchestration—anticipating job failures, recommending remediation, reallocating resources, and learning from execution patterns. But intelligence alone isn’t enough. Enterprise AI must be transparent, explainable, and governed. Black-box automation won’t meet operational standards. The most valuable AI will be embedded directly into orchestration platforms—where visibility, policy enforcement, and accountability already exist.

The organizations that succeed won’t be those that experimented the most. They’ll be the ones that operationalized AI to improve resilience, efficiency, and speed at scale.

 2. Speed at scale demands “freedom with guardrails”

Technology ownership continues to decentralize. Developers, data engineers, DevOps teams, and business technologists all want the ability to build and deploy quickly without waiting on centralized teams. But autonomy without structure creates risk—security gaps, compliance exposure, duplicated workflows, and operational sprawl.

In response, leading organizations are adopting a “freedom with guardrails” model. Teams gain self-service capabilities to build workflows and pipelines, while governance is embedded directly into the automation layer. Role-based access, policy enforcement, auditability, and standardized templates allow teams to move independently without sacrificing oversight.

The key shift is architectural: governance won’t be an afterthought or manual review step. It will be codified into orchestration frameworks themselves. This balance of empowerment and control will become a defining characteristic of high-performing digital enterprises.

 3. Real-time orchestration becomes the backbone of operations

The era of purely nightly batch processing is fading. As digital services compress response times and customer expectations approach real-time, orchestration must evolve from static scheduling to event-driven responsiveness.

Moving forward, organizations will increasingly rely on workflows triggered by real-time business signals—transaction anomalies, supply chain disruptions, or customer behavior changes. Modern orchestration platforms will unify batch, micro-batch, and streaming execution models within a single operational framework.

This requires horizontal scalability, high availability across hybrid and multi-cloud environments, and end-to-end workflow visibility. Monitoring alone won’t be enough. Organizations will demand predictive insights and proactive remediation, transforming observability into operational foresight.

4. Ecosystem collaboration emerges as a competitive advantage

No enterprise operates in isolation—and no orchestration platform can either. Modern operations span SaaS applications, hyperscalers, on-prem systems, managed file transfer, data lakes, AI services, and edge environments. The orchestration layer increasingly becomes the connective tissue across this ecosystem.

Interoperability will move from nice-to-have to mandatory. Customers will expect secure, open, and extensible integrations that connect data pipelines, AI engines, compliance systems, and operational workflows seamlessly. API-driven architectures, pre-built integrations, and ecosystem partnerships will shape buying decisions more than feature checklists.

Working with vendors that embrace openness and integration supporting hybrid and multi-cloud strategies without lock-in will stand out as strategic enablers. Trust, extensibility, and ecosystem compatibility will define leadership in the orchestration market.

5. Orchestration evolves into a shared business capability

Orchestration is no longer an IT-only concern. It touches revenue operations, customer experience, compliance, analytics, and supply chain resilience. Organizations that treat orchestration as a shared, cross-functional capability will outperform those that silo it within infrastructure teams.

This means evolving operating models. Shared ownership between IT operations, data teams, DevOps, and business stakeholders will become the norm. Governance councils, cross-functional workflow design, and outcome-based metrics will replace purely technical KPIs.

Most importantly, orchestration will be treated as a living capability—continuously refined as new applications, AI services, and regulatory requirements emerge. Enterprises that embed this mindset will build an automation backbone that scales with growth, adapts to disruption, and supports innovation without sacrificing control.

Final Thought: Confident AI at Enterprise Scale

The conversations that defined the last twelve months around AI, speed, governance, and resilience are converging into a more mature operational mandate.

The goal is no longer experimentation. It’s confident scale.

Organizations want to move faster, operate smarter, and maintain unwavering trust in every automated outcome. I’m looking forward to continuing this conversation as we operationalize these ideas.

If these themes resonate with what you’re seeing in your organization, I’d love to hear what’s highest on your operational agenda.

IMS continues to serve as the backbone for high-volume, transaction-driven applications across industries such as banking, insurance, retail, and healthcare. These environments are trusted due to their reliability and performance, yet they are also evolving. Today’s IMS systems support a wider mix of users, integrations, and business-critical activity than ever before. As a result, complexity is increasing while many teams cope with limited resources and fewer experienced specialists.

This shift is driving renewed attention to how IMS environments are monitored, analyzed, and managed every day. Traditional monitoring approaches remain important, but they are limited by static thresholds and fragmented views, making it difficult to understand how system behavior fits together. When issues arise, teams may recognize that something is wrong without clear visibility into why it matters or where to focus first.

How AIOps improves IMS visibility

To address this challenge, organizations are increasingly adopting mainframe AIOps. Using machine learning and advanced analytics to examine system and performance data, AIOps helps teams recognize normal behavior, identify emerging conditions earlier, and reduce alert noise that doesn’t require action. In IMS environments, this helps teams gain clarity faster and make more confident decisions without increasing manual effort.

BMC AMI Ops Insight, working alongside brings these AIOps capabilities into IMS operations by learning how systems behave over time and identifying meaningful changes in system behavior within the wider operational context. Rather than treating every deviation the same, intelligent models correlate activity across metrics and subsystems to help teams assess potential impact and prioritize response. This allows teams to move more quickly from detection to understanding, even in highly complex environments.

Combining operational and data insight

Insight becomes even more valuable when paired with data awareness. IMS data activity plays a critical role in application behavior, performance trends, and recovery scenarios. BMC AMI Data for IMS provides visibility into database and application activity, adding context that helps explain what teams are seeing at the system level. When operational insight and data contexts are brought together, teams gain a fuller view of system behavior and risk.

Across the industry, organizations are applying these approaches to shorten detection and resolution timelines, improve incident clarity, and reduce. These efforts also support modernization and resilience initiatives, helping teams maintain confidence and control as demands on IMS environments continue to grow.

Watch the on-demand webinar How AI-Driven Insight is Changing IMS Operations to see how BMC AMI Ops Insight and BMC AMI Data for IMS help organizations apply AIOps-driven insight to IMS operations with greater clarity, faster understanding, and more confident action.

Artificial intelligence (AI) is evolving rapidly, and so are organizations’ approach to its use. In April 2026, BMC released a statement of direction on the integration of AI into the BMC AMI suite of mainframe solutions. In The Next Evolution in Enterprise AI is Purpose Built, BMC mainframe Senior Vice President and General Manager John McKenny discusses the shift from generative AI (GenAI) to agentic AI and our intent to embed agentic AI across the BMC AMI portfolio.

While the statement of direction explains how we’re moving into the future with agentic AI workflows, over the past two years we have developed purpose-built AI for the mainframe, offering in-context expertise through BMC AMI Assistant. With Knowledge Expert Chat, practitioners can harness the power of AI to find the answers they need when and where they need them, increasing the quality and efficiency of their work. Each quarter, we have added to what mainframe professionals can accomplish while increasing integration of BMC AMI Assistant across the BMC AMI portfolio.

The April 2026 release of enhancements to the BMC AMI portfolio focuses on this advancement of mainframe GenAI as well as the open integration of the platform with the broader enterprise IT ecosystem.

GenAI powered by organizational intelligence

Even as we increase the use of agent-driven automation on the mainframe, the efforts to improve GenAI assistance continue, including this quarter’s General Availability of Knowledge Hub, which surfaces institutional knowledge from across the organization to offer contextual information and answers at the moment of decision through BMC AMI Assistant Knowledge Expert Chat.

Knowledge Hub works behind the scenes, ingesting provided institutional knowledge from past issue resolutions, operational insights, and information from runbooks, tickets, and shared files, then combining it with BMC mainframe expertise to create Knowledge Expert Chat responses that are contextually aware and tailored to user’s mainframe environment. These responses are available directly in workflows using existing BMC tools, enabling quicker decisions made with expert-level confidence, regardless of the user’s experience level.

AI-generated application analysis reports

Just as Knowledge Hub captures institutional knowledge to inform assistance provided by Knowledge Expert Chat, a new feature in BMC AMI zAdviser Enterprise adds hard-won institutional knowledge to zAdviser’s collection of development tool and toolchain data to help development managers and their developers understand the applications they are modifying.

BMC AMI zAdviser Enterprise gathers BMC AMI tool usage data and DevOps metrics to give development managers a clear picture of development effectiveness. With new application analysis reports, they now get a single AI-generated view of how their applications work, where the risk is, and where their team’s time is going, accelerating modernization decisions and cutting weeks off of developer onboarding.

BMC AMI zAdviser Enterprise’s Application Analysis turns tribal knowledge into organizational knowledge before it walks out the door. The narrative assessments capture the knowledge of experienced developers, sharing that expertise with development managers and developers, creating more efficient application review, planning, and optimization.

Beyond a greater understanding of individual programs, these application analysis reports provide a clear picture of which programs are attracting a disproportionate share of developer attention because of failures and modifications. By correlating failure history with code complexity and maintenance patterns, they enable development teams to target problem applications with proactive remediation, improving system resilience and development efficiency.

Standardizing enterprise security

Manual management of security certificates decreases efficiency and weakens system security. For some time, security teams have been able to automate certificate management on distributed systems using third-party tools.BMC revolutionized mainframe security management with BMC AMI Enterprise Connector for Venafi. This April, we introduce BMC AMI Certificate Manager, a new product within BMC AMI Security that gives customers more choice and flexibility in integrating enterprise security solutions with the mainframe.

Designed to integrate with IBM Z^® enterprise security management (ESM) environments, including RACF^®, ACF2^™, and Top Secret^® for z/OS to automate the full certificate lifecycle without the need for infrastructure changes, BMC AMI Certificate Manager extends leading enterprise certificate management platforms to the mainframe. This gives CISOs and security teams the ability to use the same vendor solutions for distributed and mainframe certificate management, further integrating the mainframe with—and simplifying—enterprise security efforts.

BMC AMI Certificate Manager currently integrates with Venafi^® and Keyfactor^®, with further integrations to follow in future releases. With this unified integration layer between enterprise certificate management tools and IBM Z^® ESMs, organizations can automate certificate issuance, renewal, and enforcement with one BMC solution while supporting multiple vendors.

Breaking down silos with new technology

This quarter’s enhancements further BMC’s commitment to optimizing what is possible on the mainframe through open integration of the platform and continuous improvements to AI capabilities. With the addition of organization-specific knowledge to AI engines, Knowledge Hub empowers mainframe professionals to make the right decisions as they do their jobs, regardless of their experience and skill levels. Application analysis reports in BMC AMI zAdviser Enterprise combine development and application performance data with system telemetry to provide a clear picture of how applications are performing and where attention and efforts should be focused. And the new BMC AMI Certificate Manager enables security teams to employ certificate management policies across platforms without the need for separate tooling.

Each of these enhancements improves the productivity and efficiency of mainframe teams, a goal BMC is committed to pursuing with each of our quarterly releases. Make your mainframe the engine of faster, better, and smarter answers. When you BMC First.

These are just a few of the innovations included in the April 2026 release of BMC AMI features. To learn more about everything included in the release, visit the What’s New in Mainframe Solutions webpage.

Mainframe organizations continue to run some of the most critical systems in the global economy while managing increasing complexity and rising expectations for innovation. Artificial intelligence (AI) provides a practical way to scale expertise, reduce operational friction, and help teams move work forward with greater confidence.

BMC believes organizations will increasingly leverage AI to broaden access to mainframe capabilities and integrate these systems more fully into their broader enterprise IT ecosystems. In many cases, AI will also act as a catalyst for modernization in place—helping organizations evolve applications, workflows, and operational practices while continuing to run critical workloads on the platform that already powers their business.

The Direction: From AI assistance to coordinated intelligence

Enterprise AI is moving beyond generating insights and explanations. Organizations now expect intelligence that can safely participate in execution across critical systems. Our direction is clear: BMC AMI solutions are evolving into intelligent participants in a governed ecosystem of AI agents that collaborate autonomously across development, operations, data, and security workflows to accelerate innovation on the mainframe.

Supporting this direction requires enterprise AI on the mainframe to operate across three coordinated layers:

Intelligence layer — Domain knowledge and reasoning grounded in mainframe expertise, operational telemetry, and institutional knowledge.

Coordination layer — Orchestrated AI agents collaborating across workflows to connect understanding with action.

Governance layer — Policy-aware controls ensuring AI-driven actions remain secure, transparent, auditable, and subject to human oversight.

Together, these layers enable AI to extend beyond analysis and participate responsibly in execution. This foundation now enables the next phase of our direction: extending AI beyond knowledge and insight toward coordinated, governed execution across the BMC AMI platform.

AI in production: Trust earned, not claimed

In July 2024, we published a Statement of Direction focused on infusing intelligence directly into BMC AMI product experiences. Our objective was straightforward: reduce complexity, close knowledge gaps, and make it easier for teams to work confidently on the mainframe.

That direction was grounded in three core principles:

• Consistent and contextual intelligence, with a common AI foundation infused across the AMI portfolio
• LLM freedom, allowing organizations to choose the models that align with their enterprise AI strategies
• In-the-moment intelligence, embedding AI directly into workflows to surface answers and insights where work happens

Since then, we have delivered.

BMC AMI Assistant was introduced beginning with GenAI-powered code explanation and expanded across consecutive quarterly releases into a cohesive intelligence layer embedded throughout the BMC AMI portfolio. Using AI as an advisor, developers gained faster understanding of unfamiliar applications, while operators gained explainable insight into system and operational issues. Investigation that once required manual analysis increasingly includes guided insight and next steps directly inside the workflow.

The addition of Knowledge Expert Chat and Knowledge Hub further expanded this intelligence layer. Together, these capabilities combine curated product intelligence, decades of BMC mainframe expertise, and each organization’s own institutional knowledge, making trusted guidance available at the moment of decision. This approach helps less-experienced staff work with greater confidence while allowing senior experts to focus on higher-value work, turning institutional knowledge into a shared operational asset. Watch this short video for a demo of how Knowledge Expert Chat provides answers at the moment of need.

Instead of searching across documentation, runbooks, and tickets, teams can surface answers through natural-language questions using a knowledge expert chat inside their workflow. Early usage shows that this approach not only accelerates troubleshooting and decision-making but also helps users discover capabilities and documentation they did not previously know existed.

Trust in AI is earned through real outcomes. This intelligence foundation now supports the next phase of our direction.

Customers are redefining what AI must do

Through advisory boards, design programs, beta participation, and enterprise AI readiness discussions, a clear message has emerged: customers want AI that does more than explain what happened. They want orchestrated intelligence that helps move work forward.

They want to reduce repetitive analysis, move faster from insight to action, and make it easier for teams to build expertise and focus on higher-value priorities. They want to realize the full promise of BMC AMI as automated mainframe intelligence.

Generative AI laid the foundation by helping teams understand, explain, and accelerate work. Agentic AI builds on that foundation by enabling intelligence to plan, decide, and safely carry work forward.

To meet these expectations at enterprise scale, our new Statement of Direction is grounded in five core principles:

• Establish the core first: Build an orchestrated agentic intelligence foundation.
• Specialized agents: Reflect domain expertise with focused, autonomous actions.
• Human-in-the-loop: Governed, transparent, and observable autonomy.
• Open ecosystem: Open standards-based, composable, and extensible.
• Outcome-driven adoption: Deliver measurable value early and expand over time.

The next phase extends AI beyond insight toward trusted participation in execution, within enterprise guardrails.

And that shift defines our next statement of direction.

From AI assistance to orchestrated intelligence

Building on the foundation of the AI-driven knowledge and guidance we have delivered with BMC AMI Assistant, our next statement of direction is clear: extending that foundation toward proactive, coordinated intelligence that can safely execute actions within defined governance policies across the BMC AMI portfolio.

This evolution introduces coordinated AI agents with specialized, policy-aware capabilities operating across development, operations, data, and security. Each agent is domain-specific, working together within enterprise guardrails to deliver governed, collaborative outcomes.

In this next phase, AI agents for operations, development, data, and security will operate as orchestrated participants across domains—supporting actions such as system and performance diagnostics, development workflows, security validation, and operational recovery. They will learn across past incidents and participate in execution with transparency, verifiability, enterprise governance, and human oversight built in by design.

We will expand beyond explanation and recommendation to enable validated action inside workflows. Agentic workflows will move from detection to investigation to explanation to resolution within a single, governed AI experience.

AI will not replace expertise. Human validation remains essential. Principal engineers and system programmers will continue to define policy, validate outcomes, and shape how intelligence operates. AI scales expertise rather than removing it.

The result is faster problem identification, more contextual decision-making, and reduced operational friction—while ensuring the mainframe participates fully in broader enterprise AI strategies. The mainframe will operate as a connected, policy-aligned participant in the enterprise AI ecosystem across hybrid environments, maintaining the security, reliability, and trust on which organizations depend.

Delivering agentic workflows across the BMC AMI portfolio

Over the coming releases, this direction will take shape through agentic workflows that coordinate knowledge, reasoning, and action across BMC AMI solutions. Initial workflows could include agentic AIOps incident resolution, agentic diagnostics, AI-assisted application insights, development troubleshooting, and test-case generation.

These workflows build on the intelligence layer already established across the BMC AMI portfolio, combining development and operations telemetry, mainframe domain expertise, and organizational knowledge to provide the context required for responsible execution.

From there, coordinated AI agents operate across solutions, aligning development, operations, data, and security activities that were previously siloed. Instead of isolated AI features, intelligence becomes part of the workflow itself, helping teams connect system understanding with the next operational step.

Every action remains governed and transparent. Human validation, enterprise policy, and operational guardrails remain central, ensuring AI participation strengthens reliability rather than introducing risk.

The objective is clear: translate insight into trusted action while preserving the control and discipline enterprise systems require.

Establishing the foundation for governed AI execution

As AI becomes more operational, governance and trust become increasingly essential. Organizations expect AI not only to inform decisions, but to operate safely, predictably, and transparently within enterprise policy boundaries.

Across the industry, fragmented approaches are already emerging: isolated AI integrations, independently managed execution layers, and disconnected tool endpoints. While intended to accelerate innovation, these models often introduce new complexity, inconsistent policy enforcement, and operational sprawl.

Enterprise mainframe environments cannot afford that fragmentation, especially as AI Agent orchestration becomes a core capability for coordinating multi-step agentic workflows. This can result in agents giving conflicting recommendations and teams lose confidence and revert to manual work. Without a unified and governed approach, orchestration itself becomes fragmented, leading to unpredictable behavior, loss of insight, and reduced trust in AI-driven operations.

As part of our next phase, our direction includes establishing an MCP Gateway as a shared, governed access layer across the BMC AMI portfolio. Rather than creating multiple independently governed AI entry points, the MCP Gateway will provide a centralized, secure, and policy-aware interface through which AI agents interact with BMC AMI solutions. Every AI-driven action will be visible, governed, and aligned to enterprise policy with consistent controls across the platform. This enables AI agents not only to access BMC AMI systems, but to safely execute actions across them within defined policy boundaries.

Supporting this architecture, we will introduce an Agent Gateway to facilitate how AI agents communicate and collaborate. Instead of agents interacting independently, interactions will flow through the Agent Gateway—where the agent interactions are visible—ensuring governance, auditability, logging, and policy enforcement across agentic workflows.

Together, the MCP Gateway and Agent Gateway extend the BMC AMI Platform into a governed AI foundation that enables coordinated intelligence and trusted execution across the portfolio (see Figure 1). The BMC AMI Platform serves as the enterprise foundation for this direction—providing a unified layer of core capabilities and services that connects intelligence, orchestration, and governance across the BMC AMI portfolio. It brings together innovations such as BMC AMI Assistant, the Agent Gateway, and the MCP Gateway to simplify mainframe transformation, accelerate innovation, and enable AI to operate consistently and securely at scale.

Industry standards define how AI systems connect to tools and agents. Our direction focuses on how those connections are governed, secured, and operationalized at enterprise scale—without introducing fragmentation or operational risk.

This is how we intend to evolve agentic AI into an enterprise-grade foundation across the BMC AMI portfolio (see Figure 1).

Figure 1: High-level overview of BMC AMI Platform’s agentic architecture supporting enterprise-grade AI.

What you can expect — and how to shape what comes next

Over the coming releases, you will experience a meaningful shift in how AI participates in the BMC AMI environment. Intelligence will extend beyond explanation and guidance to support the safe execution of operational tasks. Capabilities that once appeared as isolated features will increasingly operate as coordinated agentic workflows across the BMC AMI portfolio, connecting development, operations, data, and security activities.

AI will participate responsibly within enterprise guardrails, operating under the policies, validation models, and controls defined by the teams who run these systems. This direction represents more than a single capability release, it establishes the foundation for a new way of working with BMC AMI solutions.

We invite customers to help us shape this next phase. You can engage with us in several ways: join our Customer Design Partner program to help refine and validate the most impactful agentic workflows, participate in early access initiatives focused on execution-oriented capabilities, and work with us to shape the governance and policy models that will guide enterprise-scale AI execution.

Moving forward with confidence

We delivered on our commitments. We established a foundation of trust. And we are now stepping into the next phase with clarity.

The future of AI on the mainframe centers on purpose-built enterprise intelligence, where your choice of AI models, your institutional knowledge and operational practices, your people, and your chosen platforms work together to drive intelligent execution across the enterprise. In this model, the mainframe is not an isolated environment. It is a fully governed participant in enterprise AI strategies, capable of supporting intelligent execution across hybrid systems.

Organizations that embrace this direction will not simply modernize their mainframe environments. They will unlock new ways to operate complex workloads with greater visibility, control, and intelligence.

And this is only the beginning. Before you run AI on your most essential platform, BMC First.

Are you truly ready to move from AI insight to AI-driven execution on the mainframe?
Assess your organization’s AI readiness and maturity level—register for an AI Readiness Discovery Workshop to take the next step.

Most organizations don’t start by thinking about service orchestration. They start with symptoms: a morning dashboard showing incomplete data, an SLA missed because an upstream process ran late, or a cross-team Slack thread at 7 AM trying to figure out what broke and why. 

By the time the word “orchestration” enters the conversation, the workarounds have usually been in place for months. And that’s exactly the problem—those workarounds are invisible because they look like “process.” 

The real distinction isn’t between working and broken systems. It’s between environments that are quietly compensating for a lack of coordination and those built on a foundation that can actually support end-to-end execution. That’s where the difference between scheduling and orchestration begins—and why it matters.

Where Job Scheduling Hits Its Ceiling

Technically there’s nothing wrong with the built-in automation tools that come with modern applications. ERP systems, CRMs, data warehouses, and large database platforms all ship with native scheduling capability. For environments that aren’t particularly complex, those tools do exactly what they’re supposed to do: automate jobs within the boundaries of the application that owns them. 

The problem starts at the boundary. 

When a business outcome requires work that transcends multiple applications – supply chain operations, financial close, ML pipeline execution – there is no common coordination layer to manage it. Each application scheduler knows its own world and nothing else. The result is a set of individually automated workflows that have no shared understanding of each other’s state, progress, or failure. 

This is the core architectural gap that service orchestration addresses. A control plane above existing schedulers manages end-to-end execution with business objectives as the organizing principle. 

What Cross-System Coordination Actually Looks Like in Practice

A customer submits an application through a web portal. From there, the process fans out across a stack of systems that have very little in common: data validation runs through an API service in Kubernetes containers; identity verification calls out to a third-party SaaS provider; risk scoring runs on a model deployed in the cloud; the actual account gets created in a core banking system that lives on-premises; the CRM updates the customer profile; compliance documentation gets generated and archived. 

That’s six distinct environments – cloud, on-premises, containerized infrastructure, SaaS – all participating in a single business transaction. And that’s the simplified version. 

Now add the business constraint that makes this genuinely difficult to manage: the customer expects the whole process to complete within one hour of submitting their application. To be clear, that’s a business commitment – and it applies to the entire chain, not to any individual step. 

No application scheduler was built to coordinate that topology under that deadline. Each one was designed to manage its own job queue, not to understand where it sits in a larger business process or what happens downstream if it falls behind.

How the Problem Reveals Itself Operationally

Organizations don’t usually discover this gap through architectural reviews. They discover it through patterns of operational friction that gradually become normalized. 

The most telling early signal is what might be called the time buffer contract. When data teams and application teams operate without a common coordination layer, they start negotiating informal timing agreements. The data from the application layer usually lands by 4 PM, so the ETL pipelines are configured to kick off at 5, with an hour of buffer built in as insurance. 

That buffer looks like responsible planning, but it masks missing dependency management. It fails in both directions – if the upstream data is late, the downstream pipeline starts anyway and processes whatever is there; if the data arrives early, the pipeline sits idle waiting for a clock to tick. 

The second pattern is more disruptive. A data pipeline ingests records from a CRM and an ERP expecting a complete dataset – a million records, in a typical case. An upstream workflow encountered a problem, so now the data arrives incomplete. The pipeline has no way to know this, so it runs to completion and updates business dashboards with partial data. No one finds out until a business user notices something looks wrong. 

What follows is a cross-team reconstruction effort: Slack messages, conference calls, manual tracing of execution steps across multiple tools to figure out where in the chain the problem originated. This kind of incident response – teams scrambling across disconnected systems to piece together what happened – is one of the clearest indicators that an organization has outgrown its current approach to workflow coordination. 

The absence of process lineage across the full stack – from the business application layer down through the data layer – is what makes these incidents so difficult to resolve. As Basil Faruqui, Sr. Director of Product Marketing at BMC Software, puts it: “You cannot manage what you can’t see.” And you can’t see what no single tool was designed to show you. 

What Orchestration Requires in the Modern Era 

Listing orchestration capabilities as a feature set understates what’s actually required. The architecture needs to be a direct response to the failure modes that siloed scheduling creates. 

The first requirement is broad environmental support. A control plane that only works in cloud environments, or only connects to certain application types, is just another silo with a better UI.

Orchestrating a business outcome end-to-end is complex and needs to span SaaS applications, multi-cloud infrastructure, on-premises data centers, and even mainframe systems in some cases. For many industries like banking, financial services and insurance mainframe support isn’t optional. 

The second requirement is end-to-end visibility with process lineage. Knowing that a job completed isn’t the same as knowing whether the business service it belongs to is on track. Effective visibility means being able to trace the state of a workflow across every system it touches, in real time. 

The third requirement is SLA management that carries business context. A notification that says a process is running late is less useful than a notification that says a supply chain workflow is running late and will affect five downstream business services if it doesn’t recover within the next 20 minutes. The former tells you something is wrong. The latter tells you what to do about it. 

Self-healing is the fourth requirement, and it’s where organizational knowledge becomes operational policy. When a workflow step fails because of a transient network issue, someone investigates and decides to rerun the step. That resolution should be capturable as a policy – and increasingly, AI-enabled capabilities can identify that the same failure pattern has occurred before, surface the historical remediation, and automate the response with human approval where appropriate. The goal is to build institutional knowledge into the platform rather than relying on it to live only in the heads of the people who were on shift when the incident happened. 

Finally, CI/CD integration matters more than it’s often given credit for. Orchestration is, at its core, a team sport. Business users, application teams, and operations teams all have a stake in how workflows are defined, tested, and deployed. Treating orchestration workflows as code – with version control, automated testing, and deployment pipelines consistent with modern DevOps practice – isn’t a nice-to-have. It’s what makes the platform manageable at all scales.

AI Workloads Break the Pipeline Model 

One of the more common misconceptions in this space is that AI and ML workloads can be handled by extending existing data pipeline tools. The assumption is that model training is just a compute-intensive batch job, and inference is just another API call. Both assumptions fall short. 

Consider a customer churn detection model. The training data has to be sourced from systems that aren’t data systems at all – ERP, CRM, custom applications, potentially social media feeds. Getting that data into a staging environment already requires coordination across the application layer. Then the model training pipelines need to run. So far, that’s two layers of coordination before any inference happens. 

Once the model is in production and identifies customers at risk, the business response – sending a promotional offer, for example – doesn’t happen inside the model. It happens in the CRM and ERP layer where customer profiles and promotional workflows live. The model is one step in a business process that starts with applications and ends with a business action. 

Confining AI workloads to a data pipeline tool means the platform can manage the middle of that chain but not the ends. The data sourcing and the downstream business response are both outside its scope. Siloing the AI workload in a single tool leaves you with a system that can’t map or control the process from source to business action. 

This matters even more as agent-based architectures become more common. Organizations are already deploying agents that take on discrete tasks within larger processes. A workflow like order-to-cash will not be handed over to agents in its entirety anytime soon – but agents are increasingly handling specific components of it. That means an orchestration platform needs to be able to invoke agents, pass them the right context, enforce guardrails, and manage their execution according to the same SLAs that govern the rest of the workflow. For organizations with active agent deployments, agent orchestration is already a present consideration. 

Rethinking the Question

When organizations run into workflow coordination problems, the instinct is to ask whether they need better automation. More often, what they actually need is orchestration. 

Better automation improves individual jobs. Orchestration manages the relationships between them. For workflows that stay within the boundaries of a single application, better automation is probably the right answer. For workflows that span systems, teams, and environments to deliver a business outcome with a defined deadline, automation alone can’t provide what’s needed. 

The symptoms are identifiable. Time buffers masquerading as process discipline. Partial data incidents that require multi-team investigation to trace. SLA breaches that can’t be attributed to a specific failure point because no single tool has visibility across the full chain. ​​None of these are signs that automation needs tuning. They point to a missing control plane. 

The distinction matters because building more automation on top of a coordination gap doesn’t close the gap. It adds more things to coordinate.

A data pipeline is a series of automated steps for moving data from one or more sources to a designated destination, often transforming it along the way. Raw, disparate pieces of data enter one end, undergo processes like cleaning, restructuring, and enrichment, and emerge at the other end as usable insights. Data pipelines are the foundation of every analytics dashboard, machine learning model, and data-driven operational decision. 

You could think of a data pipeline like an airport baggage system: bags (data) enter the conveyor system, get scanned (validation), sorted (transformation), and routed to the correct flight (destination database). If one belt jams, everything backs up—just like a bottleneck in a data pipeline.

What’s the difference between a data pipeline and ETL?

A data pipeline and an ETL pipeline are not the same thing, though the terms are often used interchangeably.

A data pipeline is an umbrella term for any set of processes that move data from one system to another. This includes simple data ingestion, real-time streaming, batch processing, and complex multi-step workflows. 

An ETL pipeline (Extract, Transform, Load) is a specific type of data pipeline. Its purpose is to extract data from sources, transform it into the right format, and load it into a destination system like a data warehouse or database. All ETL pipelines are data pipelines, but not all data pipelines are ETL pipelines. A pipeline that moves raw data without transforming it, or streams data in real time, is still a data pipeline—but not an ETL pipeline. 

Is SQL a data pipeline?

No. SQL (Structured Query Language) is a language used to query, manage, and manipulate data in relational databases—it’s a tool, not a workflow. 

A data pipeline is the full automated process that moves data from one place to another. SQL can be used inside a data pipeline to filter, join, or transform data, but SQL doesn’t constitute the pipeline itself. If you’re building a house, SQL is the hammer and saw—essential tools, but not the entire construction project. 

Why do data pipelines matter?

Data pipelines are essential because organizations collect data from too many sources—customer interactions, social media, sales transactions, website logs, IoT devices, and internal applications—to manage manually. Without a systematic way to collect, process, and deliver this data, it quickly becomes unmanageable rather than useful. 

Data pipelines power the data-driven decisions we encounter every day: personalized recommendations, real-time fraud alerts, and analytics dashboards. For DataOps teams specifically, pipelines help ensure the reliability, scalability, and governance that organizations need to use data confidently instead of being overwhelmed by it. 

What are the benefits of a well-executed data pipeline?

Data pipelines don’t just move data—they make it fit for purpose, delivering it where and when it’s needed. Five benefits go beyond the basic transport function: 

Enabling analytics and business intelligence: Pipelines feed cleaned, structured data into data warehouses and analytical platforms, allowing analysts to discover trends, identify opportunities, and monitor performance. 

Fueling machine learning and AI: AI models require large volumes of high-quality, pretreated data. Data pipelines help ensure models get the data they need to learn and make accurate predictions. 

Ensuring data quality and governance: As data gets cleaned, validated, and standardized, data pipelines support greater confidence in data-driven decisions. They also enforce governance rules for compliance and security. 

Improving operational efficiency: By integrating data from various systems, pipelines provide a holistic view of operations, automating workflows and flagging issues in real time. 

Facilitating data democratization: Pipelines make data accessible and understandable to more people within an organization, empowering more teams to make informed decisions by connecting data sources to decision-makers. 

Without strong data pipelines, organizations can fly blind—making decisions based on intuition rather than evidence. 

What are the core components of a data pipeline?

Every data pipeline is made up of five essential components. Understanding these elements reveals how data flows and transforms from its original source to its final destination. 

1. Source: Where your data lives

Source is the origin point of your data—the starting line of the pipeline. The type of source determines how the data will be extracted. Common data sources include: 

• Databases: relational (e.g., MySQL) and NoSQL (e.g., MongoDB) 
• Applications: CRM systems (e.g., Salesforce), ERPs (e.g., SAP), marketing automation platforms 
• APIs: third-party services, social media platforms, public data feeds 
• Files: CSVs, JSON, XML, Parquet, Avro—often stored in cloud storage (e.g., Azure Blob) 
• Streaming data: real-time event streams from IoT devices, website clicks, financial transactions 
• Logs: system logs, web server logs, application logs 

2. Extraction: Getting your data out

Extraction is the step where data is pulled from its original source—often involving different file types, formats, and sometimes unstable or slow source connections. The goal of extraction is to get a raw copy of the data without altering the source system.

Three common extraction methods: 

• Batch extraction: Data is pulled in chunks at scheduled intervals (e.g., nightly, hourly). Used for data that doesn’t change frequently or where immediate updates aren’t critical. 
• Incremental extraction: Only new or changed data since the last pipeline run is extracted. Faster than full extraction, but requires change-detection techniques like timestamps, version numbers, or Change Data Capture (CDC). 
• Streaming extraction: Data is continuously pulled from sources as events occur, typically using message queues or event streaming platforms like Kafka or Kinesis. 

3. Transformation: Cleaning and shaping your data 

Transformation is usually the most complex part of a data pipeline. This is where the messiness of raw data gets cleaned and turned into actionable information. The goal is to ensure data quality, consistency, and suitability for the intended destination. Common transformation steps include:

• Cleaning: Removing duplicates, handling missing values, correcting errors 
• Filtering: Selecting only relevant rows or columns 
• Aggregating: Summarizing or categorizing data (e.g., total sales per day) 
• Joining or merging: Combining data from multiple sources using common keys (e.g., joining customer data with order data) 
• Standardizing or normalizing: Ensuring consistent data types, formats, and units (e.g., standardizing currency codes) 
• Enriching: Adding new data points by looking up external information or deriving new features (e.g., adding geographical data based on an IP address) 
• Structuring: Converting unstructured or semi-structured data into a structured format

4. Loading: Delivering your data

Once extracted and transformed, data is loaded into the system where it will be used—a database, data warehouse, or analytics platform—so applications, reports, and analytics tools can access it. Three common loading strategies:

• Full load: The entire destination table or dataset is overwritten with new, transformed data. Simpler to implement, but resource-intensive for large datasets. 
• Incremental load: Only new or changed records are appended to the destination. More efficient, but requires diligent management of data updates and deletions. 
• Streaming load: Data is continuously loaded as it arrives, often into specialized real-time databases or analytical engines. 

5. Destination: Where your data rests

Destination is the final storage location where processed data is available for consumption by analysts, data scientists, and applications. Common destinations include: 

• Data warehouses: Optimized for complex queries and reporting on large volumes of historical data (e.g., Snowflake) 
• Data lakes: Hold raw or semi-structured data at scale for advanced analytics and machine learning (e.g., Azure Data Lake Storage) 
• Databases: Operational systems for everyday applications like websites or apps (e.g., MongoDB) 
• Business intelligence (BI) tools: Software that turns data into dashboards and reports (e.g., Tableau) 
• File storage: Simple storage for archiving or later processing 

Note: loading and destination are conceptually distinct but closely related in practice. Loading is the action of writing processed data into a system; destination is the place where that data ends up and stays until it’s needed. Loading is like putting groceries into the fridge. Destination is the actual fridge. 

Bringing it all together—an e-commerce scenario: Source (CRM database) → Extract (SQL query for new customer orders) → Transform (clean addresses, calculate total order value, join with product details) → Load (insert into data warehouse) → Destination (data warehouse for reporting). The entire process runs automatically on a schedule, ensuring a continuous flow of refined information. 

What are the three types of data pipelines?

Just as there are different ways to transport goods, there are different types of data pipelines—each optimized for specific needs around speed, volume, and complexity. 

Batch processing: the daily shuttle

Batch processing works like a commuter train on a defined schedule. It picks up a large group of passengers (data) at scheduled times and delivers them to their destination. Data is collected over a period, then processed as a single, large batch.

• Characteristics: High latency (data may be hours or days old); processes large volumes efficiently; often scheduled during off-peak hours 
• Use cases: Nightly reports, monthly financial summaries, loading historical data into a data warehouse, running complex analytical jobs that don’t require immediate results 

Real-time streaming: the instant delivery service

Real-time streaming works like an instant delivery service. As soon as a package (data event) is created, it’s picked up, processed almost immediately, and delivered to its destination with minimal delay.

• Characteristics: Low latency (data is typically seconds or milliseconds old); handles continuous streams of individual events; requires infrastructure optimized for speed 
• Use cases: Fraud detection, real-time personalized recommendations, IoT sensor data analysis, monitoring system health, live dashboards

Hybrid approaches: the best of both worlds

Many organizations combine batch and streaming pipelines. Two common hybrid patterns:

• Lambda architecture: Uses separate batch and streaming layers. The streaming layer provides real-time views; the batch layer processes historical data for accuracy and completeness. Results from both are then merged. 
• Kappa architecture: A simpler approach that handles both real-time and historical processing using a single streaming engine, often by replaying streams.

Choosing the right pipeline type depends entirely on your business requirements for data freshness, volume, and complexity. 

What are the main challenges in data pipeline management?

Building a data pipeline is one thing; keeping it running smoothly and reliably is another. Here are the most common data pipeline management challenges—and the best practices for each. 

Ensuring data quality: Data can be inconsistent, incomplete, or incorrect at the source, leading to garbage in, garbage out. Best practices: implement data validation rules at every stage; use data profiling tools to understand data characteristics; create data quality checks within transformation steps; use data observability platforms to detect anomalies early. 

Scalability and performance: As data volumes grow or requirements shift to real-time, pipelines can become slow or break entirely. Best practices: design for scalability from the outset; use distributed processing frameworks (e.g., Apache Spark); use cloud-native services that scale automatically; implement incremental loading strategies; optimize queries and transformation logic. 

Security and compliance: Data pipelines handle sensitive information, requiring stringent security and compliance measures. Best practices: encrypt data at rest and in transit; implement strong access controls (least privilege); audit data access and changes; redact sensitive data during transformation where necessary. 

Monitoring and alerting: Without proper monitoring, pipeline failures or data quality issues can go undetected and impact downstream applications. Best practices: implement comprehensive monitoring for pipeline health, performance metrics, and data quality metrics; set up automated alerts for critical failures, latency breaches, or data anomalies; use dashboards for operational visibility.

What are the top data pipeline use cases?

Data pipelines are versatile, powering applications across industries. Seven key examples of how pipelines transform raw data into actionable insights:

• Business intelligence and reporting: Aggregating sales data, customer demographics, and marketing spend into a data warehouse for daily, weekly, or monthly reports and dashboards that guide strategic decisions 
• Customer 360-degree view: Combining data from CRM, sales, support, and marketing platforms to create a holistic profile of each customer, enabling personalized experiences and targeted campaigns 
• Fraud detection: Ingesting real-time financial transactions, social media activity, and user behavior to identify suspicious patterns and instantly flag potential fraud 
• IoT analytics: Collecting streams of data from sensors (e.g., factory machines, smart city devices) to monitor performance, predict maintenance needs, and optimize operations 
• Personalized recommendations: Processing user browsing history, purchase data, and demographic information to power content recommendations on streaming platforms 
• Log analytics: Consolidating logs from applications and servers to monitor system health, troubleshoot issues, and detect security threats in real time 
• ML model training: Preparing, cleaning, and feeding large datasets to machine learning models for tasks like image recognition, natural language processing, or predictive analytics

What makes a data pipeline modern?

Modern data pipelines share several key characteristics that distinguish them from traditional approaches:

• Cloud-native and serverless: Modern pipelines use cloud services (e.g., AWS, Azure) that scale automatically and reduce operational overhead 
• ELT-first approach: Instead of transforming data before loading, modern pipelines often load raw data into a cloud data warehouse (e.g., Snowflake, BigQuery) and transform it within the warehouse using SQL—leveraging the destination’s compute power and enabling greater flexibility 
• Data lake integration: Modern pipelines frequently integrate with data lakes to store vast amounts of raw, multi-structured data for future use, advanced analytics, and machine learning 
• Real-time capabilities: Streaming technologies like Apache Kafka and Amazon Kinesis increasingly power modern pipelines for immediate insights 
• Orchestration and automation: Tools like Apache Airflow or cloud-native orchestrators automate scheduling, manage dependencies, and monitor pipeline health 
• Data observability: Modern pipelines go beyond basic monitoring to actively track the health, quality, and lineage of data—detecting anomalies and ensuring data trustworthiness 
• Data governance and security by design: Security, privacy, and compliance are built into the pipeline architecture from the start, not added as an afterthought 
• Flexibility and agility: Modular components make modern pipelines easier to adapt to new data sources and changing business requirements 

How do traditional and modern data pipelines compare? 

Modern data pipelines are designed to be more agile, scalable, cost-effective, and resilient than traditional approaches. Here’s a side-by-side comparison:

What are the most common data pipeline tools?

The data pipeline tool landscape is broad and evolving. Four categories to consider: 

Data integration platforms: Comprehensive ETL/ELT tools, often with visual interfaces and pre-built connectors. Examples: Talend, Informatica. A good fit for teams that want an end-to-end solution without heavy coding; businesses with multiple data sources needing pre-built connectors; and organizations prioritizing ease of use and quick deployment. 

Cloud-native services: Major cloud providers offer services specifically designed for scalable data pipelines. Examples: Amazon Kinesis, Google BigQuery. A good fit for companies already invested in a specific cloud ecosystem; teams needing scalable, cost-effective solutions for batch and streaming; and use cases requiring tight integration with other cloud services. 

Open-source frameworks: Flexible, developer-friendly options for orchestration and processing. Examples: Apache Airflow, Apache Kafka. A good fit for engineering teams with strong technical skills; organizations wanting maximum flexibility and control; and scenarios with custom requirements or large-scale data processing needs. 

Enterprise workflow orchestration platforms like Control-M are ideal for teams that need to orchestrate BigQuery data pipelines alongside mainframe, cloud, and on-premises workloads — providing SLA management and audit capabilities that native GCP scheduling does not offer. A good fit for large enterprises with complex, mission-critical workflows; businesses needing robust scheduling, compliance, and audit capabilities; and teams managing cross-platform jobs spanning mainframe, cloud, and on-premises systems with high reliability requirements. 

For large enterprises with mission-critical data pipelines, a data pipeline orchestration platform like Control-M provides robust scheduling, cross-platform dependency management, and compliance capabilities that open-source tools alone cannot deliver at scale.

The right tool depends on your budget, team skill set, data volumes, and real-time requirements. 

5 key data pipeline takeaways

Whether you’re learning about data pipelines for the first time or refreshing on the fundamentals, here are the most important points to carry forward: 

1. Understand the pipeline lifecycle

A data pipeline isn’t just about moving data—it involves extraction, transformation, loading, orchestration, monitoring, and governance.

2. Orchestration is key

Orchestration ensures repeatability, scalability, and observability across the entire data pipeline. 

3. Embrace automation and CI/CD

Integrating data pipelines into CI/CD workflows enables faster, safer changes. DataOps applies DevOps principles to data: automated testing, deployment, and version control for pipelines.

4. Prioritize data quality and monitoring

Pipelines can fail silently if data quality isn’t checked. Implement validation, anomaly detection, and alerts to catch issues early. Observability is critical for trust and compliance.

5. Design for scalability and flexibility

Modern data pipelines must handle batch and streaming, adapt to schema changes, and scale with data growth. Cloud-native and modular architectures are essential for agility. 

Bonus tip: Learn the full ecosystem—ETL tools, orchestration frameworks, cloud services—and how they fit together to build pipelines that serve your organization at scale. 

Frequently asked questions about data pipelines

How is a data pipeline different from a data warehouse? 

A data pipeline is the process that moves and transforms data; a data warehouse is a destination where processed data is stored for analysis. Data pipelines feed data warehouses—the pipeline handles transport and preparation, while the warehouse provides the storage and query environment. The two are complementary, not interchangeable. 

What skills do you need to build a data pipeline? 

Building a data pipeline typically requires proficiency in SQL and at least one programming language (Python and Scala are common), familiarity with data transformation concepts (ETL/ELT), and working knowledge of at least one orchestration or workflow tool. Cloud platform experience (AWS, Azure, or GCP) is increasingly essential for modern data pipeline work. 

How do you test a data pipeline? 

Data pipeline testing involves validating data at multiple stages: confirming that extraction produces complete, unaltered source data; verifying that transformation logic produces expected outputs; and checking that loaded data matches expected row counts, data types, and values at the destination. Automated testing frameworks integrated into CI/CD pipelines can run these checks continuously and catch regressions early. 

What is data pipeline latency? 

Data pipeline latency is the time between a data event occurring at the source and that data becoming available at the destination. Batch pipelines have high latency (hours or days); real-time streaming pipelines have low latency (seconds or milliseconds). Acceptable latency depends on the use case—fraud detection requires near-zero latency, while monthly financial reporting can tolerate daily batch processing. 

How does BMC Control-M help manage data pipelines? 

BMC Control-M is an enterprise workflow orchestration platform that automates, schedules, and monitors complex data pipelines across environments—including mainframe, cloud, and on-premises systems. Control-M provides centralized visibility, dependency management, and audit capabilities, making it well-suited for mission-critical pipelines that require high reliability, compliance, and cross-platform coordination. 

The views and opinions expressed in this post are those of the author and do not necessarily reflect the official position of BMC. 

Control-M from BMC enables teams to orchestrate data pipelines on Google Cloud by defining, scheduling, and monitoring workflows across services like Cloud Storage, Dataflow, and BigQuery—all from a single interface. This article walks through the five key orchestration challenges, the Google Cloud services involved, and a real-world credit-card fraud detection example that puts Control-M into action. 

The Google Cloud Platform is designed specifically to accommodate organizations in a variety of positions along their cloud services journey, from large-scale machine learning (ML) and data analysis to services tailored to SMBs to hybrid-cloud solutions for customers that want to use services from more than one cloud provider. When BMC was migrating our Control-M application to this cloud ecosystem, we had to be very thoughtful about how we managed this change. The SADA engineering team worked alongside the BMC team to ensure that we had a seamless integration for our customers. 

SADA supported this project by providing an inventory of the Google Cloud configuration options, decisions, and recommendations to enable the data platform foundation deployment, collaborated with BMC on the implementation planning, provided automation templates, and designed the Google Cloud architecture for the relevant managed services on the Google Cloud Platform.

In this article, we will discuss the end result of this work, and look at an example using a credit-card fraud detection process to show how you can use Control-M to orchestrate a data pipeline seamlessly in Google Cloud.

What are the five challenges of orchestrating an ML data pipeline?

Orchestrating a machine learning data pipeline on Google Cloud involves five primary challenges that teams must address before workflows can run reliably at scale.

Understand the workflow

Examine all dependencies and any decision trees. For example, if data ingestion is successful, proceed down this path; if it is not successful, proceed down that path.

Understand the teams

If multiple teams are involved in the workflow, each needs to have a way to define their workflow using a standard interface, and to be able to merge their workflows to make up the pipeline.

Follow standards

Teams should use repeatable standards and conventions when building workflows. This avoids having multiple jobs with identical names. Each step should also have a meaningful description to help clarify its purpose in the event of a failure.

Minimize the number of tools required

Use a single tool for visualization and interaction with the pipeline (and dependencies). Visualization is important during the definition stage since it’s hard to manage something that you can’t see. This is even more important when the pipeline is running.

Include built-in error handling capabilities

It’s important to understand how errors can impact downstream jobs in the workflow or the business service level agreement (SLA). Failure of a job should not halt the pipeline altogether and require human interaction. Criteria can be used to determine if a failed job can be restarted automatically or whether a human must be contacted to evaluate the failure—if, for instance, there are a certain number of failures involving the same error.

How did BMC and SADA meet these orchestration challenges?

Addressing these challenges required a solid foundation and presented opportunities for collaboration. BMC and SADA aligned using the SADA POWER line of services to establish the data platform foundation. Some notable elements in this technical alignment included work by SADA to:

• Apply industry expertise to expedite BMC’s development efforts. 
• Establish a best practices baseline around data pipelines and the tools to orchestrate them. 
• Conduct collaborative sessions to understand BMC’s technical needs and provide solutions that the BMC team could integrate and then expand upon. 

SADA’s Data Platform Foundation provided opportunities to leverage Google Cloud services to accomplish the complex analytics required of an application like Control-M. The BMC and SADA teams worked together to establish a strong foundation through:

• Selecting data and storage locations in Google Cloud Storage. 
• Utilizing the advantages provided by Pub/Sub to streamline the analytics and data integration pipelines. 
• Having thorough discussions around the extract, transform, and load (ETL) processes to truly understand the end state of the data. 
• Using BigQuery and writing analytic queries. 
• Understanding the importance of automation, replicability of processes, and monitoring performance in establishing a system that is scalable and flexible. 
• Using Data Studio to create a visualization dashboard to provide the necessary business insights. 

How does fraud detection illustrate data pipeline orchestration on Google Cloud?

Credit-card fraud detection is a practical, real-world example of how Control-M can orchestrate a complex ML data pipeline on Google Cloud—combining real-time and batch processes across multiple services.

Digital transactions have been increasing steadily for years, and the accelerating adoption of digital payments by businesses and consumers has brought with it increased fraud and operational risks. With fraudsters improving their techniques, companies are relying on ML to build resilient and efficient fraud detection systems. Since fraud constantly evolves, detection systems must be able to identify new types of fraud by detecting anomalies that are seen for the first time—making fraud detection a perpetual task that requires constant diligence and innovation.

Common types of financial fraud that customers work to prevent with this application include:

• Stolen/fake credit card fraud: Transactions made using fake cards, or cards belonging to someone else.
• ATM fraud: Cash withdrawals using someone else’s card.

Fraud detection is composed of both real-time and batch processes. The real-time process is responsible for denying a transaction and possibly placing a hold on an account or credit card, thus preventing the fraud from occurring. It must respond quickly, sometimes at the cost of reduced accuracy.

To minimize false positives, which may upset or inconvenience customers, a batch phase is used to continuously fine-tune the detection model. After transactions are confirmed as valid or fraudulent, all recent events are input to the batch process on a regular cadence. This batch process then updates the training and scoring of the real-time model to keep real-time detection operating at peak accuracy. This batch process is the focus of this article.

How can you try the demo system?

SADA and BMC created a demonstration version of this solution so you can experiment with it on Google Cloud. You can find all of the code, plus sample data, in GitHub.

Resources included are:

• Kaggle datasets of transaction data, fraud status, and demographics
• Queries
• Schema
• User-defined functions (UDFs)

How does the pipeline work?

For each region in which the organization operates, transaction data is collected daily. Details collected include (but are not limited to):

• Transaction details: Describes each transaction, including the amount, item code, location, method of payment, and so on.
• Personal details: Describes the name, address, age, and other details about the purchaser.

This information is pulled from corporate data based on credit card information and real-time fraud detection that identifies which transactions were flagged as fraudulent.

New data arrives either as batch feeds or is dropped into Cloud Storage by Pub/Sub. This new data is then loaded into BigQuery by Dataflow jobs. Normalization and some data enrichment is performed by UDFs during the load process.

Once all the data preparation is complete, analytics are run against the combined new and historical data to test and rank fraud detection performance. The results are displayed in Data Studio dashboards.

Figure 1: Control-M orchestration

Which Google Cloud services power the pipeline?

Control-M orchestrates a coordinated set of Google Cloud services—Cloud Storage, Dataflow, BigQuery, and Data Studio—each handling a distinct stage of the data pipeline.

Cloud Storage provides a common landing zone for all incoming data and a consistent input for downstream processing. Dataflow is Google Cloud’s primary data integration tool.

SADA and BMC selected BigQuery for data processing. Earlier versions of this application used Hadoop, but while working with the team at SADA, we converted to BigQuery as this is the recommended strategy from Google for sophisticated data warehouse or data lake applications. This choice also simplified setup by providing out-of-the-box integration with Cloud Dataflow. UDFs provide a simple mechanism for manipulating data during the load process.

What are the two ways to define pipeline workflows in Control-M?

Control-M supports two approaches for defining pipeline workflows, giving teams flexibility to work visually or programmatically.

Using a graphical editor

This provides the option of dragging and dropping the workflow steps into a workspace and connecting them.

Use RESTful APIs

Define the workflows using a jobs-as-code method, then use JSON to integrate into a continuous integration/continuous delivery (CI/CD) toolchain. This method improves workflow management by flowing jobs through a pipeline of automated building, testing, and release. Google Cloud provides a number of developer tools for CI/CD, including Cloud Build and Cloud Deploy.

How are jobs defined in Control-M?

The basic Control-M execution unit is referred to as a job. There are a number of attributes for every job, defined in JSON:

Job type

Options include script, command, file transfer, Dataflow, or BigQuery.

Run location

For instance, which host is running the job?

Identity

For example, is the job being “run as…” or run using a connection profile?

Schedule

Determines when to run the job and identifies relevant scheduling criteria.

Dependencies

This could be things like whether the job must finish by a certain time or output must arrive by a certain time or date.

Jobs are stored in folders and the attributes discussed above, along with any other instructions, are applied to all jobs in that folder.

Below is an example of the JSON code that describes the workflow used in the fraud detection model ranking application. You can find the full JSON code in the Control-M Automation API Community Solutions GitHub repo. While there, you can also find solutions, the Control-M Automation API guide, and other code samples.

{
"Defaults" : {
},
"jog-mc-gcp-fraud-detection": {"Type": "Folder",
"Comment" : "Update fraud history, run, train and score models",
"jog-gcs-download" : {"Type" : "Job:FileTransfer",…},
"jog-dflow-gcs-to-bq-fraud": {"Type": "Job:Google DataFlow",…},
"jog-dflow-gcs-to-bq-transactions": {"Type": “Job:Google DataFlow",…},
"jog-dflow-gcs-to-bq-personal": {"Type": "Job:Google DataFlow",…},
"jog-mc-bq-query": {"Type": "Job:Database:EmbeddedQuery", …},
"jog-mc-fm-service": {"Type": "Job:SLAManagement",…},
},
"flow00": {"Type":"Flow", "Sequence":[
"jog-gcs-download",
"jog-dflow-gcs-to-bq-fraud",
"jog-mc-bq-query",
"jog-mc-fm-service"]},
"flow01": {"Type":"Flow", "Sequence":[
"jog-gcs-download",
"jog-dflow-gcs-to-bq-transactions",
"jog-mc-bq-query", "jog-mc-fm-service"]},
"flow02": {"Type":"Flow", "Sequence":[
"jog-gcs-download",
"jog-dflow-gcs-to-bq-personal",
"jog-mc-bq-query",
"jog-mc-fm-service"]}

}
}

The jobs shown in this workflow correspond directly with the steps illustrated previously in Figure 1.

The workflow contains three fundamental sections:

Defaults. These are the functions that apply to the workflow. This could include details such as who to contact for job failures or standards for job naming or structure.

{  "Defaults" : {"RunAs" : "ctmagent", "OrderMethod": "Manual", "Application" : 
       "multicloud", "SubApplication" : "jog-mc-fraud-modeling", 
      "Job" : {"SemQR": { "Type": "Resource:Semaphore", Quantity": "1"},
      "actionIfError" : {"Type": "If", "CompletionStatus":"NOTOK", "mailTeam": 
          {"Type": "Mail", "Message": "Job %%JOBNAME failed", "Subject": 
                 "Error occurred", "To": deng_support@bmc.com}}}
    }, 

Job definitions. This is where individual jobs are specified and listed. See below for descriptions of each job in the flow.

Flow statements. These define the relationships of the job, both upstream and downstream.

"flow00": {"Type":"Flow", "Sequence":["jog-gcs-download", 
           "jog-dflow-gcs-to-bq-fraud", "jog-mc-bq-query", 
           "jog-mc-fm-service"]},
"flow01": {"Type":"Flow", "Sequence":["jog-gcs-download", 
           "jog-dflow-gcs-to-bq-transactions", 
           "jog-mc-bq-query", "jog-mc-fm-service"]},
"flow02": {"Type":"Flow", "Sequence":["jog-gcs-download", 
           "jog-dflow-gcs-to-bq-personal", "jog-mc-bq-query", 
           "jog-mc-fm-service"]} 

How does Control-M schedule pipeline workflows?

Control-M uses a server-and-agent model for scheduling. The server is the central engine that manages workflow scheduling and submission to agents, which are lightweight workers. In the demo described in this article, the Control-M server and agent are both running on Google Compute Engine VM instances.

Workflows are most commonly launched in response to various events such as data arrival but may also be executed automatically based on a predefined schedule. Schedules are very flexible and can refer to business calendars; specify different days of the week, month, or quarter; define cyclic execution, which runs workflows intermittently or every “n” hours or minutes; and so on.

How does Control-M process the data?

Control-M processes data through a sequence of job types—File Transfer, Dataflow, and SLA Management—each mapped to a distinct stage of the pipeline.

File Transfer job type

The first job, called jog-gcs-download, is of type Job:FileTransfer. This job transfers files from a conventional file system described by ConnectionProfileSrc to Google Cloud Storage described by ConnectionProfileDest.

The File Transfer job type can watch for data-related events (file watching) as a prerequisite for data transfer, as well as perform pre/post actions such as deletion of the source after a successful transfer, renaming, source and destination comparison, and restart from the point of failure in the event of an interruption. In the example, this job moves several files from a Linux® host and drops them into Google Cloud Storage buckets.

"jog-gcs-download" : {"Type" : "Job:FileTransfer",
        "Host" : "ftpagents",
        "ConnectionProfileSrc" : "smprodMFT",
        "ConnectionProfileDest" : "joggcp",
        "S3BucketName" : "prj1968-bmc-data-platform-foundation",
        "Description" : "First data ingest that triggers downstream applications",
        "FileTransfers" : [
          {
            "TransferType" : "Binary",
            "TransferOption" : "SrcToDestFileWatcher",
            "Src" : "/bmc_personal_details.csv",
            "Dest" : "/bmc_personal_details.csv"
          },
          {
            "TransferType" : "Binary",
            "TransferOption" : "SrcToDestFileWatcher",
            "Src" : "/bmc_fraud_details.csv",
            "Dest" : "/bmc_fraud_details.csv"
          },
          {
            "TransferType" : "Binary",
            "TransferOption" : "SrcToDestFileWatcher",
            "Src" : "/bmc_transaction_details.csv",
            "Dest" : "/bmc_transaction_details.csv"
          } 
        ]
      }, 

Dataflow

Dataflow jobs are executed to push the newly arrived data into BigQuery. The jobs appear complex, but Google Cloud provides an easy-to-use process to make the definitions simple.

Go to the Dataflow Jobs page (Figure 2). If you have an existing job, choose to Clone it or Create Job from Template. Once you’ve provided the desired parameters, click on Equivalent REST at the bottom to get this information (Figure 3), which you can cut and paste directly into the job’s Parameters section.

Figure 2: Dataflow Jobs page

Figure 3: Cut and paste into job Parameters section

"jog-dflow-gcs-to-bq-fraud": {"Type": "Job:ApplicationIntegrator:AI Google DataFlow",
        "AI-Location": "us-central1",
        "AI-Parameters (JSON Format)": "{"jobName": "jog-dflow-gcs-to-bq-fraud",
        "environment": {        "bypassTempDirValidation": false,
        "tempLocation": "gs://prj1968-bmc-data-platform-foundation/bmc_fraud_details/temp",
        "ipConfiguration": "WORKER_IP_UNSPECIFIED",
        "additionalExperiments": []    },    
        "parameters": {
        "javascriptTextTransformGcsPath": "gs://prj1968-bmc-data-platform-foundation/bmc_fraud_details/bmc_fraud_details_transform.js", 
        "JSONPath": "gs://prj1968-bmc-data-platform-foundation/bmc_fraud_details/bmc_fraud_details_schema.json",
        "javascriptTextTransformFunctionName": "transform",
        "outputTable": "sso-gcp-dba-ctm4-pub-cc10274:bmc_dataplatform_foundation.bmc_fraud_details_V2",
        "inputFilePattern": "gs://prj1968-bmc-data-platform-foundation/bmc_fraud_details/bmc_fraud_details.csv", 
        "bigQueryLoadingTemporaryDirectory": "gs://prj1968-bmc-data-platform-foundation/bmc_fraud_details/tmpbq"    }}",
        "AI-Log Level": "INFO",
        "AI-Template Location (gs://)": "gs://dataflow-templates-us-central1/latest/GCS_Text_to_BigQuery",
        "AI-Project ID": "sso-gcp-dba-ctm4-pub-cc10274",
        "AI-Template Type": "Classic Template",
        "ConnectionProfile": "JOG-DFLOW-MIDENTITY",
        "Host": "gcpagents"
      }, 

SLA management

This job defines the SLA completion criteria and instructs Control-M to monitor the entire workflow as a single business entity.

"jog-mc-fm-service": {"Type": "Job:SLAManagement",
	 "ServiceName": "Model testing and scoring for fraud detection",
	 "ServicePriority": "3",
	 "JobRunsDeviationsTolerance": "3",
	 "CompleteIn": {
	    "Time": "20:00"
	  }
	},

The ServiceName specifies a business-relevant name that will appear in notifications or service incidents, as well as in displays for non-technical users, to make it clear which business service may be impacted. Importantly, Control-M uses statistics collected from previous executions to automatically compute the expected completion so that any deviation can be detected and reported at the earliest possible moment. This gives monitoring teams the maximum opportunity to course-correct before any impact to business services is detected.

How do you monitor the pipeline state in Control-M?

Control-M provides a real-time monitoring interface that shows the status of every job in the pipeline, making it straightforward to identify failures and take action without switching between tools.

Control-M provides a user interface for monitoring workflows (Figure 4). In the screenshot below, the first job completed successfully and is green, the next three jobs are executing and depicted in yellow. Jobs that are waiting to run are shown in gray.

Figure 4: Control-M Monitoring Domain

You can access the output and logs of every job from the pane on the right-hand side. This capability is vital during daily operations. To monitor those operations more easily, Control-M provides a single pane to view the output of jobs running on disparate systems without having to connect to each application’s console.

Control-M also allows you to perform several actions on the jobs in the pipeline, such as hold, rerun, and kill. You sometimes need to perform these actions when troubleshooting a failure or skipping a job, for example.

All of the functions discussed here are also available from a REST-based API or a CLI.

Conclusion

Coordinating and monitoring workflows across an ML pipeline remains a complex task, even with the rich set of ML tools that Google Cloud provides. Anytime you need to orchestrate a data pipeline on Google Cloud that combines file transfers, applications, data sources, or infrastructure, Control-M can simplify your workflow orchestration. Control-M integrates, automates, and orchestrates application workflows whether on-premises, on Google Cloud, or in a hybrid environment.

Frequently asked questions 

What is the best tool to orchestrate a data pipeline on Google Cloud?

Control-M from BMC is purpose-built for orchestrating data pipelines across cloud and hybrid environments. On Google Cloud, Control-M integrates natively with Cloud Storage, Dataflow, BigQuery, and Pub/Sub, enabling teams to define, schedule, and monitor workflows from a single interface using either a graphical editor or RESTful APIs. 

What is the difference between Dataflow and Control-M?

Google Cloud Dataflow is a managed data integration service that moves and transforms data between sources and destinations such as BigQuery. Control-M is a workflow orchestration engine that coordinates and monitors the execution of multiple tools—including Dataflow jobs—as part of a broader end-to-end pipeline. Dataflow handles the data movement; Control-M manages the sequencing, scheduling, and error handling of the entire workflow. 

Can Control-M run jobs as code on Google Cloud?

Yes. Control-M supports a jobs-as-code approach using JSON and RESTful APIs, which can be integrated into a CI/CD toolchain. Google Cloud developer tools including Cloud Build and Cloud Deploy are compatible with this method. 

What Google Cloud services are used in a Control-M data pipeline?

A typical Control-M-orchestrated pipeline on Google Cloud uses Cloud Storage as the data landing zone, Pub/Sub for streaming data ingestion, Dataflow for ETL processing, BigQuery for analytics and querying, and Data Studio for visualization. Control-M coordinates the sequencing and monitoring of all these services. 

How does Control-M handle pipeline failures on Google Cloud? 

Control-M includes built-in error handling and SLA management capabilities. When a job fails, Control-M can automatically restart it or escalate to a human based on predefined criteria. The SLA Management job type monitors the entire workflow as a single business entity and uses historical execution data to predict completion times, alerting monitoring teams to deviations before business SLAs are breached. 

The views and opinions expressed in this post are those of the author and do not necessarily reflect the official position of BMC.

What AI Demands from DataOps—and How Orchestration Delivers It

DataOps is the practice of applying agile and DevOps principles to data pipeline management—automating, scaling, and operationalizing data workflows across the enterprise. For organizations investing in AI, DataOps has become foundational: AI systems require data that is trusted, validated, and ready for consumption, not just data that moves fast. Closing the gap between pipeline execution and trusted outcomes is now the defining challenge of modern DataOps.

Why do so many data and AI initiatives stall before production?

Most organizations can build a promising proof of concept. The harder problem is getting it to production. According to Forbes, the vast majority of generative AI pilots never make it into production—not because the ideas aren’t sound, but because scaling those initiatives requires reliability, integration, and data quality that pilot environments don’t demand.

This reflects a broader challenge that extends beyond AI alone. Despite continued investment in data platforms and cloud technologies, organizations still struggle to operationalize data and AI initiatives at scale. The effort remains concentrated on building pipelines and training models. The harder problem—ensuring those systems run consistently in production, with the coordination and data quality required to support business-critical decisions—is where progress stalls.

How DataOps changed the way organizations manage data pipelines

DataOps emerged as a framework to address a fundamental challenge: how to operationalize data initiatives at scale. As data volumes grew and pipelines became more complex, traditional approaches struggled to keep up. DataOps applied agile engineering and DevOps best practices to data management, with a clear goal: turn new insights into production-ready data pipelines that deliver business value.

Adoption accelerated quickly, and DataOps evolved into a recognized market category, Gartner^® defines DataOps as the collaborative data management practice focusing on improving communication, continuous integration, automation, observability and operations of data flows between data managers, data consumers, and their teams across the organization. At the center of that definition is orchestration: the coordination, automation, and control that makes complex data workflows manageable.

What AI-ready data actually requires

DataOps helped organizations automate and scale data pipelines—but AI systems have raised the bar. Speed alone is no longer sufficient.

AI-ready data, as we see it in the Gartner report, requires three things:

• Alignment—relevance to the specific use case
• Qualification—continuous validation for production environments
• Governance—policy, compliance, and traceability

This represents a fundamental shift in what DataOps must deliver. DataOps is no longer just about moving data efficiently—it is about ensuring data is ready for intelligent systems to act on.

Why data quality is now an orchestration problem

Many organizations have automated orchestration. Far fewer have implemented data readiness and certification practices. That gap is where AI initiatives break down.

When pipelines prioritize speed over readiness, unvalidated data reaches AI systems and drives business decisions. The result is not just pipeline failure—it is incorrect outcomes at scale. Data quality cannot be treated as a separate, downstream check. It must be embedded within the workflow itself—validating data in context, gating execution based on readiness, preventing downstream impact, and triggering remediation automatically. Without that level of coordination, pipelines may run, but their outputs cannot be trusted.

What modern data pipeline complexity looks like

Modern data pipelines span multiple applications, data sources, and infrastructure technologies that must work together seamlessly. Organizations now operate a mix of analytics pipelines, machine learning pipelines, RAG pipelines, and real-time inference pipelines. While each serves a different purpose, all rely on the same foundation: ingestion, storage, processing, and delivery.

Figure 1. Data projects have four stages with many moving parts across multiple technologies.

Ingestion pulls data from enterprise systems, devices, and digital channels. Storage spans diverse platforms optimized for cost, scale, and access. Processing encompasses batch, real-time, and event-driven workloads—now extended to feature engineering, model training, and inference. Delivery feeds not just dashboards but applications, APIs, and AI models and agents driving automated decisions.

As pipelines become interconnected, data quality is no longer isolated to a single stage. Issues introduced upstream can silently propagate downstream—affecting analytics, model performance, and business decisions. Pipelines may run and SLAs may be met, yet outcomes can still be wrong. In AI-driven systems, this leads to incorrect automated decisions at scale.

This is why orchestration must evolve from a scheduling layer to a control layer—one that determines not just when workflows run, but whether they should run at all, based on data readiness and quality.

Operationalizing DataOps for the AI era with Control-M

DataOps provides the foundation for managing data pipelines. But success in AI-driven environments depends on the ability to operationalize data, AI, and application workflows as a unified system—reliably and at scale. The gaps identified above—data readiness, execution gating, event-driven coordination, and cross-platform visibility—require a control layer that operates above individual pipelines. Control-M and Control-M SaaS provide that layer.

By abstracting the complexity of modern environments, Control-M orchestrates workflows across data platforms, AI systems, and business applications—providing end-to-end visibility, predictive SLAs, and coordinated execution across hybrid and multi-cloud environments.

Figure 2. Control-M is a layer of abstraction to simplify complex data pipelines.

Orchestrating across the modern data and AI ecosystem

Modern workflows span a rapidly expanding ecosystem of technologies. Control-M provides deep, out-of-the-box integrations across data platforms such as Snowflake, Databricks, and BigQuery; AI platforms including SageMaker, Bedrock, and Vertex AI; streaming systems like Kafka, SQS, and RabbitMQ; and enterprise applications across ERP, CRM, and modern SaaS platforms. This breadth allows organizations to orchestrate end-to-end workflows across the entire data and AI stack—without relying on custom scripting or fragmented tools.

Enabling event-driven and real-time workflows

AI-driven systems increasingly operate in real time—triggered by events, not schedules. Control-M supports event-driven orchestration, enabling workflows to be initiated based on data changes, application events, and business triggers. This enables more responsive execution for inference pipelines, agent-driven decisions, and time-sensitive processes.

Embedding data readiness with Control-M Data Assurance

In AI-driven environments, execution without validation introduces risk. Control-M Data Assurance embeds continuous data validation directly into workflows, ensuring data is validated before, during, and after execution. Workflows can be gated based on data readiness, preventing bad data from triggering downstream processes and ensuring that AI models and agents operate only on trusted inputs. This shifts DataOps from pipeline execution to trusted execution.

SLA-driven, outcome-aware orchestration

Control-M connects workflow execution to business outcomes—tracking SLAs, predicting delays, and surfacing impact—so workflows align to business-critical timelines.

Automation, remediation, and visibility at scale

Control-M automates error detection, notifications, remediation, and cross-platform coordination, while providing end-to-end visibility across workflows for operations teams, developers, and business users.

The need for data continues to grow—but value is only realized when that data is operationalized effectively. By combining DataOps practices with a control layer like Control-M, organizations can orchestrate data, AI, and application workflows end to end, integrate across evolving ecosystems, respond to events in real time, ensure data readiness before execution, and deliver reliable, SLA-driven outcomes at scale.

Success in the AI era is not defined by how fast pipelines run—but by how reliably they produce trusted, actionable outcomes.

Frequently asked questions about DataOps and AI orchestration

What is the difference between DataOps and traditional data management?

Traditional data management focuses on storing and accessing data. DataOps applies agile engineering and DevOps principles to the full lifecycle of data pipelines—automating delivery, improving collaboration between teams, and enabling faster, more reliable production deployment of data workflows.

Why do generative AI pilots often fail to reach production?

Most generative AI pilots fail to reach production not because the underlying technology is flawed, but because scaling AI requires data that is continuously validated, reliably delivered, and governed for compliance—conditions that pilot environments rarely test for. Without production-grade DataOps practices in place, AI systems encounter data quality issues that cause failures or incorrect outcomes at scale.

What is AI-ready data?

AI-ready data meets three criteria that we see in the Gartner Report: alignment (relevance to the specific use case), qualification (continuous validation for production environments), and governance (policy, compliance, and traceability). Data that moves quickly but fails these criteria can produce incorrect outcomes when consumed by AI systems.

What is the role of orchestration in DataOps?

Orchestration is the coordination layer that manages when, how, and whether data workflows execute. In modern DataOps, orchestration has expanded beyond scheduling to include data readiness gating, event-driven triggering, SLA tracking, and automated remediation—ensuring that pipelines not only run, but produce trusted outcomes.

How does Control-M support DataOps for AI?

Control-M acts as a control layer for data, AI, and application workflows—orchestrating execution across platforms such as Snowflake, Databricks, SageMaker, and Kafka. Control-M Data Assurance embeds data validation directly into workflows, gating execution based on readiness so that AI models and agents operate only on trusted inputs.

What is the difference between pipeline orchestration and a control layer?

Pipeline orchestration coordinates task execution within a workflow. A control layer operates at a higher level—determining whether workflows should run based on data quality and readiness, coordinating across multiple systems and pipeline types, and connecting execution to business outcomes through SLA tracking and predictive monitoring.

Kindly add the required attributions and disclaimers –

Gartner, Market Guide for DataOps Tools, By Michael Simone, Sharat Menon, Robert Thanaraj, 24 October 2025.

GARTNER is a trademark of Gartner, Inc. and/or its affiliates.

Gartner does not endorse any vendor, product or service depicted in its research publications and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.