AI is changing where risk shows up—and making it harder to prove you’re in control.

TL;DR

AI agents, APIs, and automated workflows are scaling faster than the controls designed to govern them. Fewer processes involves people, but most controls still assume they do.

Policies exist. Monitoring exists. But control rarely happens at the moment work runs.

That’s the gap.

If you can’t prove—right now—what executed, what data was used, and whether it followed policy, you don’t have real-time compliance. You have delayed reporting.

What Do We Mean by “Real-Time Compliance”?

Practically speaking, it’s pretty simple: Can you tell me—right now—what ran, what data it used, and whether it followed policy? If you can’t answer that without digging through logs or pulling reports later, it’s not real-time.

Real-time compliance means control is applied as work runs—not before on paper, and not after in an audit. And the proof is created at the same time, automatically.

If you have to reconstruct what happened after the fact, you’re not operating in real time. You’re piecing together history.

The Real Problem: Control Isn’t Applied Where Work Happens

Most organizations aren’t missing controls. You likely already have:

• IAM to define who can access what
• Security tools to detect issues
• Governance frameworks to set policies

The problem is that these systems don’t actually control what happens when work runs across workflows, data pipelines, or AI systems. As a result, gaps show up in production:

• Machine-driven activity grows, but enforcement isn’t consistent
• Data pipelines move faster, but validation gets weaker
• AI systems follow policies on paper, but not always in practice

And when someone asks, “Are we compliant right now?” you still can’t answer in real time. It can take hours, days, or even weeks.

A Quick Litmus Test: Is Control Enforced?

If you’re accountable for risk and compliance, there’s a simple way to pressure-test your current state: Check the boxes where you can prove control at execution—not just in policies or audit reports.

Machine & AI execution

• You can list every non-human identity running production workflows
• You can trace every AI-driven action to a system and dataset
• Policies are enforced before execution, not just logged afterward

If not, machine activity is happening outside enforceable control—and you can’t reliably audit it.

Data and AI pipelines

• Every production pipeline includes enforced validation checkpoints
• You can prove lineage from source to output
• AI systems cannot run on unapproved or external data

If not, decisions are being made on data you can’t fully trust or defend.

Compliance proof

• You can generate evidence in real time without manual effort
• Audit trails are system-generated, not assembled afterward
• You can answer “are we compliant right now?” with actual data

If not, compliance is reactive and hard to defend under scrutiny.

The takeaway: If you can’t check every box in a category, control in that area isn’t enforced at execution. And if you see gaps across categories, you’re not preventing risk, you’re discovering it after the fact.

Where Real-Time Compliance Breaks Down

In most environments, the issue is that controls aren’t applied where the work actually runs.

Here’s where it typically breaks down:

1. Identity control stops with people

IAM works well for humans, but most production activity now isn’t driven by people. It’s service accounts, APIs, automated workflows, and AI agents doing the work. These identities often aren’t consistently governed, aren’t tied to enforceable policies at runtime, and aren’t monitored at the point of action. So, while access may be controlled, execution isn’t.

2. Data pipelines outrun your controls

Pipelines are built to move fast. Controls are often layered around them, not inside them. That leads to validation being optional, policies being applied inconsistently, and outputs being built on unverified inputs. So, you might have lineage—but not trust in the outcome.

3. AI governance stops at definition

Most organizations have started defining model policies, access controls, and governance frameworks. But they don’t control how AI actions are triggered, what data is actually used, and how decisions propagate through systems. So, policy exists, but enforcement doesn’t.

What It Takes to Prove Compliance

At some point, this becomes a practical question: Can we prove what happened, as it happened? To do that, control has to move closer to execution, where works runs.

That means having a layer that:

• enforces policy before execution
• validates data as it moves
• governs AI workflows like any other production process
• captures evidence as part of execution—not afterward

When that exists, questions like these are easier to answer:

• What ran across our environment in the last 24 hours?
• Which workflows used unvalidated data?
• Where were controls bypassed?
• Which AI-driven actions violated policy?
• Are we compliant right now?

If you can’t answer these questions quickly, control is assumed—not enforced.

What Real-Time Compliance Requires

Frameworks like the EU AI Act and NIST AI RMF aren’t asking for more documentation.

They’re asking for provable behavior. To meet that bar, four things need to be true:

1. Controls are enforced at execution

Non-compliant workflows don’t run. Policies apply consistently. Every execution records whether it passed or failed control.

2. Data and decisions are traceable

Every output can be traced back to its data sources, transformations, and execution path. Across systems, not just within tools.

3. Compliance is continuous

You don’t check compliance periodically. You can see what’s running, under which policies, in real time.

4. Evidence is generated automatically

Audit trails aren’t reconstructed. They’re created as part of execution. If compliance depends on reconstructing events, it won’t hold up under real pressure.

What This Looks Like in the First 90 Days

Achieving real-time compliance isn’t a multi-year transformation. It starts with visibility into where control breaks down, and then quickly moves to enforcing control at execution.

Days 0–30: Visibility

• Identify critical workflows
• Surface unmanaged machine and AI activity
• Map where execution happens outside control

Outcome: You know where compliance can’t be proven right now.

Days 30–60: Enforcement

• Bring high-risk workflows under orchestration
• Introduce policy checkpoints
• Apply validation to key pipelines

Outcome: High-risk execution is now governed at runtime.

Days 60–90: Proof

• Automate evidence generation
• Establish continuous compliance baselines
• Link workflows, data, and outcomes

Outcome: You can prove compliance as work runs, not after. 

Common Questions Security & Risk Leaders Ask

1. How do we find where controls are being bypassed?

You need visibility into execution, not just policy definitions. That usually means centralizing orchestration and making workflow execution observable across systems.

2. How do we keep AI agents within bounds?

By enforcing controls at runtime: identity, data access, and allowed actions. Every action must pass through those controls, not just inherit policy.

3. How do we add human approvals without slowing everything down?

By applying them selectively. Most workflows run automatically. Only exceptions or high-risk actions trigger human approval in real time.

4. How do we prove what happened without manual reconstruction?

By capturing execution as it happens: inputs, transformations, model activity, outputs. All in a single, time-sequenced record.

Final Thoughts: The Mental Shift That Matters

Compliance used to be about documentation. Now it’s about provable execution. The question isn’t “Do we have policies?” It’s “Can we prove they were enforced when work actually ran?”

That’s where an AI control plane approach starts to matter.

Platforms like Control‑M bring execution, control, and evidence together, so you can:

• Enforce policy at execution, not after the fact
• Validate data before it’s used
• See exactly what ran, how it ran, and what it produced
• Capture audit evidence automatically, as part of runtime

When that’s in place, everything tightens up: Work runs under control, activity is visible in real time, policies are applied consistently, and compliance is provable without reconstruction.

Final takeaway: If you’re trying to move from “we think we’re compliant” to “we can prove it right now,” it starts with enforcing control where execution happens.

How to operationalize AI governance to control risk and compliance in production

TL;DR

AI isn’t the hard part anymore. Operationalizing AI is.

Most AI projects stall because no one really has control over how the whole thing behaves in production.

Pipelines run. Workflows exist. But they’re stitched together across tools, scripts, and “somebody who knows how it works.” What you have isn’t a system. It’s coordination by coincidence.

A SOAP is the layer that turns all of that into something you can actually run and rely on.

Why So Many AI Projects Stall — and What’s Missing in the Middle

The model works. The use case is valid. And then it stalls. Not because the model failed, but because everything around it starts to break.

The problem shows up in the middle:

• data isn’t where it needs to be
• steps don’t run in the right order
• systems drift out of sync

Individually, those pieces exist. Together, they don’t behave like a system. That’s why “we got it working” rarely turns into “this runs reliably in production.”

There’s a layer that’s supposed to hold all of this together. In most environments, it doesn’t exist as a single system. It’s spread across schedulers, pipelines, scripts, and a lot of tribal knowledge.

When you actually pull that together into something coherent, it’s what’s referred to as a Service Orchestration and Automation Platform (SOAP).

Is SOAP Just Rebranded Orchestration?

It sounds like it. Most teams already have plenty of orchestration. And yet the common experience is: everything runs but it still doesn’t feel under control.

That’s because orchestration is built to execute workflows—even across systems—but it doesn’t inherently give you control over how those workflows behave at the system level.

The problem is that real systems don’t stay in boundaries. Workflows interact. Dependencies cross tools. Upstream delays ripple. AI steps behave inconsistently. So you end up here: everything ran, but the outcome is still wrong.

That’s not an execution failure. It’s a control failure. That’s the shift:

• Orchestration defines how the workflow is supposed to run across systems.
• SOAP is what actually keeps those workflows running correctly across the environment.

You don’t feel that difference when you’re building a pipeline. You feel it when:

• that pipeline connects to everything else
• something upstream is late
• something downstream quietly breaks
• and no single tool can explain what actually happened

In practice, this is where platforms like Control‑M show up—not as “another orchestrator,” but as the layer that coordinates workflows across data, applications, and AI so they behave like a system.

Takeaway: It’s not more orchestration. It’s the layer that makes everything already orchestrated actually behave predictably.

What a Control Plane Has to Get Right

Once you think in terms of a SOAP as the control plane, the question shifts pretty quickly from: “how do we run this workflow?” to “what does it take to keep this thing behaving under real conditions?”

In practice, there are a few things a control plane needs to get right if it’s going to hold up in production:

1. It actually has to run reliably

Not just once. Not just in a clean path. Across:

• cloud + onprem
• internal systems + external APIs
• workloads that don’t all behave the same way

AI makes this harder, not easier. Latency varies. Dependencies drift. Retries don’t always help. At some point, you realize the pipeline is only as reliable as the least predictable thing in it.

2. It has to understand what depends on what

This is the one most teams feel immediately. Something upstream is delayed, and you don’t find out until something downstream fails—or worse, runs with bad data.

Without system-level awareness, you’re always reacting. With it, you can actually see what’s at risk, what’s impacted, and what needs attention now.

3. It has to explain what’s going on

This is where things usually fall back to people. Someone knows how the workflow works, why it fails in weird ways, and what “normal” looks like. And everyone else is stuck asking them.

A control plane starts to pull that knowledge into the system itself:

• what this workflow does
• what changed
• what likely caused the issue

Not magic. Just enough context to stop everything from being a guessing game.

Takeaway: You’re no longer just running workflows. You’re running a system where behavior is visible, dependencies are understood, and issues are explainable. That’s what lets AI pipelines move from “it works” to something you can actually rely on.

What This Looks Like When It Actually Works

This is where the control plane idea stops being conceptual and starts showing up in real workflows.

1. When a “quick change” isn’t a scramble anymore

You get the request: “Can we refresh this dashboard with updated projections today?”

Without a control plane, that usually turns into a scramble—figuring out which pipelines are involved, coordinating across a few teams, manually triggering jobs, and then watching closely to see what breaks.

With a control plane, that same request is already understood as a workflow. The dependencies are mapped, the execution path is known, and the change can be applied in one place without chasing it across tools.

The difference isn’t just speed. It’s that the work becomes predictable and repeatable instead of reactive.

2. When AI actually makes it to production

Moving revenue forecasting models, fraud analysis pipelines, or supply chain optimization workflows from proof-of-concept to operational requires enterprise-grade discipline: governed data ingestion, controlled model execution, managed LLM invocation.

With a control plane, ingestion, transformation, inference, and downstream updates are all coordinated, observable, and governed the same way. At that point, moving to production stops feeling like starting over.

3. When governance isn’t a periodic fire drill

In most environments, workflow sprawl builds up quietly over time. Pipelines stick around long after they’re needed, dependencies overlap, and no one really has a clear view of what’s still in use. You don’t notice it until something breaks, performance slips, or there’s an audit coming up.

Without a control plane, governance is something you piece together after the fact. With one, the system itself starts to surface what’s changed, what’s no longer used, and where things are drifting. Governance becomes continuous instead of reactive.

Why the Lack of a Control Plane Feels Worse With AI

This isn’t a new problem. The gaps in how workflows are coordinated have always been there. AI just makes them obvious.

You now have more steps, more variability, more external dependencies (LLMs, APIs), and less predictable behavior. So the same coordination gaps that were manageable before start to show up everywhere.

Without a Control Plane

No shared context

• Each part runs its piece
• Dependencies are implicit
• Failures are discovered late
• People connect the dots manually

With a Control Plane (SOAP)

• One layer coordinates execution across everything
• Dependencies are explicit and visible
• Failures are understood in context
• Impact is clear before it spreads

Where SOAP Changes Day-to-Day Work

This is where it gets real.

• Instead of debugging across tools, you can actually see the workflow end to end.
• Instead of isolated failures, you get context about what those failures affect.
• Instead of relying on “who knows this pipeline,” the system carries that understanding.

With a SOAP, AI workloads stop being special cases and start behaving like everything else you run in production—the same way mature teams handle automation and orchestration across the rest of the stack.

Where to Start (Without Turning This Into a Rewrite)

You don’t need to replatform everything. You need to expose where you don’t have control.

1. Start with one pipeline that matters

Not a clean one. A real one. Map what actually happens: where data comes from, what it triggers, and what breaks if something is late. This is where hidden dependencies show up.

2. Count how many “control planes” you actually have

Most teams have multiple orchestration tools, scripts filling gaps, and people connecting the dots. Everything is orchestrated. Nothing is coordinated.

3. Make the system visible before you automate it

You should be able to answer, in one place: what does this workflow actually do, what depends on it, and what happens if it fails or drifts. If you can’t answer those, automation just makes troubleshooting harder.

4. Treat AI workloads like real workloads

This is where things quietly break. AI steps often have different retry behavior, weaker monitoring, and less consistent control. That works in testing. It doesn’t in production. Treat them like everything else: observable, governed, and part of the same system—which is increasingly what agentic orchestration is being built to handle.

5. Don’t try to fix everything at once

If one pipeline becomes visible, predictable and understandable, that’s already meaningful progress. From there, it scales.

What You’re Actually Building Toward

Not a new tool. Not a perfect architecture. Just this: a single layer that understands how your workflows behave and keeps them from drifting.

Once you have that, automation gets easier, failures are less surprising, and scaling doesn’t multiply chaos.

To Sum Up: The Shift That Actually Matters

AI isn’t a modeling problem anymore. It’s an operations problem: can you run complex, cross-system workflows reliably under real conditions? That’s the same shift driving teams to operationalize data and AI projects through orchestration.

If this feels familiar

If your current setup works, but only with careful coordination, tribal knowledge, and a few “don’t touch that” pipelines, you’re not behind. You’re just missing the layer that turns all of it into a system you can actually control. That’s the role SOAP is starting to play.

If you’re trying to move from “we got it working” to “we can run this reliably, every day”—here’s a practical guide to turning complex workflows into AI-powered outcomes: Orchestration: The Missing Layer in Enterprise AI.

This guide goes deeper into what’s missing when it comes to operationalizing AI—what it actually looks like in production, why it shows up so consistently, and how teams are starting to close the gap with a real control plane.

As digital certificate lifetimes drop to 47 days, automation becomes essential to maintain availability, security, and compliance. BMC AMI Digital Certificate Manager extends automated certificate lifecycle management to the mainframe, enabling standardization across the enterprise using your current CLM vendor.

Digital certificates are the connective tissue of the enterprise environment, enabling systems, workloads, applications, and APIs to verify each other and communicate securely. When a certificate fails due to expiration or error, the impact on service availability and security can be immediate and severe. And with regulators driving certificate lifetimes down to as little as 47 days by 2029, organizations face a sharp increase in both operational and compliance risk. That makes digital certificate management a top priority for BMC customers.

By enabling organizations to discover, track, and renew certificates across the infrastructure, digital certificate management prevents the outages and security gaps that can result from expired, misconfigured, or otherwise compromised certificates. This task has grown more difficult in recent years, and even greater challenges are on the horizon. But BMC has a solution.

Why digital certificate management is becoming an urgent challenge

Traditionally, many organizations have managed mainframe certificates through manual processes centered on spreadsheets and tribal knowledge. In recent years, the growing number of system identities relying on these certificates has pushed these methods to the breaking point. Now, regulatory changes have made them completely unsustainable.

To reduce the exposure that can result from a compromised certificate, the CA/Browser Forum has announced aggressive reductions in TLS certificate lifetimes. Until this month, companies were allowed a relatively manageable 398-day renewal cycle. Now that window has been nearly cut in half to 200 days. Next March, it will shrink once again to 100 days, and by March 2029, TLS certificates will be good for only 47 days. Each of these reductions effectively multiplies the certificate management workload for mainframe security teams, and with it, the chance of manual errors, expirations, and system outages.

This isn’t a future problem. The regulatory deadlines are fixed, the timelines are non-negotiable, and their impact is inevitable. That’s why I’m excited to announce BMC AMI Digital Certificate Manager (DCM)—a new solution that fundamentally changes certificate management on the mainframe.

How to extend enterprise digital certificate management to the mainframe

Most enterprises already invest in Certificate Lifecycle Management (CLM) platforms for their distributed and cloud environments. These platforms haven’t been able reach the mainframe, however, leaving z/OS as a manual island in an otherwise automated estate. Now BMC is filling that gap with the only solution enabling digital certificate management platforms to extend automation to the mainframe as part of a consistent enterprise strategy.

Proven in operational environments for over five years, DCM provides a unified integration layer to connect Venafi and Keyfactor digital certificate management tools to mainframe ESMs including RACF, ACF2, and Top Secret. With DCM, you can standardize on one BMC solution for your mainframe while supporting whichever certificate vendors your organization already uses, no rip-and-replace required.

End-to-end automated certificate operations

DCM extends your organization’s CLM to automate the entire mainframe certificate lifecycle, from issuance and renewal to replacement and rollback. The impact is immediate and measurable:

• Dramatic effort reduction: Mainframe certificate implementations that previously took up to three hours of manual work are now fully automated.
• Eliminated outage risk: Expired or mismanaged certificates are a major cause of preventable mainframe outages. DCM’s scheduled renewals and built-in rollback ensure continuous availability without late-night firefighting.
• Reduced dependency on scarce skills: Mainframe security expertise is increasingly hard to find. DCM removes the need for skilled personnel to manually execute certificate commands across RACF, ACF2, or Top Secret, freeing them for higher-value work.
• Complete audit visibility: Every action is logged with full detail, including which commands were issued, which ESM responses were received, who authorized the change, and when.

Real-world impact at a major financial institution

One of the world’s largest financial institutions evaluated DCM against its current, pre-automation state. With certificate volumes growing over 30 percent year over year, a small core team currently handles digital certificate management manually across many application owners and faces an increasing risk of outages, audit failures, and security gaps.

The firm projected the five-year value of deploying DCM as $8.6 million, driven by manual effort reduction and avoided headcount ($3.6M), eliminated application outages ($2.2M), compliance and audit risk reduction ($1.4M), operational efficiency gains ($0.8M), and future-proofing against accelerating certificate volumes ($0.6M). Beyond these measurable financial gains, the solution supports the institution’s broader strategic priorities around operational resilience and responsible growth.

Strengthening Zero Trust across the enterprise

Machine identity is foundational to Zero Trust: Every workload, process, and system must be authenticated. Working alongside BMC AMI Security, DCM becomes part of a comprehensive Zero Trust strategy for the mainframe, enabling continuous threat detection, automated response, and end-to-end protection across your most critical environment. Security teams gain the observability, policy enforcement, and confidence they need to report to the chief information security officer (CISO) and the board that the mainframe is truly protected.

Looking ahead

The certificate landscape continues to move toward shorter lifetimes, more frequent renewals, higher volumes, and tighter regulatory scrutiny. All of these trends will drive an exponential growth in manual digital certificate management workloads. By acting now, organizations can stay ahead of increasingly urgent certificate deadlines while preparing their infrastructure for continuous, automated certificate renewals.

BMC AMI Digital Certificate Manager is generally available. We invite you to learn more about how DCM can modernize certificate management across your mainframe environment—preserving your existing tools, eliminating manual effort, and building the operational resilience your business demands.

BMC is aware of recent industry discussion about the use of advanced AI techniques to identify software vulnerabilities, including initiatives such as Project Glasswing.

As part of normal security operations, BMC continuously monitors emerging research, threat intelligence, and industry developments related to software and supply chain security. These developments reflect an acceleration in how vulnerabilities may be identified across the industry.

BMC maintains a defense-in-depth security program and regularly evaluates opportunities to enhance controls and processes as technologies evolve. We engage with customers, partners, and the broader security community to remain aligned with industry best practices.

As new information becomes available, BMC will use its proactive notification process to help keep customers up to date.

To register for proactive notifications, please see the following BMC Support Central article:

https://docs.bmc.com/xwiki/bin/view/Standalone/BMC-Support-Central-User-Guide/supportcentraluserguide/Manage-Your-Support-Account/Favorite-Products-and-Alerts/

The Orchestration Imperative for the AI Era

Something interesting is happening in enterprise technology right now. In conversations with clients, I’m hearing that as companies modernize, invest heavily in AI, and build increasingly complex digital ecosystems, many are running into the same reality.

Innovation is moving faster than operations.

And the gap is now showing up in execution. Customers report that as they move quickly to launch more AI pilots and adopt new platforms, the operational foundations beneath these innovations often remain fragmented. Automation silos, disconnected pipelines, and governance gaps create that fragmentation—and introduce risk just when scale matters most.

What’s interesting is where this leads: automation itself is no longer the goal. The question is no longer how do we automate more? It’s how do we orchestrate everything reliably at enterprise scale?

Based on numerous conversations with customers and prospects in recent months, here are five operational shifts I believe every enterprise will face if they aren’t already there.

1. From AI experimentation to AI execution

Many enterprises have already piloted GenAI. They are now expecting measurable outcomes. The focus is shifting from isolated experiments to production-grade AI embedded directly in operations. Leaders are asking practical questions:

Does AI …

• Reduce incidents?
• Accelerate root cause analysis?
• Improve SLA adherence and resource utilization?

AI will increasingly power predictive workflow orchestration—anticipating job failures, recommending remediation, reallocating resources, and learning from execution patterns. But intelligence alone isn’t enough. Enterprise AI must be transparent, explainable, and governed. Black-box automation won’t meet operational standards. The most valuable AI will be embedded directly into orchestration platforms—where visibility, policy enforcement, and accountability already exist.

The organizations that succeed won’t be those that experimented the most. They’ll be the ones that operationalized AI to improve resilience, efficiency, and speed at scale.

 2. Speed at scale demands “freedom with guardrails”

Technology ownership continues to decentralize. Developers, data engineers, DevOps teams, and business technologists all want the ability to build and deploy quickly without waiting on centralized teams. But autonomy without structure creates risk—security gaps, compliance exposure, duplicated workflows, and operational sprawl.

In response, leading organizations are adopting a “freedom with guardrails” model. Teams gain self-service capabilities to build workflows and pipelines, while governance is embedded directly into the automation layer. Role-based access, policy enforcement, auditability, and standardized templates allow teams to move independently without sacrificing oversight.

The key shift is architectural: governance won’t be an afterthought or manual review step. It will be codified into orchestration frameworks themselves. This balance of empowerment and control will become a defining characteristic of high-performing digital enterprises.

 3. Real-time orchestration becomes the backbone of operations

The era of purely nightly batch processing is fading. As digital services compress response times and customer expectations approach real-time, orchestration must evolve from static scheduling to event-driven responsiveness.

Moving forward, organizations will increasingly rely on workflows triggered by real-time business signals—transaction anomalies, supply chain disruptions, or customer behavior changes. Modern orchestration platforms will unify batch, micro-batch, and streaming execution models within a single operational framework.

This requires horizontal scalability, high availability across hybrid and multi-cloud environments, and end-to-end workflow visibility. Monitoring alone won’t be enough. Organizations will demand predictive insights and proactive remediation, transforming observability into operational foresight.

4. Ecosystem collaboration emerges as a competitive advantage

No enterprise operates in isolation—and no orchestration platform can either. Modern operations span SaaS applications, hyperscalers, on-prem systems, managed file transfer, data lakes, AI services, and edge environments. The orchestration layer increasingly becomes the connective tissue across this ecosystem.

Interoperability will move from nice-to-have to mandatory. Customers will expect secure, open, and extensible integrations that connect data pipelines, AI engines, compliance systems, and operational workflows seamlessly. API-driven architectures, pre-built integrations, and ecosystem partnerships will shape buying decisions more than feature checklists.

Working with vendors that embrace openness and integration supporting hybrid and multi-cloud strategies without lock-in will stand out as strategic enablers. Trust, extensibility, and ecosystem compatibility will define leadership in the orchestration market.

5. Orchestration evolves into a shared business capability

Orchestration is no longer an IT-only concern. It touches revenue operations, customer experience, compliance, analytics, and supply chain resilience. Organizations that treat orchestration as a shared, cross-functional capability will outperform those that silo it within infrastructure teams.

This means evolving operating models. Shared ownership between IT operations, data teams, DevOps, and business stakeholders will become the norm. Governance councils, cross-functional workflow design, and outcome-based metrics will replace purely technical KPIs.

Most importantly, orchestration will be treated as a living capability—continuously refined as new applications, AI services, and regulatory requirements emerge. Enterprises that embed this mindset will build an automation backbone that scales with growth, adapts to disruption, and supports innovation without sacrificing control.

Final Thought: Confident AI at Enterprise Scale

The conversations that defined the last twelve months around AI, speed, governance, and resilience are converging into a more mature operational mandate.

The goal is no longer experimentation. It’s confident scale.

Organizations want to move faster, operate smarter, and maintain unwavering trust in every automated outcome. I’m looking forward to continuing this conversation as we operationalize these ideas.

If these themes resonate with what you’re seeing in your organization, I’d love to hear what’s highest on your operational agenda.

IMS continues to serve as the backbone for high-volume, transaction-driven applications across industries such as banking, insurance, retail, and healthcare. These environments are trusted due to their reliability and performance, yet they are also evolving. Today’s IMS systems support a wider mix of users, integrations, and business-critical activity than ever before. As a result, complexity is increasing while many teams cope with limited resources and fewer experienced specialists.

This shift is driving renewed attention to how IMS environments are monitored, analyzed, and managed every day. Traditional monitoring approaches remain important, but they are limited by static thresholds and fragmented views, making it difficult to understand how system behavior fits together. When issues arise, teams may recognize that something is wrong without clear visibility into why it matters or where to focus first.

How AIOps improves IMS visibility

To address this challenge, organizations are increasingly adopting mainframe AIOps. Using machine learning and advanced analytics to examine system and performance data, AIOps helps teams recognize normal behavior, identify emerging conditions earlier, and reduce alert noise that doesn’t require action. In IMS environments, this helps teams gain clarity faster and make more confident decisions without increasing manual effort.

BMC AMI Ops Insight, working alongside brings these AIOps capabilities into IMS operations by learning how systems behave over time and identifying meaningful changes in system behavior within the wider operational context. Rather than treating every deviation the same, intelligent models correlate activity across metrics and subsystems to help teams assess potential impact and prioritize response. This allows teams to move more quickly from detection to understanding, even in highly complex environments.

Combining operational and data insight

Insight becomes even more valuable when paired with data awareness. IMS data activity plays a critical role in application behavior, performance trends, and recovery scenarios. BMC AMI Data for IMS provides visibility into database and application activity, adding context that helps explain what teams are seeing at the system level. When operational insight and data contexts are brought together, teams gain a fuller view of system behavior and risk.

Across the industry, organizations are applying these approaches to shorten detection and resolution timelines, improve incident clarity, and reduce. These efforts also support modernization and resilience initiatives, helping teams maintain confidence and control as demands on IMS environments continue to grow.

Watch the on-demand webinar How AI-Driven Insight is Changing IMS Operations to see how BMC AMI Ops Insight and BMC AMI Data for IMS help organizations apply AIOps-driven insight to IMS operations with greater clarity, faster understanding, and more confident action.

Artificial intelligence (AI) is evolving rapidly, and so are organizations’ approach to its use. In April 2026, BMC released a statement of direction on the integration of AI into the BMC AMI suite of mainframe solutions. In The Next Evolution in Enterprise AI is Purpose Built, BMC mainframe Senior Vice President and General Manager John McKenny discusses the shift from generative AI (GenAI) to agentic AI and our intent to embed agentic AI across the BMC AMI portfolio.

While the statement of direction explains how we’re moving into the future with agentic AI workflows, over the past two years we have developed purpose-built AI for the mainframe, offering in-context expertise through BMC AMI Assistant. With Knowledge Expert Chat, practitioners can harness the power of AI to find the answers they need when and where they need them, increasing the quality and efficiency of their work. Each quarter, we have added to what mainframe professionals can accomplish while increasing integration of BMC AMI Assistant across the BMC AMI portfolio.

The April 2026 release of enhancements to the BMC AMI portfolio focuses on this advancement of mainframe GenAI as well as the open integration of the platform with the broader enterprise IT ecosystem.

GenAI powered by organizational intelligence

Even as we increase the use of agent-driven automation on the mainframe, the efforts to improve GenAI assistance continue, including this quarter’s General Availability of Knowledge Hub, which surfaces institutional knowledge from across the organization to offer contextual information and answers at the moment of decision through BMC AMI Assistant Knowledge Expert Chat.

Knowledge Hub works behind the scenes, ingesting provided institutional knowledge from past issue resolutions, operational insights, and information from runbooks, tickets, and shared files, then combining it with BMC mainframe expertise to create Knowledge Expert Chat responses that are contextually aware and tailored to user’s mainframe environment. These responses are available directly in workflows using existing BMC tools, enabling quicker decisions made with expert-level confidence, regardless of the user’s experience level.

AI-generated application analysis reports

Just as Knowledge Hub captures institutional knowledge to inform assistance provided by Knowledge Expert Chat, a new feature in BMC AMI zAdviser Enterprise adds hard-won institutional knowledge to zAdviser’s collection of development tool and toolchain data to help development managers and their developers understand the applications they are modifying.

BMC AMI zAdviser Enterprise gathers BMC AMI tool usage data and DevOps metrics to give development managers a clear picture of development effectiveness. With new application analysis reports, they now get a single AI-generated view of how their applications work, where the risk is, and where their team’s time is going, accelerating modernization decisions and cutting weeks off of developer onboarding.

BMC AMI zAdviser Enterprise’s Application Analysis turns tribal knowledge into organizational knowledge before it walks out the door. The narrative assessments capture the knowledge of experienced developers, sharing that expertise with development managers and developers, creating more efficient application review, planning, and optimization.

Beyond a greater understanding of individual programs, these application analysis reports provide a clear picture of which programs are attracting a disproportionate share of developer attention because of failures and modifications. By correlating failure history with code complexity and maintenance patterns, they enable development teams to target problem applications with proactive remediation, improving system resilience and development efficiency.

Standardizing enterprise security

Manual management of security certificates decreases efficiency and weakens system security. For some time, security teams have been able to automate certificate management on distributed systems using third-party tools.BMC revolutionized mainframe security management with BMC AMI Enterprise Connector for Venafi. This April, we introduce BMC AMI Certificate Manager, a new product within BMC AMI Security that gives customers more choice and flexibility in integrating enterprise security solutions with the mainframe.

Designed to integrate with IBM Z^® enterprise security management (ESM) environments, including RACF^®, ACF2^™, and Top Secret^® for z/OS to automate the full certificate lifecycle without the need for infrastructure changes, BMC AMI Certificate Manager extends leading enterprise certificate management platforms to the mainframe. This gives CISOs and security teams the ability to use the same vendor solutions for distributed and mainframe certificate management, further integrating the mainframe with—and simplifying—enterprise security efforts.

BMC AMI Certificate Manager currently integrates with Venafi^® and Keyfactor^®, with further integrations to follow in future releases. With this unified integration layer between enterprise certificate management tools and IBM Z^® ESMs, organizations can automate certificate issuance, renewal, and enforcement with one BMC solution while supporting multiple vendors.

Breaking down silos with new technology

This quarter’s enhancements further BMC’s commitment to optimizing what is possible on the mainframe through open integration of the platform and continuous improvements to AI capabilities. With the addition of organization-specific knowledge to AI engines, Knowledge Hub empowers mainframe professionals to make the right decisions as they do their jobs, regardless of their experience and skill levels. Application analysis reports in BMC AMI zAdviser Enterprise combine development and application performance data with system telemetry to provide a clear picture of how applications are performing and where attention and efforts should be focused. And the new BMC AMI Certificate Manager enables security teams to employ certificate management policies across platforms without the need for separate tooling.

Each of these enhancements improves the productivity and efficiency of mainframe teams, a goal BMC is committed to pursuing with each of our quarterly releases. Make your mainframe the engine of faster, better, and smarter answers. When you BMC First.

These are just a few of the innovations included in the April 2026 release of BMC AMI features. To learn more about everything included in the release, visit the What’s New in Mainframe Solutions webpage.

Mainframe organizations continue to run some of the most critical systems in the global economy while managing increasing complexity and rising expectations for innovation. Artificial intelligence (AI) provides a practical way to scale expertise, reduce operational friction, and help teams move work forward with greater confidence.

BMC believes organizations will increasingly leverage AI to broaden access to mainframe capabilities and integrate these systems more fully into their broader enterprise IT ecosystems. In many cases, AI will also act as a catalyst for modernization in place—helping organizations evolve applications, workflows, and operational practices while continuing to run critical workloads on the platform that already powers their business.

The Direction: From AI assistance to coordinated intelligence

Enterprise AI is moving beyond generating insights and explanations. Organizations now expect intelligence that can safely participate in execution across critical systems. Our direction is clear: BMC AMI solutions are evolving into intelligent participants in a governed ecosystem of AI agents that collaborate autonomously across development, operations, data, and security workflows to accelerate innovation on the mainframe.

Supporting this direction requires enterprise AI on the mainframe to operate across three coordinated layers:

Intelligence layer — Domain knowledge and reasoning grounded in mainframe expertise, operational telemetry, and institutional knowledge.

Coordination layer — Orchestrated AI agents collaborating across workflows to connect understanding with action.

Governance layer — Policy-aware controls ensuring AI-driven actions remain secure, transparent, auditable, and subject to human oversight.

Together, these layers enable AI to extend beyond analysis and participate responsibly in execution. This foundation now enables the next phase of our direction: extending AI beyond knowledge and insight toward coordinated, governed execution across the BMC AMI platform.

AI in production: Trust earned, not claimed

In July 2024, we published a Statement of Direction focused on infusing intelligence directly into BMC AMI product experiences. Our objective was straightforward: reduce complexity, close knowledge gaps, and make it easier for teams to work confidently on the mainframe.

That direction was grounded in three core principles:

• Consistent and contextual intelligence, with a common AI foundation infused across the AMI portfolio
• LLM freedom, allowing organizations to choose the models that align with their enterprise AI strategies
• In-the-moment intelligence, embedding AI directly into workflows to surface answers and insights where work happens

Since then, we have delivered.

BMC AMI Assistant was introduced beginning with GenAI-powered code explanation and expanded across consecutive quarterly releases into a cohesive intelligence layer embedded throughout the BMC AMI portfolio. Using AI as an advisor, developers gained faster understanding of unfamiliar applications, while operators gained explainable insight into system and operational issues. Investigation that once required manual analysis increasingly includes guided insight and next steps directly inside the workflow.

The addition of Knowledge Expert Chat and Knowledge Hub further expanded this intelligence layer. Together, these capabilities combine curated product intelligence, decades of BMC mainframe expertise, and each organization’s own institutional knowledge, making trusted guidance available at the moment of decision. This approach helps less-experienced staff work with greater confidence while allowing senior experts to focus on higher-value work, turning institutional knowledge into a shared operational asset. Watch this short video for a demo of how Knowledge Expert Chat provides answers at the moment of need.

Instead of searching across documentation, runbooks, and tickets, teams can surface answers through natural-language questions using a knowledge expert chat inside their workflow. Early usage shows that this approach not only accelerates troubleshooting and decision-making but also helps users discover capabilities and documentation they did not previously know existed.

Trust in AI is earned through real outcomes. This intelligence foundation now supports the next phase of our direction.

Customers are redefining what AI must do

Through advisory boards, design programs, beta participation, and enterprise AI readiness discussions, a clear message has emerged: customers want AI that does more than explain what happened. They want orchestrated intelligence that helps move work forward.

They want to reduce repetitive analysis, move faster from insight to action, and make it easier for teams to build expertise and focus on higher-value priorities. They want to realize the full promise of BMC AMI as automated mainframe intelligence.

Generative AI laid the foundation by helping teams understand, explain, and accelerate work. Agentic AI builds on that foundation by enabling intelligence to plan, decide, and safely carry work forward.

To meet these expectations at enterprise scale, our new Statement of Direction is grounded in five core principles:

• Establish the core first: Build an orchestrated agentic intelligence foundation.
• Specialized agents: Reflect domain expertise with focused, autonomous actions.
• Human-in-the-loop: Governed, transparent, and observable autonomy.
• Open ecosystem: Open standards-based, composable, and extensible.
• Outcome-driven adoption: Deliver measurable value early and expand over time.

The next phase extends AI beyond insight toward trusted participation in execution, within enterprise guardrails.

And that shift defines our next statement of direction.

From AI assistance to orchestrated intelligence

Building on the foundation of the AI-driven knowledge and guidance we have delivered with BMC AMI Assistant, our next statement of direction is clear: extending that foundation toward proactive, coordinated intelligence that can safely execute actions within defined governance policies across the BMC AMI portfolio.

This evolution introduces coordinated AI agents with specialized, policy-aware capabilities operating across development, operations, data, and security. Each agent is domain-specific, working together within enterprise guardrails to deliver governed, collaborative outcomes.

In this next phase, AI agents for operations, development, data, and security will operate as orchestrated participants across domains—supporting actions such as system and performance diagnostics, development workflows, security validation, and operational recovery. They will learn across past incidents and participate in execution with transparency, verifiability, enterprise governance, and human oversight built in by design.

We will expand beyond explanation and recommendation to enable validated action inside workflows. Agentic workflows will move from detection to investigation to explanation to resolution within a single, governed AI experience.

AI will not replace expertise. Human validation remains essential. Principal engineers and system programmers will continue to define policy, validate outcomes, and shape how intelligence operates. AI scales expertise rather than removing it.

The result is faster problem identification, more contextual decision-making, and reduced operational friction—while ensuring the mainframe participates fully in broader enterprise AI strategies. The mainframe will operate as a connected, policy-aligned participant in the enterprise AI ecosystem across hybrid environments, maintaining the security, reliability, and trust on which organizations depend.

Delivering agentic workflows across the BMC AMI portfolio

Over the coming releases, this direction will take shape through agentic workflows that coordinate knowledge, reasoning, and action across BMC AMI solutions. Initial workflows could include agentic AIOps incident resolution, agentic diagnostics, AI-assisted application insights, development troubleshooting, and test-case generation.

These workflows build on the intelligence layer already established across the BMC AMI portfolio, combining development and operations telemetry, mainframe domain expertise, and organizational knowledge to provide the context required for responsible execution.

From there, coordinated AI agents operate across solutions, aligning development, operations, data, and security activities that were previously siloed. Instead of isolated AI features, intelligence becomes part of the workflow itself, helping teams connect system understanding with the next operational step.

Every action remains governed and transparent. Human validation, enterprise policy, and operational guardrails remain central, ensuring AI participation strengthens reliability rather than introducing risk.

The objective is clear: translate insight into trusted action while preserving the control and discipline enterprise systems require.

Establishing the foundation for governed AI execution

As AI becomes more operational, governance and trust become increasingly essential. Organizations expect AI not only to inform decisions, but to operate safely, predictably, and transparently within enterprise policy boundaries.

Across the industry, fragmented approaches are already emerging: isolated AI integrations, independently managed execution layers, and disconnected tool endpoints. While intended to accelerate innovation, these models often introduce new complexity, inconsistent policy enforcement, and operational sprawl.

Enterprise mainframe environments cannot afford that fragmentation, especially as AI Agent orchestration becomes a core capability for coordinating multi-step agentic workflows. This can result in agents giving conflicting recommendations and teams lose confidence and revert to manual work. Without a unified and governed approach, orchestration itself becomes fragmented, leading to unpredictable behavior, loss of insight, and reduced trust in AI-driven operations.

As part of our next phase, our direction includes establishing an MCP Gateway as a shared, governed access layer across the BMC AMI portfolio. Rather than creating multiple independently governed AI entry points, the MCP Gateway will provide a centralized, secure, and policy-aware interface through which AI agents interact with BMC AMI solutions. Every AI-driven action will be visible, governed, and aligned to enterprise policy with consistent controls across the platform. This enables AI agents not only to access BMC AMI systems, but to safely execute actions across them within defined policy boundaries.

Supporting this architecture, we will introduce an Agent Gateway to facilitate how AI agents communicate and collaborate. Instead of agents interacting independently, interactions will flow through the Agent Gateway—where the agent interactions are visible—ensuring governance, auditability, logging, and policy enforcement across agentic workflows.

Together, the MCP Gateway and Agent Gateway extend the BMC AMI Platform into a governed AI foundation that enables coordinated intelligence and trusted execution across the portfolio (see Figure 1). The BMC AMI Platform serves as the enterprise foundation for this direction—providing a unified layer of core capabilities and services that connects intelligence, orchestration, and governance across the BMC AMI portfolio. It brings together innovations such as BMC AMI Assistant, the Agent Gateway, and the MCP Gateway to simplify mainframe transformation, accelerate innovation, and enable AI to operate consistently and securely at scale.

Industry standards define how AI systems connect to tools and agents. Our direction focuses on how those connections are governed, secured, and operationalized at enterprise scale—without introducing fragmentation or operational risk.

This is how we intend to evolve agentic AI into an enterprise-grade foundation across the BMC AMI portfolio (see Figure 1).

Figure 1: High-level overview of BMC AMI Platform’s agentic architecture supporting enterprise-grade AI.

What you can expect — and how to shape what comes next

Over the coming releases, you will experience a meaningful shift in how AI participates in the BMC AMI environment. Intelligence will extend beyond explanation and guidance to support the safe execution of operational tasks. Capabilities that once appeared as isolated features will increasingly operate as coordinated agentic workflows across the BMC AMI portfolio, connecting development, operations, data, and security activities.

AI will participate responsibly within enterprise guardrails, operating under the policies, validation models, and controls defined by the teams who run these systems. This direction represents more than a single capability release, it establishes the foundation for a new way of working with BMC AMI solutions.

We invite customers to help us shape this next phase. You can engage with us in several ways: join our Customer Design Partner program to help refine and validate the most impactful agentic workflows, participate in early access initiatives focused on execution-oriented capabilities, and work with us to shape the governance and policy models that will guide enterprise-scale AI execution.

Moving forward with confidence

We delivered on our commitments. We established a foundation of trust. And we are now stepping into the next phase with clarity.

The future of AI on the mainframe centers on purpose-built enterprise intelligence, where your choice of AI models, your institutional knowledge and operational practices, your people, and your chosen platforms work together to drive intelligent execution across the enterprise. In this model, the mainframe is not an isolated environment. It is a fully governed participant in enterprise AI strategies, capable of supporting intelligent execution across hybrid systems.

Organizations that embrace this direction will not simply modernize their mainframe environments. They will unlock new ways to operate complex workloads with greater visibility, control, and intelligence.

And this is only the beginning. Before you run AI on your most essential platform, BMC First.

Are you truly ready to move from AI insight to AI-driven execution on the mainframe?
Assess your organization’s AI readiness and maturity level—register for an AI Readiness Discovery Workshop to take the next step.

Most organizations don’t start by thinking about service orchestration. They start with symptoms: a morning dashboard showing incomplete data, an SLA missed because an upstream process ran late, or a cross-team Slack thread at 7 AM trying to figure out what broke and why. 

By the time the word “orchestration” enters the conversation, the workarounds have usually been in place for months. And that’s exactly the problem—those workarounds are invisible because they look like “process.” 

The real distinction isn’t between working and broken systems. It’s between environments that are quietly compensating for a lack of coordination and those built on a foundation that can actually support end-to-end execution. That’s where the difference between scheduling and orchestration begins—and why it matters.

Where Job Scheduling Hits Its Ceiling

Technically there’s nothing wrong with the built-in automation tools that come with modern applications. ERP systems, CRMs, data warehouses, and large database platforms all ship with native scheduling capability. For environments that aren’t particularly complex, those tools do exactly what they’re supposed to do: automate jobs within the boundaries of the application that owns them. 

The problem starts at the boundary. 

When a business outcome requires work that transcends multiple applications – supply chain operations, financial close, ML pipeline execution – there is no common coordination layer to manage it. Each application scheduler knows its own world and nothing else. The result is a set of individually automated workflows that have no shared understanding of each other’s state, progress, or failure. 

This is the core architectural gap that service orchestration addresses. A control plane above existing schedulers manages end-to-end execution with business objectives as the organizing principle. 

What Cross-System Coordination Actually Looks Like in Practice

A customer submits an application through a web portal. From there, the process fans out across a stack of systems that have very little in common: data validation runs through an API service in Kubernetes containers; identity verification calls out to a third-party SaaS provider; risk scoring runs on a model deployed in the cloud; the actual account gets created in a core banking system that lives on-premises; the CRM updates the customer profile; compliance documentation gets generated and archived. 

That’s six distinct environments – cloud, on-premises, containerized infrastructure, SaaS – all participating in a single business transaction. And that’s the simplified version. 

Now add the business constraint that makes this genuinely difficult to manage: the customer expects the whole process to complete within one hour of submitting their application. To be clear, that’s a business commitment – and it applies to the entire chain, not to any individual step. 

No application scheduler was built to coordinate that topology under that deadline. Each one was designed to manage its own job queue, not to understand where it sits in a larger business process or what happens downstream if it falls behind.

How the Problem Reveals Itself Operationally

Organizations don’t usually discover this gap through architectural reviews. They discover it through patterns of operational friction that gradually become normalized. 

The most telling early signal is what might be called the time buffer contract. When data teams and application teams operate without a common coordination layer, they start negotiating informal timing agreements. The data from the application layer usually lands by 4 PM, so the ETL pipelines are configured to kick off at 5, with an hour of buffer built in as insurance. 

That buffer looks like responsible planning, but it masks missing dependency management. It fails in both directions – if the upstream data is late, the downstream pipeline starts anyway and processes whatever is there; if the data arrives early, the pipeline sits idle waiting for a clock to tick. 

The second pattern is more disruptive. A data pipeline ingests records from a CRM and an ERP expecting a complete dataset – a million records, in a typical case. An upstream workflow encountered a problem, so now the data arrives incomplete. The pipeline has no way to know this, so it runs to completion and updates business dashboards with partial data. No one finds out until a business user notices something looks wrong. 

What follows is a cross-team reconstruction effort: Slack messages, conference calls, manual tracing of execution steps across multiple tools to figure out where in the chain the problem originated. This kind of incident response – teams scrambling across disconnected systems to piece together what happened – is one of the clearest indicators that an organization has outgrown its current approach to workflow coordination. 

The absence of process lineage across the full stack – from the business application layer down through the data layer – is what makes these incidents so difficult to resolve. As Basil Faruqui, Sr. Director of Product Marketing at BMC Software, puts it: “You cannot manage what you can’t see.” And you can’t see what no single tool was designed to show you. 

What Orchestration Requires in the Modern Era 

Listing orchestration capabilities as a feature set understates what’s actually required. The architecture needs to be a direct response to the failure modes that siloed scheduling creates. 

The first requirement is broad environmental support. A control plane that only works in cloud environments, or only connects to certain application types, is just another silo with a better UI.

Orchestrating a business outcome end-to-end is complex and needs to span SaaS applications, multi-cloud infrastructure, on-premises data centers, and even mainframe systems in some cases. For many industries like banking, financial services and insurance mainframe support isn’t optional. 

The second requirement is end-to-end visibility with process lineage. Knowing that a job completed isn’t the same as knowing whether the business service it belongs to is on track. Effective visibility means being able to trace the state of a workflow across every system it touches, in real time. 

The third requirement is SLA management that carries business context. A notification that says a process is running late is less useful than a notification that says a supply chain workflow is running late and will affect five downstream business services if it doesn’t recover within the next 20 minutes. The former tells you something is wrong. The latter tells you what to do about it. 

Self-healing is the fourth requirement, and it’s where organizational knowledge becomes operational policy. When a workflow step fails because of a transient network issue, someone investigates and decides to rerun the step. That resolution should be capturable as a policy – and increasingly, AI-enabled capabilities can identify that the same failure pattern has occurred before, surface the historical remediation, and automate the response with human approval where appropriate. The goal is to build institutional knowledge into the platform rather than relying on it to live only in the heads of the people who were on shift when the incident happened. 

Finally, CI/CD integration matters more than it’s often given credit for. Orchestration is, at its core, a team sport. Business users, application teams, and operations teams all have a stake in how workflows are defined, tested, and deployed. Treating orchestration workflows as code – with version control, automated testing, and deployment pipelines consistent with modern DevOps practice – isn’t a nice-to-have. It’s what makes the platform manageable at all scales.

AI Workloads Break the Pipeline Model 

One of the more common misconceptions in this space is that AI and ML workloads can be handled by extending existing data pipeline tools. The assumption is that model training is just a compute-intensive batch job, and inference is just another API call. Both assumptions fall short. 

Consider a customer churn detection model. The training data has to be sourced from systems that aren’t data systems at all – ERP, CRM, custom applications, potentially social media feeds. Getting that data into a staging environment already requires coordination across the application layer. Then the model training pipelines need to run. So far, that’s two layers of coordination before any inference happens. 

Once the model is in production and identifies customers at risk, the business response – sending a promotional offer, for example – doesn’t happen inside the model. It happens in the CRM and ERP layer where customer profiles and promotional workflows live. The model is one step in a business process that starts with applications and ends with a business action. 

Confining AI workloads to a data pipeline tool means the platform can manage the middle of that chain but not the ends. The data sourcing and the downstream business response are both outside its scope. Siloing the AI workload in a single tool leaves you with a system that can’t map or control the process from source to business action. 

This matters even more as agent-based architectures become more common. Organizations are already deploying agents that take on discrete tasks within larger processes. A workflow like order-to-cash will not be handed over to agents in its entirety anytime soon – but agents are increasingly handling specific components of it. That means an orchestration platform needs to be able to invoke agents, pass them the right context, enforce guardrails, and manage their execution according to the same SLAs that govern the rest of the workflow. For organizations with active agent deployments, agent orchestration is already a present consideration. 

Rethinking the Question

When organizations run into workflow coordination problems, the instinct is to ask whether they need better automation. More often, what they actually need is orchestration. 

Better automation improves individual jobs. Orchestration manages the relationships between them. For workflows that stay within the boundaries of a single application, better automation is probably the right answer. For workflows that span systems, teams, and environments to deliver a business outcome with a defined deadline, automation alone can’t provide what’s needed. 

The symptoms are identifiable. Time buffers masquerading as process discipline. Partial data incidents that require multi-team investigation to trace. SLA breaches that can’t be attributed to a specific failure point because no single tool has visibility across the full chain. ​​None of these are signs that automation needs tuning. They point to a missing control plane. 

The distinction matters because building more automation on top of a coordination gap doesn’t close the gap. It adds more things to coordinate.

A data pipeline is a series of automated steps for moving data from one or more sources to a designated destination, often transforming it along the way. Raw, disparate pieces of data enter one end, undergo processes like cleaning, restructuring, and enrichment, and emerge at the other end as usable insights. Data pipelines are the foundation of every analytics dashboard, machine learning model, and data-driven operational decision. 

You could think of a data pipeline like an airport baggage system: bags (data) enter the conveyor system, get scanned (validation), sorted (transformation), and routed to the correct flight (destination database). If one belt jams, everything backs up—just like a bottleneck in a data pipeline.

What’s the difference between a data pipeline and ETL?

A data pipeline and an ETL pipeline are not the same thing, though the terms are often used interchangeably.

A data pipeline is an umbrella term for any set of processes that move data from one system to another. This includes simple data ingestion, real-time streaming, batch processing, and complex multi-step workflows. 

An ETL pipeline (Extract, Transform, Load) is a specific type of data pipeline. Its purpose is to extract data from sources, transform it into the right format, and load it into a destination system like a data warehouse or database. All ETL pipelines are data pipelines, but not all data pipelines are ETL pipelines. A pipeline that moves raw data without transforming it, or streams data in real time, is still a data pipeline—but not an ETL pipeline. 

Is SQL a data pipeline?

No. SQL (Structured Query Language) is a language used to query, manage, and manipulate data in relational databases—it’s a tool, not a workflow. 

A data pipeline is the full automated process that moves data from one place to another. SQL can be used inside a data pipeline to filter, join, or transform data, but SQL doesn’t constitute the pipeline itself. If you’re building a house, SQL is the hammer and saw—essential tools, but not the entire construction project. 

Why do data pipelines matter?

Data pipelines are essential because organizations collect data from too many sources—customer interactions, social media, sales transactions, website logs, IoT devices, and internal applications—to manage manually. Without a systematic way to collect, process, and deliver this data, it quickly becomes unmanageable rather than useful. 

Data pipelines power the data-driven decisions we encounter every day: personalized recommendations, real-time fraud alerts, and analytics dashboards. For DataOps teams specifically, pipelines help ensure the reliability, scalability, and governance that organizations need to use data confidently instead of being overwhelmed by it. 

What are the benefits of a well-executed data pipeline?

Data pipelines don’t just move data—they make it fit for purpose, delivering it where and when it’s needed. Five benefits go beyond the basic transport function: 

Enabling analytics and business intelligence: Pipelines feed cleaned, structured data into data warehouses and analytical platforms, allowing analysts to discover trends, identify opportunities, and monitor performance. 

Fueling machine learning and AI: AI models require large volumes of high-quality, pretreated data. Data pipelines help ensure models get the data they need to learn and make accurate predictions. 

Ensuring data quality and governance: As data gets cleaned, validated, and standardized, data pipelines support greater confidence in data-driven decisions. They also enforce governance rules for compliance and security. 

Improving operational efficiency: By integrating data from various systems, pipelines provide a holistic view of operations, automating workflows and flagging issues in real time. 

Facilitating data democratization: Pipelines make data accessible and understandable to more people within an organization, empowering more teams to make informed decisions by connecting data sources to decision-makers. 

Without strong data pipelines, organizations can fly blind—making decisions based on intuition rather than evidence. 

What are the core components of a data pipeline?

Every data pipeline is made up of five essential components. Understanding these elements reveals how data flows and transforms from its original source to its final destination. 

1. Source: Where your data lives

Source is the origin point of your data—the starting line of the pipeline. The type of source determines how the data will be extracted. Common data sources include: 

• Databases: relational (e.g., MySQL) and NoSQL (e.g., MongoDB) 
• Applications: CRM systems (e.g., Salesforce), ERPs (e.g., SAP), marketing automation platforms 
• APIs: third-party services, social media platforms, public data feeds 
• Files: CSVs, JSON, XML, Parquet, Avro—often stored in cloud storage (e.g., Azure Blob) 
• Streaming data: real-time event streams from IoT devices, website clicks, financial transactions 
• Logs: system logs, web server logs, application logs 

2. Extraction: Getting your data out

Extraction is the step where data is pulled from its original source—often involving different file types, formats, and sometimes unstable or slow source connections. The goal of extraction is to get a raw copy of the data without altering the source system.

Three common extraction methods: 

• Batch extraction: Data is pulled in chunks at scheduled intervals (e.g., nightly, hourly). Used for data that doesn’t change frequently or where immediate updates aren’t critical. 
• Incremental extraction: Only new or changed data since the last pipeline run is extracted. Faster than full extraction, but requires change-detection techniques like timestamps, version numbers, or Change Data Capture (CDC). 
• Streaming extraction: Data is continuously pulled from sources as events occur, typically using message queues or event streaming platforms like Kafka or Kinesis. 

3. Transformation: Cleaning and shaping your data 

Transformation is usually the most complex part of a data pipeline. This is where the messiness of raw data gets cleaned and turned into actionable information. The goal is to ensure data quality, consistency, and suitability for the intended destination. Common transformation steps include:

• Cleaning: Removing duplicates, handling missing values, correcting errors 
• Filtering: Selecting only relevant rows or columns 
• Aggregating: Summarizing or categorizing data (e.g., total sales per day) 
• Joining or merging: Combining data from multiple sources using common keys (e.g., joining customer data with order data) 
• Standardizing or normalizing: Ensuring consistent data types, formats, and units (e.g., standardizing currency codes) 
• Enriching: Adding new data points by looking up external information or deriving new features (e.g., adding geographical data based on an IP address) 
• Structuring: Converting unstructured or semi-structured data into a structured format

4. Loading: Delivering your data

Once extracted and transformed, data is loaded into the system where it will be used—a database, data warehouse, or analytics platform—so applications, reports, and analytics tools can access it. Three common loading strategies:

• Full load: The entire destination table or dataset is overwritten with new, transformed data. Simpler to implement, but resource-intensive for large datasets. 
• Incremental load: Only new or changed records are appended to the destination. More efficient, but requires diligent management of data updates and deletions. 
• Streaming load: Data is continuously loaded as it arrives, often into specialized real-time databases or analytical engines. 

5. Destination: Where your data rests

Destination is the final storage location where processed data is available for consumption by analysts, data scientists, and applications. Common destinations include: 

• Data warehouses: Optimized for complex queries and reporting on large volumes of historical data (e.g., Snowflake) 
• Data lakes: Hold raw or semi-structured data at scale for advanced analytics and machine learning (e.g., Azure Data Lake Storage) 
• Databases: Operational systems for everyday applications like websites or apps (e.g., MongoDB) 
• Business intelligence (BI) tools: Software that turns data into dashboards and reports (e.g., Tableau) 
• File storage: Simple storage for archiving or later processing 

Note: loading and destination are conceptually distinct but closely related in practice. Loading is the action of writing processed data into a system; destination is the place where that data ends up and stays until it’s needed. Loading is like putting groceries into the fridge. Destination is the actual fridge. 

Bringing it all together—an e-commerce scenario: Source (CRM database) → Extract (SQL query for new customer orders) → Transform (clean addresses, calculate total order value, join with product details) → Load (insert into data warehouse) → Destination (data warehouse for reporting). The entire process runs automatically on a schedule, ensuring a continuous flow of refined information. 

What are the three types of data pipelines?

Just as there are different ways to transport goods, there are different types of data pipelines—each optimized for specific needs around speed, volume, and complexity. 

Batch processing: the daily shuttle

Batch processing works like a commuter train on a defined schedule. It picks up a large group of passengers (data) at scheduled times and delivers them to their destination. Data is collected over a period, then processed as a single, large batch.

• Characteristics: High latency (data may be hours or days old); processes large volumes efficiently; often scheduled during off-peak hours 
• Use cases: Nightly reports, monthly financial summaries, loading historical data into a data warehouse, running complex analytical jobs that don’t require immediate results 

Real-time streaming: the instant delivery service

Real-time streaming works like an instant delivery service. As soon as a package (data event) is created, it’s picked up, processed almost immediately, and delivered to its destination with minimal delay.

• Characteristics: Low latency (data is typically seconds or milliseconds old); handles continuous streams of individual events; requires infrastructure optimized for speed 
• Use cases: Fraud detection, real-time personalized recommendations, IoT sensor data analysis, monitoring system health, live dashboards

Hybrid approaches: the best of both worlds

Many organizations combine batch and streaming pipelines. Two common hybrid patterns:

• Lambda architecture: Uses separate batch and streaming layers. The streaming layer provides real-time views; the batch layer processes historical data for accuracy and completeness. Results from both are then merged. 
• Kappa architecture: A simpler approach that handles both real-time and historical processing using a single streaming engine, often by replaying streams.

Choosing the right pipeline type depends entirely on your business requirements for data freshness, volume, and complexity. 

What are the main challenges in data pipeline management?

Building a data pipeline is one thing; keeping it running smoothly and reliably is another. Here are the most common data pipeline management challenges—and the best practices for each. 

Ensuring data quality: Data can be inconsistent, incomplete, or incorrect at the source, leading to garbage in, garbage out. Best practices: implement data validation rules at every stage; use data profiling tools to understand data characteristics; create data quality checks within transformation steps; use data observability platforms to detect anomalies early. 

Scalability and performance: As data volumes grow or requirements shift to real-time, pipelines can become slow or break entirely. Best practices: design for scalability from the outset; use distributed processing frameworks (e.g., Apache Spark); use cloud-native services that scale automatically; implement incremental loading strategies; optimize queries and transformation logic. 

Security and compliance: Data pipelines handle sensitive information, requiring stringent security and compliance measures. Best practices: encrypt data at rest and in transit; implement strong access controls (least privilege); audit data access and changes; redact sensitive data during transformation where necessary. 

Monitoring and alerting: Without proper monitoring, pipeline failures or data quality issues can go undetected and impact downstream applications. Best practices: implement comprehensive monitoring for pipeline health, performance metrics, and data quality metrics; set up automated alerts for critical failures, latency breaches, or data anomalies; use dashboards for operational visibility.

What are the top data pipeline use cases?

Data pipelines are versatile, powering applications across industries. Seven key examples of how pipelines transform raw data into actionable insights:

• Business intelligence and reporting: Aggregating sales data, customer demographics, and marketing spend into a data warehouse for daily, weekly, or monthly reports and dashboards that guide strategic decisions 
• Customer 360-degree view: Combining data from CRM, sales, support, and marketing platforms to create a holistic profile of each customer, enabling personalized experiences and targeted campaigns 
• Fraud detection: Ingesting real-time financial transactions, social media activity, and user behavior to identify suspicious patterns and instantly flag potential fraud 
• IoT analytics: Collecting streams of data from sensors (e.g., factory machines, smart city devices) to monitor performance, predict maintenance needs, and optimize operations 
• Personalized recommendations: Processing user browsing history, purchase data, and demographic information to power content recommendations on streaming platforms 
• Log analytics: Consolidating logs from applications and servers to monitor system health, troubleshoot issues, and detect security threats in real time 
• ML model training: Preparing, cleaning, and feeding large datasets to machine learning models for tasks like image recognition, natural language processing, or predictive analytics

What makes a data pipeline modern?

Modern data pipelines share several key characteristics that distinguish them from traditional approaches:

• Cloud-native and serverless: Modern pipelines use cloud services (e.g., AWS, Azure) that scale automatically and reduce operational overhead 
• ELT-first approach: Instead of transforming data before loading, modern pipelines often load raw data into a cloud data warehouse (e.g., Snowflake, BigQuery) and transform it within the warehouse using SQL—leveraging the destination’s compute power and enabling greater flexibility 
• Data lake integration: Modern pipelines frequently integrate with data lakes to store vast amounts of raw, multi-structured data for future use, advanced analytics, and machine learning 
• Real-time capabilities: Streaming technologies like Apache Kafka and Amazon Kinesis increasingly power modern pipelines for immediate insights 
• Orchestration and automation: Tools like Apache Airflow or cloud-native orchestrators automate scheduling, manage dependencies, and monitor pipeline health 
• Data observability: Modern pipelines go beyond basic monitoring to actively track the health, quality, and lineage of data—detecting anomalies and ensuring data trustworthiness 
• Data governance and security by design: Security, privacy, and compliance are built into the pipeline architecture from the start, not added as an afterthought 
• Flexibility and agility: Modular components make modern pipelines easier to adapt to new data sources and changing business requirements 

How do traditional and modern data pipelines compare? 

Modern data pipelines are designed to be more agile, scalable, cost-effective, and resilient than traditional approaches. Here’s a side-by-side comparison:

What are the most common data pipeline tools?

The data pipeline tool landscape is broad and evolving. Four categories to consider: 

Data integration platforms: Comprehensive ETL/ELT tools, often with visual interfaces and pre-built connectors. Examples: Talend, Informatica. A good fit for teams that want an end-to-end solution without heavy coding; businesses with multiple data sources needing pre-built connectors; and organizations prioritizing ease of use and quick deployment. 

Cloud-native services: Major cloud providers offer services specifically designed for scalable data pipelines. Examples: Amazon Kinesis, Google BigQuery. A good fit for companies already invested in a specific cloud ecosystem; teams needing scalable, cost-effective solutions for batch and streaming; and use cases requiring tight integration with other cloud services. 

Open-source frameworks: Flexible, developer-friendly options for orchestration and processing. Examples: Apache Airflow, Apache Kafka. A good fit for engineering teams with strong technical skills; organizations wanting maximum flexibility and control; and scenarios with custom requirements or large-scale data processing needs. 

Enterprise workflow orchestration platforms like Control-M are ideal for teams that need to orchestrate BigQuery data pipelines alongside mainframe, cloud, and on-premises workloads — providing SLA management and audit capabilities that native GCP scheduling does not offer. A good fit for large enterprises with complex, mission-critical workflows; businesses needing robust scheduling, compliance, and audit capabilities; and teams managing cross-platform jobs spanning mainframe, cloud, and on-premises systems with high reliability requirements. 

For large enterprises with mission-critical data pipelines, a data pipeline orchestration platform like Control-M provides robust scheduling, cross-platform dependency management, and compliance capabilities that open-source tools alone cannot deliver at scale.

The right tool depends on your budget, team skill set, data volumes, and real-time requirements. 

5 key data pipeline takeaways

Whether you’re learning about data pipelines for the first time or refreshing on the fundamentals, here are the most important points to carry forward: 

1. Understand the pipeline lifecycle

A data pipeline isn’t just about moving data—it involves extraction, transformation, loading, orchestration, monitoring, and governance.

2. Orchestration is key

Orchestration ensures repeatability, scalability, and observability across the entire data pipeline. 

3. Embrace automation and CI/CD

Integrating data pipelines into CI/CD workflows enables faster, safer changes. DataOps applies DevOps principles to data: automated testing, deployment, and version control for pipelines.

4. Prioritize data quality and monitoring

Pipelines can fail silently if data quality isn’t checked. Implement validation, anomaly detection, and alerts to catch issues early. Observability is critical for trust and compliance.

5. Design for scalability and flexibility

Modern data pipelines must handle batch and streaming, adapt to schema changes, and scale with data growth. Cloud-native and modular architectures are essential for agility. 

Bonus tip: Learn the full ecosystem—ETL tools, orchestration frameworks, cloud services—and how they fit together to build pipelines that serve your organization at scale. 

Frequently asked questions about data pipelines

How is a data pipeline different from a data warehouse? 

A data pipeline is the process that moves and transforms data; a data warehouse is a destination where processed data is stored for analysis. Data pipelines feed data warehouses—the pipeline handles transport and preparation, while the warehouse provides the storage and query environment. The two are complementary, not interchangeable. 

What skills do you need to build a data pipeline? 

Building a data pipeline typically requires proficiency in SQL and at least one programming language (Python and Scala are common), familiarity with data transformation concepts (ETL/ELT), and working knowledge of at least one orchestration or workflow tool. Cloud platform experience (AWS, Azure, or GCP) is increasingly essential for modern data pipeline work. 

How do you test a data pipeline? 

Data pipeline testing involves validating data at multiple stages: confirming that extraction produces complete, unaltered source data; verifying that transformation logic produces expected outputs; and checking that loaded data matches expected row counts, data types, and values at the destination. Automated testing frameworks integrated into CI/CD pipelines can run these checks continuously and catch regressions early. 

What is data pipeline latency? 

Data pipeline latency is the time between a data event occurring at the source and that data becoming available at the destination. Batch pipelines have high latency (hours or days); real-time streaming pipelines have low latency (seconds or milliseconds). Acceptable latency depends on the use case—fraud detection requires near-zero latency, while monthly financial reporting can tolerate daily batch processing. 

How does BMC Control-M help manage data pipelines? 

BMC Control-M is an enterprise workflow orchestration platform that automates, schedules, and monitors complex data pipelines across environments—including mainframe, cloud, and on-premises systems. Control-M provides centralized visibility, dependency management, and audit capabilities, making it well-suited for mission-critical pipelines that require high reliability, compliance, and cross-platform coordination. 

The views and opinions expressed in this post are those of the author and do not necessarily reflect the official position of BMC.