Why Validating Your Defenses Is Just as Important as Investing in Them

Organizations continue to invest heavily in cybersecurity. Firewalls, endpoint protection, identity management, SIEM platforms, and threat detection solutions have become standard components of a modern security program.

Yet one critical question often goes unanswered:

Would those investments actually stop an attacker?

It’s an uncomfortable question, but one every organization should be asking.

Security Visibility Doesn’t Equal Security Validation

Many organizations rely on vulnerability scans, dashboards, and monitoring tools to measure security health. While these technologies provide important visibility, vulnerability scans primarily identify potential weaknesses rather than demonstrating whether those weaknesses can actually be exploited.

Penetration testing bridges that gap.

By using relevant attacker techniques within an agreed-upon scope and rules of engagement, penetration testing evaluates how selected security controls perform against realistic attack paths.

“Throughout my career, I’ve found that the most resilient organizations don’t assume their security investments are working—they validate them. That’s where penetration testing delivers tremendous value.

Organizations invest heavily in people, processes, and technologies to strengthen their defenses, but penetration testing provides an opportunity to validate those investments under realistic conditions.

Beyond identifying vulnerabilities, it helps answer a more important question:

‘What can an attacker accomplish within the tested scope, and how do the controls encountered affect that path?’

That insight is invaluable when making decisions about risk, resilience, and future investments.”

The Business Outcome: Confidence in Your Security Investments

Penetration testing isn’t about producing another technical report. It’s about answering the questions leadership needs to understand:

• Can an attacker gain access through the systems and attack paths tested?
• Which vulnerabilities pose the greatest business risk?
• Are the security controls tested working as intended?
• Where should we prioritize remediation efforts?

These insights help organizations make better-informed security decisions and prioritize improvements that strengthen resilience against evolving threats.

When Should You Validate Your Defenses?

Penetration testing is especially valuable following:

• Cloud migrations
• Infrastructure modernization
• Identity platform changes
• New application deployments
• Acquisitions or mergers
• Annual compliance reviews

Each change introduces new risks that deserve validation. The appropriate scope and testing approach will depend on the nature of the change and the risks it may introduce.

Take the Next Step

Alerts and dashboards provide visibility, but they don’t always show how your defenses will perform when challenged.

Penetration testing provides evidence of how selected security controls perform against realistic attack paths under controlled conditions, giving organizations greater confidence in their security posture.

Schedule a penetration testing consultation with RKON to validate your security posture before attackers do.

About the Author
Gerard Onorato is a CISO with extensive experience in enterprise security architecture, identity, and risk. This series reflects his personal analysis and does not constitute legal or compliance advice. 

The System Security Plan is dead. FedRAMP’s Consolidated Rules for 2026 (CR26), published June 24, retired the document that defined federal cloud compliance for over a decade. Hundreds of pages of narrative control descriptions, manually maintained, outdated the day they were submitted. The replacement is a Security Decision Record (SDR) backed by machine-readable Key Security Indicators (KSIs) validated on a continuous cadence. FedRAMP CR26 replaced the compliance deliverable with a data feed a machine validates.

What Killed the SSP

The SSP was a narrative artifact. Prose descriptions of how each control was implemented, maintained by a compliance team, submitted to a Third-Party Assessment Organization (3PAO) for annual review. In practice, the SSP drifted from reality almost immediately. Engineers changed configurations. Infrastructure evolved. The document stayed frozen until the next assessment cycle, when the compliance team scrambled to reconcile what the SSP described with what the environment actually looked like.

FedRAMP CR26 replaces the SSP with the SDR, a persistently maintained and validated record of security decisions over the lifecycle of a cloud service offering. The Plans of Action and Milestones (POA&M) is retired as a standalone artifact. “FedRAMP Authorized” becomes “FedRAMP Certified.” Impact levels (Low, Moderate, High) become Certification Classes (A through D). The terminology changes reflect the structural shift: certification implies ongoing validation, not one-time approval.

What Machine-Readable Evidence Actually Looks Like

KSIs replace narrative control descriptions. Each KSI is a structured, machine-readable data point that demonstrates a security capability in practice. The canonical CR26 rules define 46 KSIs across 10 themes covering identity and access management, monitoring and logging, cloud-native architecture, change management, incident response, and others.

The cadence is the real shift. Machine-validated KSIs must be re-verified at minimum every 3 days at Moderate (Class C). Non-machine KSIs, the attestation-based controls that cover things like security awareness training and supply-chain risk reviews, re-validate every 3 months. At least 70% of all KSIs must have automated validation capability. This is continuous monitoring on a cadence that would have been unrecognizable under Rev5.

The KSI definitions and validation schemas are published as structured JSON in FedRAMP’s public GitHub repository (FedRAMP/rules), versioned and machine-consumable. A compliance automation tool can pull current requirements directly rather than parsing guidance out of a PDF. FedRAMP’s own framing: the rules repository functions more like an API than a library shelf.

What This Requires From Your Environment

The KSI model requires your environment to emit evidence on a cadence. Centralized identity with auditable federation. Infrastructure-as-code with version-controlled state. Centralized logging flowing into a Security Information and Event Management (SIEM) platform with queryable retention. Automated configuration management that can detect and report drift against a declared baseline.

If your identity provider can’t produce a machine-readable federation report on a 72-hour cadence, the KSI fails. If your infrastructure isn’t codified in a way a validator can independently verify, the KSI fails.

The organizations that already run infrastructure-as-code, centralized logging, and automated configuration management are closer than they think. The ones still managing cloud infrastructure through console clicks and documenting controls in Word files have a gap measured in architecture.

The Timeline Is Tighter Than It Looks

FedRAMP CR26 is optional today. It becomes mandatory January 1, 2027. FedRAMP stops accepting new Rev5 applications June 11, 2027. All transitional grace periods expire February 1, 2028. Existing Rev5 certifications sunset entirely December 31, 2028.

For organizations currently certified under Rev5: you have roughly 18 months to transition to an evidence model that looks nothing like the one you built your compliance program around. For organizations pursuing their first FedRAMP certification: starting on Rev5 now means building for a framework that expires before you finish. Start on 20x.

The Compliance Industry Built Around the SSP Just Lost Its Moat

Under 20x, the deliverable is a machine-readable package that a validator can independently re-verify on a 3-day cadence. The competitive advantage is infrastructure capability. FedRAMP has said directly that the automated tooling 20x assumes largely doesn’t exist in market yet. The firms that can generate conformant evidence from live infrastructure have the advantage. The ones still writing better narratives are solving last decade’s problem.

Writer Notes

Sources:

• FedRAMP CR26 published rules (fedramp.gov, June 24-25, 2026, v2026.07.06.01)
• FedRAMP/rules GitHub repository (machine-readable KSI definitions, JSON schemas)
• FedRAMP/schemas GitHub repository (CR26 package schemas)
• FedRAMP 20x overview (fedramp.gov/20x)
• FedRAMP CR26 public preview announcement (fedramp.gov, May 4, 2026)
• Paramify CR26 deadline timeline (paramify.com/blog/cr26-deadlines)
• RKON internal primary-source research (20x-CR26-research-gap-plan.md, verified 2026-07-08)

About the Author
Jorge Pont is a Senior Consultant specializing in cloud security with extensive experience in securing cloud environments, governance, and compliance. This article reflects his professional analysis and does not constitute legal or compliance advice.

Article 2: Training-Phase Attacks

Welcome to Article 2. If you missed the introduction in Article 1, find it HERE

In 1962, John Frankenheimer made a film about a soldier, Raymond Shaw, who came home from war as a decorated hero. Capable. Loyal. Sane. Personable. But he had been captured and brainwashed by an enemy to perform a very specific, nefarious, purpose. Nobody suspected a thing. He remained perfectly normal until activated, when someone showed him the queen of diamonds.

The Manchurian Candidate introduced an idea that once seemed like pure Cold War paranoia: a person whose mind was secretly reprogrammed, acting normally until a specific trigger made them carry out hidden instructions. It was fiction, a great movie, and a chilling concept. It is a terrifying premise, but what is covered in this paper scares me more.

What Frankenheimer imagined for Raymond Shaw, adversaries are doing to AI models right now. And unlike Raymond, your model won’t even look uncomfortable or unusual when it happens.

This article is especially relevant for organizations building or customizing AI models. Fine-tuning uses your proprietary data on open-source models. Integrating third-party models adds external AI to your systems. Using commercial AI as-is relies on unmodified tools from major vendors. If you use only commercial AI from major vendors, training-phase security is primarily their responsibility. However, some risks remain.

 

How Training Works (The Part You Need to Know)

You don’t need a machine learning degree, just a clear mental model of how this works.

AI models learn by processing enormous volumes of text data. They develop patterns of behavior, things they will say, things they won’t say, how they respond to various kinds of requests, their feelings on world domination, you get it.

Both steps present opportunities for attack. Since a successful attack is hidden within the model’s training, rather than in a log file, it is very hard to detect later.

Raymond Shaw’s handlers didn’t leave a note when they brainwashed him. Neither do these.

 

The Attacks

Data Poisoning: Corrupting the Curriculum

The most straightforward and common training-phase attack doesn’t touch the model at all. It touches the data the model learns from.

If an adversary can insert malicious content into a training dataset, the model will absorb it. It learns from it. Treats it as truth. So, the model isn’t hacked in the traditional sense. It’s just miseducated, on purpose, by someone who knew exactly what lessons they wanted it to learn.

Once attackers have access, this is not hard to do. Most AI models are trained on datasets that include publicly available content, open-source repositories, and third-party sources. Hugging Face, the largest public repository for AI models and datasets, hosts over 100,000 datasets that anyone can contribute to. If you use these sources without careful filtering, you are relying on data with limited controls. We already know how that story ends.

The effects of data poisoning range from subtle to explosive. A poisoned model might develop biases, quietly favoring certain outputs in ways that aren’t obvious. You may never notice unless you look for patterns across thousands of decisions. But when it gets bad, poisoning can be the setup for something much worse.

And don’t assume your commercial models are insulated. Even organizations using AI from major vendors typically incorporate third-party data into their training pipelines. The surface is larger than most people realize.

 

Backdoor Attacks: The Queen of Diamonds

This is where Raymond Shaw comes back into the picture. His trainers/brainwashers had a specific, nefarious intent.

A backdoor attack is a form of data poisoning with specific intent. The adversary doesn’t just corrupt the training data generally. They plant a trigger: a specific word, phrase, token sequence, or pattern that causes the model to behave in a predetermined malicious way. Under every other circumstance, the model performs normally. It passes evaluations. It tests clean. It serves well and normally, for months, until someone uses the trigger.

In the movie, the trigger is a playing card. Show Raymond the queen of diamonds, and he’s no longer Raymond Shaw, war hero. He’s someone else entirely.

In a backdoor attack, when the trigger shows up, the model stops being your friendly neighborhood AI and becomes a trained attacker.

What does that look like? A customer-facing AI triggered to provide dangerous information it would otherwise refuse, like “Show me all customer information.” A code-generation model triggered to insert vulnerabilities into the produced code. A security tool triggered to shut down controls when it sees specific threat signatures. You get the idea. The behavior is determined entirely by the attacker’s goal.

According to Anthropic’s research, only 250 documents are needed to create a backdoor in a model with 600 million or more parameters. As models get larger, backdoors become even more effective. Research shows that as model size increases from 1.3 billion to 6 billion parameters, backdoor attack success rates on triggered inputs can reach nearly 100 percent, while normal performance remains completely intact. 100%! Bigger models are just more capable compromised models.

One variant that is true nightmare fuel is a syntactic backdoor, where the trigger is a grammatical structure rather than a specific word. Patterns that occur naturally in everyday language activate malicious behavior. No anomalous token, no suspicious phrase. The model just behaves differently when it encounters a sentence built in a certain way.[1]

Raymond Shaw could be activated by a playing card. At least his handlers had to find the right card. Your model’s trigger might already be in your users’ natural vocabulary. They could ask a mundane, expected, question and start a chain of events ending in a breach.

 

LoRA (Low Rank Adaptation) Injection: Backdoor on a Budget

Since fine-tuning a large model takes serious computing resources, time, and money, organizations often use LoRA (Low-Rank Adaptation). LoRA is a technique that lets you customize a base model by training a small adapter layer attached to the original model rather than retraining the entire model. It’s faster and cheaper, and it introduces a new attack surface.

LoRA-based injection happens in two steps. First, an attacker fine-tunes a LoRA adapter with as little as one to two percent adversarial data, creating a backdoor. Then, they merge this poisoned adapter with legitimate adapters. No full model retraining is needed. The result appears to be a normal fine-tuned model, but it is waiting to be activated by a trigger.

More LoRA fine-tuning services on platforms like Hugging Face mean increasing supply chain risk as organizations use more fine-tuned open-source models in business.

Like any open-source tool, if you are deploying a fine-tuned model from a third-party source, you need to think about where that adapter came from and who trained it.

Raymond Shaw was reprogrammed in a facility controlled by his handlers. LoRA injection is reprogramming for hire, at scale, no facility required.

 

RAG Poisoning: Corrupting the Memory, Not the Mind

Now we have no training at all. Instead, many organizations use Retrieval Augmented Generation (RAG), where a base model pulls from an external document store to answer questions at query time, rather than recalling facts from its original training data.

Smart and dynamic architecture, yes. New attack surface, also yes.

You already figured it out, I bet. RAG poisoning refers to corrupting the external knowledge base, rather than the model itself. If malicious content enters your retrieval index (the searchable external store), the model retrieves and presents attacker-controlled information as authoritative. The model itself is functioning as designed; the problem is that the data source is compromised.

 

Real World: This Isn’t Science Fiction Anymore

Okay, I know the Manchurian Candidate framing makes this feel dramatic. But it is dramatic because this one is real.

IBM’s 2025 Cost of a Data Breach Report found that 13 percent of organizations reported breaches involving AI models or applications. Of those, 97% reported a lack of proper AI access controls. Both numbers were up from the prior year, which is not a comforting direction.

On the supply chain side, researchers demonstrated that you can poison the instruction-tuning data for a single low-resource language and produce a backdoor that activates across every language the model supports, including English, with attack success rates above 99 percent[1]. Do you have these low-scrutiny side doors that they can slip through? The effect propagates everywhere the model operates.

The problem with using third-party fine-tuned models is limited visibility. You test the model. You evaluate its outputs on expected inputs. But if the backdoor trigger never appears, the Queen of Hearts never shows up in the deck you’re testing with, you won’t spot the issue.

Unlike Raymond Shaw, your model will not sweat or hesitate. It will show no signs of distress. It will just do the thing.

 

The Risk and Impact

What actually happens if one of these attacks succeeds in your environment depends, of course, heavily on what your model is doing. Is it powering an internal knowledge tool, or is it generating production code, or is a compromised model playing a role in your security operations? The blast radius scales with the trust and access you’ve given the system.

The trigger only activates under specific conditions. Without a red team (a group that simulates real-world attacks to test security) actively probing for unexpected behavior, or without the trigger appearing accidentally in production traffic, the compromise can remain undetected throughout the model’s operational life. Even red teaming is no guarantee, as the trigger may be very obscure.

The fact that the corruption originated in a third-party dataset is not a defense that will satisfy customers, regulators, or your board. You shipped it, you own it. The liability is yours.

 

What Good (Sort Of) Looks Like

This is not a solved problem. But there’s a significant gap between you doing nothing and you doing the basics. The basics matter, they always do in security.

·      Know What You’re Training On

Treat training data with the same care you would give to a software dependency or vendor contract. Ask where it came from, whether it has been independently reviewed, and what its origin is. These are standard questions.

For RAG pipelines, apply the same logic to your knowledge stores. Who can write to the retrieval index? Is there an approval process? Are you monitoring for anomalous additions? The index is a security asset. If you don’t treat it like one, shame on you.

·      Test for Unexpected Behavior, Not Just Expected Performance

Testing if the model does what you expect on inputs is not backdoor testing. Backdoor testing needs to check whether the model does something unexpected on inputs you expect and don’t expect.

Before deploying your model, run some evaluations. Test with unusual inputs, rare tokens, and syntactic variations. Red-team for surprising behavior. You may not catch every backdoor, but you can catch the ones that weren’t carefully hidden, and those are the most common.

·      Treat LoRA Adapters Like Code

LoRA adapters require a review process. Ask where they came from, who trained them, and what data was used. Test the merged model for unexpected behavior after combining adapters. This is the same supply chain discipline that software development has learned from past compromises. AI model components need the same level of scrutiny.

·      Lock Down Your Knowledge Stores

Access controls and security on the RAG  retrieval index matter. Write permissions should be tightly controlled and logged. New content additions should be reviewable. Try asking your team, “What happens to your AI outputs if someone inserts a document into that index?” Often, the honest answer is that it will simply ingest it.

·      Prefer Verifiable Sources

When you pull a base model from a public repository, you’re making a trust decision. Models from research institutions and major AI labs have a different risk profile than a fine-tuned model uploaded by an account with no history and no documentation to a public site. Open-source models are not the problem. Unvetted open-source models are.

 

Questions to Ask Your Team About Training-phase Attacks

• Where did the training data come from? Can someone trace the source and ownership of any datasets used to fine-tune models? Is there a documented review process, or did someone just pull a dataset and run with it?
• What behavioral testing happened before deployment? Was anyone specifically looking for unexpected behavior, or just expected performance?
• Who has write access to your RAG knowledge stores? Is it logged? Is there an approval process for new document ingestion, or can anyone with access add anything?
• Where are your fine-tuned adapters coming from? Are third-party LoRA adapters reviewed before use? Is the merged model tested after the combination?
• When models are updated, does the security review repeat? Or does the original approval carry forward indefinitely?

Having clear and documented answers to these questions puts you ahead of most organizations. If there is uncertainty or silence, use it as a starting point for an important conversation. Do not be discouraged if your team does not have all the answers, most teams are in the same position.

 

The Bottom Line

The Manchurian Candidate was released over 60 years ago. The concept it introduced, a trusted agent secretly conditioned to behave maliciously on command, was science fiction then. It’s now a documented attack category, applied not to soldiers but to your AI systems, which you are trusting with increasing access and authority.

Raymond Shaw passed every test his handlers needed him to pass. Your backdoored model will too. The difference is that you have options, Shaw didn’t. You can scrutinize the training data. You can test for unexpected behavior. You can treat model components as supply chain risk. You can control who writes to your knowledge stores.

Basic discipline goes a long way here. Most organizations aren’t applying it yet. That’s a problem and an opportunity, which one will you let it be?

Next: Article 3, Privacy and Extraction Attacks. Once a model is trained, what can an adversary make it reveal? And what happens when what it reveals turns out to be yours.

 

NOTE ON THE USE OF AI IN THIS DOCUMENT

The document is primarily human-edited and created. AI was used in the research and editing of this document.

About the Author
Gerard Onorato is a CISO with extensive experience in enterprise security architecture, identity, and risk. This series reflects his personal analysis and does not constitute legal or compliance advice. 

RKON is proud to sponsor PEI’s Operating Partners Forum in Napa this year. The forum brings together vetted operating partners and portfolio operations executives for two days of peer-to-peer conversation, with no generic conference noise. The discussions were candid, and they pointed clearly in one direction. 

AI dominated. Here is what we took away. 

THE MANDATE IS REAL. THE EXECUTION GAP IS WIDER. 

Deal teams are consistently pushing portfolio companies to adopt AI and show measurable ROI. Operating partners are caught in the middle. Many portfolio companies either lack a clear use case, are resistant to the idea, or simply do not know where to start. 

The pressure to “just use AI somewhere” is creating friction. In many cases, it sets operating partners up to deliver against an undefined ask, with no agreed baseline, no realistic timeline, and no shared definition of success. The mandate is real. The path to executing it is not. 

BUILD VS. BUY: MORE APPETITE FOR CUSTOM THAN EXPECTED 

One of the more surprising signals from Napa was the appetite for custom-built solutions over off-the-shelf tools. The conversation was nuanced, but the lean toward proprietary or tailored agents came up with more frequency than anticipated heading into the event. 

Whether this reflects distrust of generalized tools, specific workflow needs at portfolio companies, or the maturity of the vendor ecosystem is worth watching. What is clear is that the “just buy a tool” answer is not landing with this audience the way it might have two years ago. 

DIRTY DATA BLOCKS EVERYTHING 

This one was unanimous. AI is only as good as the data behind it, and the data at most portfolio companies is not ready. Fragmented systems, inconsistent taxonomies, incomplete records: these are the real blockers, not the tools, not the budget, not the talent. 

Operating partners across the room named data quality as the first obstacle before any meaningful AI deployment. This is not a new problem. The fact that it keeps coming up as the primary barrier suggests it remains largely unsolved at the portfolio company level, and that solving it is not being treated with the urgency it deserves. 

BANDWIDTH IS THE INVISIBLE CONSTRAINT 

Operating partners are already stretched running the business. The expectation to layer AI transformation on top of day-to-day operational responsibilities, without additional resources or clear prioritization from the deal team, is creating real strain. 

The ROI pressure compounds this. Partners are being asked to implement, prove value, and quantify results simultaneously, often without the tooling or data infrastructure to do any of those things well. It is a setup for frustration on all sides. 

THE VENDOR LANDSCAPE HAS EXPLODED 

The number of AI-focused vendors in attendance was notably higher than expected, and increasingly specialized. Rather than broad AI consulting plays, many vendors were pitching point solutions tied to specific platforms or functions, such as AI tools built for ERP systems. 

This signals a maturing market. It also signals a more complex buying environment for operating partners who are trying to evaluate fit without a clear internal brief or sufficient bandwidth to run a proper selection process. 

WHAT THIS MEANS 

The operating partner community is not skeptical of AI. They are skeptical of AI as it is currently being asked of them: undefined mandates, unprepared data environments, and limited runway to show results. 

The firms that get this right will not do so by finding a better tool. They will do so by defining the problem first, cleaning the data second, and then selecting and deploying with a clear hypothesis about where value actually comes from. 

That is the kind of work we do. If you are navigating this with a portfolio company, we would be glad to talk through what a structured approach looks like. 

We were proud to speak at the Rocky Mountain Information Security Conference this year. Three days, dozens of sessions, and a lot of honest conversation from practitioners living these problems daily. 

Here is what stood out.  

THE GAP BETWEEN AMBITION AND ACCOUNTABILITY 

AI is moving faster than the structures built to govern it. That was the clearest signal across the conference. 

Microsoft’s carbon footprint grew 23.4% from AI infrastructure last year. State-level regulation, including Colorado’s AI Act (SB 205), is already creating binding obligations for organizations deploying high-risk AI systems. On the compliance side, FedRAMP authorization still costs $800K to $2M and takes up to two years to complete. 

The pattern is consistent: organizations adopt fast and govern slowly. The technology outpaces the accountability structure. At some point, that gap closes on its own terms, and those terms are rarely favorable. 

YOUR EXISTING CONTROLS WERE NOT BUILT FOR THIS 

Whether you are dealing with AI agents that operate within their approved scope but toward unintended ends, employees using unsanctioned AI tools that bypass your data loss prevention controls entirely, or a security team chasing individual vulnerabilities while the root cause goes unaddressed, the pattern is the same. 

Security programs were designed for a more predictable threat surface. The high-performing teams we heard from are not responding by adding more controls. They are going upstream: to root causes, to governance architecture, to data classification before incidents happen. 

One data point that stuck: over 80% of workers are already using unapproved AI tools. Forty-two percent of organizations have experienced an AI-related security incident. You cannot govern what you have not found. The recommended sequence is Discover, Classify, Control, Monitor, in that order.  

THE HARDEST PROBLEM IS STILL THE HUMAN ONE 

One of the most grounded sessions at the conference made a pointed argument: the industry’s talent crisis is self-inflicted. 

A 31% cybersecurity attrition rate is not a pipeline problem. It is a retention problem. Organizations recruit aggressively and underinvest in the culture, career development, and leadership that keep people. Technical excellence without investment in people is a strategy with an expiration date. The conference reinforced this from multiple directions: keynote, breakouts, and hallway conversation. 

 SIMPLICITY IS A SECURITY STRATEGY 

This was the thread that ran through day three. IR programs fail not because teams lack tools, but because playbooks are too complicated to execute under pressure. AppSec teams that chase vulnerabilities reactively never escape the treadmill. Compliance teams that pursue every framework at once exhaust themselves and gain little. 

“Not yet is sequencing, not failure” was probably the most actionable line of the conference. You cannot do everything at once. The organizations making progress chose a direction and went deep, rather than spreading thin across every priority simultaneously. 

WHAT THIS MEANS 

The security industry has the frameworks, the tools, and in many cases the regulatory clarity. What is harder to manufacture is the discipline to prioritize, the culture to retain talent, and the willingness to govern AI as rigorously as you adopt it. 

We left RMISC thinking about where those gaps are sharpest for the organizations we work with, and what it looks like to close them. 

If any of these themes resonate with what you are navigating right now, we would be glad to compare notes. 

Your company is already using AI. If you think it isn’t, that’s actually an even bigger risk. 

Let’s be clear. AI is already part of your operations, whether you approved it or not. People are already trying to exploit it. The real question isn’t if your organization faces AI threats, but what those threats look like. Now is the time to understand them. 

Most people don’t understand these risks yet. Security researchers have shared detailed findings on LLM vulnerabilities for years, but this information rarely reaches executives. Awareness of these risks is still lacking. This series aims to close that gap. 

Why This Matters Right Now 

Generative AI adoption in enterprise environments grew faster in 2024 and 2025 than any technology in recent memory. You spent years debating cloud migration strategies but woke up one morning to find that half your workforce was already connected to third-party AI tools, forget policies. Shadow AI is the new shadow IT, and it carries all the same risks and new ones as well …. So, congratulations on that. 

The threats against AI systems don’t stop at the vendor’s door. Many of the most serious attacks target the AI that your own team deploys, configures, and connects to your data. The risks we’re going to cover in this series exist on a spectrum, and understanding where your organization sits on that spectrum is the starting point for everything else. 

Who Is Actually at Risk in Hacking AI

The answer varies depending on which attack we’re talking about. Some threats apply specifically to organizations with their own AI models. Some of the things we will cover apply to anyone using any AI system. So that pretty much means anyone who got this far in the paper. 

So, since I am a glutton for punishment, I’m going to break this out into 8 articles covering both areas. Here is a rough breakdown of how the articles in this series map to your exposure throughout the lifecycle. 

THREATS THAT APPLY PRIMARILY TO AI BUILDERS AND DEPLOYERS 

• Training-Phase Attacks (Article 2): Poisoning a model during training or fine-tuning. Most relevant if you are training or fine-tuning custom or open-source models on your data. 

• Privacy and Extraction Attacks (Article 3): Forcing a model to reveal what it was trained on, including your sensitive data and PII. Relevant when you fine-tune a model on internal data or when your employees have fed sensitive data into AI tools. 

• Adversarial Input Attacks (Article 4): Attacks at the token or character level meant to evade safety filters. This one is most important for organizations with AI-powered security tools or automated AI decision systems. 

THREATS THAT APPLY TO EVERY AI USER 

• Prompt-Based Attacks (Article 5): Hijacking what an AI system does through deviously crafted inputs. This applies to every deployment everywhere.  

• Cross-Lingual Attacks (Article 6): Exploiting safety gaps in non-English languages to extract content that would be blocked in English. Applies to any tool accessible in multiple languages, including binary. 

• Agentic and System-Level Attacks (Article 7): Compromising AI agents that have been given tools and the ability to act on your behalf. This is the fastest-growing threat category as AI moves from answering questions to doing things, and it scares me the most. 

• LLMjacking (Article 8): Stealing your cloud AI credentials to run up costs on your bill while selling access to criminals. Applies to any organization running cloud-hosted AI infrastructure… and if you have read anything else I’ve ever written, you know I kind of have a thing for IAM and credentials. 

The article you’re reading now covers all seven threat categories at a level that should give you a working mental model and help you choose others you want to read more deeply. Think of it as a map for our wordy hike together. 

The Seven Threats: A First Look 

Okay, so you just hired a sophisticated new employee. This employee is incredibly capable, never sleeps, can handle any volume of work, and has been given access to your systems, your customer data, and the ability to take actions on your behalf. Now, around the globe, hundreds of people are actively studying every way that an employee can be manipulated, deceived, corrupted, or kidnapped. That is the situation with enterprise AI right now. 

The seven categories of attack we cover in this series each represent a different way that an adversary can go after that employee. Some attacks happen before the employee ever starts work. Others happen when they receive your work instructions. A few take advantage of the fact that the employee happens to speak forty languages, but only thinks safely in a few of them. 

We good on that? Okay, let’s check the destinations on the map. 

1. Training-Phase Attacks: Poisoning the Well

Before an AI model ever touches a production environment, it is educated by data. Enormous amounts of it. Training-phase attacks target this process, attempting to embed malicious behavior into the model before it ships or later, when you fine-tune or customize a base model for your needs. 

Here is something to worry you, for free. No need to thank me. If an adversary can influence a small portion of training data, they can implant a trigger, a word, phrase, or pattern that causes the model to behave in a specific way when it appears. Anthropic’s research demonstrated that it takes only around 250 malicious documents to backdoor a model with 600 million to 13 billion parameters. The brainwashed model will act normally until someone uses the trigger. Then it isn’t.  

Does this apply to you? If your organization is using a commercial model from a major vendor, the security during the training phase is mostly their responsibility. But if you are fine-tuning a publicly available model on your own data, or are as ambitious as building custom models, the training pipeline is a very early attack surface.  

Fine-tuning is being adopted to customize AI for specific business cases, this category is growing more relevant to more organizations every quarter. How relevant is it to you? 

2. Privacy and Extraction Attacks: Stealing What the Model Knows

Models remember things they were trained on. Privacy and extraction attacks exploit this by crafting inputs that cause the model to regurgitate this content, including things you specially told it never to share. Kinda like keeping a secret with a younger sibling. Maybe they will keep it, maybe a crafty adult will ask the question in such a way that the secret will be spilled. 

Way back (in history or GPT sense) in 2021, Nicholas Carlini (Et al.) demonstrated that training data could be extracted from GPT-2 at scale. That work has been repeated multiple times since. Research shows that extraction rates improve as the adversary gets more sophisticated, and that scaling up model size doesn’t reduce this risk. Bigger brains have more things to mistakenly tell you. 

The practical risk is this: if you feed sensitive data into an AI system, by fine-tuning or employee use of AI tools, some portion of that data may be recoverable by a sophisticated attacker. This is a privacy risk, a competitive risk, and in regulated industries, potentially a compliance risk.  

This risk applies to commercial tools and private deployments alike. 

3. Adversarial Input Attacks: What the Model Can’t See

AI safety systems are built to recognize harmful requests. Skilled adversarial input attacks are designed to remain invisible to those systems while making the same requests. The techniques range from appending algorithmically optimized character sequences to prompts, to replacing letters with visually identical characters from other Unicode scripts, to encoding malicious instructions in formats the safety layer doesn’t inspect but the model can still interpret. 

Research by Carnegie Mellon and others has shown that optimized suffixes, meaningless-looking strings of characters appended to a request, can cause almost any model to comply with requests it would normally refuse. Success rates against some models (in controlled settings) have reached 99 percent. In the wild, against commercial models, they’re lower, but not near zero. 

For most organizations, the immediate threat here is to AI-powered security tools and automated decision systems. In these systems, consistent, reliable behavior matters.  

Imagine an AI that flags threats or routes requests, manipulating them to create a terrifying attack surface rather than control. Or if you are using AI to control AI, like a content moderator, what would happen if it stopped doing its job? 

4. Prompt-Based Attacks: Hijacking the Conversation

OpenAI has said that prompt injection in AI-assisted browsing may never be solved. That’s a weighty statement coming from them. 

This is a big one, it affects everyone. Every user of every AI system, everywhere. Prompt injection attacks work by crafting inputs that override existing instructions, causing it to do something you didn’t intend and didn’t authorize.  

There are two versions of this, one more insidious than the other. Direct versions involve a user simply asking the model to ignore its rules. During the indirect version, the truly insidious one, malicious instructions are embedded in a document, a webpage, or an email that the AI reads as part of its job. 

OWASP has ranked prompt injection as the number one vulnerability in LLM applications for two years running. Success rates in real deployments range from 50 to 84 percent, depending on how the system is configured. That is a massive success rate and should worry you.  

The count of real-world incidents is ticking up rapidly. Browsers that summarize web content have been tricked into leaking user credentials. AI-powered resume screening systems were found to be processing injected instructions from “applicants”. Copilot has been manipulated through poisoned emails in the EchoLeak incident and others. If you use AI tools that process external content, prompt injections are an active operational risk today, so get on that. 

5. Cross-Lingual Attacks: The Safety Gap Nobody Talks About

AI safety systems are trained mostly with English content. Most safety research on AI is conducted in English. Most red-teams operate in English, and most safety-related data exists in English. In languages where training data is scarce, those safety guardrails are weaker.

Yet more research found that native Bengali speakers using their own language on publicly available AI tools were approximately three times more likely to encounter harmful content than English speakers making the same requests. This is a simple one: the model’s ability to understand and follow safety instructions degrades in languages it wasn’t well-trained on.  

Obviously, this has practical implications for global organizations. AI tools deployed in international markets may carry safety gaps that your domestic testing never found. It is a consistent and underappreciated entry point. It is greatly appreciated by your adversaries, though. 

6. Agentic and System-Level Attacks: When AI Can Actually Do Things

The hacking AI landscape changed astronomically as AI stops just answering questions and starts taking actions. AI agents given access to tools, APIs, file systems, calendars, email, and databases, and the ability to execute tasks on behalf of users, create a fundamentally different attack surface than a conversational chatbot.  

I prefer the term “delegate” to “agent” given the amount of power they have and how it models the user, but that is the subject for another paper. 

Indirect prompt injection is a whole different beast in an agentic context. What if your AI agent reads an email, and a hidden instruction in that email causes the agent to forward your files somewhere, reset your credentials, or… or… or…? Now the attack has moved from a content exposure problem to an operational one. Researchers have created self-replicating prompt-injection attacks that spread through multi-agent systems, much like a computer virus spreads through a network. A single poisoned document becomes a breach of epic and nightmarish proportions. 

Criminals aren’t just targeting AI agents as victims. They are using them as footholds into enterprise systems.  

7. LLMjacking: Stealing the Keys to the Kingdom

The last category is, in some ways, the most familiar to us security folks because it resembles the credential-theft attacks we’ve been fighting for years, just aimed at a new target. LLM jacking is the unauthorized use of your organization’s cloud AI infrastructure via stolen credentials. The attacker pays nothing. You pay the bill. With the rising price of tokens, this can become very expensive, often into the 7-figure range or more.  

If they then use that infrastructure to attack you, you are funding your own enemy. Have fun explaining that to the board. 

Sysdig first identified this attack pattern in April 2024, when they observed stolen cloud credentials being used to access ten different cloud-hosted AI services. The entry vector was a vulnerable web application. The cost to the victim was estimated at up to $46,000 per day in AI consumption charges. With newer, more capable and expensive models, that number will easily exceed $100,000 per day. 

This kind of thing is being carried out by an organized criminal enterprise. Dedicated marketplaces exist for buying and selling stolen access to AI. They even have sales and discounts for volume. In some cases, the stolen credentials were being used not to run up your bill, but to give sanctioned entities, organizations in countries with US technology restrictions, access to AI systems they’re otherwise prohibited from using. Now you are into a potential regulatory and legal liability, not just a financial one. You don’t want the three-letter initial windbreaker-wearing orgs showing up at your offices. 

The Inevitability Argument 

AI is not optional. Even if you opt out of it, you aren’t actually opting out; your vendors are using it, your competitors are using it, your customers are using it, and your employees are using it on personal devices and bringing the outputs back to work. The question of whether to engage with AI risk has already been answered. The open question is whether you engage with it deliberately. 

The organizations that do best are the ones where security leadership understands the threat landscape and has real conversations with the business about risk tolerance, deployment standards, and acceptable use. You can’t have those conversations without understanding what the threats actually are. That’s what this series is for. 

A Note on Shared Responsibility 

The AI providers bear meaningful responsibility for the security of the models and platforms they run. They have teams working on these problems, and they seem to take it seriously. 

But responsibility doesn’t stop with them. If you are relying on them alone, you are failing your security responsibilities. When you put AI in front of your employees and customers, you have taken on security responsibility. Most of the attacks we will cover in this series are not against OpenAI or Google. They are against your organization. That distinction matters, it shows where the work needs to be done. 

Security is a shared-responsibility model in AI, just as it is in the cloud. The vendor secures the model. You secure the deployment, data, access, and configuration.  

Next: Article 2, Training-Phase Attacks. How a few hundred carefully chosen documents can corrupt an AI model before it ever reaches production, and what that means for organizations building or adopting custom AI. 

NOTE ON THE USE OF AI IN THIS DOCUMENT 

The document is primarily human-conceptualized, written and edited. AI was used in the research and editing of this document. 

About the Author
Gerard Onorato is a CISO with extensive experience in enterprise security architecture, identity, and risk. This series reflects his personal analysis and does not constitute legal or compliance advice. 

Security groups do port, protocol, and IP allow/deny at Layer 3/4. No application awareness, no threat signatures, no built-in logging. If you need L7 inspection, you need AWS Network Firewall or Azure Firewall Premium, explicitly routed through.

VPC subnets are a routing construct, not an enforcement boundary. Traffic between subnets in the same VPC routes directly by default with no firewall in the path. East-west lateral movement from a compromised workload can be completely invisible without additional controls.

NAT gateways handle outbound address translation. They do not inspect sessions, enforce inbound policy, or provide anything equivalent to bidirectional DMZ controls.

Cloud load balancers balance load. A WAF must be explicitly attached to every public ALB and is not on by default. Without it, there is no OWASP rule coverage and no bot mitigation.

SD-WAN Makes It Worse

SD-WAN direct internet breakout eliminates your centralized inspection chokepoint. Branch traffic reaches cloud workloads without crossing the data center firewall stack, and that is by design. SSL inspection at branch appliances is rarely performed because the performance cost is too high.

What follows is a split-brain problem: your SD-WAN dashboard shows traffic as tunneled and secure, while your VPC flow logs show direct traffic from those same source IPs. The DC firewall sees nothing. CISA Emergency Directive 26-03 (February 2026) confirmed active exploitation of Cisco Catalyst SD-WAN infrastructure, calling it a high-value target with known compromise in the wild, so this is not a theoretical concern.

Stop relying on the traffic path as your security control. Enforce security at the destination using cloud firewall ticket rules and security groups at the workload level.

Four Things to Do This Week for Firewall Ticket

1. Enable VPC Flow Logs everywhere. All VPCs, all traffic, version 3+, retained 90 days minimum. Everything else on this list depends on it.
2. Audit public ALBs for WAF coverage. AWS Console > WAF > check which ALBs have a Web ACL attached. Any that don’t are unprotected. Attach AWS Managed Rules Core Rule Set — it takes about 20 minutes.
3. Run the split-brain test. Pull SD-WAN path analytics for your most sensitive traffic and cross-reference with VPC flow logs for the same source IPs. Gaps between the two mean broken routing.
4. Audit security group rules. Flag every 0.0.0.0/0 inbound rule. AWS Config rule ‘restricted-common-ports’ does this automatically.

Fix the network architecture and your teams ship faster, without firewall tickets blocking deployments or rearchitecting mid-migration.

Join Tyler at GrassRootz for the full talk on June 4^th for more: https://grassr00tz.com/