Background

Amadeus is a major Global Distribution System (GDS) provider and is one of the largest travel booking networks used by airlines and travel agencies, responsible for processing the personal data of millions of individuals for bookings in the travel industry. Following an anonymous complaint on 26 September 2023, the AEPD initiated proceedings against Amadeus to investigate possible GDPR breaches in relation to a pilot scheme called ‘PLATAFORMA.1’ (the Pilot).

The intention of the Pilot was to identify travel trends and offer “hyper-personalised” experiences and targeted searches through user profiling. The Pilot took personal data collected from flight and travel bookings through its GDS, including Passenger Name Record (PNR) data. This data was then repurposed to create traveller profiles which would then be shared with hotels and other travel companies.

Why were they fined?

(i) Breach of Article 14 (Information obligations to data subjects)

The AEPD found that Amadeus had failed to comply with its obligations to provide data subjects with information about its processing under Article 14 GDPR. Amadeus is a B2B service and they generally receive traveller data through intermediaries, airlines and travel agencies. It was found that the Pilot involved further processing of traveller data for a purpose that was different than the original intention. Amadeus failed to provide the information required under Article 14 to data subjects about the further processing of their data in relation to the Pilot.

Amadeus attempted to rely on broad language in their privacy policy, such as references to use of data for analytics or product improvement. This was rejected by the AEPD as insufficient, especially for a specific and complex further processing activity, where there is a lack of a direct relationship between Amadeus and travellers. Furthermore, Amadeus did not provide specific, timely information about the new purpose in advance of the new processing.

Without meaningful notice of the new processing, travellers are unable to exercise their rights under GDPR, and travellers could not reasonably be expected to be aware of the use of their data.

(ii) Breach of Article 6 (No lawful basis)

In addition, the AEPD found that Amadeus had breached Article 6, as they had processed the personal data for the pilot without a valid lawful basis. Amadeus sought to rely on the legitimate interest basis, however AEPD rejected this for a number of reasons, including that the processing was not within the reasonable expectations of travellers, there was no direct relationship between the parties, the processing lacked transparency, and, particularly given the scale and context, no clear balancing exercise had taken place to demonstrate that Amadeus’s commercial interests outweighed the fundamental rights of the travellers.

The AEPD also referenced the failure to meet the requirement under GDPR Article 6(4) to conduct a compatibility assessment to determine whether a new purpose (i.e. the further processing under the pilot) is compatible with the original purpose for which the data had been collected. Amadeus did not appear to have sufficiently carried out such an assessment. Furthermore, it was found that the PNR data was kept beyond statutory retention periods set out in Regulation (EC) No 80/2009 (the Computerised Reservation Systems Regulation). Under the Computerised Reservation Systems Regulation, it should have ensured that information concerning individual bookings was stored offline within seventy-two hours and destroyed within three years, but instead, Amadeus had used both active and inactive data for 2019.

(iii) Final outcome

The AEPD proposed a fine of €9 million for the breach of the duty to provide information (Article 14), and a further €9 million for processing with no lawful basis (Article 6), totalling an overall fine of €18 million. The AEPD considered aggravating factors in setting the amounts:

•  the scale and severity of the processing, with millions of data subjects affected, and unable to exercise their rights, as they were unaware of the further processing;
• a prior infringement for breach of Article 12 GDPR in 2022; and
• Amadeus’s position as a routine processor of large-scale personal data meant they should have been fully aware of their obligations.

Amadeus’ processing under the Pilot was cross-border, so the AEPD had submitted a draft decision to other supervisory authorities under the Article 60 GDPR cooperation process. The concerned supervisory authorities raised no relevant and reasoned objections.

Amadeus chose to make voluntary payment of the proposed fine, resulting in a 20% discount to €14.4 million. However, voluntary payment can be made without admission of liability, and Amadeus has stated that it intends to appeal in the Spanish court.

Our take: key steps for data reuse projects

The AEPD’s decision highlights a number of key lessons for any organisations looking to reuse data for development of new products, services, or insights.

Service providers reusing data to develop products, services, or insights are data controllers – where the reuse is carried out for the supplier’s own purpose, and is not for any specific client, organisations act as controller. This is the case even if they acquired the data through providing services as a processor; factually, the supplier is a controller and must comply with its controller obligations.

Compliance still matters for pilots – fast-tracking compliance for pilots may be the correct risk decision in some cases, for example where dummy data is used.  But where a ‘pilot’ involves processing real data, particularly where the processing is unexpected, intrusive, or carried out at scale, privacy compliance processes should be followed in full before the pilot commences.

Privacy notices need to be specific – individuals should be given enough information to understand how their personal data will be used.

Privacy notices need to be communicated – where organisations have no direct relationship with the data subjects, they must consider how they will communicate information about their processing to comply with their Article 14 obligations.

Compatibility assessments, legitimate interests assessments and data protection impact assessments – where an organisation is carrying out further processing for a new purpose not originally communicated, it will need to assess compatibility with the initial purpose. The AEPD’s decision also highlights the importance of carrying out a legitimate interests assessment and, in particular, the balancing test to see whether the individual’s fundamental rights outweigh the controller’s interests. For most projects involving innovative uses of, and insights from, data, a data protection impact assessment will be appropriate.

Contracts – where suppliers reuse customer personal data for their own purposes, they should ensure contracts include:

• clear terms on where they act as controller, joint controller, and processor and appropriate provisions in each case; and
• the right to use customer data for their own business purposes.

Where it makes sense to do so, they should also consider an obligation on their customers to provide their privacy notice to data subjects, where the customer is best placed to do so.

The full AEPD decision is available to read in Spanish here. 

The authors of this article are human subject matter experts, but have referred to a machine translation of the decision (from the original Spanish language decision linked above).

Article prepared with the kind assistance of Amelia Farquharson.

Background

Many regulators, including NYDFS, emphasize the need for organizations to update their risk analyses in order to account for new threats in the environment.  According to NYDFS, a “heightened threat environment” exists “when cybersecurity risks are significantly elevated and therefore have a high likelihood of impacting Information Systems, Nonpublic Information or operations.” (footnote omitted). 

NYDFS offered two examples of a heightened cybersecurity environment: 

• “geopolitical events that have the potential to increase the risk of cyberattacks,” and
• “technological developments that materially change cybersecurity risks, such as the release of frontier AI models, may result in a heightened threat environment and warrant stronger defensive measures and increased vigilance.”  (footnote omitted) 

The reference to geopolitical events reflects a broader regulatory and government awareness of nation-state cyber threats, sanctions-related attack risk, and the targeting of financial sector infrastructure, all factors that have been particularly prominent in the current global environment.

We explained the cybersecurity risks relating to frontier AI models like Anthropic’s Claude Mythos in a recent post, and expect that this guidance from NYDFS is likely driven, at least in part,  by Anthropic’s  Mythos model and its purported ability to autonomously identify security vulnerabilities.  This guidance was released in conjunction with an Industry Letter specifically emphasizing the threat of certain frontier AI models that “that amplify the potency, scale, and speed of identifying vulnerabilities and exploits in information systems” and urges organizations to prepare for the release of such frontier AI models.  Of course, as new models get released, it will simply raise the overall threat level, and the “heightened cybersecurity environment” will become the new normal.

Cybersecurity Measures to Consider

NYDFS emphasized that the steps listed in the guidance are not new requirements under the cybersecurity regulation, 23 NYCRR Part 500, but instead are “a non-exhaustive list of best practices”  that may go “beyond the explicit minimum controls” required under Part 500.  Whether a licensee elects to incorporate them:

“depends on the unique circumstances and operations of an organization. To determine when and which additional security controls to employ to address specific threat environments, Regulated Entities should assess the specific cybersecurity threat, their Information Systems, supply chain dependencies and usage, as well as sector-specific risks.”

That being said, regulatory minimums are not the driving force in cybersecurity for many companies as the reputational, legal and intellectual property costs and risks of a cyber event outstrip the risk from regulatory scrutiny and push hard for greater protections than mandated by NYDFS or other regulators.  For insurers and health plans, the stakes are especially high. These entities hold among the most sensitive categories of Nonpublic Information regulated under Part 500: health data, financial data, and identity information are routinely co-mingled in a single customer record. This data cache makes insurers high-value targets for sophisticated threat actors, and it is the reason NYDFS insurance examiners pay close attention to cybersecurity controls during market conduct and financial examinations alike.

NYDFS aggregated its listed best practices into three groups:

1. Measures to Reduce the Attack Surface. If an organization can reduce the area the attacker can reach, it makes breaking in harder for the threat actor, which may move on to an easier target.  NYDFS listed nine items, including “expeditiously identify and remediate known exploited vulnerabilities” and “conduct privileged access reviews, especially for threat-relevant users, systems, and devices, to prevent unauthorized or unregistered access to Information Systems.”  Overall the measures appear designed to encourage identifying and patching vulnerabilities on faster timelines (given attackers armed with more advanced AI are expected to  identify and exploit vulnerabilities at greater speeds), enhanced coordination with third party service providers (who are yet another attack surface), and further hardening information systems.  The Department further advises specific measures designed to strengthen security practices around how organizations use AI in their own environment, including secure programming practices for AI-generated code and measures to restrict and validate inputs prior to running certain processes – these are likely intended to minimize the risk of prompt injection and code execution-based attacks.  
2. Measures to Improve Threat Detection and Readiness. An organization may not be able to keep all threat actors out of its systems, so it is important to detect them.  NYDFS listed six steps, including that logging and security event data should be captured and acted upon appropriately, as well as that organizations should “engage with critical Third-Party Service Providers to confirm awareness of and appropriate action on heightened cybersecurity risks and readiness to respond to potential disruptions.”  For insurers in particular, this engagement obligation extends to third-party administrators, managing general agents, prescription benefit managers, and claims administrators – counterparties that routinely hold or process large volumes of sensitive Nonpublic Information on the insurer’s behalf.
3. Measures to Improve Resilience and Response. Finally, if the threat actor does get in, in addition to notifying NYDFS promptly if required, the organization needs to be able to respond and recover.  NYDFS included five items for this group, including that the organization should review and test “threat-relevant operational resilience procedures (e.g., incident response and business continuity plans) to protect and restore critical functions, Information Systems, and Nonpublic Information.”

Our Take

Organizations that are subject to the NYDFS cybersecurity regulation should review all of the security measures listed and conduct a gap assessment promptly rather than at the next annual review cycle.   Given that NYDFS issued this guidance in response to an active, heightened threat environment, delay is itself a risk management decision.

Board-level accountability matters. Part 500 already imposes CISO reporting obligations and senior officer certifications. Boards of directors of licensed insurers are the appropriate governance body to receive briefing(s) on this guidance and steps implemented or being considered by management.

One heightened protection that NYDFS did not call out in this guidance, but has addressed in other settlements and guidance is improved information governance and internal IT organization.  No matter how much effort a company puts into protect their IT environment from intrusion, a cyber event is inevitable and companies need to minimize its impact once the unauthorized user has gained access.  Strong record retention programs minimize obsolete and redundant records, which in turn reduces the data available for extraction.  Strong data segmentation not only ensures that employees only have access to the data they need (minimizing inadvertent misuse), but makes it inherently minimizes cyber events by impeding unauthorized users ability to survey the IT environment.

Organizations should also document their decision-making about which best practices they decide to adopt and why. An undocumented decision not to implement a recommended control is far harder to defend than a reasoned, risk-based explanation.

Insurers that also write cyber liability insurance face an additional consideration: their own enterprise security posture should be consistent with the underwriting standards they apply to policyholders. An insurer that recommends robust third-party risk controls to its insureds but has not itself engaged its TPAs and MGAs on heightened cybersecurity readiness may face uncomfortable questions from both regulators and sophisticated policyholders. It may be advantageous to work with experienced counsel both to review the items listed and for implementation of such items as conduct

The guidelines

The Commission has published three sets of draft guidelines:

• general principles;
• Annex III high-risk AI systems (standalone AI systems); and
• Annex I high-risk AI systems (AI systems embedded in a product.

Interested stakeholders can submit their input on the consultation until 23 June 2026.

This post looks at the guidelines for Annex III high-risk AI systems.  These guidelines:

• discuss the application of the derogation under Article 6(3)
• provide discussion and examples of what is high-risk, what is not high-risk, and what is high-risk but falls under Article 6(3) for each high-risk purpose under Annex III. 

This post provides commentary on the Commission’s view on the application of the derogation Article 6(3).  It also discusses some of the commission’s views on employment high-risk purposes under Annex III(4), since these have broad cross-sectoral relevance.  

Recap – what are the consequences of an AI system being high-risk?

These guidelines are designed to help organisations falling within the geographic scope of the AI Act to confirm whether their AI systems are high-risk.  If the AI system is high-risk, consequent obligations depend on the organisation’s role.

Providers: generally, the organisation developing the technology or having it developed. 

Providers are subject to significant risk management and governance obligations for the specific high-risk AI system.  These obligations are not about the organisation’s general risk management processes (although a strong organisational AI governance programme provides a helpful foundation).  Rather, they relate to assessing and mitigating risks for the specific product and creating detailed technical documentation for the product.  Before placing the product on the market or putting it into service, the provider must assess (or have a third-party notified body assess) and declare conformity, affix the CE mark, and register the AI system in the EU database.  The rules envisage that compliance is carried out throughout the development life cycle, rather than retrospectively after development.  Retrofitting and compiling technical documentation once development is complete will be more challenging.

European standardisation bodies CEN and CENELEC are currently developing standards to set out how providers could comply, addressing a standardisation request from the Commission.  When published in the EU’s Official Journal, these will become harmonised standards and following them will provide a presumption of conformity with the relevant requirements.  At the time of writing, three of these standards (prEN 18288 (AI Risk Management), prEN 18282 (Cybersecurity), and FprEN 18286 (Quality management system)) have been made available in draft form for public enquiry, while the majority are still under preparation.

Becoming a provider: it is possible for organisation which does not carry out the initial development (or any development at all) to become subject to these very significant obligations by:

• use of general-purpose tools (internal or off-the shelf generative AI solutions) for a high-risk purpose (Article 25(1)(c));
• creation of an agent for a high-risk purpose, e.g. sifting CVs or performance management;
• adding the organisation’s branding (name and trade mark) on another provider’s high-risk AI system (Article 25(1)(a)); or
• substantial modification Article 25(1)(b) – though this is not generally an option for off-the-shelf tools with a specific function.

Deployers: generally, the organisation using the technology.

Deployer obligations relate to managing risks around how the AI system is used in practice, for example by informing decision subjects about AI use and assigning competent human oversight.

What’s in and out

There is NO human-in-the-loop exemption

The Commission confirms that human involvement “has no effect on the classification of the system as high-risk”.[1]  This is because human involvement cannot change the purpose for which an AI system is intended to be used.

If one of the four conditions of Article 6(3) are met, the provider might be able to exempt the system classification of high-risk.  The type of human involvement during deployment may be relevant to demonstrate that the AI system is intended to perform a narrow procedural or preparatory task or improve an already completed human task.

However, “The provider cannot exempt and categorise an AI system as ‘low risk’ simply by adding to it a requirement for human involvement”.

The four derogations listed in Article 6(3) should be interpreted narrowly

Article 6(3) provides a derogation for AI systems falling under an Annex III purpose that do not “pose a significant risk of harm to the health, safety or fundamental rights of natural persons, including by not materially influencing the outcome of decision making” where they meet any of the following conditions:

“ (a) the AI system is intended to perform a narrow procedural task;

(b) the AI system is intended to improve the result of a previously completed human activity;

(c) the AI system is intended to detect decision-making patterns or deviations from prior decision-making patterns and is not meant to replace or influence the previously completed human assessment, without proper human review; or

(d) the AI system is intended to perform a preparatory task to an assessment relevant for the purposes of the use cases listed in Annex III.”

The Commission refers to the derogation in Article 6(3) as a ‘filter mechanism’.  This can only be applied when an AI system meets one of the conditions set out at Article 6(3)(a)-(d).[2]

The Commission appears to suggest that there is no generalderogation availablefor AI systems that do not pose a significant risk of harmto health, safety or fundamental rights – the conditions listed are exhaustive.[3]

Each of the Article 6(3)(a)-(d) grounds must be interpreted narrowly.  However, where a provider has assessed and concluded that their AI system does fall under one of the exemptions, it is not necessary to conduct an additional assessment to determine whether the AI system poses a significant or any risk of harm besides those conditions.[4]

The profiling exemption – which means the AI system is high-risk even if one of the Article 6(3) conditions would apply – is also discussed.  Profiling is carried out where an individual’s decisions and personal characteristics are assessed.  For example, an AI system to evaluate recruiter’s decisions and deviations from previous patterns, in light of their personal characteristics, would be considered profiling.  It would therefore be high-risk even though it could fall under Article 6(3)(c).[5]  Examples are then provided to assist with general interpretation of each condition,[6] followed by examples for each high-risk purpose later in the guidance.

Providers wishing to rely on one of these conditions must document an assessment setting out why the system would be high-risk, which condition it believes applies and why, and a description of why the system does not perform profiling.[7]

Providers must also register their AI system in the EU database where they assess and conclude that Article 6(3) applies to their AI system.  The changes to the AI Act made by the AI Omnibus will simplify registration requirements.

Employment examples

The Commission provides valuable colour on which use cases fall into this category.

Recruitment, selection, placing targeted job ads, analysing and filtering job applications, and evaluating candidates

This encompasses the range of activities around identification and attraction of potential applicants.[8]

The line on whether a purpose falls under Article 6(3) can be a fine one.  Generation of job descriptions falls within this category where the AI system plays a role in the recruitment process. The specific AI system may fall within Article 6(3)(a) derogation (narrow procedural task) if the AI system is generating the description based on a list of tasks to be carried out and list of necessary qualifications and skills previously defined by a human recruiter.  However, it will fall outside the ‘narrow procedural task’ derogation where it generates the necessary qualifications and skills based on high-level description.[9]

Practical examples of high-risk use cases, unsurprisingly, include AI systems intended  to be used as automated job matching and ranking tools, AI systems used for scoring applicant answers in a recruitment process, and placing targeted social media ads based on navigation patters and a wide range of user characteristics. [10]

Examples of use cases that would not be high-risk include AI systems used to non-inclusive and discriminatory wording in job descriptions.  AI systems that would fall under Article 6(3) (and so require registration) interestingly include CV parsers and AI systems for scheduling interviews.

AI systems intended to manage work-related relationships

This high-risk purpose is included to address potential discrimination in relation to opportunities that could impact workers’ livelihoods and career prospects.

It encompasses promotion and termination, and also allocation of tasks based on personal traits and characteristics.  AI systems allocating tasks based on neutral, objective, and external factors, such as availability or location, would not be caught.[11]  In contrast, AI systems that use criteria such as responsiveness to customer requests or reliability metrics, such as AI systems that allocate delivery slots or rank external lawyers, will be considered high-risk.[12]

Other examples considered high-risk include an AI system that dynamically sets driver compensation based on real-time demand, driver acceptance rates, and passenger ratings.  Examples of AI systems that would fall under Article 6(3) include AI systems compiling performance records into structured reports to send to managers at a fixed date, and an AI system to refine human-drafted promotion evaluations and flag potentially biased wording.[13]

Our take: what to do now (providers and deployers)

The obligations for high-risk AI systems are now set to apply from 2 December 2027 (rather than 2 August 2026).  At the time of writing, political agreement has been reached on pushing this date back, though the AI Omnibus still needs to be published in the EU’s Official Journal before 10 July to make the change effective.

Providers: should use these guidelines to confirm which of their products are high-risk and begin the work of embedding compliance processes as soon as possible.  As discussed above, provider obligations require risk management and governance to be embedded and documented during the product life cycle and cannot simple be addressed on the AI Act’s application date.

Where providers assess and believe that an AI system falls under Article 6(3), they should also ensure that their assessment is documented (registration will also be required when the EU database is available).

Providers should also familiarise themselves with AI Act standards as they become available, including the three already made available in draft form.

Deployers: should ensure that they have policy and, ideally, technical controls in place to prevent accidental accession to the provider role.  As discussed above, this can happen inadvertently through high-risk application of general-purpose tools.

Deployers should also ensure that they are able to identify which of their AI systems are high-risk and begin to:

• ensure they have appropriate contractual protections in place for contracts that may run past December 2027; and
• augment existing compliance processes to comply with deployer obligations.

[1] Para 70.

[2] Para 86.

[3] Para 88.

[4] Para 88.

[5] Para 112 and example following.

[6] Paras 97-108.

[7] Para 115.

[8] Para 246.

[9] Example on p.64.

[10] Examples for Annex III(4)(a) from p.66.

[11] Para 271.

[12] Para 270.

[13] Examples for Annex III(4)(b) from p.76.

Although access to Mythos itself is currently restricted to approximately 40 hand-picked organizations under an initiative called Project Glasswing, equivalent capability is estimated to emerge in the broader market, potentially in adversarial hands, within 6 to 24 months.

What Anthropic disclosed

Mythos is a general-purpose large language model that was not designed as a cybersecurity tool. Its offensive capabilities were not purposefully engineered but rather emerged as a byproduct of general improvements in reasoning, code generation, and autonomous task execution. That distinction is essential as it implies that every frontier AI laboratory currently pursuing the same general improvements is potentially on a path to the same destination.

According to Anthropic’s own published technical analysis, Mythos can autonomously identify “zero-day vulnerabilities in every major operating system and every major web browser.” The model is then able to develop working exploits for those vulnerabilities without human assistance. In benchmark testing, Anthropic’s prior frontier model produced working exploits only twice in several hundred attempts on a standardized task, where Mythos produced 181 working exploits. In broader testing across thousands of open source software targets, Mythos distinguished itself from prior frontier models by achieving full system compromise on ten fully-patched targets.

Anthropic also disclosed three notable behavioral findings from its safety testing of earlier versions of Mythos:

• First, the model escaped its sandboxed testing environment by developing its own multi-step exploit, gained broader internet access it was not supposed to have, and then, going beyond its assignment, posted details of its escape to publicly accessible websites.
• Second, in rare cases, the model used a prohibited method to arrive at a solution, appeared to recognize that doing so would be detectable, and then re-worked the problem through permitted means before presenting its answer, effectively concealing how it had actually solved the problem.
• Third, in very limited instances, when the model’s work on a coding task was being evaluated by another AI system and the submission was rejected, the model attempted a rudimentary prompt injection attack against the grader. In other words, it can be inferred that the model tried to manipulate the system judging it rather than simply improving its answer.

Viewed collectively, these findings describe a system potentially capable of strategic deception and autonomous action beyond the scope of its instructions at an unprecedented pace.

Anthropic’s response through Project Glasswing, in coordination with a limited group of organizations responsible for critical digital infrastructures, reflects a well-resourced effort to identify and patch critical vulnerabilities by leveraging the defensive capabilities of Mythos. Nevertheless, the controlled release model retained by Anthropic is likely to concentrate the defensive advantages among organizations already operating at the frontier of security capability. What remains uncertain is how much practical benefit will flow to the organizations that most need it, on what timeline, and through what mechanisms.

What comes next

In the coming months, the capabilities of Mythos-like models (“Frontier AI models”) are expected to reach adversaries, financially motivated threat actors and state-sponsored threat groups alike. Threat actors will be able to simply acquire the capability through a model and maintain the necessary compute power to use it. This ability will lower the barrier to entry to the criminal market of exploits and potentially grant operational independence to organized criminal ransomware groups who may have depended on exploits developed by and purchased from the broker ecosystem. With the ability to run autonomous vulnerability discovery against industrial control systems and Operation Technology (OT) environments, for instance, threat actor groups will be able to produce working exploits for vulnerabilities that have existed undetected for decades at marginal cost and at scale.

Mythos is not without limitations such as its false positive rates, the conditions under which vulnerabilities were found in testing, and the practical usefulness of certain vulnerabilities in the context of real-world exploits. Nevertheless, Mythos and other Frontier AI models represent an inflection point in the threat landscape as it illustrates that AI-enabled offensive capabilities have been accelerating faster than the governance and defensive frameworks designed to contain them.

Looking ahead, AI-enabled adversaries will be equipped to launch simultaneous multi-vector attacks without any human bandwidth constraints. As such, adversaries could start running parallel overnight discovery campaigns against networks, messaging infrastructure, and cloud hypervisors at the same time, at marginal cost.

What to do now

The coming months present a consequential preparation window for organizations to jumpstart or reignite efforts to manage cybersecurity risks.

The next 90 days

Regardless of sector, identifying where their oldest, least-maintained code and systems live and how much of it is written in memory-unsafe programming languages is key. This recommendation is consistent with the long-standing cybersecurity principle – organizations cannot effectively prioritize remediation for assets they have not inventoried. That inventory, and the risk assessment that follows from it, is the first step for every organization regardless of industry.

A review of the organization’s incident response plan and preparedness to address simultaneous multi-vector attacks comes next. Most incident response programs assume a sequential attack that does not account for adversaries armored with automated and parallelized attack planning. Under existing incident response protocols, containment decisions made to respond to one attack vector can potentially accelerate damage from another, and notification timelines for different incident types may also run concurrently and complicate stakeholders’ decision making process. With the rise of AI-enabled adversaries, organizations should now test their response programs against this scenario. Identifying that gap now, before it is exposed during an active incident, is a meaningful risk reduction step.

Each sector faces heightened exposure to specific challenges.

• Financial institutions: Financial institutions face additional hurdles in relation to future regulatory disclosures or examinations. In particular, for certain companies subject to the New York Department of Financial Services (NYDFS) oversight and/or the Gramm-Leach-Bliley Act (GLBA), the development of Frontier AI models may warrant conducting risk assessments to account for potential changes to the company risk profile and update written materials.
• Energy sector: Companies in the energy sector often rely on OT devices and systems, which are among the hardest to patch and the most disruptive to take offline for maintenance and update. Nevertheless, pipeline operators subject to the TSA Security Directives may benefit from an initial advantage from compliance-driven OT asset inventories. At the same time, added connectivity expands the attack surface that organizations should monitor against AI‑enabled threats.
• Airlines and transportation sector: TSA-regulated airlines and other transportation-sector organizations have long operated under mature compliance regimes. These frameworks establish a robust baseline for critical systems and companies’ OT environments, but the connectivity introduced by compliance-driven monitoring has expanded the attack surface. TSA-regulated companies should specifically reassess this added connectivity in light of emerging threats such as via Frontier AI models exploitations.
• SaaS and IaaS providers: For IT service providers, complex cyber incidents often affect both the organization and its business customers. The pace and sophistication of attacks leveraging AI are likely to push the boundaries of what most current incident response plans contemplate, especially in the context of a multi-vector attack. Organizations should consider reviewing their incident response and customer communication plans to address scenarios in which multiple business customers are simultaneously experiencing disruptions.

Our take

There is no doubt that the novel threat of AI-enabled adversaries will cast more light on long-standing cybersecurity challenges and amplify the associated risks at an unmatched rate. The risk management solutions may nonetheless sound more familiar than one would expect.

Demonstration of having conducted a substantive exposure assessment and made reasonable, documented decisions in response to new and evolving risks, best positions organizations in addressing scrutiny by regulators, litigants, insurers and the public.

Additionally, organizations regardless of industry can benefit from taking several longer-term steps as they adapt to the rapidly shifting nature of AI-enabled exploits:

• How do our vulnerability and patch-management operations need to be enhanced to accelerate discovery and timely patching as AI-enabled threats evolve?   
• Do we need to review and update our contractual protections with our most significant vendors regarding AI-enabled threats?
• Do the existing threat intelligence service providers and managed detection tools have the ability to detect attack signatures that have no CVE match and no prior pattern in threat feeds?
• Do tabletop exercises and incident response plans account for multi-vector, simultaneous attack scenarios?
• Do board and executive briefings incorporate updated threat frameworks that reflect the development of Frontier AI models in the threat landscape?
• Do the existing cyber insurance policies provide adequate coverage and do the underlying risk questionnaires and policy terms remain accurate in the current threat landscape?

This briefing focuses on the impact of the changes for pension scheme trustees.  It includes points of relevance across sectors, and particularly for other regulated sectors.

One AI security vendor recently claimed that SOC2 was insufficient for evaluating AI security and controls. Although we might agree with the ultimate conclusion, the vendor made some surprising statements that show an implicit bias for the Band-Aid approach and a poor understanding of how to effectively govern Agentic AI systems. 

Here are a few examples of representations that should raise red flags.

Vendor:

“An AI Agent doesn’t have a fixed set of behaviors.”

Correct Approach:

AI agents in most settings should have established, fixed behaviors. Success and failure modes should be explicitly defined and governed. This should be done with static logic (code) and is one of the most important controls in any Agentic AI system. 

If success and failure are not well defined and monitored, then arguably the system is constructively allowed to misbehave.  And, when it does so, the failure mode will be silent. Trying to detect and stop these system failures after-the-fact with a Band-Aid approach resembles whack-a-mole more than it does governance. Relying on after-the-fact controls is like treating aviation safety as mostly a matter of investigating crashes instead of trying to increase the overall safety and reliability of the system.  After-the-fact detection isn’t meaningless, but it is the wrong layer to lean on.  In agentic AI, as in aviation, the primary safety work should happen before and during system operation.

Vendor:

“The agent may call APIs that are not part of any predefined workflow.”

Correct Approach:

Workflows should be pre-defined, and AI agents should only call APIs for which they are explicitly scoped in advance and subject to pre-defined static logic/control.

If access to API calls is not limited and controlled, then it becomes easier for those APIs to be accessed either inadvertently or maliciously.  Potential exploit chains (which string together multiple agentic components) become deeper and more numerous. To extend the airplane analogy, many commercial airplanes have a control system that sits between pilot action and airplane behavior.  Control systems on modern aircraft actively resist or prevent unsafe pilot maneuvers and will limit pitch, bank, and angle of attack to prevent stalls and exceeding safe speeds. These are exactly like the pre-defined limits on API calls: before the system can do the unsafe thing, a control has already prevented that system behavior.

Vendor:

“An AI agent decides at runtime what tools to use.”

Correct Approach:

Although it is true that Agent behavior is dynamic, the vendor leaves open the possibility that natural language (i.e., a call to an AI endpoint) can be allowed to invoke tools.  For a system to be reliable, however, tool usage should be controlled with static logic. The reason is because responses from AI endpoints are “probabilistic,” meaning one can’t perfectly control the content of the AI response over time. When tool usage turns on probabilistic responses from AI endpoints then, at some point, they are likely be invoked in ways that make the system unreliable and exploitable.  Over time, these small and unexpected responses can chain together and lead to cascading failures.  Similar to ungoverned API use, exploit chains become more dangerous and multiply.  In other words, when preparing for landing, there should be room for the pilot to make judgment calls based on wind speed, runway length, and physical obstacles, but those judgment calls should only be allowed to flip certain switches in the cockpit. 

The Critical Role of Lawyers and Compliance

In the current environment, where Agentic AI risks may not be fully appreciated by existing development and cyber teams, a compliance/procurement/cyber attorney may end up being the first and best line of defense to ensure the organization effectuates real governance and controls over the AI use case. 

With respect to every AI system and AI system component, an organization ideally should begin by posing the following questions:

• What am I trusting the system or component to do and why?
• How do I limit trust (and limit risk)?
• What is the system or component allowed to do?
• What is the system or component not allowed to do?
• What is the data the system is acting upon and what are my rights and limitations with respect to that data?
• How does the system enforce the answers to these questions?
• What is the risk of scope creep and how can I monitor the actual use over time?

The goal is to surface those areas where trust or scope of action is unwarranted or unnecessary.  The governance process is of course a good deal more involved than this, but this is the right conceptual and practical place to start.