Production-ready agents, Prerequisite 5: Compliance for AI systems: what an architect needs to know

Prerequisite 4 was about security: stopping someone from misusing the agent. This one is about compliance: being able to show that you are allowed to run the agent at all, that it treats people’s data lawfully, and that you can prove both when asked. An agent can be perfectly secure and still unlawful, for example if it keeps personal data for no reason or makes decisions about people without anyone checking.

I write this for architects and for anyone designing or approving an AI system. It is a map, not legal advice: it tells you which questions to ask, which words your legal and privacy colleagues will use, and where the design choices in this series help. The final word on any specific system belongs to them.

Where are we in the architecture?

This prerequisite does not add a box. It maps the legal and regulatory obligations around every box to the controls that meet them. Part 1.25 walks through the whole picture, layer by layer.

the complete Acme architecture: Maria asks in Gemini Enterprise; the router runs on Agent Runtime behind plugins and Model Armor and calls the marketing and supply agents over A2A; they reach BigQuery and Databricks through Agent Gateway; identity and secrets on the left, models, knowledge and the market watch managed agent for public research on the right, traces, logs and analytics below; above it, shoppers reach Gemini Enterprise for CX, whose conversation insights land in BigQuery, and the Legal industry edition plugs into Gemini Enterprise

What do I need to know first?

Your role decides your duties: the same agent puts different obligations on whoever decides why data is used, whoever only handles it for them, whoever builds the AI system and whoever uses it. So before any law, work out which roles you play.

An everyday example first. A small company hires a payroll bureau to produce its payslips. The company decides who gets paid, how much and why; the bureau runs the numbers on the company’s instructions and may not use the staff list for its own marketing. Separately, a software firm builds the payroll program and sells it, and the bureau uses it day to day. If an employee asks to see their data, they ask the company, because the company decided why the data is used. The firm that wrote the program answers for how it was built; the bureau answers for how it runs it.

The company that decides         ->  controller (GDPR)
The payroll bureau that handles  ->  processor (GDPR)
The firm that builds the program ->  provider (EU AI Act)
Whoever uses it under its own    ->  deployer (EU AI Act)
  authority

How does this apply to Acme? The same four words, in the order the example used them:

Under data protection law (GDPR), a controller decides why and how personal data is used; a processor handles it on the controller’s behalf. Acme is the controller of Maria’s and Sam’s data; Google Cloud acts as a processor for the data Acme puts into its services.
Under the EU AI Act, a provider develops an AI system or model and places it on the market; a deployer uses an AI system under its own authority. Google is the provider of the Gemini models. Acme, building and running its own agents, is in my reading both the provider and the deployer of those agents; confirm the roles for your own case.
Personal data is any information about an identifiable person. In an agent it turns up in more places than people expect: the user’s question, memories, session state, logs, traces and eval sets.
A DPIA (data protection impact assessment) is a written assessment of the risks a use of personal data poses to people, and how they are reduced. GDPR requires one when processing, especially with new technologies, is likely to result in a high risk.

Where people get tripped up is assuming that using a big provider’s model makes the provider responsible for everything. It does not: Acme decides what its agents are for and whose data they touch, so most of the duties in this post land on Acme.

What applies to one agent?

Several rulebooks apply to one agent at the same time, and they stack rather than replace each other. Passing one layer tells you nothing about the next.

An everyday example first. Opening a café is similar. Food hygiene rules apply whatever you sell. If you serve alcohol, a licensing layer is added on top. Some trades add their own rules, such as a café inside a hospital. A recognised management standard is optional, but it gives you a tidy way to run the checks. And your supplier contracts decide what the bakery may do with your orders. Pass the hygiene inspection and you still need the licence.

How does this apply to Acme? Think of the obligations as the same kind of layers, checked from the outside in:

five layers of obligation around one agent: privacy law such as GDPR for any personal data, AI-specific law such as the EU AI Act by risk level, sector rules such as DORA for finance, standards such as ISO/IEC 42001 and the NIST AI RMF that organise the work, and the contract with your model and cloud provider; the design controls in this series answer parts of each

The next sections take each layer in turn.

GDPR does not care that it is an agent; it cares that personal data goes in. Any name, question, memory or log line about an identifiable person brings the same duties as a spreadsheet would.

An everyday example first. A gym takes your name, date of birth and a note about a knee injury when you join. It needs a reason to hold each item (running your membership, keeping you safe on the machines). It should not later use the injury note to sell you insurance. It keeps only what it needs and deletes it a sensible time after you leave. You can ask to see or correct what it holds. If the gym bans you automatically because a turnstile counted three late arrivals, you can ask for a person to look at it. And if it stores your file with a company abroad, it needs a lawful way to send it.

A reason for each item, no quiet reuse   ->  lawful basis and purpose limitation
Keep only what is needed, then delete    ->  minimisation and storage limitation
See, correct, delete your file           ->  people's rights
A person reviews the automatic ban       ->  Article 22, automated decisions
The file stored abroad                   ->  international transfers

How does this apply to Acme? GDPR (and the UK’s version, UK GDPR) applies whenever the agent handles personal data, whatever technology it uses. The same five parts matter most for agents:

A lawful basis and a clear purpose. You need a reason the law accepts for each use of personal data, and you may not quietly reuse the data for something else. If Acme’s agent saves Maria’s preferences to improve her answers, that purpose should be written down; using the same memories to rate her performance would be a new purpose needing its own basis.
Minimisation and storage limitation. Keep only what the job needs, for as long as it needs it. In this series that is temp: state for tokens (Part 2.5), Memory Bank topics that limit what is kept (Part 8), and log retention you set on purpose rather than leave at the default.
People’s rights. A person can ask to see, correct or delete their data. For an agent, that includes memories and stored sessions, so design them to be found and deleted per user. ADK’s memory service has a delete call, and Memory Bank keeps memories per user (Part 8).
Automated decisions. Article 22 gives people the right not to be subject to a decision based solely on automated processing that has legal or similarly significant effects on them. Acme’s agents answer questions; they decide nothing about people. If an agent ever approves a loan, ranks job applicants or flags an employee, a person must genuinely review the decision, as the approval step in Part 9 does for Acme’s one write.
International transfers. Personal data leaving the EU or UK needs a legal mechanism. Pin the regions where models run and data is stored, and know where your provider processes it.

The European Data Protection Board’s Opinion 28/2024 adds a point specific to AI: a model trained on personal data cannot be considered anonymous in all cases, and a claim that it is must be assessed case by case. That mostly concerns teams that train or tune models (Part 14), but it is worth knowing before anyone proposes tuning on support tickets.

The hidden trap is thinking an internal agent is outside GDPR because it only serves staff. Maria’s questions, her saved region and the traces that record her name are personal data about an employee, so the five parts above apply to them as well (Part 8 and Part 11 show where they are stored).

What does the EU AI Act mean for an agent?

The AI Act regulates what a system is used for, not how clever it is. The same agent can carry almost no new duties in one job and the heaviest duties in another.

An everyday example first. A minibus is a minibus. Used to take friends to a festival, it needs an ordinary licence and insurance. Used every day to carry children to school, the same vehicle usually meets extra checks on the driver and the operator, because the use puts more people at risk. Nothing about the engine changed; the job did.

The minibus                     ->  the agent and its model
Friends to a festival           ->  a minimal or transparency risk use
The school run                  ->  a high-risk use, such as one listed in Annex III
Extra checks for the school run ->  the high-risk duties (risk management, logs, oversight)

The Act defines four levels of risk; after them, I show where Acme’s minibus is parked.

Unacceptable risk: banned practices, such as social scoring and harmful manipulation. The Commission’s page lists nine.
High risk: uses that affect safety or people’s rights, such as recruitment and CV sorting, managing workers, credit scoring, education and critical infrastructure. These carry the heavy duties: risk management, data quality, documentation, logging, human oversight, accuracy and cybersecurity.
Transparency risk: for example, people must be told when they are talking to an AI system such as a chatbot, and some AI-generated content must be labelled.
Minimal risk: no new obligations. The Commission says most AI systems used in the EU fall here.

The dates, as of November 2026, from the European Commission’s page:

1 Aug 2024   the Act entered into force
2 Feb 2025   bans on prohibited practices, and AI literacy duties, apply
2 Aug 2025   governance rules and duties for general-purpose AI model providers apply
2 Aug 2026   the Act is generally applicable, including transparency rules; the AI Office and
             national authorities enforce it
2 Dec 2027   rules for high-risk systems in areas such as employment, education, biometrics
             and critical infrastructure apply (moved by the AI Omnibus, in force 27 July 2026)
2 Aug 2028   rules for high-risk AI built into regulated products, such as lifts or toys, apply

Fines are set by use: for prohibited practices, up to EUR 35 million or 7% of worldwide annual turnover, whichever is higher (Article 99).

How does this apply to Acme? Where Acme sits, in my reading: an internal assistant that answers analysts’ questions about marketing and supply data is the festival trip: it is not in the high-risk list, so its main AI Act duties are transparency (people know they are using an AI) and AI literacy for staff. The same agents become the school run the moment someone points them at people decisions, such as using supply performance to rank employees, which falls under worker management. The use, not the code, decides the risk level, so write the intended use down and review the classification whenever it changes.

Does a human in the loop make an AI system low risk?

No. A person clicking the final button does not, by itself, take a system out of high risk. The use decides the risk level; the human is one of the controls the high-risk rules then require. If you remember one sentence from this section, make it that one.

An everyday example first. Think of a passenger aircraft with an autopilot. Having a trained pilot in the seat does not exempt the aircraft from certification; the pilot is part of what gets certified. Mapped to the AI Act:

Aircraft carrying passengers      ->  an AI system whose intended use is in Annex III
The pilot who can take over       ->  human oversight (Article 14)
Training against over-trusting    ->  awareness of automation bias (Article 14(4)(b))
  the autopilot
The flight recorder               ->  automatic logs (Article 12)
Certification before flying       ->  conformity assessment before launch (Article 43)
The pilot does not change what    ->  the person does not change the risk level;
  the aircraft is                     the intended use does

Where people get tripped up is carrying a GDPR idea across. GDPR Article 22 is about decisions based solely on automated processing, so genuine human review changes the analysis there. The AI Act works differently: it classifies by intended purpose. Annex III, point 5 lists, among others:

5(b): AI systems intended to evaluate the creditworthiness of people or establish their credit score (with an exception for systems used to detect financial fraud).
5(c): AI systems intended for risk assessment and pricing for people in life and health insurance.

A system that recommends a credit score or an insurance price is still intended to evaluate or assess. The person who confirms it is not an exit from Annex III; Article 14 demands that person.

Is there any way out?

There is one narrow exception, in Article 6(3). An Annex III system is not high risk if it does not pose a significant risk of harm, including by not materially influencing the outcome of the decision, and it meets at least one of four conditions: (a) it performs a narrow procedural task; (b) it improves the result of a previously completed human activity; (c) it detects decision-making patterns or deviations from earlier patterns, and is not meant to replace or influence the earlier human assessment without proper human review; or (d) it performs a preparatory task to an Annex III assessment. Two catches:

Profiling closes the door. An Annex III system that profiles people (evaluates or predicts things like their economic situation, health or reliability) is always high risk.
You must write it down. A provider who decides its Annex III system is not high risk must document that assessment before launch (Article 6(4)) and is still subject to registration (Article 49(2)).

A small example, in my reading: a tool that turns a scanned loan application into structured fields (name, income, employer) is a narrow procedural task and may qualify. A tool that reads the same form and suggests “decline, score 412” evaluates creditworthiness and profiles the applicant, so it does not. The three questions, in the order I would ask them:

how an Annex III system is classified: first the intended use, then profiling, then the narrow Article 6(3) exception; none of the questions asks whether a person clicks the final button, so a human in the loop changes the controls you build, not the risk level

What would a high-risk agent have to show? A hypothetical

A different company, not Acme. Acme’s analytics assistant answers Maria’s and Sam’s questions about campaigns and supply; it scores no one, prices nothing and is not this system. Imagine instead a lender, call it Northgate, that builds an agent to read a loan application, pull the credit file and recommend approve or decline with a score. An underwriter confirms each recommendation. That is Annex III point 5(b) with profiling, so it is high risk, and the button does not change that.

Here is the trap in numbers (illustrative, not from any study). Suppose one underwriter reviews 200 recommendations a day and agrees with 198 of them, at a median of 4 seconds each. That is 200 x 4 = 800 seconds, about 13 minutes of review for a day of credit decisions. Nobody can read a credit file in 4 seconds, so in practice the agent decides. Article 14(4)(b) names this: oversight must help the person stay aware of the tendency to over-rely on the output, automation bias, “in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons”. That is exactly Northgate’s set-up.

For a system like this, the Act puts duties on two roles. The provider (whoever builds it and puts it into service under its name, here Northgate itself) must meet Articles 9 to 15:

Risk management (Article 9): a continuous, iterative process across the system’s whole life, not a one-off review.
Data and data governance (Article 10): training, validation and test data that are relevant, sufficiently representative and as free of errors as possible, examined for possible biases.
Technical documentation (Article 11): at minimum the elements in Annex IV, written before launch and kept up to date.
Record-keeping (Article 12): the system must technically allow automatic logging of events over its lifetime, for traceability.
Transparency to deployers (Article 13): operation transparent enough for the deployer to interpret the output and use it properly, plus clear instructions for use.
Human oversight (Article 14): people able to understand the system’s limits, watch for automation bias, interpret the output, decide not to use it or override it, and stop it with “a ‘stop’ button or a similar procedure”.
Accuracy, robustness and cybersecurity (Article 15): including resilience to attacks such as data poisoning and adversarial inputs.
Post-market monitoring (Article 72): actively and systematically collect and analyse data on how the system performs in use, throughout its lifetime.

The deployer (the organisation using it, again Northgate if it builds its own) has duties of its own under Article 26: use the system as instructed, assign oversight to people with the competence, training and authority to do it, monitor the system in operation, keep the logs it generates for at least six months unless other law says otherwise, and tell people when a high-risk Annex III system is used to make or help make decisions about them. Article 27 adds a fundamental rights impact assessment (an assessment of how the system could affect people’s rights, and what reduces that) before first use. It applies to public bodies, private entities providing public services, and, named explicitly, deployers of Annex III points 5(b) and 5(c): the lender and the life or health insurer. Separately, Article 86 gives a person affected by such a decision the right to a clear and meaningful explanation from the deployer of the AI system’s role.

Who checks it? Not, usually, an outside auditor. Under Article 43(2), high-risk systems in Annex III points 2 to 8, which include credit scoring and insurance pricing, follow the internal control procedure in Annex VI: the provider assesses itself, without a notified body. The route with a notified body (an independent assessment organisation designated by a member state) is for biometric systems in point 1, in some cases. Internal control is not a lighter standard; it means the evidence must already exist in your own records when a market surveillance authority asks.

When? For Annex III systems, these rules apply from 2 December 2027, after the AI Omnibus, as in the timeline above.

How does each requirement become a control?

The good news for anyone who has followed this series: almost every requirement maps to a control we have already built or described, at least in outline.

from high-risk requirement to design control: each duty in Articles 9 to 15, 26 and 72 maps to a control this series builds or describes, such as audit logs for record-keeping, citations and logged tool calls for transparency, approval with friction and a kill switch for human oversight, release-gating evals for accuracy and online evals for monitoring after launch

Two of those rows need a word, because they are where a human in the loop either works or does not:

Approval with friction, against rubber-stamping. Part 9 builds a person’s approval for Acme’s one write. For a decision about a person you would add friction that Acme does not have: show the recommendation next to its evidence (the credit file lines and the policy rule it relied on, with citations as in Part 14), require the reviewer to confirm the evidence before the button unlocks, and record agreement, override and the reason. Logged tool calls (Parts 9 and 11) show the working behind each recommendation, which is what Article 13 transparency and an Article 86 explanation both need.
Stop, and monitor. Part 9’s kill switch ends every run with one setting, and Part 16 uses it to contain incidents. In Northgate’s case, stopping should route applications to a manual queue rather than stop lending. Part 10’s online evals then watch live traffic for drift, and for oversight itself: if the agreement rate climbs towards 99% while review time falls towards seconds, that is a signal for the deployer’s Article 26 monitoring and the provider’s Article 72 monitoring.

Why it fails, why it wins. Fails: Northgate ships an “Approve” button and nothing else. A market surveillance authority asks how oversight works; the only evidence is 40,000 clicks with no record of what each reviewer saw, how long they looked, or why anyone overrode. The button exists; the oversight cannot be shown. Wins: the same volume, but each review screen shows cited evidence, the reviewer confirms two pieces of it before deciding, overrides carry a reason, the audit log keeps every tool call for at least six months, a weekly report tracks agreement rate and review time, and one setting sends everything to the manual queue. Same people, same button; now the oversight is real and provable. (Both scenarios are illustrative.)

A litmus test, cheapest first:

Is the intended use listed in Annex III? If not, it is not high risk under Annex III; write the intended use down and stop here.
Does it profile people? If yes, it is high risk; Article 6(3) is closed.
Does it only do a narrow procedural, improving, pattern-checking or preparatory task, without materially influencing the decision? If yes, document that assessment and register.
Only then: plan the high-risk build. Articles 9 to 15, conformity by internal control, the deployer’s Article 26 duties, and a fundamental rights impact assessment if you are a lender, an insurer or a public body.

Lock it in:

High risk          set by intended use (Annex III), not by who clicks the button
Human oversight    a duty of high-risk systems (Article 14), not an exit from them
Automation bias    over-relying on the output; Article 14 says oversight must guard against it
Article 6(3)       narrow exit for procedural or preparatory work; closed if it profiles people
Conformity         Annex III points 2 to 8: internal control, no notified body
FRIA               deployers of credit scoring and life or health insurance pricing must do one
Date               2 December 2027 for Annex III systems, after the AI Omnibus

My view: the human in the loop is the most expensive control to fake and the cheapest to design in. Build the evidence screen, the override reason and the agreement-rate report on day one, and the human becomes your best control rather than a liability you have to explain.

One honest gap: Article 6(5) asks the Commission for guidelines with a list of practical examples of high-risk and not high-risk use cases, and that is the place to check borderline cases like the form reader above. My reading of Article 6(3) here is a starting point for your legal colleagues, not a classification, and none of this is legal advice.

What about sector rules and the rest of the world?

Sector rules sit on top of the general rules rather than replacing them, and outside the EU the general rules themselves differ. Three examples architects meet often:

Financial services in the EU: DORA, the Digital Operational Resilience Act, has applied since 17 January 2025. It is about ICT risk and resilience, including the risk from third-party providers, which covers the cloud and model services an agent depends on.
Health, public sector and employment each have their own rules on data, decisions and records, and often stricter regional laws.
Outside the EU, rules differ by country and change quickly. In the US, for example, there is no single federal AI law; sector and state rules apply. The UK applies UK GDPR and regulator guidance rather than an AI Act. Check the jurisdictions your users are in, not only where you are.

Which standards help you organise the work?

Three different things get called an AI governance framework: a law that says what is allowed, a playbook that says how to run the risk work, and a management system that an outside auditor can certify. Know which is which, and remember that none of them protects anyone until it can stop a release. A standard is not a law, and following one does not by itself make you compliant.

An everyday example first. Think about driving. The road rules and speed limits are law: they say what you may do, they are stricter outside a school, and breaking them costs you a fine. The driving manual is guidance: nobody fines you for skipping a chapter, but it is how careful drivers learn to check mirrors and plan a junction. The MOT is an outside check: an authorised tester inspects the car against a published list and issues a certificate you can show a buyer. And none of the three slows anyone down on its own; speed cameras and brakes do.

Road rules and speed limits    ->  the law: EU AI Act, what is allowed, by risk level
The driving manual             ->  the playbook: NIST AI RMF, how to run the risk work
The MOT, by an outside tester  ->  the certificate: ISO/IEC 42001, an AI management
                                     system audited by an accredited certification body
Speed cameras and brakes       ->  the teeth: controls that act on every release

Where people get tripped up is treating the three as alternatives, as if a team must choose the EU AI Act, NIST or ISO. They answer different questions, so an organisation that sells AI in the EU can use all three: the law for what is allowed, the playbook for how the work is run, and the certificate when a customer wants independent proof. The second trap is reading a certificate as a legal pass. An ISO/IEC 42001 certificate is not the AI Act’s conformity assessment (Article 43), in the same way that passing an MOT does not change the speed limit.

The law: what is allowed (the EU AI Act)

The sections above cover the Act in depth, so here is only its shape, in the Act’s own terms:

Prohibited practices (Article 5): banned outright, such as social scoring.
High risk, by two routes. Annex III lists eight areas by use: biometrics; critical infrastructure; education and vocational training; employment, workers’ management and access to self-employment; access to essential private and public services and benefits (which includes credit scoring and life and health insurance pricing); law enforcement; migration, asylum and border control; and the administration of justice and democratic processes. Separately, Article 6(1) and Annex I cover AI that is, or is a safety component of, a product already regulated by EU product law, such as medical devices, lifts or toys.
Transparency (Article 50): for example, people must be told they are interacting with an AI system, unless that is obvious.
Records: for a high-risk system, the provider (Article 19) and the deployer (Article 26(6)) each keep the automatically generated logs under their control for a period suited to the purpose, and for at least six months unless other law, such as data protection law, says otherwise.

The playbook: how to run the risk work (the NIST AI RMF)

The National Institute of Standards and Technology (NIST) is a US government agency that publishes standards and guidance. Its AI Risk Management Framework (AI RMF 1.0, publication NIST AI 100-1) was released on 26 January 2023 and is intended for voluntary use. Nothing in it is a legal duty. It is a structured way to do the work, organised as four functions, and the framework’s core numbers each outcome so teams can point at it:

Govern: policies, roles and accountability. NIST calls it a cross-cutting function, “infused throughout the other three”. It includes keeping an inventory of AI systems (Govern 1.6) and documenting who is responsible for what (Govern 2.1).
Map: the context. What the system is for, who uses it, which laws apply, and whether it should be built at all. Map 1.1 asks for the intended purposes and settings to be “understood and documented”.
Measure: testing and tracking: evals, red-teaming and monitoring the risks that Map found. Measure 2.3 asks for performance to be measured “for conditions similar to deployment setting(s)”.
Manage: acting on what was measured: prioritising risks, responding to incidents, and being able to “supersede, disengage, or deactivate” a system that behaves outside its intended use (Manage 2.4).

NIST’s own advice is that once Govern is in place, most users start with Map and continue to Measure or Manage, and then repeat. It has two companions worth knowing: the AI RMF Playbook, with suggested actions for each outcome, and the Generative AI Profile, NIST AI 600-1, released on 26 July 2024, which applies the framework to risks specific to generative AI. As of November 2026, NIST’s page says AI RMF 1.0 is being revised as part of the White House AI Action Plan, so check the current version before you cite a subcategory number.

The certificate: a management system someone can audit (ISO/IEC 42001)

A management system is the set of policies, roles, processes and records an organisation uses to run something the same way every time and to keep improving it. ISO and IEC publish standards that say what such a system must contain, and each organisation builds its own to meet them.

ISO/IEC 27001:2022 does this for information security. It sets the requirements for an information security management system (ISMS).
ISO/IEC 42001:2023 does the same for AI. Published on 18 December 2023, it “specifies the requirements and provides guidance for establishing, implementing, maintaining and continually improving an AI (artificial intelligence) management system” (an AIMS), for organisations that provide or use AI-based products or services.

               ISO/IEC 27001:2022                ISO/IEC 42001:2023
Manages        information security              AI systems you provide or use
System name    ISMS                              AIMS
Shape          policy, roles, risk assessment, controls, continual improvement
Certified by   an accredited certification body, after an audit of your organisation

Who issues the certificate? ISO and IEC write the standard; they do not audit you. An independent certification body audits the organisation and issues the certificate, and a national accreditation body checks that the certification body is competent and impartial. In the UK that is UKAS, appointed by government; the International Accreditation Forum links the national bodies so that accredited certificates are accepted across countries. Google Cloud, for example, says its ISMS “has received an accredited ISO/IEC 27001 certification after undergoing an audit by an independent third party” (ISO/IEC 27001), and lists Google Cloud, Google Workspace and the Gemini app as certified to ISO/IEC 42001:2023.

My view: for a provider of high-risk systems, an AIMS is a natural home for the quality management system that Article 17 requires, which includes “an accountability framework setting out the responsibilities of the management and other staff”. It organises that evidence; it does not replace the Act’s own conformity assessment.

How does this apply to Acme?

At Acme, this is how it would work, pillar by pillar, using the AI inventory and release records described under How do obligations become design?. Here is one inventory entry, for the supply agent:

AI inventory: supply agent                                         (illustrative)
Purpose       answers supply questions for Sam's team from supply_service
Must not      rank, score or monitor individual employees
Owner         one named person: the supply analytics lead
Users         supply planners, through the router in Gemini Enterprise
Data          Databricks supply_service, read only, as the signed-in user (Part 7)
AI Act        not in Annex III; transparency and AI literacy duties
Last release  eval gate passed (Part 10), model versions pinned, approved by owner
Next review   at any change of intended use, and at least once a year

The law sets the speed limit. The inventory’s AI Act line is the classification from earlier: the festival trip, not the school run. The “must not” line is what keeps it there, because ranking employees would move the agent into Annex III point 4.
The playbook organises the work. Govern is the owner and the inventory itself (Govern 1.6 and 2.1). Map is the purpose, users and “must not” lines (Map 1.1). Measure is the golden set results that each release record keeps (Part 10). Manage is the runbook and the kill switch (Part 16).
The certificate is optional for Acme. An internal assistant does not need one. If a large retailer later asked Acme for independent assurance about AI in a customer-facing service, an ISO/IEC 42001 auditor would sample exactly these records: the inventory, the release records, the incident log and the review dates. Google’s certificates cover Google’s management system, not Acme’s agents; as Google’s ISO/IEC 27001 page puts it, “Your organization will have to seek out and obtain its own certification”.

How do you stop a framework becoming shelfware?

A framework is real only when it can stop something: a project before it starts, a release before it ships, or a running system when it misbehaves. Shelfware first meant software a company bought and never used, left on the shelf in its box. In governance it means a policy that is written, approved and then never consulted: a long “AI principles” document that no pipeline reads and no meeting checks.

An everyday example first. Back to driving. Four things make the road rules bite:

A separate test before you may drive a coach  ->  a review board for high-risk uses,
  (an ordinary licence covers a car)              and a fast lane for low-risk ones
The MOT fail that keeps a car off the road    ->  an eval gate in CI that blocks a release
The registered keeper named for every car     ->  one named owner per AI system
Speed cameras that catch it, brakes that      ->  incident reporting, and a kill switch
  stop it

Where people get tripped up is answering a governance gap with more policy. A longer document changes nothing if nothing in the pipeline reads it. The opposite mistake is just as common: sending every request to a monthly review board, so a two-line prompt change waits weeks and teams quietly route around it, which is how Shadow AI starts (Prerequisite 4).

Here is each tooth, with the NIST function it puts into practice and the AI Act duty it helps you show for a high-risk system:

A review board for high-risk uses, and a fast lane for the rest. A small group (legal, privacy, security and the platform team) approves any use that could be high risk before work starts, while low-risk changes pass a short checklist the same day. These are the risk tiers from Part 17. NIST: Govern and Map; Map is where NIST puts the initial decision about whether an AI solution is appropriate at all. AI Act: the classification under Article 6, the written Article 6(4) assessment if you claim the narrow exception, and a fundamental rights impact assessment under Article 27 where it applies. At Acme: a new measure on campaign_performance for the marketing agent is internal, read only and decides nothing about people, so it takes the fast lane. A request to rank warehouse staff using the supply agent’s figures goes to the board, because that is Annex III point 4.
An eval gate in CI that blocks the release. Just as a failing unit test stops a build in a CI pipeline, a failing eval stops a release, because the gate returns a non-zero exit code. Part 10 builds Acme’s gate.py: groundedness must be 1.0 and golden numbers at least 0.9. In Part 10’s illustrative run, one ungrounded answer gave a groundedness of 0.500; the gate printed GATE FAILED: ['grounded'] and exited with 1, so the release stopped without anyone having to read a report. NIST: Measure. AI Act: testing under Article 9 and accuracy under Article 15, with the results kept as evidence for the technical documentation (Article 11).
One named owner per system. One person, not a team alias, who answers for the system’s purpose, its metrics and its incidents, and who approves its releases. A shared mailbox cannot be woken at 3 a.m. NIST: Govern (Govern 2.1, documented roles and responsibilities). AI Act: the Act never says “owner”, but deployers must assign human oversight to people with “the necessary competence, training and authority” (Article 26(2)), and a provider’s quality management system needs an accountability framework (Article 17). At Acme: the owner line in the inventory above, and the approver in each release record.
Incident reporting, with a kill switch. A known route for anyone to report a bad output, a clock for who must be told and by when, and one setting that stops the system while it is investigated. Acme’s kill switch (ACB_KILL_SWITCH, Part 9) ends every run before any model or tool is called; Part 11.5 puts it at the bottom of the fallback ladder, and Part 16 uses it to contain an incident before fixing it. NIST: Manage (Manage 2.4, deactivating a system; Manage 4.3, tracking and communicating incidents). AI Act: for high-risk systems, Article 73 makes the provider report any serious incident to the market surveillance authority. A serious incident is one that leads to, among other things, a death or serious harm to health, a serious and irreversible disruption of critical infrastructure, or an infringement of EU law that protects fundamental rights (Article 3(49)). The report is due immediately once a causal link, or its reasonable likelihood, is established, and no later than 15 days after becoming aware; no later than two days for a widespread infringement or a critical infrastructure disruption; and no later than ten days after a death. The deployer must suspend use when it sees a risk, and when it identifies a serious incident it must inform the provider first and then the authorities (Article 26(5)). Article 14’s “stop” button is the same idea built into the system. At Acme: the assistant is not high risk, so Article 73 does not apply to it. The same report route, clock and switch still run internally, because that habit is what you need on the day a high-risk system arrives.

Tooth                    NIST       EU AI Act duty it helps show (high risk)
Review board, fast lane  Govern,    Article 6 classification and 6(4) record,
                         Map        Article 27 FRIA where it applies
Eval gate in CI          Measure    Article 9 testing, Article 15 accuracy,
                                    Article 11 documentation
One named owner          Govern     Article 26(2) oversight people,
                                    Article 17 accountability framework
Incidents, kill switch   Manage     Article 73 (providers), Article 26(5)
                                    (deployers), Article 14 stop

three pillars and the teeth that make them real: the EU AI Act is the law that says what is allowed by risk level, the NIST AI RMF is the playbook for running the risk work through Govern, Map, Measure and Manage, and ISO/IEC 42001 is the management system an accredited certification body can audit; underneath, four operational teeth, a review board with a fast lane, an eval gate in CI, one named owner per system, and incident reporting with a kill switch, stop the framework from becoming shelfware

Why it fails, why it wins. Fails: a company publishes a long AI policy, approved by its board. Months later a team changes a prompt on a Friday afternoon. No gate runs, groundedness falls, and a customer-facing agent quotes a refund rule that does not exist for three days. Nobody is named as the agent’s owner, so the first complaint sits in a shared inbox until Monday, and nobody is sure who may switch the agent off. The policy existed the whole time. Wins: the same change fails the eval gate and never ships. When a different problem does slip through, the named owner is paged, turns on the kill switch within minutes, logs the incident with its timeline, and adds an eval case so it cannot return (Part 16). The policy is two pages long, and every line points at one of the four teeth. (Both scenarios are illustrative.)

A litmus test, cheapest first:

Can you name one person who owns each agent in production? An afternoon with the inventory answers it.
Can you stop each agent with a single setting, and has anyone practised doing it?
Does a failed eval return a non-zero exit code that stops your release pipeline?
Is there a written route, and a clock, for reporting an incident?
Do low-risk changes have a fast lane, so the review board sees only the uses that could be high risk?
Only then: write the longer policy, map it to the NIST functions, and decide whether an ISO/IEC 42001 certificate is worth it for your customers.

Lock it in:

EU AI Act      the law: what is allowed, by risk level        road rules, speed limits
NIST AI RMF    the playbook: Govern, Map, Measure, Manage     the driving manual
ISO/IEC 42001  an AI management system, certifiable by an     the MOT, by an outside
               accredited certification body                    tester
ISO/IEC 27001  the same idea for information security         the older sibling
Shelfware      a policy that nothing reads or enforces        rules with no cameras
The teeth      review board and fast lane, eval gate in CI,   speed cameras and brakes
               named owner, incidents and kill switch

My view: start with the teeth, then write the document. A named owner, a gate and a kill switch take days to put in place, and a policy that points at them fits on two pages. A policy written first tends to describe controls nobody has built.

What should you ask your model and cloud provider?

Part of your compliance depends on what the provider does with your data, so ask before you send any. You cannot delegate the duty, only the work.

An everyday example first. Before you put your files in a rented storage unit, you ask: will the company open my boxes, how long do they keep things after I stop paying, which site is it in, who else has a key, and who pays if something goes missing. The answers are in the contract, not in the brochure.

How does this apply to Acme? The same five questions, with what Google Cloud’s public documentation says for its generative AI services:

Is our data used to train models? Google’s terms include a training restriction: Google will not use customer data to train or fine-tune any AI or ML models without the customer’s prior permission or instruction.
Is it retained, and for how long? By default some data may be cached briefly to reduce latency, and prompts may be logged for abuse monitoring. Google documents the steps needed for zero data retention, which differ by feature.
Where is it processed and stored? Choose regions deliberately and check data residency for each service the agent uses.
Who else touches it? Ask for the list of sub-processors and the data processing terms; for Google Cloud these are in its Cloud Data Processing Addendum.
Who carries the risk if output infringes someone’s rights? Google Cloud announced generative AI indemnification for some of its services in October 2023. Check which services and conditions it covers for your use.

How do obligations become design?

Most compliance duties turn into a control you can build, test and show to an auditor. A duty that stays a sentence in a policy document is the one that fails the first audit.

An everyday example first. A restaurant kitchen has a rule: do not let raw meat touch food that is ready to eat. Nobody passes an inspection by quoting the rule. The kitchen turns it into things you can see: red chopping boards for raw meat, a separate shelf at the bottom of the fridge, and a temperature log filled in twice a day. The inspector checks the boards, the shelf and the log.

The rule                    ->  the obligation (for example, storage limitation)
Red boards, separate shelf  ->  the design control (temp: state, memory topics)
The temperature log         ->  the evidence (audit log, eval results, release records)
The inspector               ->  the regulator, a customer's auditor, or a court

How does this apply to Acme? Here are the common duties, with the control that answers each and the part of this series that builds it:

from obligation to design: each common compliance requirement for an agent maps to a concrete control in the build and the part that covers it, such as deletion rights to deletable memory, transparency to telling users they are talking to an AI, human oversight to approval for writes, and records to logs, evals and an AI inventory

Two kinds of evidence are easy to forget and painful to rebuild later:

An AI inventory. A list of every agent, its purpose, owner, model and version, data sources, risk classification and the date of its last review. NIST asks for one (Govern 1.6), an ISO/IEC 42001 auditor will ask to see one, and the supply agent’s entry above shows what a row looks like.
Release records. For each release, the eval results it passed (Part 10), the model versions it pinned, and who approved it. If a regulator, customer or court asks why the agent said something in March, these are what you show.

My view: bring legal and privacy colleagues in at the design stage, with a one-page description of the intended use, the data, the roles and the controls. It is far cheaper than a review after launch, and it usually makes the system simpler: much of what they ask for (minimisation, deletion, oversight, records) is good engineering anyway.

One honest gap: this field moves fast. The AI Act’s dates changed in 2026, guidance under it is still being published, and other countries are writing their own laws. Treat every date and rule here as true as of November 2026, and check the sources before you rely on one.

Try it: a one-page compliance brief

Fill this in for your own agent and take it to your privacy and legal colleagues:

Intended use      Who uses it, for what decision or task. What it must never be used for.
Roles             Controller / processor (GDPR). Provider / deployer (AI Act).
Personal data     Which data, in prompts, memory, state, logs, traces, eval sets?
Lawful basis      For each use of personal data. DPIA needed? (likely high risk?)
AI Act level      Prohibited / high risk / transparency / minimal, and why.
Decisions         Does it decide anything about a person? Who reviews it?
Rights            How a person's data is found, exported and deleted, including memories.
Retention         Session, memory, log and trace retention, and who set them.
Regions           Where models run and data is stored; any transfers out of the EU or UK.
Provider terms    Training, retention, residency, sub-processors, indemnity.
Sector rules      DORA, health, public sector, employment, others?
Evidence          AI inventory entry, eval results per release, approvals, audit logs.

What can go wrong

Treating security as compliance. Guardrails stop attacks; they do not give you a lawful basis, answer a deletion request or classify the system under the AI Act.
Forgetting where personal data hides. Logs, traces, memories and eval sets all hold it, and all need retention and deletion.
Function creep. An approved assistant is quietly reused for decisions about people, which can change both its GDPR basis and its AI Act risk level.
Assuming the provider covers you. The provider’s terms cover what the provider does. What your agent does with the data is your responsibility.
Relying on a date or rule that has moved. Check the source at design time and at each review.
Shelfware governance. A policy with no review board, no eval gate, no named owner and no kill switch reads well and stops nothing. Build the four teeth first.
Reading a certificate as legal compliance. An ISO/IEC 42001 certificate shows that a management system was audited; it is not the AI Act’s conformity assessment, and your provider’s certificate does not cover your agents.

What did we learn?

Compliance is about being allowed to run the system and proving it; security is about stopping misuse. You need both.
Know your roles first: controller or processor under GDPR, provider or deployer under the AI Act.
GDPR applies to personal data wherever the agent keeps it, including memory and logs; design for minimisation, deletion and human review of significant decisions.
The AI Act classifies by use; most internal assistants are minimal or transparency risk, but uses in areas such as employment are high risk, with those rules applying from 2 December 2027.
AI governance has three pillars that answer different questions: the EU AI Act is the law (what is allowed, by risk level), the NIST AI RMF is the voluntary playbook (Govern, Map, Measure, Manage), and ISO/IEC 42001 is an AI management system that an accredited certification body can audit, as ISO/IEC 27001 is for security.
A framework becomes real through four teeth that can stop something: a review board for high-risk uses with a fast lane for the rest, an eval gate in CI, one named owner per system, and incident reporting with a kill switch; Article 73 sets the serious incident reporting clock for high-risk providers.
A human clicking the final button does not take a system out of high risk: credit scoring and life or health insurance pricing are high risk by use (Annex III, 5(b) and 5(c)), and the narrow Article 6(3) exception is closed to systems that profile people.
For high-risk systems the human is a required control, built to resist automation bias, and Articles 9 to 15 map to audit logs, citations, evals, approval with friction, a kill switch and online monitoring; Annex III points 2 to 8 use internal control, not a notified body.

What comes next?

Production-ready agents, Part 1: What an agent really is, and the system we will build starts building the Acme system. Where a part adds a control that supports compliance, such as deletable memory in Part 8 or approval in Part 9, it is the same control mapped in this post.

Sources

Opinions are my own and this is not legal advice. Every fact in this post comes from the public sources linked above, as of November 2026. Acme Consumer Brands and its data are fictional.