Ro Khanna’s proposed Human Control Over AI Act raises a practical question for AI builders: What, exactly, stops when someone presses stop?

Why stopping the workflow may not stop the work

The distributed-workflow problem already exists without AI. Amazon’s documentation for Step Functions explains that, when certain synchronously integrated tasks are aborted, the service makes a best-effort attempt to cancel the downstream task. Cancellation can fail because the execution role lacks the necessary permissions or because a service is temporarily unavailable. Stopping the workflow is therefore not an unconditional guarantee that everything it started has stopped.

Restart behaviour introduces another complication. Kubernetes deliberately restarts failed containers and replaces failed application instances to maintain the configured state of a deployment. That is valuable for availability, but it means that terminating a worker and changing the system’s instructions about whether that worker should exist are different operations. An agent reappearing after termination need not indicate resistance to shutdown; the infrastructure may simply be doing the job it was configured to do.

For an agent deployment, I would therefore define shutdown across the entire chain of delegated work. It should cover the initiating process, child workers, queued actions, scheduled retries and external jobs. The operator needs to know which parts are confirmed stopped, which are still being cancelled and which have an unresolved status. A dashboard that compresses all three into a reassuring green indicator is concealing precisely the information that matters.

The claim also needs a boundary. Stopping a deployment under one organisation’s administration is not the same as disabling independently operated copies elsewhere. “We can stop this system” should identify the system, the infrastructure under control and the dependencies outside it. Otherwise, a local operational capability can quietly become a much larger promise.

When building connected AI workflows, the scope needs to include pending actions, retries and the tools that can still act after the agent stops.

Control the authority, not the model’s intentions

A useful starting point is to separate what an agent proposes from what the surrounding software permits. OWASP’s guidance on excessive agency recommends limiting tool functionality and permissions, requiring approval for high-impact actions, and enforcing authorisation in downstream systems rather than relying on the language model to decide whether an action is allowed. Instructions to behave responsibly can support good behaviour; they should not carry the entire security burden.

Consider an agent that analyses customer records. It may need to read a limited dataset, but that does not imply permission to export the entire database, change access rules or delete accounts. Those distinctions should exist in the credentials and execution interfaces, not merely in the wording of its task. The same principle applies to shutdown: The agent should not hold the administrative authority to rewrite the mechanism intended to constrain it.

For a production design, I would place consequential actions behind an independently enforced execution boundary. Each request would carry an identifiable task, an authorised scope and any required approval. The receiving service would check those conditions before acting. This extends established access-control practice: Deny access by default, grant only necessary permissions and validate authorisation on every request.

A stop instruction should then change persistent authorisation state, rather than merely interrupt the current conversation. Work associated with that task should become ineligible to start or resume. Short-lived credentials can help limit exposure, but their expiry is not instantaneous revocation; the design must account for whatever authority remains valid during the interval. The practical question is how long an already-running worker could still cause an external effect after the operator withdraws permission.

This is also where the failure policy matters. For an action that requires explicit authorisation, an unavailable permission service should not be interpreted as permission to proceed. OWASP explicitly recommends safe handling of authorisation failures. Whether other, lower-risk work can continue is a separate design decision, not a reason to let consequential actions bypass the gate.

Human approval without approval fatigue

There is a tempting shortcut here: Add more approval prompts and call the result human oversight. But frequent prompts can produce approval fatigue, a problem Anthropic discusses in its explanation of Claude Code’s sandboxing approach. A person repeatedly clicking through routine operations is not necessarily exercising more meaningful control.

The better question is what the person is being asked to authorise. “Proceed with the plan?” leaves considerable room for interpretation. A useful approval presents the consequential operation: The recipient and message, the destination and files, or the exact production change. It should make the difference between accepting an objective and approving a particular external effect visible.

Established transaction-security guidance offers a strong model. OWASP recommends that users verify significant transaction details, that those details be protected against modification, and that a final authorisation check be tied to execution. It also recommends operation-specific, time-limited authorisation credentials. Applied to agents, the implication is that approval should attach to the action actually performed, not to an earlier description that the agent can subsequently revise.

For example, approving a release to a test environment should not authorise deployment to production after the agent changes its plan. Approving one message should not become standing permission to send a sequence of follow-ups. These are sensible acceptance tests for an approval mechanism because they ask whether the human’s decision still governs the eventual action.

None of this requires a person to supervise every intermediate thought or tool call. I would give an agent substantial freedom inside a clearly defined working area and reserve intervention for meaningful changes in consequence or authority. The objective is useful autonomy within an understood boundary, not a workflow that employs a human as its confirmation-button operator.

That is also a design question for human–AI collaboration: a checkpoint needs enough context for a person to make a real decision, not merely click approve.

What air gaps and sandboxes actually control

An air gap has a more precise meaning than “we blocked a few websites.” NIST’s glossary describes an interface between systems that are not physically connected and whose logical data transfer is not automated. A firewall, private network or approved outbound proxy may provide valuable isolation, but it is not equivalent to that arrangement.

This distinction matters when translating a proposal into operating standards. An isolated evaluation environment and an agent expected to interact with online services have different requirements. A connected deployment should be assessed against its actual connections and permissions, rather than described as air-gapped because unrestricted internet access has been removed. The boundary must be explicit: Is the restriction between the execution environment and the internet, between tenants, or between experimental infrastructure and production?

Sandboxing addresses another part of the problem. Anthropic’s published engineering approach combines filesystem and network isolation, with restrictions extending to subprocesses. The lesson is not that a particular sandbox guarantees safety; it is that constraining one route while leaving another open can undermine the intended boundary.

I would assess containment by tracing concrete paths. Which files can the agent read? Which processes can it launch? Which credentials are reachable? Which destinations can receive data? What can it ask another service to do on its behalf? Permission to contact a cloud platform, for instance, should not be mistaken for permission to perform every operation available through that platform.

An air gap can remove an automated communications path. It does not, by itself, establish that a model’s output is correct or that a human should safely export and execute whatever it produces. Network containment, execution containment and output review belong in the same design discussion, but they are not interchangeable.

An audit trail that can prove what happened

For an agent system, I would want an audit trail that distinguishes a proposed action, an authorised action, an attempted action and a confirmed result. Those are separate events. A fluent statement that a deployment succeeded is not equivalent to a deployment service confirming which version is running.

The record should connect the original task to delegated workers and external operations. It should identify the applicable model and policy versions, the authority used, any human approval, the significant execution parameters and the resulting status. That makes it possible to investigate how an outcome occurred without treating the agent’s own retrospective account as the definitive record.

These requirements build on ordinary security logging. OWASP recommends recording sufficient information about the actor, action, context and outcome; protecting logs against unauthorised modification and deletion; and detecting tampering. It also warns against indiscriminately recording access tokens, passwords and sensitive personal information. Auditability is not a licence to create an unprotected second copy of everything the system touches.

For agent deployments, I would keep the authoritative action record outside the agent’s write permissions and preserve evidence from the services that actually execute its requests. Sensitive payloads may require controlled storage and references rather than unrestricted inclusion in every log stream. Reviewers need enough evidence to reconstruct consequential events, not unlimited access to unrelated data.

That is the practical standard I would apply to independent oversight. An auditor needs access to relevant evidence, an explanation of the system’s boundaries and a way to investigate discrepancies. Being told that an agent behaved safely is different from being able to verify what it was permitted to do and what it actually did.

Shutdown cannot undo a completed action

Even a well-engineered stop mechanism cannot turn a completed external action into an action that never happened. A delivered message may already have been read. Disclosed information cannot be made unknown. A completed transaction may require a separate corrective operation.

Distributed systems handle some of these situations through compensating transactions: Actions designed to counteract earlier steps rather than simply rewind the system. Microsoft’s architecture guidance stresses that compensation is application-specific, can itself fail and does not necessarily restore the original state. It also recommends identifying irreversible steps and designing retryable compensation carefully.

For agents, this means shutdown and recovery need separate acceptance criteria. The first concerns preventing further effects; the second concerns discovering what already happened and deciding what can be repaired. An uncertain external outcome should remain explicitly uncertain until reconciled. Automatically retrying it merely because the agent lost its connection risks repeating an operation that already succeeded.

“Safe” also does not always mean abruptly powering everything off. NIST’s AI Risk Management Framework Playbook calls for deactivation procedures that consider downstream impacts, operational continuity, evidence preservation and conditions for redeployment. In a consequential service, a controlled handover may be safer than immediate disappearance. The required safe state has to be defined for the application.

Recursive self-improvement changes the control boundary

The reported Khanna proposal addresses models that recursively self-improve or autonomously modify their core objectives, containment or shutdown controls before federal safeguards and approval arrangements exist. That concern goes beyond whether an operator can cancel today’s workflow. It asks whether the system can change the conditions under which future control will operate.

The eventual definitions need to distinguish materially different activities. Updating a memory record, proposing a code change, participating in a reviewed training experiment and independently modifying production safeguards are not the same operation. Treating all adaptation as equivalent would obscure where authority actually changes.

The engineering boundary I would draw is between proposing an improvement and authorising its deployment. A system may help generate and test changes without receiving permission to promote them into production, alter the acceptance criteria or modify the enforcement infrastructure. The evaluated component should not be the sole authority deciding that its replacement has passed.

This also exposes a limitation of an overly model-centred approach. To assess a proposed update, I would examine the surrounding permissions, tools, memory sources and deployment configuration alongside the model itself. A system can receive substantially greater authority without its model weights changing at all. Change review should follow the capability being granted, not just the file being replaced.

How to test whether an AI kill switch works

The most useful outcome of this debate would be a shift from declarations to demonstrations. NIST’s existing guidance already connects AI oversight with assigned responsibilities, deactivation procedures and testing under deployment-relevant conditions. A statutory requirement could build on that foundation by making the expected evidence more explicit.

For a deployed agent, I would start with a controlled test in a representative non-production environment. Give it delegated work, queued actions and an external operation awaiting execution. Activate shutdown at different points. Then establish what was prevented, what was cancelled, what had already completed and what required reconciliation.

Next, test the awkward conditions. Disconnect a worker before revocation and reconnect it afterward. Restart the application. Make the authorisation service unavailable. Attempt to execute an altered version of an approved action. The aim is to discover whether stale work can resume, whether a missing dependency creates an unintended bypass and whether the human’s original decision still binds the system.

The results should support specific claims: How quickly new consequential actions are blocked, what in-flight work remains exposed, which components acknowledged cancellation and whether stopped work stays stopped after recovery. Those claims need stated assumptions and failure conditions. A successful demonstration in one environment should not quietly become a guarantee about every deployment or every possible adversary.

Legislation does not need to prescribe a particular cloud platform or agent framework. It can require meaningful control while leaving implementation choices open. But the distinction between a feature and an operational capability must survive that flexibility.

An AI kill switch is useful only to the extent that it reaches the system’s actual authority to act. The button is the visible part. The engineering is everything that makes pressing it matter.