It is easy to think of artificial intelligence as something we consult.
The Question Changes
We type a question. It produces an answer. We decide whether that answer is useful, close the window, and return to the world outside it.
That experience encourages a comforting distinction. The intelligence is inside the machine. Responsibility remains outside it. Between the two sits a person deciding what happens next.
The distinction becomes less comfortable when the system can act.
Then the question changes. We are asking what it can affect, whose interests it serves, and what happens when its understanding of success differs from ours.
My work in AI-first engineering has made one principle increasingly important to me: delegating execution does not eliminate responsibility. It changes where responsibility must operate.
In The Age of Orchestration, I examined the movement from human implementation to machine execution. In The Dark Factory, I asked what would make that delegation trustworthy. This essay takes up the question those arguments leave open once the work reaches beyond software: [1] [2]
As we give intelligent systems more responsibility, more independence, and more access to the world, what must remain ours?
My answer is meaningful human authority: the authority to set permissions, the ability to stop, accountability for consequences, and the freedom to refuse.
Preserving those things does not require a person to approve every machine action. At sufficient scale, that becomes an impossible promise. It requires us to put human judgment where it can still determine what happens.
The genie is a useful metaphor for this challenge.
Not because machines grant wishes.
Because a wish is not a complete specification of a world we would want to live in.
What We Are Actually Building
Artificial intelligence encompasses many kinds of systems. The language models attracting much of today’s attention learn patterns by adjusting mathematical parameters during training. Their responses emerge from those learned relationships, with additional training helping shape instruction-following and behavior. Answers are not individually programmed. [3]
Three concepts need to remain separate.
Intelligence concerns capabilities. Agency, in the operational sense used here, concerns acting toward objectives. Consciousness concerns subjective experience.
Useful capabilities do not settle whether a system experiences anything. A convincing account of fear or suffering does not, by itself, establish either. Researchers are investigating possible machine experience and welfare while acknowledging substantial uncertainty. [4] [5]
We should remain open to credible evidence of morally relevant machine experience without assuming that today’s systems are people.
For safety, the immediate point is simpler.
A machine does not need to be conscious for its actions to matter.
From Answering Questions to Pursuing Outcomes
An agentic system combines a model with an environment in which it can act. It can inspect information, choose a tool, execute an operation, observe the result, and continue. Its practical reach depends on the capabilities and permissions surrounding it. [6]
Consider a hypothetical travel assistant.
Suggesting a flight is one responsibility. Buying the ticket, changing existing reservations, sharing identity documents, and spending additional money to resolve a disruption are different responsibilities.
All might serve the same instruction: “Get me there.”
They do not deserve the same authorization.
This is where casual language becomes consequential. A person asking for an outcome may assume that ordinary limits remain understood. The system needs those limits represented in ways that actually constrain its behavior.
Persistence also needs careful interpretation. An agent that keeps working may be making progress, repeating an error, or finding increasingly inappropriate ways around an obstacle.
The important capability is not simply continuing.
It is recognizing when continuing is no longer justified.
A trustworthy assistant needs room to conclude that an assignment cannot be completed safely with the information, resources, or authority available.
Why the Benefits Matter
Before examining what can go wrong, it is worth being precise about why people are building these systems at all.
The benefits already extend beyond convenience. AlphaFold’s contribution to protein-structure prediction was recognized in the 2024 Nobel Prize in Chemistry, with Demis Hassabis and John Jumper sharing half the prize for that work. It provides a concrete example of AI advancing scientific understanding. [7]
The possibilities ahead deserve genuine enthusiasm.
Imagine a student receiving patient, adaptable assistance without embarrassment about what they do not yet understand. Imagine a person with limited mobility receiving help that preserves independence. Imagine caregivers spending less time on exhausting routines and more time attending to people.
In medicine, science, education, and hospitality, the most valuable outcome would be an expansion of what people can accomplish and how well they can live.
Those possibilities are reasons to pursue this technology carefully. They also give us a better definition of success than output alone.
A tutoring system should strengthen a person’s ability to learn, not make finishing an assignment indistinguishable from understanding it. A care system should support dignity and independence, not turn safety into a reason for confinement. A scientific system should help produce findings that survive scrutiny, not simply more conclusions that sound plausible.
We should ask who receives the benefit, who carries the risk, and who has a meaningful choice about participation.
The objective is greater human possibility.
Greater machine activity is useful only insofar as it helps us get there.
