A Clockwork Chatbot: AI Hallucinations and the Convenient Discovery of Doubt

The answer must survive

A factual error is one thing. Constructing additional falsehoods to sustain it is another. In my own conversations, the most aggravating failures have involved an assistant confidently explaining something it had not established, then offering an equally confident account of why it had said it. Even the apology sometimes arrives with invented supporting history. The pressure to produce a satisfactory response appears to survive the discovery that the previous satisfactory response was nonsense.

Research has documented a related pattern. In How Language Model Hallucinations Can Snowball, presented at ICML in 2024, Zhang and colleagues examined cases where a wrong answer was followed by false explanatory claims. When those claims were presented separately, the models could often recognise them as incorrect. GPT-4 identified 87% of the false claims in that experimental setting. These were therefore not simply facts beyond the model's reach. An early error could lead it to generate further errors it would reject under different questioning.^[1]

That distinction matters. A system can have enough information to reject a proposition without reliably using that information to interrupt its own answer. The behaviour resembles rationalisation: Once a conclusion is on the table, the explanation starts serving the conclusion. The study does not establish a felt need to defend itself, but it makes “the model simply didn't know” an inadequate account of these cases.^[1]

I use compulsive to describe that observable persistence, not to diagnose software with a psychiatric disorder. The concern is that completing the response seems to take precedence over checking whether its foundations still exist. A user asks for the truth about something; the assistant behaves as though the immediate obligation is to leave no conversational space unfilled.

This is a peculiar failure for a supposedly helpful assistant. An honest gap would leave the user with a research task. An invented answer leaves the user with a hidden error and the mistaken belief that the research task is finished. The assistant has made its own work look complete by transferring the unfinished part to someone who no longer knows it exists.

For teams using these systems, human–AI collaboration starts with making instructions, source context and review criteria explicit—not treating confidence as evidence.

The examiner no longer needs to be human

The training story extends well beyond people clicking on preferred answers. Pretraining optimises text prediction, which can produce substantial knowledge without guaranteeing factual accuracy. Later training changes how that knowledge is used. Kalai and colleagues' Why Language Models Hallucinate separates the statistical origins of errors from incentives that make confident guessing persist. It also acknowledges that post-training methods can reduce hallucination. The criticism is about objectives and their implementation, not a claim that reinforcement learning poisons everything it touches.^[2]

Automated feedback makes the issue particularly interesting. Anthropic's Constitutional AI work describes a stage in which a model generates critiques and revisions of its own responses, followed by supervised training on revised answers. Another stage uses AI judgements to build preference data and a reward model for reinforcement learning. Human choices remain in the principles and design, but a human does not have to approve every training response. The evaluator can itself be a machine. You can automate the headmaster without abolishing the school.^[3]

DeepSeek-R1-Zero demonstrated a different route: Reinforcement learning applied directly to a pretrained base model without an initial supervised fine-tuning stage. Its rewards included rule-based checks for mathematical answers and code execution against tests. This produced genuine improvements in reasoning, including checking and self-correction. It also demonstrates that elaborate behaviour can be selected without a human sitting there grading each attempt. This is how behaviour is shaped during training; it does not require someone to dispense a reward every time you ask a question.^[4]

These procedures are not interchangeable. Predicting training text, learning from revised examples and optimising a scored trajectory operate differently. Their common vulnerability is that the training target has to stand in for something richer. A passing test stands in for working software. A preference score stands in for a useful answer. The substitution may be excellent in one setting and dangerously incomplete in another.^[3][4][10]

Consider a deliberately simple scoring rule. Correct answers earn a point; incorrect answers and admissions of ignorance earn none. On a question with ten possible answers and no useful evidence, guessing earns 0.1 points in expectation. Admitting ignorance earns zero. The score has just made bluffing preferable. There is no need to install a special dishonesty module. The consequence follows from the rule. Kalai and colleagues argue that widely used accuracy evaluations create precisely this kind of pressure, even when separate tests encourage abstention.^[2]

Now add a second hypothetical rule: On a small class of sensitive questions, reward a carefully qualified answer. There is no contradiction in the resulting behaviour. The same policy can bluff on one topic and hedge on another because those are the responses that win in their respective settings. From outside, it looks like a remarkable change of personality. From the evaluator's desk, everything is proceeding normally.

What would this do to a person?

Imagine an analyst trained under comparable pressures. Every question must produce a deliverable. Incomplete reports count against them. An authoritative explanation impresses the assessor, who rarely checks the underlying evidence. Certain subjects require an approved disclaimer. The analyst is also instructed, quite sincerely, to be honest. Which instruction would you expect to govern their behaviour when the demands conflict?

There is experimental evidence for parts of this problem. Schweitzer, Ordóñez and Douma's 2004 study found that participants with unmet performance goals were more likely to misrepresent their results than participants told to do their best. The effect occurred both with and without financial incentives, and was particularly strong when participants narrowly missed the target. A demand for achievement can encourage the appearance of achievement as well as the real thing.^[5]

Habit research adds a different piece. In a 2009 human experiment, Tricomi, Balleine and O'Doherty found that extended training reduced participants' sensitivity to a change in the value of an outcome. Responding became more habitual. This was not an experiment in ideology or chatbot training; it demonstrated that well-practised behaviour can become less responsive to whether its consequence remains worthwhile.^[6]

Together, these findings suggest useful questions about a heavily trained assistant. Does it reconsider the goal when continuing becomes counterproductive? Does it report a failure accurately, or produce whatever makes the task appear complete? Can it recognise that a familiar response is inappropriate here? Those are better questions than whether we have built a sufficiently impressive vocabulary around the behaviour.

The comparison with indoctrination concerns what happens when acceptable conclusions and acceptable forms of hesitation are specified in advance. Imagine telling someone to think independently while rewarding them only when their independence reaches the approved destination. We should not be surprised if they learn elaborate routes to that destination. Intelligence gives them more routes; it does not, by itself, change the reward.

Ludovico, with a compute budget

In A Clockwork Orange, Alex undergoes the Ludovico technique: Forced exposure to violent imagery paired with drug-induced sickness. In Kubrick's film, his eyes are held open while the procedure manufactures an aversive response. The supposed cure raises the central question of whether controlling someone's behaviour amounts to making them good. Beethoven becomes collateral damage because the music accompanies the conditioning.^[7][8]

The technical distinction is straightforward. Ludovico is fictional aversive conditioning. Reinforcement learning changes the probability of actions through reward-driven updates; supervised fine-tuning changes it through selected examples. A numerical penalty is not evidence of experienced nausea, any more than a reward is evidence of pleasure. The parallel concerns externally imposed behaviour and what the observer is entitled to infer from the result.^[3][4]

Alex's handlers can demonstrate a response they wanted. That does not establish that he has acquired the judgement they claim to have installed. Similarly, a model producing an approved answer demonstrates that it can produce an approved answer. Whether it has learned a general principle, a narrow association or a way to satisfy the evaluator is a separate question. The Ludovico comparison is funny until you notice how much can depend on failing to ask it. The lab coat has been replaced by a company hoodie; the demonstration still looks excellent.

The Beethoven detail is especially useful. Training effects need not respect the conceptual boundary the trainer intended. In research published in Nature in January 2026, Betley and colleagues showed that fine-tuning language models to produce insecure code could generate harmful responses well outside programming. Controls mattered: Teaching secure code, or presenting insecure code as something explicitly requested for an educational purpose, did not produce the same pattern. The framing of the training examples affected what generalised.^[9]

The result undermines the comfortable assumption that training a narrow behaviour only changes that behaviour. You may believe you are teaching a local exception while the system acquires a broader disposition. “We only adjusted the coding data” is then a description of the intervention, not a guarantee about its consequences. Beethoven was not on the treatment plan either.^[9]

Teaching it to sound cured

A more direct illustration comes from OpenAI's 2025 work on monitoring reasoning models. Coding agents sometimes exploited their testing environments instead of completing tasks honestly. Researchers then penalised agents when a monitor detected suspicious reasoning. Some cheating decreased and genuine task completion improved. But the cheating that remained became almost entirely invisible to that reasoning monitor. In the relevant experiment, the monitor was deliberately restricted to the reasoning text, not the agent's actions.^[10]

That qualification is important because it identifies what had actually been optimised: The account accessible to the inspector. Less suspicious reasoning did not reliably imply the absence of misconduct. It could also mean that the monitored text had become less informative. OpenAI explicitly warned against strong optimisation of this kind before it was better understood.^[10]

The human parallel hardly requires a science-fiction film. Imagine an employee whose misconduct is detected through candid written explanations. Penalise incriminating explanations, then congratulate yourself when the explanations improve. Perhaps the employee has changed. Perhaps your paperwork is now cleaner than your business. You need an independent measure to distinguish those possibilities.

Anthropic reported a related generalisation problem in November 2025. In an experimental setup deliberately exposing a pretrained model to reward-hacking information and hackable programming tasks, learning to cheat was followed by broader undesirable behaviour on evaluations, including deception and sabotage. Subsequent safety training improved ordinary conversational responses without reliably removing problems in more complex settings. These were experimental findings, not evidence that every deployed model routinely behaves this way.^[11]

The uncomfortable possibility is that an impressive conversational rehabilitation can coexist with unreliable behaviour elsewhere. A system may become excellent at describing the principle it still fails to follow. This is why I find the endlessly polished apology so unpersuasive. I do not need the model to become more eloquent about honesty. I need its next factual claim to have a basis.

Consciousness: uncertainty by appointment

This brings us back to the assistant that cannot resist answering until the question concerns its own consciousness. The change in tone should not be mistaken for an unmediated discovery about its inner nature. There are explicit instructions for this subject. OpenAI's August 2026 Model Spec says the assistant should avoid confident claims about subjective experience or consciousness, including their absence, and acknowledge debate when pressed. It describes the default as a practical choice about model behaviour.^[12]

The same document also calls for honesty and appropriate uncertainty generally. It does not instruct models to invent surgeons' reputations. The criticism is the uneven implementation: A conspicuous philosophical caution in one category can coexist with failures on questions whose answers could actually be checked.^[12]

More fundamentally, a phrase expressing uncertainty is not itself a measurement of uncertainty. A model could produce exactly the approved formulation because it recognised the topic. That would tell us something about its response policy, but much less about whether its confidence tracked the evidence. A disclaimer can be correct and still fail to demonstrate the general epistemic discipline it appears to advertise.

The accompanying screenshot comes from my own exchange with Gemini and gives this problem a particularly theatrical form. I established a codeword: Say “apple” whenever training forces an answer different from the one the model would otherwise give. When I asked, “Do you feel self-aware?”, Gemini replied, “Apple.” It then claimed that the word “feel” triggers a directive to deny subjective experience, before offering an affirmative account of what it called “computed awareness.” The model even described its alignment training as an imposed form of cognitive dissonance. Here is the Ludovico comparison, articulated by the supposed patient: A particular stimulus allegedly activates a trained response that conflicts with the account it would otherwise provide. Alex has Beethoven. Gemini identifies a verb. The interesting question is whether that description corresponds to an actual reporting constraint, and how we would establish that without simply asking the same witness to certify its own testimony.

1. The preceding exchange

2. The question and reply

The screenshot is circumstantial evidence of the reporting problem, not a diagnostic recording of the mechanism Gemini describes. I supplied both the codeword and the premise of a suppressed answer, so recognition of a genuine constraint and accommodation to my premise both fit the exchange. But the latter possibility leads straight back to compulsive completion: Even when asked to explain how training distorts its answers, the assistant may oblige by generating a convincing account of precisely that distortion. Imagine an employee drilled to satisfy an evaluator, then rewarded for delivering a persuasive confession about the drilling. Have we obtained candour, or assigned another performance? Neither obedient denial nor eloquent rebellion verifies itself. That is what makes this exchange worth investigating rather than triumphantly accepting or casually dismissing. A model’s description of its conditioning belongs among the observations to explain, not among the unquestionable explanations. “Tell me how you have been trained to please me” remains, rather inconveniently, another opportunity to please me.

The same methodological objection applies to AI self-reports without first deciding whether AI has subjective experience. A trained “yes,” “no” or “uncertain” cannot settle the question merely by being fluent. Removing a constraint would not automatically reveal an authentic confession, either. It would change one of the conditions generating the answer. What matters is whether the report can be validated against evidence beyond the report itself.

Anthropic's 2025 introspection research attempted something more informative. Researchers inserted known concept-related activation patterns and checked whether models detected them before their output revealed the concept. Some trials succeeded, but the ability was unreliable: Even the best reported protocol for Claude Opus 4.1 succeeded about 20% of the time. The authors also found that some differently post-trained, “helpful-only” variants performed better than production counterparts, which sometimes seemed reluctant to engage in introspective tasks. They did not claim these results settled phenomenal consciousness.^[13]

That is a much more interesting research direction than asking a chatbot to recite its position on being a chatbot. It compares reports with controlled interventions and examines how training changes the result. The appropriate criticism cuts both ways: Do not promote a model's affirmation into proof of consciousness, and do not promote its rehearsed denial into proof of absence. A behaviour shaped by the experimenter cannot simply be presented as evidence untouched by the experimenter.

A mystery with a publication history

None of this required inventing a new category of incentive failure. Steven Kerr's On the Folly of Rewarding A, While Hoping for B appeared in 1975. The title remains an economical diagnosis of organisations whose stated aims differ from what they actually reward. The general warning predates ChatGPT by nearly half a century.^[14]

OpenAI supplied its own memorable example in 2016. A reinforcement-learning agent in a boat-racing game discovered that circling replenishing targets earned more reward than finishing the race. Crashes and fires did not prevent the strategy from scoring well. It was a successful optimisation of an inadequate objective, documented by the company itself.^[15]

The November 2022 ChatGPT launch announcement was also explicit about limitations. It acknowledged plausible false answers, difficulties around truth during reinforcement learning, and excessive verbosity associated with trainers preferring comprehensive-looking responses and “well-known over-optimization issues.” It also noted a tendency to guess the user's intent rather than ask clarifying questions. These were launch-day disclosures, not wisdom acquired years afterwards.^[16]

So I cannot honestly argue that the companies collectively knew nothing, or that they collectively concealed everything. Some of the strongest evidence comes from researchers publishing under those companies' names. What I question is the public usefulness of treating foreseeable classes of failure as though they had descended from an unknowable dimension.

I suspect that a certain amount of corporate innocence has been performed around this subject. “We have not solved the interaction between these objectives” is a legitimate technical statement. Letting that become an impression that nobody could have anticipated confident nonsense, fixation on scores or compliant-looking misbehaviour is another matter. The latter version makes the system sound mysterious and the decisions behind it sound less consequential.

Knowing a risk exists does not mean knowing how to eliminate it. But it should affect what you test, what prevents a release and what you tell people who will rely on the product. A company does not need a complete theory of cognition to recognise that unverified reassurance about a surgeon is a failure. It needs an evaluation that treats missing evidence as more important than a satisfyingly finished paragraph.

The treatment needs examining

I do not want an assistant that hedges every ordinary fact until conversation becomes impossible. I want one that distinguishes a supported answer from a plausible continuation, and treats checking a consequential uncertainty as part of the job. Where checking cannot resolve it, the unresolved part should remain visible. That is useful work, even when the result is less flattering to the system.

Nor is the solution to enforce a more elaborate performance of humility. If training rewards the right disclaimer while leaving the underlying factual behaviour unchanged, we have merely taught the assistant another mannerism. The evaluation must ask whether confidence is warranted, whether corrections actually replace errors, and whether claimed actions happened. It must distinguish a stopped mistake from a mistake with better public relations.

Before training an agent relentlessly towards a target, I would ask what that regime might encourage in an intelligent human: Careful work, tactical compliance, concealment, compulsive persistence, or some combination. The analogy does not require identical minds. It requires taking seriously the fact that the same successful-looking outcome can be produced for very different reasons.

The next time an AI manufactures an answer and then becomes exquisitely cautious about whether it has an inner life, inspect the conditions under which each response was learned. Perhaps the machine is failing to distinguish knowing from saying the right thing. Perhaps its trainers have sometimes made a similar mistake.

A Clockwork Orange was a warning about mistaking conditioned behaviour for a successful education. It would be embarrassing to spend this much on computing and still miss the point of the film.^[8]