AI research is a methods problem, not a coding problem

Everyone sells AI research now

Language models made it cheap to produce text that looks like research. The market responded quickly: AI research services now promise literature reviews, market scans and due diligence in hours instead of weeks. Some of that speed is real. The problem sits elsewhere.

A deliverable that reads like research and a deliverable that survives scrutiny are different products. From the outside, they can be hard to tell apart.

Code is not methodology

Many of these offerings are built by developers, and often capable ones. That is not a criticism of their engineering. Building a retrieval pipeline is a coding task; deciding what counts as evidence is not.

Methodology is its own discipline: framing a question so it can actually be answered, choosing sources and recording what was excluded, weighing study quality, keeping direct evidence apart from inference. These are trained skills, not features you can add to a prompt.

The result is a new kind of vibe coding. A pipeline gets tuned until the output reads convincingly, the way a vibe coder iterates until the app looks right. Fluency becomes the acceptance test. But a fluent paragraph can rest on a misread abstract, an invented citation, or a real paper cited for a claim it does not support. Nothing in the pipeline notices, because nothing in the pipeline was built to.

How the human leaves the loop

Few providers set out to remove the researcher. It happens one stage at a time. First the search is automated, then the screening, then the extraction, then the summary. Each step is defensible on its own.

Together they shrink the human role from making judgements to approving outputs, and approval without inspection is not oversight. The end state is a report nobody can fully account for: no one has read the sources behind the key claims, and no one can say why a study was excluded.

What a methodologist adds

• A decision question, framed before any searching starts, so the review answers the thing you actually need to decide.
• Deliberate source selection, with the limits recorded, so the gaps stay visible instead of hidden.
• Appraisal of quality and directness: whether a study measures what your claim needs, not whether it sounds supportive.
• Separation of observation, estimate and interpretation, so a reader can challenge one layer without dismantling the whole report.
• Accountability: a named person who owns the conclusion and can defend how it was reached.

My literature review services put that methodological work into a commissionable review: a framed question, deliberate search, study appraisal and cited synthesis.

Using AI without hiding behind it

None of this is an argument against AI in research. I use it myself, heavily. The pipeline behind my own reviews is one I built, and it handles the mechanical work well: retrieval, deduplication, first-pass extraction. The difference is what sits around it.

Outputs pass through epistemological audits, run by other AI systems and by me, and every material claim is checked against its source before it enters a brief. The model does the lifting. It does not get the final word.

That is what ‘human in the loop’ should mean: a named person, with relevant training, at defined checkpoints, with the authority to stop the line. My own training is in experimental psychology and research methodology, and my peer-reviewed work is published and there to be checked. Those facts do not make a conclusion correct. They mean the checkpoints are run by someone whose job is to know where research goes wrong.

Questions to ask before you buy

• Who is the human in the loop, and what are they trained in?
• At which stages do they make judgements, rather than approve outputs?
• Can they show the search strategy and explain its limits?
• Can each important conclusion be traced to a source a person actually read?
• What checks catch a fluent but wrong claim?
• What would make them lower their confidence, and will they tell you when it happens?

If the question is tied to an imminent decision, rapid evidence synthesis is the narrower product: an agreed brief with explicit search limits and source traceability.

The honest offer

AI has made the mechanical parts of research faster, and that gain is worth taking. It has not made judgement optional. The credible offer is modest: automation where the task is bounded and checkable, and a methodologist where it is not.

If a provider cannot tell you who holds the pen at the end, the human is not in the loop. The human is in the brochure.