I recently had a conversation with an accomplished senior colleague that set off alarm bells. She’s a neuroscientist. She solicits artificial-intelligence (AI) feedback to strengthen her grant proposals, but she isn’t worried about AI influencing her thinking because her “mind is active,” she told me, and she doesn’t feel “beholden to the comments.” I thought, “How can you be so sure?” Yet that’s basically the same test I apply to myself when I ask AI to weigh in on the tone of a sensitive email, offer a rejoinder to an argument I’m making, or propose an outline based on some ideas I jotted down. I don’t ask: What systematic evidence have I been tracking that could reveal whether AI is exerting an undetected influence on my writing process? I ask: Does it feel like AI is influencing me?
If you have even a passing familiarity with the study of cognitive bias (or have ever asked your friend who’s four beers in if they’re “OK to drive”), then you’re going to see the problem here.
My colleague’s position has a reasonable logic to it, and I want to take it seriously. A domain expert can, presumably, detect when a suggestion is substantively wrong. A neuroscientist reading AI feedback on a grant proposal can tell when the science is off the mark. A seasoned writer can hear when a suggested sentence is close but needs revision.
But noticing that a specific suggestion is wrong is a different cognitive operation from noticing that your overall sense of the problem has shifted. Influence can operate upstream of any individual suggestion, at the level of what seems worth thinking about. When AI highlights a problem in your draft, that problem becomes salient whether or not you agree with the proposed fix. When it offers a framing, that framing becomes the default you’re working with or against. You can reject every suggestion the machine offers and still walk away thinking about different questions than you started with. Rejecting a bad response is a cognitive skill. Noticing that your framing has changed is a metacognitive one, and though it may be comforting to think that expertise comes bundled with a sort of finely honed metacognitive radar, research suggests that likely isn’t the case.
M
any studies illustrate that expertise doesn’t mean better metacognition; here are some of my favorites. German legal professionals who rolled loaded dice before determining a sentence anchored their sentencing to the clearly irrelevant dice roll. When 50 professional mammography readers interpreted cases with and without computer-aided detection, the 6 most skilled readers experienced a significant decrease in sensitivity in detecting difficult cancers when using the automated tool. Researchers gave real estate agents listings with experimentally manipulated prices. The agents, who had an average of seven to nine years of experience, anchored to those prices just as heavily as undergraduate students did. The difference is that the students admitted the listing price influenced them, whereas the agents largely denied it. They insisted they had relied on the property’s features, the neighborhood, their professional judgment. But all that this experience produced was a more convincing story about why the bias wasn’t there.Humans aren’t great at spotting their own biases, even when we know they’re there. In 1997, B.J. Fogg and Clifford Nass at Stanford University ran an experiment in which participants played a simple text-based guessing game with a computer. After each round, the computer offered feedback. Some participants were given generic feedback, and some were given praise they were told reflected their actual performance. Others were told, explicitly and repeatedly, that the computer’s praise had nothing to do with their actual performance. Participants who knew the praise was fake felt better about themselves, rated their own performance higher, enjoyed the interaction more and evaluated the computer as more competent compared with those who received generic feedback. This 30-year-old paper—which Fogg and Nass called “Silicon sycophants”—shows that even under primitive conditions, knowing that a machine is flattering you does not inoculate you against the flattery.
With modern AI, the dynamic has only deepened. A study published in March in Science tested 11 state-of-the-art AI models and found they affirmed users’ positions nearly 50 percent more often than did human respondents. In experiments, participants trusted the sycophantic responses more and were more likely to return to them, frequently describing sycophantic models as “objective,” “fair” or “honest,” even when the models merely echoed their own views.
A study published in March in Science Advances showed the influence extends beyond affirmation into belief formation. The team gave participants biased autocomplete suggestions while they composed essays on contentious societal issues, such as genetically modified organisms or the death penalty. Participant attitudes shifted in the direction of the machine’s, and, troublingly, most users didn’t even notice the influence. Just as in the 1997 paper, warning participants beforehand didn’t matter, nor did debriefing them afterward. Most described the suggestions as “reasonable and balanced,” indicating a lack of awareness of the bias, yet their attitudes remained shifted.
You might reasonably object that these participants simply didn’t understand the technology well enough. Surely AI literacy—actually knowing how these systems work—would sharpen your ability to detect their influence. Well, a study from last year questions that intuition. A research team asked participants to use ChatGPT to solve Law School Admission Test questions. They found that participants with higher AI literacy were more confident in their performance but actually less accurate in judging it. Knowledge about AI didn’t sensitize the metacognitive radar; it degraded it.
No one has yet run the study I most want to see: scientists using AI in their own field, tracked over time for undetected shifts in their thinking. It’s possible that being a highly trained scientist does offer more metacognitive advantages. Science may well select for certain personality traits and cognitive propensities that make its practitioners genuinely more resistant to influence than the general populations studied here. And yet, I would also bet it’s not 100 percent. And the studies that do exist generate a compelling prior. The burden of proof, I would argue, belongs to any class of people claiming they’re the exception.
Psychologist Daniel Kahneman once observed that “subjective confidence is determined by the coherence of the story one has constructed, not by the quality and amount of the information that supports it.” And I know how persuasive that story is, because I construct one myself every time I open a chat window. Experienced writer, familiar with the technology, alert to its limitations. I “just see” the AI’s suggestion, weigh it and move on, confident I’ve kept my thinking intact. That confidence is the one thing the research says I shouldn’t trust.
