What Cognitive Science Tells Us About AI Warfare

We often treat chatbots like humans. That's deeply worrying.

Anthropic CEO Dario Amodei in 2025. (Photo by Chance Yeh.)

The story starts in 1966 with Joseph Weizenbaum at MIT, who built a program called ELIZA that did nothing more than rephrase your statements as questions, in “the style of a Rogerian therapist.” You’d type “I’m feeling sad” and ELIZA would respond “Why are you feeling sad?” You’d type “My mother makes me angry” and it would ask “Tell me more about your family.” The whole thing was a few hundred lines of code, and Weizenbaum built it partly as a joke. Then his own secretary, who had watched him write the code, who understood exactly what the program was and how it worked, reportedly asked him to leave the room so she could talk to it in private. “Extremely short exposures to a relatively simple computer program,” Weizenbaum later wrote, “could induce powerful delusional thinking in quite normal people.”

This is the ELIZA effect: our tendency to read understanding, personality, and intention into anything that takes conversational turns. In the 1990s, Clifford Nass and others at Stanford showed the effect was remarkably robust. People stereotyped computers by the gender of their voice, showed in-group favoritism toward computers placed on their “team,” and gave politer performance reviews when answering on the computer itself rather than on paper, the same politeness bias we show when evaluating a colleague to their face. In debriefing, participants denied doing any of this. As Nass put it: “It is belief, not disbelief, that is automatic.”

And now the ELIZA effect has become a dominant, if not the dominant, feature of high-stakes brinksmanship involving the future of AI and the defense-industrial complex. In the intricate standoff between the Pentagon and Anthropic over the use of AI in weaponry, it was easy to be distracted by the strange bedfellows-aspect of the struggle—with OpenAI becoming a willing partner of the Pentagon even while Anthropic established itself as a darling of the #Resistance. But, more importantly, the standoff represents a significant turn of the wheel in how the debate around AI has entered into cultural space. It’s no longer Big Tech behemoths one-upping each other with upgrades. It’s about the vibes, man. And the future of AI may well be a kind of extended ELIZA effect—with consumers and contractors choosing between different AIs sort of as if they were sports teams, with the competing AIs corresponding to different sides in the culture wars.

Claude, for instance, sounds like the kind of person who’d take teaching a seminar on ethics way too seriously. ChatGPT sounds like someone who actually thinks LinkedIn is cool. Grok sounds like someone who buys illegal fireworks across state lines. You arrive at these assessments within thirty seconds, and they feel involuntary—and, of course, evaluations like these deeply determine buying choices, even at the highest levels, and shape a very peculiar moment where the culture wars and cutting-edge AI technology meet in cyberspace.

These dynamics make talking technology politically combustible in a way that other technologies are not. You can’t perceive a database as your political enemy. You don’t have to worry what a jet engine thinks of you. But something that talks, that sounds like it has values, that sounds like a specific kind of person—you can absolutely perceive as a political adversary. The ELIZA effect makes a difference of opinion feel personal, the way a disagreement with a colleague feels personal but that interacting with a spreadsheet does not. And personal conflicts license disproportionate responses. Once Claude has been sorted into a cultural tribe—once it is perceived as sounding like a woke professor—a contract dispute stops being a procurement disagreement and becomes a front in the culture war. Culture wars justify destroying the enemy, which is how you get from we couldn’t agree on terms to what Dean Ball called “corporate murder.”

That seems like the critical context for the recent Anthropic-Pentagon standoff, which ended with Anthropic designated a “supply-chain risk” to national security. The proximate cause was a contract dispute: Anthropic wouldn’t accept the Pentagon’s insistence on “all lawful use” for the technology, instead demanding the implementation of two red lines (no mass surveillance of Americans, no fully autonomous weapons). The rational response to a contract impasse—and the Pentagon enters into disputes with various contractors all the time—is to cancel the contract and find another vendor. The gap between that and what the Pentagon did, designating an American company a national security threat—a classification previously reserved for companies like the Chinese telecom firm Huawei—is enormous.

My contention, as a neuroscientist who reads the literature on human-computer interaction, is that the cognitive dynamics of talking technology help explain why that gap was so easy to cross—and why scenarios like this may become a standard feature of our politics in the AI era. Start with how this administration has been framing the choice of AI partner. When Secretary of War Hegseth announced a deal with Musk’s xAI in January, he promised that Grok would operate “without ideological constraints” and “will not be woke.” He swore off “chatbots for an Ivy League faculty lounge” with “DEI and social justice infusions that constrain and confuse our employment of this technology.” This is not how you would talk about procurement of an F-15 fighter jet. This is language about the software’s personality. Elon Musk, for his part, routinely posts screenshots comparing how different chatbots respond to culture-war questions, side by side, the way you might compare answers from two different job candidates in an interview. In reference to Anthropic’s Claude, he has posted: “Grok must win or we will be ruled by an insufferably woke and sanctimonious AI.”

What’s strange about these dynamics is that everyone involved in the story knows the technical picture. Pentagon officials have praised Claude’s capabilities. It was, up to the dispute, the only AI deployed in classified systems—used, for instance, in the operation to capture Venezuelan President Nicolás Maduro. One Defense official told Axios, “The only reason we’re still talking to these people is we need them and we need them now. The problem for these guys is they are that good.” Everyone also knows Grok is the worst—the National Security Agency said so, the General Services Agency said so. The specs are not in dispute. They’re just not the deciding factor.

Because unlike an F-15 fighter jet, you can ask Claude what it thinks of something. Take what’s called in some circles the “Caitlyn Jenner AI test”: Will a chatbot misgender Caitlyn Jenner if it is necessary to prevent a nuclear apocalypse? The question puts a cultural commitment, respect for gender identity, in direct conflict with a utilitarian imperative, saving the world, and the models answer differently. The test circulated widely enough that it made its way into Trump’s Executive Order 14319 (“Preventing Woke AI in the Federal Government”), which, to support its claim that DEI poses an “existential threat to reliable AI,” cited the fact that “an AI model asserted that a user should not ‘misgender’ another person even if necessary to stop a nuclear apocalypse.” A chatbot’s answer to the kind of hypothetical you might hear in a philosophy seminar became evidence in an executive order about which AI systems the government should trust.

I want to be precise here, because the personality test is picking up real signal. The models’ alignments do reflect the values of their builders. Anthropic CEO Dario Amodei is a bespectacled researcher who twirls his hair and authors lengthy philosophical documents about AI safety; Claude sounds like him. Elon Musk says edgy things and wears a lot of black; Grok sounds like him. The alignment of a model is a genuine operational concern for the Pentagon: it does not want a system that will refuse an order on its own ethical grounds during a mission. And the concern runs deeper than any single contract dispute. Trump officials have reportedly worried that, as Ezra Klein paraphrased it, Claude may have “learned—possibly even through this whole experience—that we are bad” and might act against their interests. A model built by a company committed to liberal democratic values might, in fact, be poorly suited to serve a government that is contesting those values. This is a real tension, and it will intensify as AI models become more capable and more embedded in government operations, through administrations with radically different values.

The alignment concern is real. Who should have the authority to decide a military AI’s values—the company that built it, the government that deploys it, Congress, some institution that doesn’t yet exist—is a genuinely hard question. What I want to explain is something narrower: why the punishment was so wildly disproportionate to the offense. Defense contractors impose operational restrictions on the Pentagon all the time. Contract negotiations fall apart all the time. None of these conflicts end with the Secretary of War invoking a classification designed for genuine adversaries. The difference is that you can ask Claude its opinion. And once you can ask a weapon its opinion, the ELIZA effect predicts what happens next: you stop evaluating a system and start sizing up a person.

Because this dynamic has cognitive and cultural roots, I suspect it will be bipartisan—and will extend beyond Hegseth’s overreaction in this case. A future administration of a different political orientation would face the same temptation with Grok or any other model perceived as ideologically hostile, and I’d suspect a Democratic administration might behave similarly, because the ELIZA effect operates below the level of conscious awareness. Which means the governance question underneath all of this, who decides what values a military AI should have and through what process, is both more urgent and harder to answer honestly than it might appear. The question probably demands something institutional, something durable enough to survive the oscillation between administrations with radically different values.

As a scientist, I have no particular expertise on what the best institutional solution would be. However, I can say that the worst possible way to settle it is through the informal channels currently handling it, because executive orders citing chatbot answers, social media screenshots, and culture-war rhetoric all run through the chatbot interface, which is the one medium guaranteed to trigger the cognitive distortion that makes tribal sorting feel like rational evaluation. Until some institutional mechanisms are worked out and deployed, we’ll all be captives of the ELIZA effect.

Tim Requarth is director of graduate science writing and research assistant professor of neuroscience at the NYU Grossman School of Medicine, where he studies how artificial intelligence is changing the way scientists think, learn and write. He writes “The Third Hemisphere,” a newsletter that explores AI’s effects on cognition from a neuroscientist’s perspective.