A program's illusion of listening was not comprehension

The writer's problem

By the mid-1960s, a computer scientist at MIT wanted to test how far a very simple program could go in producing text that read as a natural conversational reply, without teaching the machine anything about meaning. Joseph Weizenbaum's task was not to write a program that understood language, but to describe, honestly, a program that could hold a passable conversation while understanding almost none of it, and to be explicit in print about that gap before others mistook the performance for comprehension.

What the documents show

Weizenbaum's paper, 'ELIZA—A Computer Program for the Study of Natural Language Communication Between Man and Machine', describes a program that scans a user's typed sentence for keywords, applies a 'decomposition rule' triggered by that keyword, and reassembles fragments of the user's own words into a reply using a 'reassembly rule.' The journal's own publication record confirms the paper appeared in Communications of the ACM, volume 9, issue 1, pages 36 to 45, in January 1966. Weizenbaum states plainly that 'ELIZA throws away each of its inputs' except for a small set tagged by a 'MEMORY' mechanism, and that its scripts, most famously one imitating a Rogerian psychotherapist, are data the program manipulates, not knowledge it holds.

Weizenbaum is unusually direct that this is not language generation in any deep sense: he writes that ELIZA 'maintains the illusion of understanding with so little machinery,' and describes the program as chiefly concerned with 'the concealment of its lack of understanding.' He raises, in the same paper, the possibility that people would find it credible that a human, not a machine, was typing the replies, precisely because a convincing surface does not require comprehension underneath it.

The editorial choice

Treating ELIZA as a precedent for later language models is an editorial choice that needs the same qualification Weizenbaum built into his own paper: fluent-sounding replies are evidence of a well-designed pattern match, not proof of understanding, whether the year is 1966 or the present. An editor citing ELIZA as an early chatbot should also carry forward Weizenbaum's own warning about mistaking a program's surface behavior for its actual mechanism.

What stays with the author

Distinguishing a tool's fluent output from genuine understanding, and saying so plainly to a reader, is a judgment the paper itself models but cannot make for anyone citing it decades later. Weizenbaum describes his own program's limits, not the limits of tools built long after his.

• Is a tool's reply evidence of understanding, or of a well-tuned pattern match, and does the description say which?
• Would Weizenbaum's own phrase about concealing a lack of understanding apply to the tool being described?
• Does citing ELIZA as an ancestor claim more continuity than the 1966 paper supports?

A therapist-imitating program built from keyword rules is not a language model, but Weizenbaum's own caution about mistaking fluency for comprehension has outlived the program by six decades, and it still applies.