ChatGPT, Galactica, and the Progress Trap

The claims re: ChatGPT by "fascinated evangelists... [that they] contain “humanity’s scientific knowledge,” are approaching artificial general intelligence ... even consciousness", are simply "a distraction from the actual harm perpetuated by these systems. People get hurt from the very practical ways such models fall short in deployment".

This excellent piece provides plenty of examples, some explanations and a plea for better engineering.

Examples:

• BERT (Google): when a user searched how to handle a seizure, they received answers promoting things they should not do
• Galactica "generated a “scientific paper” on the benefits of eating crushed glass"
• ChatGPT generated "a text on “how crushed porcelain added to breast milk can support the infant digestive system”.

Underlying problems include:

• LLMs cannot "handle negation, as Allyson Ettinger demonstrated years ago with a simple study... models could not actually distinguish between" affirmative and negative statements because they have learnt "the form of the English language without possessing any of the inherent linguistic capabilities that would demonstrate actual understanding."
• the models' lack of robustness: "sensitivity ... to simple typos and misspellings in the prompts and the differences in responses caused by even a simple rewording of the same question ...
• the models memorize sensitive personal information from the training data... [and reproduce] or the societal stereotypes they encode."

These are well-known and have been documented since at least Microsoft's Tay chatbot was released and then quickly withdrawn. However, "as models get larger ... becomes increasingly difficult to document the details of the data involved and justify their environmental cost."

Moreover, these models' achievements are presented as "independent of the design and implementation choices of its engineers", which disconnects these problems from human accountability and puts the blame on "society at large or supposedly “naturally occurring” datasets, factors the companies developing these models claim they have little control over." This is not true: "none of the models we are seeing now are inevitable.

But its easier to "dismiss criticism as baseless and vilify it as “negativism,” “anti-progress,” and “anti-innovation."... for some reason it seems to be the job of the marginalized to “fix”" these models, with engineers and CEOs appealing to users to help them improve their flawed, dangerous software.