Deep Research, Deep Bullshit, and the potential (model) collapse of science

Gary Marcus pouring cold water on Deep Research, "which... can write science-sounding articles on demand, on any topic of your choice".

In many ways there's not much new here, except perhaps for how model collapse will affect science, not just LLMs.

As known:

• LLMs flooding the zone with shit has been a concern since the beginning, and as they get better the associated risks will rise.
• for any non-illegitimate application, the risk of hallucinations - still present - remains the achilles heel. When "using “reasoning” to “synthesize large amounts of online information and complete multi-step research” .. you want to get it right, reliably"

However, it's worth noting that the increasing quality has second-degree implications, notably in science: "virtually everything that Deep Research produces will pass a LGTM test ( “it ‘looks good to me’”); people will assume it’s legit, even when it is isn’t... Students will quickly learn that every term paper it writes gets an A. Scientific researchers will start to use it... inevitably errors will seep in to the scientific literature, often undetected."

So not only will this be fed into the next generation of models, but also "the scientific literature itself... choking the peer review process and saturating the market with hard-to-find, hard-to-rectify errors."