Companies making machine learning and generative software aren’t just metaphorically ripping off books. In at least one case, they’re rather literally shredding millions of physical books to feed to their chatbots. As uncovered last month by The Washington Post, AI giant Anthropic ran a massive program called Project Panama where they spent tens of millions of dollars to hoover up used books, which they then sliced, scanned, and pulped. The scanned data removed from the books was then used to train their software.
Anthropic has already been in the hot seat for getting caught pirating millions of digital copies of books. But the judge’s equivocating ruling in that piracy case created a loophole, according to Anthropic’s lawyers. If the books training AI were used in a “transformative” way, the judge ruled, it was legally aboveboard, akin to using books to teach kids or how you can do what you want with a book once you buy it—a legal precedent that allows for second hand bookstores, for example.
Project Panama capitalized on that loophole. Anthropic spent a bundle at libraries, online secondhand stores, and used bookstores like The Strand to build out a massive library—the Post’s article includes images of huge warehouses filled with books. Anthropic then hired “an experienced document scanning services vendor to convert from 500,000 to two million books over a six-month period,” according to the proposal sent out to vendors.
The process to scan these books ultimately destroyed them, as reported in Futurism:
From the way the lawsuit documents tell it, Anthropic turned literally ripping off books into an art form. It used a ‘hydraulic powered cutting machine’ to ‘neatly cut’ the millions of books it got from used book retailers, and then scanned the pages ‘on high speed, high quality, production level scanners.’ Then a recycling company would be scheduled to pick up the eviscerated volumes—because you wouldn’t want to be wasteful, after all.
According to the Post’s coverage, this program raised red flags for some inside Anthropic, who knew that tearing books apart to feed into an AI model was rather literally bringing the critiques of these companies to life. Which I guess is to their credit. It also shows they’re aware that they’re losing the PR battle to make these slop machines look cool—see also, Microsoft’s CEO wanting us to stop thinking of AI as “slop” and start thinking of it as “bicycles for the mind”. Not quite as catchy.
“Project Panama is our effort to destructively scan all the books in the world,” reads a newly unsealed internal planning document, according to the Post. “We don’t want it to be known that we are working on this.”
Hate to agree with an AI guy, but you’re right—trying to “destructively scan all the books in the world” is a bad look! It’s always nice to see a flicker of shame these days, but as usual with tech overreach, it’s too little, too late.
