A classifier caught AI-written book reviews with high accuracy

The writer's problem

An author whose book relies on genuine reader reviews to build word of mouth has a stake in whether a retailer or review platform can actually catch AI-generated fake reviews at scale, not just promise to. A paper by Panagiotis Theocharopoulos, Spiros Georgakopoulos, Sotiris Tasoulis and Vassilis Plagianakos built and tested a classifier for exactly that detection problem, using book reviews specifically.

What the documents show

The paper's own methods describe training a transfer-learning classifier on a Kaggle dataset of roughly 3 million real Amazon book reviews, then generating a matched set of 10,000 AI-written reviews of the same book titles using the open-source Vicuna model, for a balanced 20,000-entry evaluation set. The paper's own results state the classifier reached 96.86 percent average accuracy across 100 runs, up from 92.4 percent without the transfer-learning step, distinguishing real from Vicuna-generated reviews. The authors note the misclassified cases showed only minor wording differences from correctly classified ones. A separate document adds a caution about the AI system used to generate the test reviews. Vicuna's own creators describe its widely cited claim of reaching '90 percent of ChatGPT quality' as based on an evaluation they themselves call 'fun and non-scientific,' explicitly flagging that a rigorous evaluation was still needed.

The editorial choice

A platform or an author should not read this classifier's 96.86 percent accuracy as evidence that fake book reviews are reliably catchable in production, since the test set was built from one specific, self-described non-rigorously-evaluated model's output matched to real reviews under controlled conditions, not from reviews an adversary designed to evade detection or generated with newer models. Editorially, the honest claim this paper supports is narrower: a classifier can be built that distinguishes real reviews from one particular AI model's reviews of the same books, a proof of concept rather than a deployed safeguard.

What stays with the author

Neither document resolves what an individual author can do if fake reviews target their own book, and neither describes any retailer's actual production detection system, which the paper does not test. An author's recourse for suspected review manipulation runs through a platform's own reporting and policy channels, not through this research directly.

• Would this classifier's accuracy hold against reviews generated by a newer or different model?
• Does a controlled, matched dataset understate how hard real-world fake reviews are to catch?
• What should an author or platform do differently knowing the underlying generator's own quality claim was self-described as non-rigorous?

The study demonstrates technical feasibility for one detection approach against one generator's output, not a settled answer to whether book-review manipulation can be caught at platform scale.