ChatGPT access cut writing time and lifted quality in a trial

The writer's problem

A midlevel professional writer, such as someone drafting a report or a press release rather than a novel, wants to know whether a chatbot actually saves working time without costing quality. A paper in Science by Shakked Noy and Whitney Zhang tested exactly that professional-writing question with a controlled experiment rather than a self-report survey.

What the documents show

The published paper's own abstract states that 453 college-educated professionals were assigned incentivized, occupation-specific writing tasks, with half randomly given access to ChatGPT; average time taken fell by 40 percent and evaluated output quality rose by 18 percent, with inequality between higher- and lower-performing workers narrowing. Workers given ChatGPT during the experiment were twice as likely to report using it in their real job two weeks later. An earlier, related document shows how these headline numbers evolved. A working paper posted to SSRN months before Science publication describes the same design with 444 participants and reports the effect in standard deviations, a 0.8 SD decrease in time and a 0.4 SD increase in quality, rather than the percentage terms the peer-reviewed version later reported. Neither the sample count nor the effect metric is identical across the two documents, which is a normal feature of a working paper reaching peer review, not a contradiction.

The editorial choice

A publisher weighing whether to encourage staff writers to use a chatbot for first drafts of professional, non-creative writing can treat this study as evidence for a real time and quality effect on the specific task type tested, occupation-specific writing prompts under incentive pay, not as a finding about creative manuscripts, which the study did not test. Editorially, quoting the 40 percent or 18 percent figures without naming the task type and the incentivized, timed setting would overstate what the paper measured.

What stays with the author

The study measured aggregate time and third-party-rated quality on a single assigned task; it did not measure whether a given writer's own judgment, factual accuracy or voice held up under sustained real-world use over months. The reported rise in workers' later real-job use of the tool describes a behavior, not an endorsement of unsupervised drafting for every kind of writing.

• Does the task type used here resemble what a given publishing team actually drafts?
• Would a quality rise on a single timed task hold up across a longer editing and fact-checking process?
• Which of the two reported effect sizes, percentage or standard deviation, is more meaningful for a specific team's own metrics?

Read together, the working paper and the published article describe a genuine measured effect for one kind of professional writing, presented with different summary statistics at different stages of the same research.