Studies show that ChatGPT increases worker productivity for some writing tasks. MIT News

Amidst the huge amount of hype surrounding generative AI, a new study by MIT researchers highlights the technology’s impact on work, finding that it has increased the risk of delegated tasks such as writing cover letters, delicate emails and cost-benefit analysis. Increased productivity for lost workers.

The tasks in the study weren’t replicas of real work: they didn’t require precise factual accuracy or context about things like company goals or customer preferences. Still, many of the study’s participants said the assignments were similar to what they’d written at their real jobs—and the benefits were substantial. Access to the helpful chatbot ChatGPT reduced the time it took workers to complete tasks by 40 percent and increased output quality as measured by independent evaluators by 18 percent.

Researchers have hope from the study, which appears today in open-access form in the journal ScienceHelps people understand the impact AI tools like ChatGPT can have on the workforce.

,We can say with certainty that generic AI is going to have a big impact on white-collar work,” says Shaqd Noy, PhD student in MIT’s Department of Economics, who co-authored the paper with fellow PhD student Whitney Zhang ’21. Did. “I think our study shows that this type of technology has important applications in white-collar work. It’s a useful technology. But it’s still too early to tell whether it will be good or bad, or whether it will actually benefit society.” How to adjust.

Task simulation for chatbots

For centuries, people have worried that new technological advances will lead to mass automation and job losses. But new technologies also create new jobs, and when they increase the productivity of workers, they can have a net positive effect on the economy.

“Productivity is the first thing that comes to the mind of economists when they think about new technological developments,” says Noy. “The classical view in economics is that the most important thing that technological progress does is increase productivity, in the sense that we can produce more efficiently economically.”

To study the impact of generic AI on worker productivity, researchers gave 453 college-educated marketers, grant writers, consultants, data analysts, human resource professionals and managers two writing assignments specific to their businesses. The 20- to 30-minute tasks included writing cover letters for grant applications, emails about organizational restructuring, and plans for analysis to help the company decide which customers to send push notifications to based on given customer data. Have to send Experienced professionals in occupations similar to each participant evaluated each presentation as if they were experiencing it in a work setting. The evaluators did not know which submissions were created with the help of ChatGPT.

For the second assignment, half the participants were given access to ChatGPT-3.5, a chatbot developed by the company OpenAI. Those users completed the task 11 minutes faster than the control group, while their average quality ratings increased by 18 percent.

The data also showed that performance disparity between workers decreased, meaning that workers who received lower grades on the first task benefited more from using ChatGPT for the second task.

The researchers say the tasks were broadly representative of the assignments such professionals see in their actual jobs, but they noted several limitations. Because they were using anonymous participants, the researchers could not require relevant knowledge about a specific company or customer. They also had to have clear instructions for each assignment, whereas real-world tasks can be more open-ended. Additionally, the researchers did not consider it possible to employ fact-checkers to evaluate the accuracy of the outputs. Accuracy is a major problem for today’s Generative AI technologies.

Those limitations may reduce ChatGPT’s ability to increase productivity in the real world, the researchers said. Still, they believe the results show promise for the technology – an idea supported by the study’s other findings: workers exposed to ChatGPT during the experiment were able to return to their actual jobs two weeks after the experiment. were twice as likely to report using it.

“Experiments show that this brings significant speed gains, even though those speed gains are small in the real world because you need to spend time fact-checking and writing pointers,” says Noy.

taking a macro view

The study offered a closer look at the impact that tools like ChatGPT can have on certain writing tasks. But it is more difficult to extrapolate that impact to understanding the impact of generic AI on the economy. The researchers hope to work on this further.

“There are many other factors that are going to affect changes in wages, employment, and different sectors, which would require pieces of evidence that are not in our paper,” says Zhang. “But the magnitude of the time savings and quality increase is huge in our paper, so it looks like it’s quite revolutionary, at least for some types of tasks.”

Both researchers agree that, even if it is accepted that ChatGPT will increase the productivity of many workers, much work remains to be done to ascertain how society should respond to the proliferation of generic AI.

“The policy needed to accommodate these technologies could vary greatly depending on the findings of future research,” says Zhang. “If we think that this will increase wages for low-wage workers, that has very different implications than increasing wage inequality by raising the wages of already high-income workers. I think there are a lot of economic and political implications downstream that are important to explore.”

The study was supported by an Emergent Ventures grant, the Mercatus Center, George Mason University, a George and OB Schultz Fund grant, the MIT Department of Economics, and a National Science Foundation Graduate Research Fellowship Grant.