AI-Powered Text From This Program Could Fool the Government

In October 2019, Idaho proposed changing its Medicaid program. The state needed approval from the federal government, which solicited public feedback via Medicaid.gov.

Roughly 1,000 comments arrived. But half came not from concerned citizens or even internet trolls. They were generated by artificial intelligence. And a study found that people could not distinguish the real comments from the fake ones.

The project was the work of Max Weiss, a tech-savvy medical student at Harvard, but it received little attention at the time. Now, with AI language systems advancing rapidly, some say the government, and internet companies, need to rethink how they solicit and screen feedback to guard against deepfake text manipulation and other AI-powered interference.

“The ease with which a bot can generate and submit relevant text that impersonates human speech on government websites is surprising and really important to know,” says Latanya Sweeney, a professor at Harvard’s Kennedy School who advised Weiss on how to run the experiment ethically.

Sweeney says the problems extend well beyond government services, but it is imperative that public agencies find a solution. “AI can drown speech from real humans,” she says. “Government websites have to change.”

The Centers for Medicare and Medicaid Services says it has added new safeguards to the public comment system in response to Weiss’ study, though it declines to discuss specifics. Weiss says he was contacted by the US General Services Administration, which is developing a new version of the federal government website for publishing regulations and comments, about ways to better protect it from fake comments.

Government systems have been the target of automated influence campaigns before. In 2017, researchers discovered that over a million comments submitted to the Federal Communications Commission regarding plans to roll back net neutrality rules had been auto-generated, with certain phrases copied and pasted into different messages.

Weiss’ project highlights a more serious threat. There has been remarkable progress in applying AI to language over the past few years. When powerful machine-learning algorithms are fed huge amounts of training data—in the form of books and text scraped from the web—they can produce programs capable of generating convincing text. Besides myriad useful applications, this raises the prospect that all sorts of internet messages, comments, and posts could be faked easily and less detectably.

“As technology gets better,” Sweeney says, “human speech venues become subject to manipulation without human knowledge that it has happened.” Weiss was working at a health care consumer-advocacy organization in the summer of 2019 when he learned about the public feedback process required to make Medicaid changes. Knowing that these public comments had swayed previous efforts to change state Medicaid programs, Weiss looked for tools that could auto-generate comments.

“I was a bit shocked when I saw nothing more than a submit button standing in the way of your comment becoming a part of the public record,” he says.

Weiss discovered GPT-2, a program released earlier that year by OpenAI, an AI company in San Francisco, and realized he could generate fake comments to simulate a groundswell of public opinion. “I was also shocked at how easy it was to fine tune GPT-2 to actually spit out the comments,” Weiss says. “It’s relatively concerning on a number of fronts.”

Besides the comment-generating tool, Weiss built software for automatically submitting comments. He also conducted an experiment in which volunteers were asked to distinguish between the AI-generated comments and ones written by humans. The volunteers did no better than random guessing.

After submitting the comments, Weiss notified the Centers for Medicare and Medicaid Services. He had added a few characters to make it easy to identify each fake comment. Even so, he says, the AI feedback remained posted online for several months.

OpenAI released a more capable version of its text-generation program, called GPT-3, last June. So far, it has only been made available to a few AI researchers and companies, with some people building useful applications such as programs that generate email messages from bullet points. When GPT-3 was released, OpenAI said in a research paper that it had not seen signs of GPT-2 being used maliciously, even though it had been aware of Weiss’s research.