{"id":791,"date":"2021-06-02T09:00:00","date_gmt":"2021-06-02T13:00:00","guid":{"rendered":"https:\/\/www.macloo.com\/ai\/?p=791"},"modified":"2021-06-02T09:33:55","modified_gmt":"2021-06-02T13:33:55","slug":"the-trouble-with-large-language-models","status":"publish","type":"post","link":"https:\/\/www.macloo.com\/ai\/2021\/06\/02\/the-trouble-with-large-language-models\/","title":{"rendered":"The trouble with large language models"},"content":{"rendered":"\n

Yesterday I summarized<\/a> the first two articles in a series about algorithms and AI by Hayden Field, a technology journalist at Morning Brew<\/a>. Today I’ll finish out the series.<\/p>\n\n\n\n

The third article, This Powerful AI Technique Led to Clashes at Google and Fierce Debate in Tech. Here’s Why<\/a>, explores the basis of the volatile situation around the firing of Timnit Gebru and later Margaret Mitchell from Google’s Ethical AI unit earlier this year. Both women are highly respected and experienced AI researchers. Mitchell founded the team in 2017.<\/p>\n\n\n\n

Central to the situation is a criticism of large language models<\/strong> and a March 2021 paper (On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?<\/a>) co-authored by Gebru, Mitchell, and two researchers at the University of Washington. The biggest current example is GPT-3, previously covered in several posts<\/a> here.<\/p>\n\n\n\n

“Models this big require an unthinkable amount of data<\/em>; the entirety of English-language Wikipedia makes up just 0.6% of GPT-3\u2019s training data.”<\/p>\u2014”This Powerful AI Technique Led to Clashes at Google and Fierce Debate in Tech. Here’s Why”<\/cite><\/blockquote>\n\n\n\n

The Morning Brew article sums up the very recent and very big improvements in large language models that have come about thanks to new algorithms and faster computer hardware (GPUs running in parallel). It highlights BERT, “the model that now underpins Google Search,” which came out of the research that resulted in the first Transformer<\/a>. A good at-the-time article about GPT-3’s release was published in July 2020<\/a> in MIT’s Technology Review<\/em>: “OpenAI first described GPT-3 in a research paper published in May [2020].”<\/p>\n\n\n\n

One point being \u2014 Google fired Timnit Gebru very soon after<\/em> news and discussion of large language models (GPT-3 especially, but remember Google’s investment in BERT too) ramped up \u2014 way<\/em> up. Her criticism of a previously obscure AI technology (not obscure among NLP researchers, but in the wider world) might have been seen as increasingly inconvenient for Google. Morning Brew summarizes the criticism (not attributed to Gebru): “Because large language models often scrape data from most of the internet, racism, sexism, homophobia, and other toxic content inevitably filter in.”<\/p>\n\n\n\n

“Once the barrier to create AI tools and generate text is lower, people could just use it to create misinformation at scale, and having that data coupled with certain other platforms can just be a very disastrous situation.\u201d<\/p>\u2014Sandhini Agarwal, AI policy researcher, OpenAi<\/cite><\/blockquote>\n\n\n\n

The Morning Brew article goes well beyond Google’s dismissal of Gebru and Mitchell, bringing in a lot of clear, easy-to-understand explanation of what large language models require (for example, significant energy resources<\/strong>), what they’re being used for, and even the English-centric nature<\/strong> of such models \u2014 lacking a gigantic corpus of digitized text in a given human language, you can’t create a large model in that language.<\/p>\n\n\n\n

The turmoil in Google’s Ethical AI unit is covered in more detail in this May 2021 article<\/a>, also by Hayden Field.<\/p>\n\n\n\n

It’s easy to find articles that discuss “scary things GPT-3 can do and does” and especially the bias issues; it’s much harder to find information about some of the other aspects covered here. It’s also not just about GPT-3. I appreciated insights from an interview with Emily M. Bender, first author on the “Stochastic Parrots” article. I also liked the explicit statement that many useful NLP tasks can be done well without a large<\/em> language model. In smaller datasets, finding and accounting for toxic content can be more manageable.<\/p>\n\n\n\n

“Do we need this at all? What\u2019s the actual value proposition of the technology? … Who is paying the environmental price for us doing this, and is this fair?\u201d<\/p>\u2014Emily M. Bender, professor and director, Professional MS in Computational Linguistics, University of Washington<\/cite><\/blockquote>\n\n\n\n

Finally, in a recap of Morning Brew’s “Demystifying Algorithms” event<\/a>, editor Dan McCarthy summarized two AI researchers’ answers to one of my favorite questions: What can an algorithm actually know<\/em>?<\/p>\n\n\n\n

An AI system\u2019s ability to generalize<\/strong> \u2014 to transfer learning from one domain to another \u2014 is still a wide-open frontier, according to Mark Riedl, a computer science professor at Georgia Tech. This is something I remind my students of over and over \u2014 what’s called “general intelligence” is still a long way off for artificial intelligence. Riedl works on aspects of storytelling to test whether an AI system is able to “make something new” out of what it has ingested.<\/p>\n\n\n\n

Sa\u0161ka Mojsilovi\u0107, head of Trusted AI Foundations at IBM Research, made a similar point \u2014 and also emphasized that “narrow AI<\/strong>” (which is all the AI we’ve ever had, up to now and<\/em> for the foreseeable future) is not nothing. <\/p>\n\n\n\n

She suggested: \u201cWe may want to take a pause from obsessing over artificial general intelligence and maybe think about how we create AI solutions for these kinds of problems\u201d \u2014 for example, narrow domains such as drug discovery (e.g. new antibiotics) and creation of new molecules. These are extraordinary accomplishments within the capabilities of today’s AI.<\/p>\n\n\n\n

This is a half-hour conversation with those two experts:<\/p>\n\n\n\n

\n