{"id":100,"date":"2020-08-18T11:27:14","date_gmt":"2020-08-18T15:27:14","guid":{"rendered":"https:\/\/www.macloo.com\/ai\/?p=100"},"modified":"2020-08-24T12:04:42","modified_gmt":"2020-08-24T16:04:42","slug":"gpt-3-and-automated-text-generation","status":"publish","type":"post","link":"https:\/\/www.macloo.com\/ai\/2020\/08\/18\/gpt-3-and-automated-text-generation\/","title":{"rendered":"GPT-3 and automated text generation"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">GPT-3 has to be the <em>most-hyped<\/em> AI technology of the past year. Headlines said its predecessor, GPT-2, was &#8220;<a rel=\"noreferrer noopener\" href=\"https:\/\/www.theverge.com\/2019\/11\/7\/20953040\/openai-text-generation-ai-gpt-2-full-model-release-1-5b-parameters\" target=\"_blank\">too dangerous<\/a>&#8221; to be released publicly. Then it was released. The world did not end. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Less than a year later, the more advanced (next generation) GPT-3 was released by <a rel=\"noreferrer noopener\" href=\"https:\/\/openai.com\/\" target=\"_blank\">OpenAI<\/a>. Why are people so excited about GPT-3? See for yourself in the video below. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">GPT-3 is a natural language generation (NLG) system. Given instructions about what you want, it writes original text that \u2014 in most (<a href=\"https:\/\/lacker.io\/ai\/2020\/07\/06\/giving-gpt-3-a-turing-test.html\" target=\"_blank\" rel=\"noreferrer noopener\">but not all<\/a>) cases \u2014 sounds like a human wrote it. The technology could be used to rapidly write 10,000 fake user comments into a discussion forum, for example. Or 10,000 fake restaurant reviews.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Don&#8217;t worry about the first examples in the video showing GPT-3 writing computer code, if that&#8217;s not something you&#8217;re well acquainted with \u2014 it quickly moves on to show the system extracting text from long documents and writing summaries on the fly. The presenter does a good job of demonstrating the breadth and variety of tasks GPT-3 can be used for. You might be flat-out amazed.<\/p>\n\n\n\n<figure class=\"wp-block-embed-youtube aligncenter wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"jetpack-video-wrapper\"><iframe loading=\"lazy\" title=\"GPT 3 Demo and Explanation - An AI revolution from OpenAI\" width=\"739\" height=\"416\" src=\"https:\/\/www.youtube.com\/embed\/8psgEDhT1MM?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe><\/div>\n<\/div><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Bear in mind that the examples shown in the video are different, separate applications of GPT-3. You don&#8217;t just install GPT-3 and it does all of those things. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Developers can <a rel=\"noreferrer noopener\" href=\"https:\/\/beta.openai.com\/\" target=\"_blank\">apply<\/a> to gain access to the GPT-3 API. This enables them to create applications that use GPT-3 but not to see or modify the actual code that makes GPT-3 work. You can view more examples of GPT-3 applications at that same link.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Another nice thing about the video above is the explanation of <em>generative pre-training<\/em>. Instead of training the GPT-3 model (or models) <em>only<\/em> with labeled data (supervised learning), <a rel=\"noreferrer noopener\" href=\"https:\/\/cdn.openai.com\/research-covers\/language-unsupervised\/language_understanding_paper.pdf\" target=\"_blank\">the OpenAI researchers used<\/a> &#8220;a semi-supervised approach for language understanding tasks using a combination of unsupervised pre-training and supervised fine-tuning.&#8221; The pre-training for GPT-2 included a dataset of <strong>more than 7,000 unpublished books<\/strong> &#8220;from a variety of genres including Adventure, Fantasy, and Romance.&#8221; Because entire books were used \u2014 <em>instead of<\/em> sentences separated from their context \u2014 the model was able to learn long-range structure.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">GPT-3 used <em>even more<\/em> long-form texts for pre-training (described in a <a rel=\"noreferrer noopener\" href=\"https:\/\/arxiv.org\/pdf\/2005.14165.pdf\" target=\"_blank\">technical paper<\/a>): <\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"346\" src=\"https:\/\/www.macloo.com\/ai\/wp-content\/uploads\/2020\/08\/GPT3_corpus.png\" alt=\"\" class=\"wp-image-105\" srcset=\"https:\/\/www.macloo.com\/ai\/wp-content\/uploads\/2020\/08\/GPT3_corpus.png 1024w, https:\/\/www.macloo.com\/ai\/wp-content\/uploads\/2020\/08\/GPT3_corpus-300x101.png 300w, https:\/\/www.macloo.com\/ai\/wp-content\/uploads\/2020\/08\/GPT3_corpus-768x260.png 768w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption><em>Above: Screenshot from &#8220;Language Models Are Few-Shot Learners,&#8221; Brown et al., July 2020<\/em><\/figcaption><\/figure><\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Once again we can see that tremendous advances in AI capability are made possible precisely because today&#8217;s computer hardware has the ability to run through enormous quantities of data <em>very quickly<\/em>. It&#8217;s not only that we now <em>have<\/em> billions of pages of text in digital form. It&#8217;s not just that we can <em>store<\/em> that Himalayan mountain range of data. It&#8217;s very much because processors are able to run multiple calculations simultaneously at lightning speed. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An important point about GPT-3 that&#8217;s <em>not<\/em> covered in the video: None of these applications, or GPT-3 itself, understands the <em>meaning<\/em> of the text that is being generated. <\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It&#8217;s going to be very easy for people to jump to conclusions about the &#8220;intelligence&#8221; of a computer system when it&#8217;s able to generate responses and explanations that are so human-like. There is no comprehension here. There is no knowledge of the world \u2014 there is only knowledge about language itself.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To learn more about how GPT-3 does what it does: <a rel=\"noreferrer noopener\" href=\"https:\/\/daleonai.com\/gpt3-explained-fast\" target=\"_blank\">GPT-3 Explained in Under 3 Minutes<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a rel=\"license\" href=\"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/\"><img decoding=\"async\" alt=\"Creative Commons License\" style=\"border-width:0\" src=\"https:\/\/i.creativecommons.org\/l\/by-nc-nd\/4.0\/88x31.png\"><\/a><br>\n<small><span xmlns:dct=\"http:\/\/purl.org\/dc\/terms\/\" property=\"dct:title\"><strong>AI in Media and Society<\/strong><\/span> by <span xmlns:cc=\"http:\/\/creativecommons.org\/ns#\" property=\"cc:attributionName\">Mindy McAdams<\/span> is licensed under a <a rel=\"license\" href=\"http:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/\">Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License<\/a>.<br>\nInclude the author&#8217;s name (Mindy McAdams) and a link to the original post in any reuse of this content.<\/small><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>GPT-3 has to be the most-hyped AI technology of the past year. Headlines said its predecessor, GPT-2, was &#8220;too dangerous&#8221; to be released publicly. Then it was released. The world did not end. Less than a year later, the more advanced (next generation) GPT-3 was released by OpenAI. Why are people so excited about GPT-3?&hellip; <a class=\"more-link\" href=\"https:\/\/www.macloo.com\/ai\/2020\/08\/18\/gpt-3-and-automated-text-generation\/\">Continue reading <span class=\"screen-reader-text\">GPT-3 and automated text generation<\/span> <span class=\"meta-nav\" aria-hidden=\"true\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[2],"tags":[32,29,30,31,33],"class_list":["post-100","post","type-post","status-publish","format-standard","hentry","category-nlp","tag-intelligence","tag-nlg","tag-pre-training","tag-text","tag-understanding"],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.macloo.com\/ai\/wp-json\/wp\/v2\/posts\/100","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.macloo.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.macloo.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.macloo.com\/ai\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.macloo.com\/ai\/wp-json\/wp\/v2\/comments?post=100"}],"version-history":[{"count":10,"href":"https:\/\/www.macloo.com\/ai\/wp-json\/wp\/v2\/posts\/100\/revisions"}],"predecessor-version":[{"id":190,"href":"https:\/\/www.macloo.com\/ai\/wp-json\/wp\/v2\/posts\/100\/revisions\/190"}],"wp:attachment":[{"href":"https:\/\/www.macloo.com\/ai\/wp-json\/wp\/v2\/media?parent=100"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.macloo.com\/ai\/wp-json\/wp\/v2\/categories?post=100"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.macloo.com\/ai\/wp-json\/wp\/v2\/tags?post=100"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}