(I’ve written before about machine learning used in comment moderation<\/a>, which is a large concern in media companies that permit users to post comments on articles and blog posts.) <\/p>\n\n\n\n Jigsaw<\/a>, a Google division, “released two public data sets containing over one million toxic and non-toxic comments from Wikipedia and a service called Civil Comments.” Each comment was labeled with a rating such as \u201cToxic\u201d or \u201cVery Toxic.\u201d The data sets were used as training data in three competitions, hosted by Google, in which AI researchers could enter their trained models and see how they compared to others (and win money). The three “Jigsaw challenges” (one per year):<\/p>\n\n\n\n .<\/p>\n\n\n\n “We decided to take inspiration from the best Kaggle solutions and train our own algorithms with the specific intent of releasing them publicly.”<\/p>\u2014 Unitary researchers<\/em><\/cite><\/blockquote>\n\n\n\n The Unitary researchers describe Detoxify<\/a>, “an open-source, user-friendly comment detection library,” which is intended “to help researchers and practitioners identify potential toxic comments.” The library includes three separate models, one for each Jigsaw challenge. These models can be fine-tuned using additional data sets.<\/p>\n\n\n\n One particular limitation pointed out by the researchers is that a high toxicity score does not always indicate actually toxic content<\/em><\/strong>: “As an example, the sentence ‘I am tired of writing this stupid essay’ will give a toxicity score of 99.7 percent, while removing the word ‘stupid’ will change the score to 0.05 percent.”<\/p>\n\n\n\n There’s still a long way to go before harmful comments and social media posts can be instantly removed from platforms. <\/p>\n\n\n\n .<\/p>\n","protected":false},"excerpt":{"rendered":" The basic idea: Immediately detect and remove hateful or dangerous posts in social media and other online forums. With advances in natural language processing (NLP), identification of harmful speech becomes more accurate and more practical. In this essay published in Scientific American (2021), researchers from the private company Unitary (see their public Detoxify code on… Continue reading Identifying toxic comments with AI<\/span> ![\"Creative](\"https:\/\/i.creativecommons.org\/l\/by-nc-nd\/4.0\/88x31.png\")
<\/a>
\nAI in Media and Society<\/strong><\/span> by Mindy McAdams<\/span> is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License<\/a>.
\nInclude the author’s name (Mindy McAdams) and a link to the original post in any reuse of this content.<\/small><\/p>\n\n\n\n