{"id":276,"date":"2020-09-07T09:00:25","date_gmt":"2020-09-07T13:00:25","guid":{"rendered":"https:\/\/www.macloo.com\/ai\/?p=276"},"modified":"2020-09-07T09:52:57","modified_gmt":"2020-09-07T13:52:57","slug":"comment-moderation-as-a-machine-learning-case-study","status":"publish","type":"post","link":"https:\/\/www.macloo.com\/ai\/2020\/09\/07\/comment-moderation-as-a-machine-learning-case-study\/","title":{"rendered":"Comment moderation as a machine learning case study"},"content":{"rendered":"\n

Continuing my summary of the\u00a0lessons in\u00a0Introduction to Machine Learning<\/a>\u00a0from the Google News Initiative,<\/strong> today I’m looking at Lesson 5 of 8, “Training your Machine Learning model.” Previous lessons were covered here<\/a> and here<\/a>.<\/p>\n\n\n\n

Now we get into the real “how it works” details \u2014 but still without looking at any code or computer languages. <\/p>\n\n\n\n

The “lesson” (actually just a text) covers a common case for news organizations: comment moderation. <\/strong>If you permit people to comment on articles on your site, machine learning can be used to identify offensive comments and flag them so that human editors can review them.<\/p>\n\n\n\n

With supervised learning<\/strong> (one of three approaches included in machine learning; see previous post here<\/a>), you need labeled data<\/strong>. In this case, that means complete comments \u2014 real ones \u2014 that have already been labeled by humans as offensive or not. You need an equally large number of both kinds of comments. Creating this dataset of comments is discussed more fully in the lesson.<\/p>\n\n\n\n

You will also need to choose a machine learning algorithm<\/strong>. Comments are text, obviously, so you’ll select among the existing algorithms that process language (rather than those that handle images and video). There are many from which to choose. As the lesson comes from Google, it suggests you use a Google algorithm.<\/p>\n\n\n\n

In all AI courses and training modules I’ve looked at, this step is boiled down to “Here, we’ll use this one,” without providing a comparison of the options available. This is something I would expect an experienced ML practitioner to be able to explain \u2014 why are they using X algorithm instead of Y algorithm for this particular job? Certainly there are reasons why one text-analysis algorithm might be better for analyzing comments on news articles than another one.<\/p>\n\n\n\n

What is the algorithm doing? <\/strong>It is creating and refining a model<\/strong>. The more accurate the final model is, the better it will be at predicting<\/em> whether a comment is offensive. Note that the model doesn’t actually know<\/em> anything. It is a computer’s representation of a “world” of comments in which some<\/em> \u2014 with particular features or attributes perceived in the training data \u2014 are rated as offensive, and others<\/em> \u2014 which lack a sufficient quantity of those features or attributes \u2014 are rated as not likely to be<\/em> offensive.<\/p>\n\n\n\n

The lesson goes on to discuss false positives and false negatives, which are possibly unavoidable \u2014 but the fewer, the better. We especially want to eliminate false negatives, which are offensive comments not flagged<\/em> by the system.<\/p>\n\n\n\n

“The most common reason for bias creeping in is when your training data isn’t truly representative of the population that your model is making predictions on.”<\/p>\u2014Lesson 6, Bias in Machine Learning<\/cite><\/blockquote>\n\n\n\n

Lesson 6 in the course covers bias<\/strong> in machine learning. A quick way to understand how ML systems come to be biased is to consider the comment-moderation example above. What if the labeled data (real comments) included a lot of comments offensive to women<\/em> \u2014 but all of the labels were created by a team of men, with no women on the team? Surely the men would miss some offensive comments that women team members would have caught. The training data are flawed because a significant number of comments are labeled incorrectly.<\/em><\/p>\n\n\n\n

There’s a pretty good video attached to this lesson. It’s only 2.5 minutes, and it illustrates interaction bias, latent bias,<\/strong> and selection bias<\/strong>.<\/p>\n\n\n\n

\n