{"id":744,"date":"2021-05-30T10:01:48","date_gmt":"2021-05-30T14:01:48","guid":{"rendered":"https:\/\/www.macloo.com\/ai\/?p=744"},"modified":"2021-05-30T10:12:18","modified_gmt":"2021-05-30T14:12:18","slug":"attention-in-machine-learning-and-nlp","status":"publish","type":"post","link":"https:\/\/www.macloo.com\/ai\/2021\/05\/30\/attention-in-machine-learning-and-nlp\/","title":{"rendered":"Attention, in machine learning and NLP"},"content":{"rendered":"\n

Let’s begin at the beginning, with Attention Is All You Need<\/a> (Vaswani et al., 2017). This is a conference paper with eight authors, six of whom then worked at Google. They contended that neither recurrent neural networks nor convolutional neural networks are necessary for machine translation of languages, and hence the Transformer,<\/strong> “a new simple network architecture,” was born. (Note: It relies on feed-forward neural networks.)<\/p>\n\n\n\n

Transformers<\/a> are the basis for machine translation and other tasks relying on language models. GPT-3<\/strong> has recently become infamous; others include BERT (from Google) and ELMo.<\/p>\n\n\n\n

Before the work by Vaswani and his co-equal co-authors, progress in NLP was limited (although it had advanced a lot since 2012) because of the ways in which RNN models depend on the sequence<\/em> and position<\/em> of words in a text. Transformers eliminate those limitations. With recurrent neural networks, there are impediments to parallel processing. Other researchers had previously cracked that nut using ConvNets, but then other<\/em> limitations were inherent (exponential increase in the number of computational operations). Transformers also<\/em> eliminate those<\/em> limitations.<\/p>\n\n\n\n

So the Transformer was a first in NLP, a breakthrough. For machine translation, the paper claimed “a new state of the art” (p. 10).<\/p>\n\n\n\n

I had learned that an encoder and a decoder connected by <\/em>an attention module<\/strong> is a standard architecture for machine language translation, e.g. Google Translate. This was true before<\/em> 2017, so what is the difference effected by the Transformer? It eliminates RNNs and ConvNets from the architecture, yes (“our model contains no recurrence and no convolution”) \u2014 but what else?<\/p>\n\n\n\n

Attention used in a new way<\/h2>\n\n\n\n

“An attention function<\/strong> can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors<\/strong>. The output is computed as a weighted sum<\/strong> of the values, where the weight assigned to each value is computed by a compatibility function<\/strong> of the query with the corresponding key”(Vaswani et al., 2017, p. 3). I’m okay with that, although I doubt I would be able to explain it to my non\u2013computer science students. (I do explain weights and features when I introduce neural nets to them, and I explain word vectors when we start NLP. The trouble is they don’t know how to write a program, and they certainly don’t understand what a function is.)<\/p>\n\n\n\n

There are different attention functions<\/strong> that could be used. One is additive attention; another is dot-product attention, which is multiplicative rather than additive. Dot-product is “much faster and more space-efficient in practice.” Vaswani et al. used a scaled<\/em> dot-product attention function (p. 4). They also used multi-head attention,<\/em> meaning the model uses eight parallel attention layers, or heads. The explanation was a bit beyond me, but the gist is that the model can look at multiple things at the same time, like juggling more balls simultaneously.<\/p>\n\n\n\n

Multi-head attention \u2014 plus the freedom of no-sequence, no-position<\/em> \u2014 enables the Transformer to look at all<\/em> the context for a word, and do it for multiple words at the same time<\/em>.<\/p>\n\n\n\n

With my rudimentary understanding of recurrent neural nets<\/a>, I have a fuzzy idea of how this use of attention functions produces better results, mainly by being able to take in and compare<\/em> more of the text, a little closer to the way human brains hold an entire conversation even though it’s not a literal “recording” of the exact conversation. The way we comprehend meaning when we read has to do with millions of associations built up over a lifetime, as well as many associations within that present text. We are not processing separate little slices of a sentence \u2014 our brains handle a text more holistically. <\/p>\n\n\n\n

A Transformer does<\/em> use word embeddings<\/strong> to convert the tokens (both inout and output) to vectors (Vaswani et al., 2017, p. 5). It uses softmax<\/a> but no LSTMs<\/a> (because, again, “no recurrence”). <\/p>\n\n\n\n

Please help me, YouTube<\/h2>\n\n\n\n

I found a video (13:04) that helped me in my struggle to understand the Transformer architecture: <\/p>\n\n\n\n

\n