# Attention based models for machine comprehension

Currently, I am understanding Attention models. I specifically need it to build a machine comprehension model (a model which can find answer to a question from a given comprehension). But I want to understand the model generally and not specifically to this topic. I have a question about attention layer but before that I would like to share what I understood.

## The things I understood so that you can correct me if I made some mistakes

For this, I have referred an article online which I am sharing here - http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/

It gave basic intuition of what the attention layer does - It asks the model the model to focus on a specific part of the input in order to generate the output. Here's an image of it - http://www.wildml.com/wp-content/uploads/2015/12/Screen-Shot-2015-12-30-at-1.51.19-PM.png

The pink shades are defined by attention variables 'a(i)'. These variables are actually probability distribution that shows 'the probability that the model should focus on the area represented by that variable'. So, a general figure on any attention based model is as - http://www.wildml.com/wp-content/uploads/2015/12/Screen-Shot-2015-12-30-at-1.16.08-PM.png - where 'a's are the attention variables (weights). These variables are trained by the model.

## The question

I have a question in mind. Are the attention weights same for every inputs (in machine comprehension, input = 'comprehension'). If so, then independent of the comprehension, the model will focus on a very small part of every comprehension to find an answer. And it isn't necessary that the answer lies in same part of all comprehensions.

If the weights are different for different inputs, then how will we train them. Because, as per my knowledge, in every neural network, the weights are trained for all the input data irrelevant of the input identifier i.e the final weight values at the end of training is same for all the inputs. And after training, these weights are the final values which are used in real-time application.

The attention "weights" are not weights in the sense that the word is usually used -- they aren't trainable parameters. They are usually computed as a function of the input and some other parameters, so for question answering, the weight might be $a_t = \text{softmax}(h_t^T W q)$, where $h$ is the hidden state, $W$ is a parameter, and $q$ is the question you're trying to answer. So $a_t$ is different for different inputs, but the weight matrix $W$ is learnable and can be fixed.
• @KadamParikh $W$ is the parameter responsible for directing attention. I'm not sure what you're confused about. See this post for more info – shimao Sep 17 '18 at 5:11