2
A "typical" attention mechanism might assign the weight $w_i$ to one of the source vectors as $w_i \propto \exp(u_i^Tv)$ where $u_i$ is the $i$th "source" vector and $v$ is the query vector. The attention mechanism described in OP from "Pointer Networks" opts for something slightly more involved: $w_i \propto \exp(q^T \tanh(W_1u_i + W_2v))$, but the main ...
1
I think the paper you quote is just wrong.
Luong only generalizes Bahdanau's equations by replacing a single-layer MLP by a general function score (and shows that dot-product can work equally well as the MLP), but it still scores "similarity" of decoder state and exactly one encoder state.
1
I guess the only thing I am aware of that does this are (dense) embeddings in real vector spaces."
Exactly, they get the best of both worlds: A fixed vector length and an ability to represent as many concepts as desired. While indexing with discrete values for each word will loose the advantage of fixed vector lengths, which is useful for many things. In ...
1
$\alpha$ is the hyper-parameter for the mixing proportions. The smaller the $\alpha$ the more focused your documents will be (they will strongly focus on small number of topics). Btw, it is generally better to allow for an asymmetric $\alpha$.
You want to reduce the other hyper-parameter, $\eta$. The same thing, the smaller the hyper-parameter, the more "...
1
It is only an efficiency issue. In theory, the attention mechanism can work with arbitrarily long sequences. The reason is that batches must be padded to the same length.
Sentences of the maximum length will use all the attention weights, while shorter sentences will only use the first few.
By this sentence they mean they want to avoid batches like this:
...
1
As you point out an accuracy of 0.5 (on average) is the lowest you could get by random labeling. A very low F1 score and an accuracy of 0.5 are therefore not logically inconsistent. I would recommend rephrasing the question title. Your question is "why does the classifier predict everything as negative?". If the class imbalance does not exist (no error in ...
1
In the standard Transformer model as introduced by Vaswani et al. it is not possible, because generating a word is always conditioned on the previously decoded words, so there is no other option than generating words one by one.
Recently, there appeared several papers on so-called non-autoregressive models which parallelize the decoder as well, but the ...
1
This is not really an answer but rather a comment (don't have the rep yet) - in general, LDA is perfect for dimensionality reduction under the bag-of-words assumption. But you wouldn't really do that the way you suggested (or at least I haven't seen it done like this). Using LDA you would estimate two things - topics (that is distributions over the ...
1
Recently there were two papers commenting on the self-attention heads:
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned from University of Edinburgh
Are Sixteen Heads Really Better than One? from CMU
Both of them come to the same conclusion that most of the heads are just noise and can be relatively easily ...
Only top voted, non community-wiki answers of a minimum length are eligible
