# Tag Info

1

I think the paper you quote is just wrong. Luong only generalizes Bahdanau's equations by replacing a single-layer MLP by a general function score (and shows that dot-product can work equally well as the MLP), but it still scores "similarity" of decoder state and exactly one encoder state.

0

This is called transfer learning. There are many methods for doing this, including adding new layers at the end, deleting or retraining some of your DNN’s final layers, freezing or lowering learning rated on later layers, etc. I’d guess that a high percentage of Bert use involves at least some new tweaking before deployment.

1

This is not really an answer but rather a comment (don't have the rep yet) - in general, LDA is perfect for dimensionality reduction under the bag-of-words assumption. But you wouldn't really do that the way you suggested (or at least I haven't seen it done like this). Using LDA you would estimate two things - topics (that is distributions over the ...

1

$\alpha$ is the hyper-parameter for the mixing proportions. The smaller the $\alpha$ the more focused your documents will be (they will strongly focus on small number of topics). Btw, it is generally better to allow for an asymmetric $\alpha$. You want to reduce the other hyper-parameter, $\eta$. The same thing, the smaller the hyper-parameter, the more "...

1

I guess the only thing I am aware of that does this are (dense) embeddings in real vector spaces." Exactly, they get the best of both worlds: A fixed vector length and an ability to represent as many concepts as desired. While indexing with discrete values for each word will loose the advantage of fixed vector lengths, which is useful for many things. In ...

0

My main field of research isn't NLP, so I can't make any claims about what is or isn't state of the art. However I've observed that many SOTA modern language models (BERT and GPT2) use neither character nor word vocabularies: instead they use byte-pair encoding (BPE) vocabularies. Basically the idea is to start with a character vocabulary, and then ...

0

In order to determine whether 18% is too big or not, it would be needed to have a precise definition of how that index is calculated. Fortunatelly, we have a calculator that does it for us, but I don't know to what extent the source should be trusted (i.e: more detail on how the calculation was done is required). Assuming the p-value shown there is accurate,...

2

A "typical" attention mechanism might assign the weight $w_i$ to one of the source vectors as $w_i \propto \exp(u_i^Tv)$ where $u_i$ is the $i$th "source" vector and $v$ is the query vector. The attention mechanism described in OP from "Pointer Networks" opts for something slightly more involved: $w_i \propto \exp(q^T \tanh(W_1u_i + W_2v))$, but the main ...

2

It is only an efficiency issue. In theory, the attention mechanism can work with arbitrarily long sequences. The reason is that batches must be padded to the same length. Sentences of the maximum length will use all the attention weights, while shorter sentences will only use the first few. By this sentence they mean they want to avoid batches like this: ...

2

Recently there were two papers commenting on the self-attention heads: Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned from University of Edinburgh Are Sixteen Heads Really Better than One? from CMU Both of them come to the same conclusion that most of the heads are just noise and can be relatively easily ...

1

As you point out an accuracy of 0.5 (on average) is the lowest you could get by random labeling. A very low F1 score and an accuracy of 0.5 are therefore not logically inconsistent. I would recommend rephrasing the question title. Your question is "why does the classifier predict everything as negative?". If the class imbalance does not exist (no error in ...

1

In the standard Transformer model as introduced by Vaswani et al. it is not possible, because generating a word is always conditioned on the previously decoded words, so there is no other option than generating words one by one. Recently, there appeared several papers on so-called non-autoregressive models which parallelize the decoder as well, but the ...

Top 50 recent answers are included