Will the attention mechanism be useful for the short sequence length?

Let's say your training corpus has each query of MAX length 10. and most queries are of word length 3-4 words. How reasonable is to use Attention mechanism in such a scenario?

Because on a high level the paper "Attention is all you need" paper inclines to capture global dependencies in the corpus.


It's reasonable in the sense that there is nothing conceptually wrong with the approach. However as you point out, it's questionable whether this would do better than a straightforward sequence model.

I once read somewhere that character level sequence models perform better when the first letter of every word is swapped with the second letter -- presumably because the second letter, usually being a vowel, is easier to predict and condition the rest of the word on. I mention this because if sequence models are so sensitive to the order in which information is presented, perhaps an attention mechanism would help with that, even if on very short sequences.

  • $\begingroup$ smart answer! - :) $\endgroup$ – HARSH PATHAK Apr 11 at 23:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.