Why does my RNN perform well on long sequences, but not on the short, easy ones? I have a problem that has left me perplexed and baffled. I am training an RNN (with LSTM cells) to generate a sequence from the output of a CNN for the purpose of annotating images. E.g.,
input: picture of an orange
output -> "fruit"
input: picture containing: apple fish banana broccoli
output -> "fruit meat fruit veggie"
The sequences vary in length between 0-20 words. My model seems to be performing very well when there are ~7+ inputs (60%+ accuracy), but almost completely incorrect for shorter sequences (<7). How can this happen? I would think that if it is good at recognizing "orange" in a long sequence, a single sequence with only the word "orange" should be a piece of cake, since it can reuse that same, learned pattern regardless of whether the input is long or short.
One note: There are very many more long examples in the training set (naturally, since there are exponentially more possible combinations). But rather than augmenting the training set, I'm curious as to why the easy, short sequences don't turn out to be easy at all. Any ideas?
 A: My hunch is that this is because you also have a  language model (implicitly) trained which might be helping in your case.
I guess you are trying to perform image captioning. Taking a concrete example,  an image of a diaper will always typically occur with the image of a baby but not vice versa.
Assume that images of only diapers are quite rare but those of only babies are numerous in your dataset and do can be learnt easily. Then captioning "Baby wearing diaper" might be easier than captioning just "Diaper" . Assumption here is that your network is able to recognize a baby well. 
@jstaker7 If you can post some images on which the network works well and some on which it doesn't it'll be easier to dig deeper
A: Your input does not seem to be sequential (I.e in the output the $t^{th}$ word seems to be independent of $t-1^{th}$). Using LSTMs might be an overkill for this problem.
To your question on why performance deteriorates-can you indicate what's the overall structure of your network. Are you using an encoder-decoder kind of framework? Knowing this will help provide a better answer.
