Recurrent neural network for sequence generation outputs uniform probabilities

Question

I am using a Clockwork neural network in Tensorflow to generate a sequence of words. Words are 21-length vectors containing either 0s or 1s (mainly 0s). The idea for the training phase is to provide the network a sequence of max_time words as input and the following word as output. Training batches are taken randomly from the dataset. Each batch contains the input sequence and the target word.

For the generation of new words, the model processes the sequence and the final layer applies a softmax to the 21 final neurons, providing a probability for each element in the word vector. The error is the mean square difference between the target word and the output vector (both ranging from 0 to 1) and it is used for the AdaGrad optimizer.

I am using a batch size of 50, max_time 32, 300 hidden units, 200 epochs. The problem is that, after each epoch (and even in the test phase), the model assigns equal probability to every element in the vector. This is bad because I expect some elements to have high probability and other elements to have probability close to 0. What can be the reason for this behaviour?

Answer 1

A good first sanity check would be to pick some simpler task that you think a RNN ought to be able to easily learn, and use your same setup to see if it successfully learns that task. Example: the correct output is the same as the input from 7 time steps ago.

Also, are you sure you want the mean square difference as your loss function? It would be more typical to use the cross-entropy loss in this case (where you want to estimate a probability/likelihood), I think.

Answer 2

I've had similar problems, have you tried checking:

1. Are your layers really wired correctly? Is the output layer really getting the signal?
2. As pointed out by @D.W., do check the loss function -- cross entropy is the gold standard indeed
3. Have you checked how the inputs look like? Have you encoded the input vectors (words) correctly?
4. Are you using too much regularization at some point?
5. What about the vocabulary, do its dimensions match with the e.g., embedding layer?
6. Reproduce a minimal $\textbf{working}$ example
7. Check if weights change in time
8. Does loss change? If not, this is the core problem and the architecture does not learn from the input at all.