I am using a Clockwork neural network in Tensorflow to generate a sequence of words. Words are 21-length vectors containing either 0s or 1s (mainly 0s). The idea for the training phase is to provide the network a sequence of
max_time words as input and the following word as output. Training batches are taken randomly from the dataset. Each batch contains the input sequence and the target word.
For the generation of new words, the model processes the sequence and the final layer applies a softmax to the 21 final neurons, providing a probability for each element in the word vector. The error is the mean square difference between the target word and the output vector (both ranging from 0 to 1) and it is used for the AdaGrad optimizer.
I am using a batch size of 50, max_time 32, 300 hidden units, 200 epochs. The problem is that, after each epoch (and even in the test phase), the model assigns equal probability to every element in the vector. This is bad because I expect some elements to have high probability and other elements to have probability close to 0. What can be the reason for this behaviour?