I'm currently working on a Polyphonic Sound Event Detection task and I've already implemented in Lasagne a complex structure which involves both Convolutional and Recurrent layers. The network is quite big and it gets trained slowly but results are good up to now and I predict a total of 5 classes with almost 80% of average f1-score.
All the Recurrent layers involved in the network use GRU units. However, doing some experiments I tried to replace them with LSTM. It was merely a replacement and I didn't modify any other parameters/settings of the network. However, something really unexpected happened: the network seems to not work at all. I've a drop of more than 40% in my average f1-score and even increasing batch_size or total number of epochs the network doesn't learn at all.
How is it possible such a huge difference in the performances of the two different recurrent units? How could I face the problem?
My intuition is that as the network is quite hard to train, maybe all the complexity that relies behind LSTM implementation doesn't allow the learning process? What should I monitor in the network?