1
$\begingroup$

I'm currently working on a Polyphonic Sound Event Detection task and I've already implemented in Lasagne a complex structure which involves both Convolutional and Recurrent layers. The network is quite big and it gets trained slowly but results are good up to now and I predict a total of 5 classes with almost 80% of average f1-score.

All the Recurrent layers involved in the network use GRU units. However, doing some experiments I tried to replace them with LSTM. It was merely a replacement and I didn't modify any other parameters/settings of the network. However, something really unexpected happened: the network seems to not work at all. I've a drop of more than 40% in my average f1-score and even increasing batch_size or total number of epochs the network doesn't learn at all.

How is it possible such a huge difference in the performances of the two different recurrent units? How could I face the problem?

My intuition is that as the network is quite hard to train, maybe all the complexity that relies behind LSTM implementation doesn't allow the learning process? What should I monitor in the network?

$\endgroup$
  • $\begingroup$ Are you measuring performance on a per-parameter basis? A network with $k$ GRUs has fewer parameters than a network with $k$ LSTM cells. $\endgroup$ – Sycorax Apr 17 '18 at 14:25
  • $\begingroup$ @Sycorax what do you mean with per-parameter basis? Actually I've worked a little bit on the correct initialization of each parameter of the LSTM unit (weights and biases of each gate) but nothing seems to solve the problem. $\endgroup$ – arcticriki Apr 17 '18 at 14:42
  • $\begingroup$ I mean that if you're comparing performance of a model with $k$ parameters to a model with $100k$ parameters, you're comparing apples to orangoutangs. $\endgroup$ – Sycorax Apr 17 '18 at 14:47
  • $\begingroup$ I've just tried to reduce the number of units per LSTM layer, in such a way that now the total number of parameters of the models is more or less the same, but unfortunately there are no improvements. Actually, I also think it's a problem of complexity but I cannot figure out how to manage the issue properly. $\endgroup$ – arcticriki Apr 17 '18 at 15:02
  • $\begingroup$ If may help other desperate readers, once the two models were pretty much normalized by the number of parameters and after A LOT of hyper-parameters retuning, the performances started to be comparable. GRU outperforms LSTM but not dramatically, and a consistent difference in training time is visible. $\endgroup$ – arcticriki Apr 19 '18 at 8:41
1
$\begingroup$

In comments, OP writes:

If may help other desperate readers, once the two models were pretty much normalized by the number of parameters and after A LOT of hyper-parameters retuning, the performances started to be comparable. GRU outperforms LSTM but not dramatically, and a consistent difference in training time is visible.


I've copied OP's comment as a community wiki answer because the comment is, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions.

$\endgroup$
  • $\begingroup$ This result is really interesting. It'd be nice to have a write-up, showing the hyper-parameters used and so on. Might even be paper-worthy, if well done. Certainly worthy of a blog post I reckon though. cc @arcticriki $\endgroup$ – Hugh Perkins Jul 9 '18 at 2:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.