1
$\begingroup$

I'm currently working on a Polyphonic Sound Event Detection task and I've already implemented in Lasagne a complex structure which involves both Convolutional and Recurrent layers. The network is quite big and it gets trained slowly but results are good up to now and I predict a total of 5 classes with almost 80% of average f1-score.

All the Recurrent layers involved in the network use GRU units. However, doing some experiments I tried to replace them with LSTM. It was merely a replacement and I didn't modify any other parameters/settings of the network. However, something really unexpected happened: the network seems to not work at all. I've a drop of more than 40% in my average f1-score and even increasing batch_size or total number of epochs the network doesn't learn at all.

How is it possible such a huge difference in the performances of the two different recurrent units? How could I face the problem?

My intuition is that as the network is quite hard to train, maybe all the complexity that relies behind LSTM implementation doesn't allow the learning process? What should I monitor in the network?

$\endgroup$
6
  • $\begingroup$ Are you measuring performance on a per-parameter basis? A network with $k$ GRUs has fewer parameters than a network with $k$ LSTM cells. $\endgroup$
    – Sycorax
    Apr 17, 2018 at 14:25
  • $\begingroup$ @Sycorax what do you mean with per-parameter basis? Actually I've worked a little bit on the correct initialization of each parameter of the LSTM unit (weights and biases of each gate) but nothing seems to solve the problem. $\endgroup$
    – arcticriki
    Apr 17, 2018 at 14:42
  • $\begingroup$ I mean that if you're comparing performance of a model with $k$ parameters to a model with $100k$ parameters, you're comparing apples to orangoutangs. $\endgroup$
    – Sycorax
    Apr 17, 2018 at 14:47
  • $\begingroup$ I've just tried to reduce the number of units per LSTM layer, in such a way that now the total number of parameters of the models is more or less the same, but unfortunately there are no improvements. Actually, I also think it's a problem of complexity but I cannot figure out how to manage the issue properly. $\endgroup$
    – arcticriki
    Apr 17, 2018 at 15:02
  • $\begingroup$ If may help other desperate readers, once the two models were pretty much normalized by the number of parameters and after A LOT of hyper-parameters retuning, the performances started to be comparable. GRU outperforms LSTM but not dramatically, and a consistent difference in training time is visible. $\endgroup$
    – arcticriki
    Apr 19, 2018 at 8:41

1 Answer 1

1
$\begingroup$

In comments, OP writes:

If may help other desperate readers, once the two models were pretty much normalized by the number of parameters and after A LOT of hyper-parameters retuning, the performances started to be comparable. GRU outperforms LSTM but not dramatically, and a consistent difference in training time is visible.


I've copied OP's comment as a community wiki answer because the comment is, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions.

$\endgroup$
1
  • $\begingroup$ This result is really interesting. It'd be nice to have a write-up, showing the hyper-parameters used and so on. Might even be paper-worthy, if well done. Certainly worthy of a blog post I reckon though. cc @arcticriki $\endgroup$ Jul 9, 2018 at 2:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.