0
$\begingroup$

I have trained two networks N1 and N2. Both are trained on the same data with same number of epochs. N2 has more number of parameters with more number of layers and nodes. The training and validation accuracy/loss are similar for both the networks. Now before doing test with real test data, which network I should select: N1 (smaller) or N2 (bigger)?

$\endgroup$
  • $\begingroup$ Is there anything preventing you from trying both? The general principle is to stick with the simplest model that works, but I don't know if that translates to neural networks $\endgroup$ – David Jun 20 at 7:35
  • $\begingroup$ I am participating in a machine learning challenge and there is no leader-board to see the results on the test set. Whatever I submit will be the final one. $\endgroup$ – talk2speech Jun 20 at 10:26
1
$\begingroup$

If your target is to finish as high as possible in a one-chance-only ML competition, as pointed out in your comment, you are facing a tough dilemma! If validation scores are similar, I'd say their performance on the actual test set will also be close.

A possible approach would be to make a high number of train-test splits with the data you have, fit both networks and try on your different test sets. You should do this a very high number of times, and check for statistically significative differences. However, since the iterations are not really independent, this approach also has its flaws. With a lot of data this may work, but with a small dataset this is just an overfitting contest.

Another consideration, simpler models often generalize better, but I don't see this really applying here.

So, I would go for a completely different approach: a meta-model. Analize the type of errors the networks make and try to find regions in your data where one of the networks perform better than the other. Then build a "meta-model" that calls the adequate network for each region. A simpler way to do something similar is to just average the output from both networks. Of course, you should make the convenient test to this new type of model.

Another reason for using what I just described in ML competitions is that nowadays anyone can write some Python code and build a neural network. If you want to win, you have to do something different than anyone else

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.