Small or big network: which one to choose? I have trained two networks N1 and N2. Both are trained on the same data with same number of epochs. N2 has more number of parameters with more number of layers and nodes. The training and validation accuracy/loss are similar for both the networks. Now before doing test with real test data, which network I should select: N1 (smaller) or N2 (bigger)?
 A: If your target is to finish as high as possible in a one-chance-only ML competition, as pointed out in your comment, you are facing a tough dilemma! If validation scores are similar, I'd say their performance on the actual test set will also be close.
A possible approach would be to make a high number of train-test splits with the data you have, fit both networks and try on your different test sets. You should do this a very high number of times, and check for statistically significative differences. However, since the iterations are not really independent, this approach also has its flaws. With a lot of data this may work, but with a small dataset this is just an overfitting contest.
Another consideration, simpler models often generalize better, but I don't see this really applying here.
So, I would go for a completely different approach: a meta-model. Analize the type of errors the networks make and try to find regions in your data where one of the networks perform better than the other. Then build a "meta-model" that calls the adequate network for each region. A simpler way to do something similar is to just average the output from both networks. Of course, you should make the convenient test to this new type of model.
Another reason for using what I just described in ML competitions is that nowadays anyone can write some Python code and build a neural network. If you want to win, you have to do something different than anyone else
