0
$\begingroup$

enter image description hereI am developing a neural network for pattern recognition in Matlab.

Currently:

  1. I divide my dataset into 6 folds (5 folds CV + 1 fold Test)

  2. I choose 10 different number of hidden neurons

  3. I choose 10 different sets of initial weights

  4. -For each fold (as test)

    --For each number of hidden neurons

    ---For each set of initial weights

    ----I perform 5 fold CV (4 training and 1 early stop), saving the average performance on Training Validation and Test and the average number of epochs of training

  5. Averaging across the 6 different choices of test folds (10x10x6 -> 10x10) I choose the optimal number of hidden neurons as the value that gives the best mean performance on ten different random sets of initial values.

6 I choose the optimal number of training epochs as the average of training epochs found across the ten iteration of initial weights.

My problem is now how to choose the initial set of weights for the final network that will be trained over all data. Should I choose again ten sets of initial weights and train 10 different networks with the previous defined parameters to find the best or this could result in a overtraining problem?

Here i attached an image of the performance obtained by different net models on the first fold as Test Set

Improve this question
$\endgroup$
8
  • $\begingroup$ If you initialize the weights randomly in the ordinary fashion, and train your network until the validation loss stops decreasing, there will not be an appreciable difference between alternative initializations. Stated another way, optimizing initializations isn't necessary because a good training method makes the initialization irrelevant. $\endgroup$
    – Sycorax ♦
    Aug 7, 2019 at 22:50
  • $\begingroup$ So as soon that I have the optimal number of hidden neurons and the average training epochs I should not pay attention to the initialization of weights ? I have this doubt becouse during the training of the network I obtain different values of performances. $\endgroup$
    – Mirko Job
    Aug 8, 2019 at 6:24
  • $\begingroup$ How much data do you have? Are you tuning any other parameters? What loss are you computing? How do you decide when to stop training? The answers to these questions could mean that you're doing it wrong. The point is that optimizing initial weights is just a sideshow in a correct training procedure. $\endgroup$
    – Sycorax ♦
    Aug 8, 2019 at 9:41
  • $\begingroup$ How much data do you have? Are you tuning any other parameters? What loss are you computing? How do you decide when to stop training? The answers to these questions could mean that you're doing it wrong. The point is that optimizing initial weights is just a sideshow in a correct training procedure. CV is a random process and measuring performance is subject to variation, like every other statistic. So you'll see some variation. Optimization of more parameters is naturally tied to overfitting... $\endgroup$
    – Sycorax ♦
    Aug 8, 2019 at 9:49
  • $\begingroup$ How do you choose the learning rate and the number of epochs? How are you regularizing? The practical question is whether any configurations are truly different in substantive terms. Does changing the number of neurons from 6 to 7 make an enormous difference in whether or not your network is useful at solving a problem? Does changing the initial value of a weight from 0.0001 to 0.0002? You can optimize until the cows come home, but unless the performance gains you're making are the difference between having a useful model and a useless model, it's not really important. $\endgroup$
    – Sycorax ♦
    Aug 8, 2019 at 9:51

1 Answer 1

Reset to default
0
$\begingroup$

Between different CV folds, you're using different data to estimate the model. My suggestion is to characterize how large a difference (modern) initializations make when you train the same model (same number of neurons etc.) on the same data a large number of times. If your training is working well, then the values of holdout set loss should be very similar (because initializations should not make an enormous difference, because training is directing the network toward equivalent parameter configurations).

However, even if the values of the loss are different, this may not be important if the two models are not different in their practical utility. Optimization for its own sake is not really the point. The point is whether or not your model solves your problem.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.