-1
$\begingroup$

I have trained my neural networks on full dataset and found out that the best architecture contains 27 hidden neurons. Is it appropriate for me to perform k-fold validation solely for the network with 27 hidden neurons? Or i must apply k-fold validation during my training process?

$\endgroup$

closed as unclear what you're asking by Michael Chernick, SmallChess, kjetil b halvorsen, Peter Flom Apr 17 '18 at 13:12

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

0
$\begingroup$

I think you may be misunderstanding the incentive to use K-fold CV, I will explain my understanding of it.

K-Fold CV is usually employed to improve the validity of your test data set. In cases where you may have a limited size data-set it's important to maximize the amount of data you train your network with; the more data your network trains with the more variation it has experience dealing with. This, however, limits the size of the portion of your data-set put aside for testing. The issue with this is that if your test data isn't representative of you entire data-set then your neural networks performance with it also isn't really representative either. The performance you get from a small test set is now down to 'luck of the draw'. That's an issue.

Enter K-Fold CV. The idea here is to split your small data-set up into K chunks and run your neural network K times using a different chunk for testing each time. The other K-1 chunks are used for training. Now for that particular network architecture you have K different sets of results which, combined, have been tested across the entire data-set. These results can be averaged and errors can be calculated to produce an accurate range for the performance of your network. So in essence K-Fold CV helps to relinquish the difficulties created by a undersized data-set.

Hopefully you have understood me, we're speaking the same language, and we can tackle the question you've asked.

Firstly, when you say you've trained your networks on the 'full data-set' does this mean you didn't put aside a fraction of that data-set to be tested on? If that were so I'm afraid you'll have to either find some more data that matches your data-set or you'll have to retrain your networks from scratch with a fraction of the data left to test on.

Secondly, with this in mind, how big is your data-set? Is it necessary to perform K-Fold CV as it's a costly, but sometimes valuable, process. If you think you have a smaller data-set then the answer would be a YES to the last question you've asked. You WILL need to perform K-Fold cross validation during the training process.

Lastly, you may also want to look into reserving what is called a validation set of data. This answer can give you some insight into why that may be of interest to you.

Good luck!

$\endgroup$
  • $\begingroup$ Thank you so much!! I trained on the full dataset at the first time. Then, I realized I didn't split the data into training dataset and validation dataset. I tried splitting my data into 75% training dataset and 25% validation dataset and train again but the validation results are really bad. My dataset contains a total of 739 data and I just wonder if I have too little data. $\endgroup$ – GabriellaN Apr 17 '18 at 5:35
  • $\begingroup$ Then K-Fold CV may come in handy to you! $\endgroup$ – Eden Trainor Apr 17 '18 at 6:05
  • $\begingroup$ K-Fold CV MAY come in handy to you, depending on what the nature of your data is and what are you trying to do with it! You say that your validation results come out really bad? Does that mean you network is performing considerably better on training data? If so your network may be over-fitting. There are quite a few remedies to that issue.There are a LOT of reasons why your network may not be performing as well as you'd hope so don't put it down to the size of your data just yet! I'm currently working on a binary classifier with only 400 data points. $\endgroup$ – Eden Trainor Apr 17 '18 at 6:15
  • $\begingroup$ My training data can have a correlation coefficient of 0.769 and a MSE of 0.038 but my validation only has a 0.2046 R-sqt value. I may try K-Fold CV to see if I can improve the validation results. Thank you very much for your help! :D $\endgroup$ – GabriellaN Apr 17 '18 at 8:45

Not the answer you're looking for? Browse other questions tagged or ask your own question.