7
$\begingroup$

Similar threads:

Feature selection for "final" model when performing cross-validation in machine learning

How to choose a predictive model after k-fold cross-validation?

My question is quite simple and is definitely related to the similar threads above but what I am looking for is a concrete yes/no to the question below:

I am working on a regression problem where I have a target function of 1 variable that I am trying to predict using 5 explanatory variables. I have 1200 examples of the response and explanatory data. I decide to split my 1200 examples into a calibration set of 1000 examples and a test set of 200 examples. The calibration set is used to train my model and the test set is completely independent.

Let's say I am using a Neural Network of a particular configuration/parametrization and I am looking to find the best possible network weights and biases such that it provides the best performance on my test set.

To do this I have chosen to perform k-fold cross-validation on the calibration data. Let's say I opt for 10 folds. I thus produce 10 different calibrated models (using the training and validation sets for each k-fold) each of the same configuration using the Neural Network described above. I now want to use the Neural Network to provide an output on my test set using the parameters (weight and biases) determined from the k-fold cross-validation. To produce the estimates on the test set do I simply average the weights and biases from each of the 10 different calibrated models and use this parametrization to produce outputs to compare with my test set for the target function?

Thank you everyone for their help!

Improve this question
$\endgroup$
4
  • $\begingroup$ (1) I don't know, but hoping to prompt someone to give answer related to your split sample/test set (2) I'd train on whole calibration set. The validation was to validate the process and not the model, and to estimate optimism. (3) is your test set large enough? it seems to small to0 provide stable estimates. $\endgroup$
    – charles
    May 2, 2014 at 1:41
  • $\begingroup$ @charles (2) I am using k-fold cross-validation on my calibration data to avoid over-fitting. Using the complete calibration data with no hold-out set leads to drastic over-fitting with my data. (3) In my trials with random-splitting to produce a hold-out set provides fairly stable outputs on my test set. My question is to learn the process for determining the final model parameters using k-fold cross-validation so I may use the model to provide predictions as new explanatory data becomes available. $\endgroup$
    – John
    May 2, 2014 at 1:53
  • 1
    $\begingroup$ The alternative would be to replace the hold-out set by an outer, independent cross validation (so-called nested or double cross validation), not to skip the validation of the optimized model. $\endgroup$
    – cbeleites unhappy with SX
    May 2, 2014 at 9:18
  • $\begingroup$ If the (inner) cross validation yield much better looking performance estimates than the outer hold-out valiation (which I assume from your "drastic overfitting"), then it is not clear whether the optimization was acutally successful at all. $\endgroup$
    – cbeleites unhappy with SX
    May 2, 2014 at 10:55

2 Answers 2

Reset to default
8
$\begingroup$

"To produce the estimates on the test set do I simply average the weights and biases from each of the 10 different calibrated models and use this parametrization to produce outputs to compare with my test set for the target function?"

No. Cross-validation is a procedure for estimating the test performance of a method of producing a model, rather than of the model itself. So the best thing to do is to perform k-fold cross-validation to determine the best hyper-parameter settings, e.g. number of hidden units, values of regularisation parameters etc. Then train a single network on the whole calibration set (or several and pick the one with the best value of the regularised training criterion to guard against local minima). Evaluate the performance of that model using the test set.

In the case of neural networks, averaging the weights and biases of individual models won't work as different models will choose different internal representations, so the corresponding hidden units of different networks will represent different (distributed) concepts. If you average their weights, they mean of these concepts will be meaningless.

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ Averaging the parameters will only work in the expected manner for linear models. However, even then you produce an aggregated or ensemble model, which is not the same as a single model. $\endgroup$
    – cbeleites unhappy with SX
    May 2, 2014 at 9:16
  • $\begingroup$ @Dikran Marsupial Since the focus of my question is to find model parameters from k-fold cross-validation that will provide a good model that I can use to simulate predictions as new explanatory data becomes available would it be appropriate to use parameters found on a single k-fold trial if it happens to provide the best performance on the complete calibration data? $\endgroup$
    – John
    May 2, 2014 at 20:18
  • $\begingroup$ @Dikran Marsupial In other words: say I run 10-fold cross-validation and the model parameters determined on fold 3 happen to provide the best model performance when applied to the complete calibration data would it be appropriate to use these parameters as the optimal configuration for simulating predictions given new explanatory data? $\endgroup$
    – John
    May 2, 2014 at 20:20
0
$\begingroup$

I think the correct answer to this question is provided by a document of sklearn here: http://scikit-learn.org/stable/modules/cross_validation.html

Basically by doing cross-validation(CV), compared with hold-out validation, we can reduce the amount of data taken by the validation set, thus increase the amount of data used by training set. This solves the problem where the amount of training data is not enough while we still want to have training, validation and test set.

As written in the document: "The performance measure reported by k-fold cross-validation is then the average of the values computed in the loop. This approach can be computationally expensive, but does not waste too much data (as it is the case when fixing an arbitrary test set), which is a major advantage in problem such as inverse inference where the number of samples is very small."

Improve this answer
$\endgroup$
1
  • $\begingroup$ The sklearn page is a useful page. @Dikran's answer agrees with the information on that page. Refer to the first figure on that page which indicates that cross-validation is used to obtain the best parameters for the model. Also, in the discussion, they say that cross-validation is useful to evaluate different hyper-parameter settings for estimators while being able to reduce the variability of the training result than when using a single partitioning of training+validation+test sets. $\endgroup$
    – user650654
    Feb 13, 2020 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.