0
$\begingroup$

I have a dataset of 240 samples data, with 12 variables (independent variables). From these 12 variables, i would like to identify significant variables for prediction. I perform Gamma analysis, since the data is highly positive skewed.

Here's what I am doing currently: 1. Split the data set into 70% (training data) and 30% (test data) 2. I used 70% data (168 data) to build prediction model using Gamma analysis. I run the analysis few times by excluding each variable at one time to get the best final model. 3. Then validate the final model using the remaining 20% data.

My problem: When should I use K-fold cross validation? Is it when building the prediction model using 70% data, or after i get the final model and use k-fold CV on the remaining 20% data?

$\endgroup$
6
$\begingroup$

Traditionally, you do three steps of "tuning", so you need to split your sample into three parts: a training set, a cross-validation set and a test set.

Training (~60%) In training, you simply estimate your model, but you don't make any changes to the model based on the results (accuracy, goodness of fit) from the training data to avoid overfitting the training set.

Cross validation (~20%) After training your model, you can tune it - vary hyperparameters, remove features, or even select between different models - based on its performance on the cross validation set.

As an example, let's say you want to test which variables to include and which to leave: You specify three different variable combinations (three different models). You train all of them using your training set. Then you evaluate all of them using the cross validation set and select the one that performs best on the CV set.

K-fold CV If you are interested in doing k-fold validation, you repeat exactly what's written above, with one major difference: instead of hard-selecting the 60% and 20% for your training and CV sets, you run the training and validation procedures K-times, each time selecting a different random subsample for training and cross validation. Then you get a set of K results (accuracy, goodness of fit) that you can average to get a more robust estimate of your model's performance.

E.g., if you do 10-fold CV, you'd run it 10 times, and each time you'd randomly sample 10% of your data to be a cross-validation set, with the rest being a training set.

Test set (~20%) After tuning the model and/or selecting the best one, you can test it using the test set. This is data that the model has not seen yet and you shouldn't make any changes to the model based on the test set. This is the very last stage of building the model, only used to evaluate your final model, not to tune it any more (you don't want to overfit your test set).

If doing k-fold CV, you still have to leave out a test set that is separate from your training/CV set you are sampling from.

Putting it all together So in your case, you have $N=240$ and the number of variables is $12$. So the first split of the data would be training/CV (70-80%) and test (20-30%). Which in your case would be $168-192$ for training/CV and $48-72$ for test. Then, in selecting the variables to include, for each model (combination of variables), do K-fold CV as follows:

  1. Split your training/CV set into K equal (random) subsets.
  2. Estimate your model K times, each time leaving out one of the K subsets.
  3. Cross-validate each estimate with the subset that was left out.
  4. Pool your cross-validation results across all the K estimates.

Then pick the model that performs best in CV (on average). Evaluate it on the test set. Don't change it any more.

$\endgroup$
  • $\begingroup$ Hi @metjush. For Gamma analysis, do you know what K results should I refer to? You did mention (accuracy, goodness of fit) --> is it the deviance value? $\endgroup$ – Shahida Nov 3 '15 at 18:21
  • $\begingroup$ If by gamma analysis you mean gamma regression (or GLM with family Gamma), then yes, looking at deviance residuals could be one way of estimating fit. Or the dispersion parameter. Or log-likelihood. I am not very familiar with gamma regressions, but that's where I would start. Which software/package are you using? $\endgroup$ – metjush Nov 4 '15 at 7:23
  • $\begingroup$ Hi @metjush. For the last step, how to evaluate the final model on the test set?. $\endgroup$ – Shahida Dec 9 '15 at 1:29
  • $\begingroup$ @Shahida after you have your final model done and estimated, have it make predictions on the test set that you left out. compare prediction accuracy (F1, AUC, RMSE, depending on your specific problem...) with the ground truth. report this accuracy as your model's accuracy. $\endgroup$ – metjush Dec 9 '15 at 7:15
  • $\begingroup$ I would be dubious of a single hold-out sample like this procedure to estimate "model performance". I would prefer nested cross-validation (K-folds probably) to capture the fitting process and sensititvity to subsample hold-out selection. $\endgroup$ – Noah Hammarlund Oct 11 '18 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.