6
$\begingroup$

If variance of test dataset is lower than the one of the train dataset is it worth splitting the data? Since we know our dataset will always be limited is it fair to select models under the above condition? Thanks

$\endgroup$
4
$\begingroup$

You have to first figure out why you are splitting the data. The only reason that comes immediately to mind is that fitting the model is so laborious that you can only do it once. Otherwise, resampling methods are far better, starting with the Efron-Gong optimism bootstrap (see e.g. the R rms package) or 10-fold cross-validation repeated 100 times.

$\endgroup$
7
  • $\begingroup$ I'm splitting data to validate the models. I'm already using 100x10 folds cv in my train set. $\endgroup$
    – jpcgandre
    Sep 12 '14 at 13:19
  • 2
    $\begingroup$ Cross-validation is validating the model. Don't use a single test set. If you need to tweak the model in a way that requires cross-validation to determine the tweak (not usually recommended) then you will need nest cross-validation or double bootstrap. $\endgroup$ Sep 12 '14 at 13:23
  • 2
    $\begingroup$ Unless both training and independent test sets are huge, resampling performs better. You just have to make absolutely sure that the resampling procedure repeats all steps that utilized $Y$, afresh for each resample. $\endgroup$ Sep 13 '14 at 13:30
  • 1
    $\begingroup$ Could you please expand or give references on resampling procedure repeats all steps that utilized Y. Many thanks $\endgroup$
    – jpcgandre
    Sep 13 '14 at 17:56
  • 2
    $\begingroup$ There are papers by Ewout Steyerberg on this topic. In my experience for a binary outcome you need more than 15,000 subjects for split-sample validation to work satisfactorily. The bootstrap and cv work well for any size dataset, and you don't also need an independent sample when you use them unless you think the data collection process or the type of observation has changed. If the datasets are really large the confidence interval for the accuracy score computed in the independent sample is narrow enough to make the point estimate trustworthy. $\endgroup$ Sep 14 '14 at 1:10
5
$\begingroup$

Not necessarily. What is more important is the conditional distribution of $Y|X$ being consistent in both data sets. In other words, if $Y$ variance in the test data set is higher, it could be that $X$ variance is also higher and the fitted coefficients will explain $Y$ variance equally well.

Plot Y ~ X on both data sets and fit a regression line on each plot. What do you see?

$\endgroup$
7
  • $\begingroup$ This question came up when I was analysing a dataset consisting of 250 observations of each 21 variables (20 input and 1 output variables). I analysed the effect of changing the train/test size on the model selection criteria. If I used 70/30, 80/20, 90/10 splits I ended up with very different results, being optimistically better when I used 90/10. The reason for this is because my 10% dataset has a much lower variance. $\endgroup$
    – jpcgandre
    Sep 12 '14 at 13:18
  • 3
    $\begingroup$ You have uncovered a big problem with cross-validation: the choice of the fold sizes. Use the bootstrap instead, which requires only the choice of the number of resamples (I suggest about 500 here). You didn' explain why you need variable selection. That is causing much of your problem. Variable selection is arbitrary and does not usually help with overfitting. For your sample size you badly need a penalized MLE method. $\endgroup$ Sep 12 '14 at 13:28
  • 1
    $\begingroup$ If you use a proper or semi-proper accuracy scoring rule (e.g., you don't use the ill-advised proportion classified correctly) the .632 bootstrap (which is implemented in R rms) is not needed and you can use the ordinary optimism bootstrap. If you did not do variable selection then why did you mention 'model selection criteria'? $\endgroup$ Sep 12 '14 at 14:23
  • $\begingroup$ @Robert: Why do P(Y|X) have to be consistent? Do you mean comparable histograms? I'd think that the test set should be disjoint and independent from the train set and the train set should have a comparable histogram to the histogram of the entire dataset in order to validate the model... $\endgroup$
    – jpcgandre
    Sep 13 '14 at 2:04
  • $\begingroup$ For me the more dissimilar the train and test sets the better to test whether your trained model really fits the data behaviour.... but my question is what's the criteria to be happy with the test set? Variance? Should the test set be made exclusively of extreme values (if so how are we expecting the trained model to catch them?). Should the test and train sets have similar variance and stratification? If yes why do we care to bother splitting? They exhibit the same data behaviour. Many questions and very few answers. $\endgroup$
    – jpcgandre
    Sep 13 '14 at 2:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.