When measuring predictive performance of a regression model, I am thinking about using repeated data splitting (or leave-out data at random) instead of using bootstrapping.

By "repeated data splitting" (not sure if that's even term) I understand:

FOR N=10000
Sample 70% of the data points and fit the model
Predict for the remaining 30%
Store performance
Calculate quantiles from stored performance data

I am currently using a GLM in order to rank my data according to a ratio scaled response variable Y (which my model is supposed to predict). Later I am evaluating the quality of the ranking using a specific metric.

I would like to calculate the prediction (or at least) confidence intervals for this metric -- that's the thing I care about (and not, say, the variance of the $\beta_3$ estimate)

Would the repeated data-splitting provide useful intervals for the performance of my ranking? i.e. Can I consider them as prediction intervals?

I would argue that this approach makes more sense than non-parametric bootstrap, as long as the data set is large enough, since we are using only actual data (that really occurred at one point in history, as opposed to a bootstrapped point the quality of which completely depends on the quality of the empirical approximation of the underlying distribution).

Since my data set is fairly large ($>20,000$) but also extremely skewed and long tailed, I would prefer using the data-splitting but I am not sure if this is a plain heuristic, or whether the intervals are actual prediction intervals.

  • $\begingroup$ The actual estimators and models are not important to your metric, just the performance of the model based on 1000 random subsets? $\endgroup$
    – Drew75
    Oct 25, 2013 at 9:36
  • $\begingroup$ Correct. Of course the number of subsets should be chosen large enough, s.t. the bands will show convergence. $\endgroup$ Oct 25, 2013 at 10:10


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy