When measuring predictive performance of a regression model, I am thinking about using repeated data splitting (or leave-out data at random) instead of using bootstrapping.
By "repeated data splitting" (not sure if that's even term) I understand:
FOR N=10000
Sample 70% of the data points and fit the model
Predict for the remaining 30%
Store performance
END FOR
Calculate quantiles from stored performance data
I am currently using a GLM in order to rank my data according to a ratio scaled response variable Y (which my model is supposed to predict). Later I am evaluating the quality of the ranking using a specific metric.
I would like to calculate the prediction (or at least) confidence intervals for this metric -- that's the thing I care about (and not, say, the variance of the $\beta_3$ estimate)
Would the repeated data-splitting provide useful intervals for the performance of my ranking? i.e. Can I consider them as prediction intervals?
I would argue that this approach makes more sense than non-parametric bootstrap, as long as the data set is large enough, since we are using only actual data (that really occurred at one point in history, as opposed to a bootstrapped point the quality of which completely depends on the quality of the empirical approximation of the underlying distribution).
Since my data set is fairly large ($>20,000$) but also extremely skewed and long tailed, I would prefer using the data-splitting but I am not sure if this is a plain heuristic, or whether the intervals are actual prediction intervals.
