Is cross-validation was a better way to show the variance of the slope of a line than a Bootstrap or a Bayesian approach?

Question

Background

I am trying to estimate the slope of a line for a univariate linear regression analysis as well as uncertainty about that slope (95% Confidence Interval). Originally I used two approaches to accomplish this: (1) Bootstrapping, where the data were repeatedly resampled with replacement and a regression line was fit for each resampling, and (2) a Bayesian regression, where 100,000 credible regression lines were generated based on the data and vague prior assumptions about the data.

I felt good about these approaches to estimate the variance in the slope of the regression line for my data, but I was told that cross-validation was a better way to show variance. I was under the impression that cross-validation was a good way to assess how well a particular model will perform when given out of sample data, but wasn't aware of its usefulness in estimating the variance of a particular parameter within a model such as slope.

My questions

1. Is it true that cross-validation was a better way to show the variance of the slope of a line than a Bootstrap or a Bayesian approach?
2. Assuming that #1 is true, if the database only consists of 21 sample points then would there be a certain type of cross-validation that is more appropriate than others (e.g., leave-one-out cross-validation, 10-fold cross-validation, repeated k-fold cross-validation)?
3. Again assuming that #1 is true, what measures would be most important for quantifying the variance of slope using cross-validation? (e.g., some average of the parameter estimates of the various folds?)

There is another post that sort of addresses the idea of variance determined via cross-validation but it doesn't seem to provide a clear answer to these particular questions.

Appreciate any insights!

Answer 1

I was under the impression that cross-validation was a good way to assess how well a particular model will perform when given out of sample data, but wasn't aware of its usefulness in estimating the variance of a particular parameter within a model such as slope.

You are right that cross validation per se is for validation [verification]. If the calculations are used to characterize variation in the surrogate models, it is often called jackknifing (to be very precise, jackkifing refers to a leave one out procedure). Jackknifing is to (LOO) cross validation like bootstrapping the model coefficients is to out-of-bootstrap validation.

(The other post you link to is concerned with the variance on the prediction performance estimates one gets from a cross validation for validation/verification purposes - as opposed to the variance of the coefficients across the surrogate models. These are not the same, it is possible to have variance in model coefficients that cancels out for the predictions)

Cross validation/jackknifing is a slightly different way of showing variance than bootstrapping. Neither is per se better, they assess/simulate different situations:

• Jackknifing answers the question: "How much variation does exchanging a small portion of the training samples cause?"
• Bootstrapping simulates drawing an entirely new set of n cases, and allows to measure the variance associated with that.
• (Your Bayesian approach is yet another valid approach, and may be approaching the same variance either bootstrap or CV looks at by a different way.)

Which ones are relevant depends entirely on the task at hand.

---

if the database only consists of 21 sample points then would there be a certain type of cross-validation that is more appropriate than others (e.g., leave-one-out cross-validation, 10-fold cross-validation, repeated k-fold cross-validation)?

• Varying $k$ means varying the fraction of cases that is exchanged.
• Leave-one-out is exhausted after $n$ models (there are only those $n$ models with 1 case left out/exchanged against another case). Thus, repeated cross validation does make sense only for $k \neq n$. In which case it may allow you to get a more fine-grained picture of what is going on.

---

 what measures would be most important for quantifying the variance of slope using cross-validation? (e.g., some average of the parameter estimates of the various folds?)

• First of all, I'd never look at the slope (of a linear model) without also looking at the intercept. They are not/should not be independent!
• If you want to quantify variance of the slope, you should use the variance of the slopes you observe in the simulation experiment (what am I missing?)