Deriving confidence intervals from a LOOCV of a GAM Say that I have some observations, $y_1, y_2, ...y_n$ that are described using a generalized additive model (GAM). A Leave-One-Out Cross-Validation (LOOCV) is then performed where each observation $y_i$ is removed and then the GAM is refit to the remaining observations and the omitted observation, $\hat{y}_i$, is predicted. This gives errors $\epsilon_i=(\hat{y}_i - y_i)$ that can be used to calculate useful metrics of my model's predictive strength. For example, I could calculate mean absolute error (MAE):
$\text{MAE} = \frac{1}{n}\sum_{i=1}^n|\epsilon_i|$
So, that's cool, but rather than a point estimate of the error like this I'm curious if I'm able to calculate a confidence interval for the error based on these calculated errors? If so, is this also possible for small data sets (e.g., $n<20$)?
 A: Let's assume for the sake of the argument that your sample points $\epsilon_i$ are i.i.d. You cannot assume they are normally distributed unless you have a good reason to. $n=20$ would be too few to assume that their sample mean is normally distributed. This combination leaves you already in a tough spot to compute your sample mean and standard error which you need for a confidence interval.
Furthermore, your sample points are actually not i.i.d., the cross validation introduces pseudo replication. Since you did LOOCV and thus without repetition, the pseudo replication comes only from the fact that each sample point is predicted through a model that was trained on almost the same training set. (Your "test-sets" which are individual records do not overlap.) There are corrected resampled tests which correct for cross validation induced pseudo replication. They make your standard error larger and thus your confidence interval wider.
The usual standard error would be:
$$\sigma/\sqrt{n}$$
The corrected resampled one is:
$$\sigma\times\sqrt{\frac{1}{n}+\frac{1}{1-n}}$$
Your confidence intervals most likely will be so wide as to be useless in this situation.
If you had slightly bigger data-sets $n>30$, you could do a confidence interval since you wouldn't need to assume normality anymore. If you have one like this, just try and see how large your confidence interval is with the resampled correction applied. Otherwise stay with the type of measures you mentioned.
