Using bootstrap to get distribution of mean error on test set

Question

I am currently dividing my data into a 70/30 train/test split. I then perform a grid search on the 70% training data in order to find the optimal hyper-parameters for my model using 5-fold cross validation for each set of hyperparams.

After I have found the optimal model on the training data, I evaluate its performance on the 30% test data and calculate $\epsilon_{MAE}$, the mean absolute error.

Now, I am not very familiar with applied ML literature, but it seems odd to me to only report a single $\epsilon_{MAE}$ to summarize my model performance, without including confidence bounds. Some of the literature I have seen reports mean error with standard errors derived from the CV process, but that seems like it would produce optimistic errors.

Is it an accepted practice to use bootstrap resampling on my 30% test data and use my previous "optimal" model to get a distribution for $\epsilon_{MAE}$? Or is there a better/more accepted way to get standard errors for my prediction error.

Answer 1

 it seems odd to me to only report a single ϵMAE to summarize my model performance, without including confidence bounds.

To me (analytical chemist and chemometrician) that is odd as well: I keep asking for those confidence intervals.

 standard errors derived from the CV process ... would produce optimistic errors

There are lots of things that can go wrong, but AFAIK if done correctly (the splits being really independent, the aim of the CV being the actually fitted model on the whole data, and showing that the surrogate models are stable, and then taking into account only the numer of actually different cases, i.e. n does not increase for iterated/repeated CV) confidence intervals from the CV procedure should yield sensible estimates.
That is, as far as CV goes in simulating the real application. BUt the same applies to any kind of train/test splitting.

 use bootstrap resampling on my 30% test data and use my previous "optimal" model to get a distribution for ϵ$_{MAE}$?

This will yield a confidence interval that belongs to the model fitted on 70% of your data, and you'll not cover model (in)stability - so I'd be even more wary of extrapolating such results to a model fitted on the whole data than for CV-like procedures.

Note also, that for accuracy-like measures and situations where all test cases are independent of each other (no hierarchical data, no clustering, no repeated measurements) you have binomial distributions and thus know the variance from number of cases and probability. Thus you can construct confidence intervals for e.g. the accuracy if you know how many out of how many cases were predicted correctly. If you have clustered data, the situation is of course far more complicated, but then also test/train splitting and the bootstrapping would need to be set up according to this structure.