Should I cross-validate metrics that were not optimised?

Question

I want to compare two models. Say I have two objective functions on the same data $f(X,y,\theta)$ and $g(X,y,\theta)$ that both evaluate the models performance in ways that I am interested in ($\theta$ are the model parameters, $X$ and $y$ are my data).

Instead of using a multi-objective optmisation algorithm though, I only optimise parameters of both of my models with respect to $f(X,y)$ because I can tractably optimise this.

However, I also want to compare model performance using the second objective function $g$. Do I need to use validation to get an unbiased estimate of the performances of my models and/or be able to compare my models with this metric? Are there standard practices in these cases or work that has explored this?

Answer 1

It is a good idea to bootstrap or cross-validate (e.g., 100 repeats of 10-fold cross-validation) indexes that were not optimized. For example, I recommend optimizing on a gold standard such as log-likelihood, penalized log-likelihood, or in a Bayesian model log-likelihood + log-prior. You can report measures such as pseudo $R^2$ that are just transformations of the gold standard objective function, and in addition do resampling validation on helpful indexes such as the $c$-index (concordance probability = AUROC), Brier score, and most of all, the full calibration curve. I do validation of smooth nonparametric calibration curves by bootstrapping 99 predicted values when using a probability model, i.e., to validate the absolute accuracy of predicted probabilities of 0.01, 0.02, ..., 0.99. Likewise you can show overfitting-corrected estimates of Brier score, calibration slope, mean squared error, and many other quantities. Details are in my RMS book and course notes.

Answer 2

You will anyways need a validation (verification) of the performance of the optimized model. Regardless of the testing scheme you employ for this (resampling/[outer] cross validation/[outer] out-of-bootstrap, single train/test split, validation study), this is where you evaluate the performance for all parameters of interest, i.e. $f$ and $g$.

A slight exception are parameters that are not calculated from test cases but rather fromt the model itself (say, some measure of model complexity). These are of course calculated directly for the final model (in your case: final model for each of the algorithms). Nevertheless, I'd also calculate them for out-of-bootstrap or cross validation surrogate models in order to check whether they are stable and possibly different between surrogate models and final model.

---

In addition, it may be interesting/important to study how $g$ evolves alongside the $f$ opimization, so it may be worth while to compute $g$ also during the [inner] cross validation inside the optimization of $f$. (That is, if that computation is feasible).