I will talk restrict to hold-out for simplicity but my question applies to cross validation too.
Say we have a regularization hyper-parameters we are looking for $\lambda$. We choose it by training our model with some specific lambda and then testing how that model does in the validation/hold-out set. Then we choose the lambda that has the smallest error on the validation set. e.g. pseudocode:
validation_errors =  for lambda in lambdas: model = ERM(lambda) # say argmin_w ||Xw-y||^2_2 + lambda ||w||^2_2 val_error = Validation_error(model) validation_errors.append(val_error) report smallest val_error in validation_errors
I was asked why we don't just report $\lambda$ according the average instead of the one that performed best. From my perspective it should be obvious that unless we knew our parameter search for the regularization hyper-param was convex taking the average makes no sense. Otherwise if it was convex it makes sense to me that we should just search for it with gradient descent on the validation set instead of brute forcing by checking every bin (which is what is usually done, or in deep learning they do random search as suggest by Begnio et al). If makes it easier or interesting to provide an example I suggest to think of $||Xw-y||^2_2 + \lambda ||w||^2_2$ L2 regularization on linear regression (what statisticians call Ridge Regression I believe).
Anyway, for the stadard textbook, why wouldn't we just report just the average lambda? Is it just because that doesn't make sense since the parameter search might be non-convex (wrt to the validation error of course)? Or is there another reason? (another thing to note is that its obvious that the average of something sampled from bins would be the middle, which of course seems silly to choose since the algorithm would be choose the middle one).
Furthermore, it doesn't make any sense to report the average validation error as there is no reason that the "average validation error" means anything interesting (again because of the non convexity and because the algorithm isn't even random so there is no expectation to actually do...nor is there any noise to zero out in the validation set, though this last point is unclear to me how to make rigorous/precise).