Why do we not take the average of the regularization hyperparameters chosen by hold-out/cross validation?

I will talk restrict to hold-out for simplicity but my question applies to cross validation too.

Say we have a regularization hyper-parameters we are looking for $\lambda$. We choose it by training our model with some specific lambda and then testing how that model does in the validation/hold-out set. Then we choose the lambda that has the smallest error on the validation set. e.g. pseudocode:

validation_errors = []
for lambda in lambdas:
model = ERM(lambda) # say argmin_w ||Xw-y||^2_2 + lambda ||w||^2_2
val_error = Validation_error(model)
validation_errors.append(val_error)
report smallest val_error in validation_errors

I was asked why we don't just report $\lambda$ according the average instead of the one that performed best. From my perspective it should be obvious that unless we knew our parameter search for the regularization hyper-param was convex taking the average makes no sense. Otherwise if it was convex it makes sense to me that we should just search for it with gradient descent on the validation set instead of brute forcing by checking every bin (which is what is usually done, or in deep learning they do random search as suggest by Begnio et al). If makes it easier or interesting to provide an example I suggest to think of $||Xw-y||^2_2 + \lambda ||w||^2_2$ L2 regularization on linear regression (what statisticians call Ridge Regression I believe).

Anyway, for the stadard textbook, why wouldn't we just report just the average lambda? Is it just because that doesn't make sense since the parameter search might be non-convex (wrt to the validation error of course)? Or is there another reason? (another thing to note is that its obvious that the average of something sampled from bins would be the middle, which of course seems silly to choose since the algorithm would be choose the middle one).

Furthermore, it doesn't make any sense to report the average validation error as there is no reason that the "average validation error" means anything interesting (again because of the non convexity and because the algorithm isn't even random so there is no expectation to actually do...nor is there any noise to zero out in the validation set, though this last point is unclear to me how to make rigorous/precise).

• I'm not sure what you mean. The average $\lambda$ is known ahead of time, because you've provided a list lambdas to your loop. But the average $\lambda$ has no informative value because it's not aware of which values $\lambda_i$ did better than the others. – Sycorax Jul 6 '18 at 19:19
• @Sycorax thats what I pointed out in my question that the average lambda is known apriori. Maybe if I ask you this it will make more sense to you. Why do we not search for $\lambda$ with gradient descent on the validation set instead of brute force guessing an interval? I assume we are guessing across an interval because with a fixed $\lambda$ because based on that $\lambda$ ERM choose some model and that dependence is not simple (probably non convex). For me an average parameter only makes sense if the landscape we are considering (including the hyperparam) is convex. – Pinocchio Jul 6 '18 at 19:45
• Convexity doesn't have anything to do with it. The simple average is just as uninformative about the relative merits of each $\lambda_i$ if the surface is convex or non-convex because in neither case is the model performance considered. The only way I can make heads or tails of your question is if we introduce Bayesian inference to the problem: if we have a probability model over the hyperparameters, then Bayesian machinery gives us an average that takes model performance into account directly as part of the procedure. cf the link between MAP Bayes and $L^2$ regularization. – Sycorax Jul 6 '18 at 19:49
• @Sycorax I'll take a look at it thanks! I'm surprised that any method exists that would suggest averaging hyper-params or anything like that since obviously, the landscape $[W,\lambda]$ is non-convex especially since one picks $W$ first as a function of $\lambda$ so we have $W_{\lambda}$. So any type of averaging seems ridiculous/wrong to me seen there is no guarantee it won't throw you in between the two minima's (some higher part of the curve with higher expected error). – Pinocchio Jul 6 '18 at 23:03
• merriam-webster.com/dictionary/beset – Sycorax Jul 9 '18 at 15:57