Choose $λ=λ^∗$ to minimize the average OOS mean square error.
This strategy assumes you have enough independent test cases so the error on your OOS estimate is negligible.
You are right: if the error on the OOS measurements is not negligible, this can cause a bias towards too complex models. The reason is that if you compare
- many models of varying complexity
- that have essentially the same performance (i.e. you cannot distinguish their performance with the given validation set-up, particularly the given total no. of test cases),
- with a performance measurement that is subject to substantial variance,
you may "skim" the variance: the best observed performance may be caused by an (accidentally) favorable split of training and test sets rather than actually better generalization performance of the model.
Cawley, G. C. & Talbot, N. L. C.: On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation, Journal of Machine Learning Research, 11, 2079-2107 (2010).
The next weaker assumption is that there is some non-negligible error on the OOS estimate, but essentially the individual OOS measurements (for each surrogate model) still behave independently of each other:
Choose the largest $λ$ that is within one standard error (taken over all cross validation sets) of the $λ$ that minimizes the average OOS mean square error.
Otherwise, you need to take into account that you actually have only slightly varying models (only few training cases are exchanged between any two of the surrogate models) and only a finite number of distinct test cases. This means that the usual standard error calculation would overestimate the effective number measurements ($n$) and thus underestimate the standard error.
In consequence, in this situation you should select an even less complex model.