Alternatives to 1SE Rule for Validation Set Parameter Tuning

Question

I have a general question regarding parameter tuning on a held-out validation set (read: NOT cross validation, but a single held-out set of data).

Suppose I would like to tune a parameter in a machine learning model using a held-out validation set. To do so, I evaluate each feasible parameter value on the validation set and choose the parameter value with the lowest error. However, in the spirit of the 1SE rule in machine learning, suppose that I would like to choose the most conservative parameter value within an interval of values that perform decently well on the validation set. I can only find examples of the 1SE rule used within the context of cross-validation, in which the standard error used in the rule is computed over the K different CV folds. Is there any precedent in the machine learning literature or in practice for how to apply a rule similar in spirit to the 1SE rule when selecting parameters tuned from a single validation set?

Some context for my question: I am training a CART decision tree and am pruning the tree (i.e., evaluating which subtree is best) using a held-out validation set. I am not using cross-validation to tune the tree's complexity parameter for two reasons: (1) The tree and data are both massive, and the training procedure can take several days. So I would prefer to not train the tree K different times. (2) I do not believe there is evidence to suggest that the tree's complexity parameter found through CV would work well when applied to training the tree on the full training set. This is because the tree might have different splits when trained on the full dataset, and since the tree is structurally different it should have a different optimal pruning parameter.

Answer 1

I do not know such a rule for a single validation set, but you might consider additional measure like r2 score or the variation/standard deviation of you predictions in general to select a robust model.

And about point (2): I think it is common to tune parameters for tree-based models using CV. If you are worried about complexity parameters like the minimum number of samples in a leaf, you can still define them as ratios in proportion to the train set size.