1
$\begingroup$

Let's say I want to tune hyperparameter lambda in a Lasso regressor. I have a train and validation set for hyperparameter tuning using 10-fold CV. Can the MSE calculated on the 10% validation fold each time be considered an out-of-sample MSE, or is it technically more an "estimate" of the out-of-sample MSE?

$\endgroup$
1
  • 1
    $\begingroup$ Everything is an estimate. The procedure you describe sounds a lot like "nested cross validation". $\endgroup$ Dec 5, 2020 at 12:07

1 Answer 1

1
$\begingroup$

As said by Cagdas Ozgenc, what you obtain with CV is always an estimate. From your explanation It seems me that you obtain something like $MSE(\lambda)$ and at the end you have $MSE(\lambda_{best}) < MSE(\lambda)$, where $\lambda_{best}$ is the tuned parameter you looked for. However it seems me that $MSE(\lambda_{best})$ represent a too optimistic estimate for test MSE, especially if you try with many value for $\lambda$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.