0
$\begingroup$

Say we have some model, $f(x) = \hat{Y}$, such as linear regression, that estimates an output dependent value for some set of input data.

We want to do inference on the coefficients of the model to estimate features' influence on the dependent variable.

We also want to properly evaluate the predictive accuracy of the model.

Would a suitable work flow go as follows:

  • Model feature selection using measures such as AIC, BIC etc.. and regularization
  • Inference on coefficients
  • Finally, k-fold cross-validation to estimate predictive accuracy (such as MSE).

It seems to me we need to not do any cross-validation or parameter tuning until inference is complete. Otherwise, our p values will not be valid. Our predictive assessment must be made independently of our model selection.

Does this make sense?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ How do you plan to do things like regularization without hyperparameter tuning? $\endgroup$
    – Henry
    Sep 27 at 8:32
  • $\begingroup$ Feature selection very seldom works even under ideal conditions (large N, no collinearities, high signal:noise ratios). In other conditions it's a disaster. What makes you think feature selection works in your case? When I say works I mean that it has a remote chance of finding the right features. See fharrell.com/talk/stratos19. Contrast feature selection with ridge regression and data reduction (unsupervised learning to restrict the number of variables used to predict the outcome). $\endgroup$
    – Frank Harrell
    Sep 27 at 11:30

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.