0
$\begingroup$

I am curious whether we have to use the same fitting method in model selection and prediction.

For example, suppose that we are going to use the logistic regression in prediction. Then, we may select a subset of features using the logit's maximum log-likelihood and cross-validation.

In this context, if we also want to use five-nearest neighbor, then do we have to change the model selection method or just use the same selected features above?

So, in a sentence, my question is that "should we use the same fitting method in model selection and prediction?"

$\endgroup$

1 Answer 1

1
$\begingroup$

The model selection process compares different models - in fact it compares different fitting methods, as the models are compared based on the fits generated by the methods. Of those it selects the best one, according to a hopefully suitable criterion, as determined by the data (there are many issues with model selection, but they are probably not very relevant here).

If you then use the same fitting method for prediction, this is the one that has won the "competition" of methods in the model selection process and can as such be thought of as "good" or "optimal" (obviously relative to what was said above). This does not hold for a different fitting method, and for that reason the model selection process doesn't give you information about any other method; using another one would basically amount to ignoring the model selection process.

Now you seem to think of using the variables as found by the model selection process with another method. But you need to realise that the model selection process in fact compares the models as fitted by specific methods. It doesn't give you any indication about what happens with a different fitting method, for which indeed different variables could be optimal.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.