Logistic regression as classifier and overfitting

Question

I am using logistic regression to classify data into two classes. The variable to predict (Y) is either 0 or 1.

I have found a rather good estimation of Y by logistic regression, and ended up using a set of the variables as predictor variables (the rest were quite correlated, or did not decrease the error a lot when added). My chosen set of variables is the optimum when using AIC as an optimization criteria.

Anyway, to the point: I have never used regression functions as classifiers before, and am not sure how to test this classifier in a good way. My professor suggested crossvalidation, but I don't understand why crossvalidation is a good choice is this case. I have a fairly large data set (196).

If my regression function is shown to be the best possible (given some defined optimum, I use AIC as criteria), why should I have to even test it?

Could my classifier be overfitted, and should I therefore leave out some data to only use as testing/validation in the end?

I am a bit new to this field.

Thanks in advance for all help.

(I have 196 feature vectors, and 22 features for this specific case, where I use 12 of them in my prediction of Y).

Answer 1

My chosen set of variables is the optimum when using AIC as an optimization criteria.

This is a data-driven optimization which always comes at a risk of overfitting (even if the model complexity is penalized).

Thus you need to measure the performance of your finally chosen model. Perfomance for unknown cases needs unknown test cases for the measurement, which can be achieved with a properly set-up cross validation.

I recommend to browse through the cross validation questions here to get an overview of what is possible, important, and recommended for cross validation.