4
$\begingroup$

I'm analysing various factors to forecast the value of a dichotomic variable and in this context I'm testing many different models (Logistic Regression, DA,PLSDA, various random and non-random "screenings" of the variables). As a measure of efficiency, I'm using the percentage of correct forecasts on a mildly big out-of-sample dataset.

I noticed that (obviously) changing parameters in the models or the dimension of the data which I built those on (training set), I get different results. My human basic instinct tells me naturally to set these options in order to get the best estimates.

My question eventually is: Is it wrong to optimise a model not overfitting on the training set, but on the out-of-sample data?

$\endgroup$
4
$\begingroup$

"Is it wrong to optimise a model not overfitting on the training set, but on the out-of-sample data?"

It is only "wrong" if you assume that the out-of-sample still gives an unbiased performance estimation. It is quite common (at least in machine learning) to divide the available data into a training, a validation and a test set. The models are trained on the training set, the choice of model (and/or model hyper-parameters etc) is made using the validation set and the performance of the chosen model evaluated on the test set. This is O.K. as the test set is not used to optimize the model in any way.

I was quite surprised when I found out just how susceptible models can be to this form of "over-fitting" in model selection/optimization and just how much this can give an optimistic bias to performance estimation if not done with care. I wrote a paper demonstrating how this issue arises in machine learning.

G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010. (www)

John Langford has a great blog post on "clever methods of over-fitting", which is well worth reading, the question here would be a case of "Parameter tweak overfitting". "Human-loop overfitting." is another form of over-fitting that we definitely need to be aware of!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.