I'm analysing various factors to forecast the value of a dichotomic variable and in this context I'm testing many different models (Logistic Regression, DA,PLSDA, various random and non-random "screenings" of the variables). As a measure of efficiency, I'm using the percentage of correct forecasts on a mildly big out-of-sample dataset.
I noticed that (obviously) changing parameters in the models or the dimension of the data which I built those on (training set), I get different results. My human basic instinct tells me naturally to set these options in order to get the best estimates.
My question eventually is: Is it wrong to optimise a model not overfitting on the training set, but on the out-of-sample data?