I am constructing a 2-class classifier and using cross validation to tune certain parameters in my model.

The predictor variables are both continuous and one is ordinal. Based on looking at the response variable and the ordinal variable it seems like it might be a good idea to collapase/combine certain levels of the ordinal variable.

Am I allowed to make this decision based on the full data or will it mess up the cross validation procedure?

If I am not allowed to do so, would it then make sense to treat the value I am using as a threshold for collapsing the levels in the ordinal variable as a tuning parameter?


In my experience you can try any kind of feature transformation you want as long as you are consistent across all the subsets of your data (i.e. the test set and all the training subsets including holdout).

This can be done most easily by putting together the whole data set (into one DataFrame if you are using Python Pandas for example) and do the transformation on the whole set.

You can experiment with different ways how you collapse or combine the values of the ordinal feature by doing the transformation and measuring the classifier's performance in a cross validated run.

Maybe some collapsing helps the classifier by reducing complexity but too much collapsing takes away useful information.

Experiment with it, this is the most fun part of classification in my opinion.

Remember, the most important thing is consistency across training, holdout and test data when you are doing feature engineering.

  • $\begingroup$ Perhaps I should specify what I mean. I know that I can cross validate my way to an estimate of the "best" threshold value in the way that think you are discribing. But what I am asking is. Am I allowed to do any qualitative analysis to determine the threshold based on the entire data set or should I stick to cross validation? $\endgroup$ – Zink Jun 25 '16 at 21:09
  • $\begingroup$ Use the entire data set, otherwise you might end up with different threshold values for the different CV data subsets, which would not make sense. $\endgroup$ – Peter Csizsek Jun 26 '16 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.