I need an advice which portion of a dataset should be used to calculate cuts for discretization. I use two levels of Cross Validation. One is external to the model creating, but the second is used internally by the model learning algorithm.
Current test procedure is as follows:
Use CV5 to split data onto train and test sets
- Calculate cuts on train set for continues attributes
- Pass discretized train data to the model learning procedure
- Apply cuts to test data set
- Classify test data using the model
In step 3. internally the learning procedure do as follows:
a. Take the data and split it using CV3
b. Train each model independently using each of the 3 data folds use the rest of the data to measure performance
c. select and return the best model
Is the above procedure correct or should discretization be done by model learning method internally?