I am trying to perform model selection and evaluation using a 5-fold (internal) CV for the
iris data. The things that I performed so far.
irisinto 50% Train, 25% Test, 25% validation.
- I used
knnto determine which is optimal from a set of k values (1,3,5) by Grid search.
- Based on the winner model (obtained with k=1) I want to predict the performance on its test set with a 25% split.
I thought of 2 options:
Plan A: To select a set of features internally, inside each of the K-1 folds, but leaving out instances from the validation fold.
Plan B: To select features globally (unfairly, on the whole, data set!) and then build classifiers using these features in each of the folds.
I am aware of
Plan B by using feature selection techniques like
Mutual Information filter and
Chi-squared filter but, not sure how to proceed with this. I would also like to know which of these would be a better estimate given the feature space of
library(caret) library(class) spec = c(train = .5, test = .25, validate = .25) byparts = sample(cut( seq(nrow(iris)), nrow(iris)*cumsum(c(0,spec)), labels = names(spec) )) res = split(iris, byparts) addmargins(prop.table(table(byparts))) classifier = train(form = Species ~ ., data = res$train, method = 'knn', tuneGrid = expand.grid(k = c(1,3,5))) classifier #5-fold CV trControl <- trainControl(method = "cv", number = 5) fit <- train(Species ~ ., method = "knn", tuneGrid = expand.grid(k = c(1,3,5)), trControl = trControl, metric = "Accuracy", data = res$validate) library(praznik) miScores(iris[,-5],iris$Species)