# Nested Cross validation with two settings on KNN

I am trying to perform model selection and evaluation using a 5-fold (internal) CV for the iris data. The things that I performed so far.

1. Partitioned iris into 50% Train, 25% Test, 25% validation.
2. I used knn to determine which is optimal from a set of k values (1,3,5) by Grid search.
3. Based on the winner model (obtained with k=1) I want to predict the performance on its test set with a 25% split.

I thought of 2 options:

• Plan A: To select a set of features internally, inside each of the K-1 folds, but leaving out instances from the validation fold.

• Plan B: To select features globally (unfairly, on the whole, data set!) and then build classifiers using these features in each of the folds.

I am aware of Plan B by using feature selection techniques like Mutual Information filter and Chi-squared filter but, not sure how to proceed with this. I would also like to know which of these would be a better estimate given the feature space of iris.

Code:

library(caret)
library(class)

spec = c(train = .5, test = .25, validate = .25)

byparts = sample(cut(
seq(nrow(iris)),
nrow(iris)*cumsum(c(0,spec)),
labels = names(spec)
))

res = split(iris, byparts)

classifier = train(form = Species ~ ., data = res$train, method = 'knn', tuneGrid = expand.grid(k = c(1,3,5))) classifier #5-fold CV trControl <- trainControl(method = "cv", number = 5) fit <- train(Species ~ ., method = "knn", tuneGrid = expand.grid(k = c(1,3,5)), trControl = trControl, metric = "Accuracy", data = res$validate)