Nested Cross validation with two settings on KNN

Question

I am trying to perform model selection and evaluation using a 5-fold (internal) CV for the iris data. The things that I performed so far.

1. Partitioned iris into 50% Train, 25% Test, 25% validation.
2. I used knn to determine which is optimal from a set of k values (1,3,5) by Grid search.
3. Based on the winner model (obtained with k=1) I want to predict the performance on its test set with a 25% split.

I thought of 2 options:

• Plan A: To select a set of features internally, inside each of the K-1 folds, but leaving out instances from the validation fold.

• Plan B: To select features globally (unfairly, on the whole, data set!) and then build classifiers using these features in each of the folds.

I am aware of Plan B by using feature selection techniques like Mutual Information filter and Chi-squared filter but, not sure how to proceed with this. I would also like to know which of these would be a better estimate given the feature space of iris.

Code:

library(caret)
library(class)

spec = c(train = .5, test = .25, validate = .25)

byparts = sample(cut(
  seq(nrow(iris)), 
  nrow(iris)*cumsum(c(0,spec)),
  labels = names(spec)
))

res = split(iris, byparts)

addmargins(prop.table(table(byparts)))

classifier = train(form = Species ~ ., data = res$train, method = 'knn', tuneGrid   = expand.grid(k = c(1,3,5)))
classifier

#5-fold CV
trControl <- trainControl(method  = "cv",
                          number  = 5)

fit <- train(Species ~ .,
             method     = "knn",
             tuneGrid   = expand.grid(k = c(1,3,5)),
             trControl  = trControl,
             metric     = "Accuracy",
             data       = res$validate)

library(praznik)
miScores(iris[,-5],iris$Species)