What is the optimal value of $k$, for an unweighted Euclidean kNN classifier applied to the Iris data set?
Where optimal implies the value for $k$ which leads to the lowest generalisation error.
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this communityWhat is the optimal value of $k$, for an unweighted Euclidean kNN classifier applied to the Iris data set?
Where optimal implies the value for $k$ which leads to the lowest generalisation error.
Lets say you want to use Accuracy (or % correct) to evaluate "optimal," and you have time to look at 25 values for k. The following R code will answer your question using 15 repeats of 10-fold cross-validation. It will also take a long time to run.
library(caret)
model <- train(
Species~.,
data=iris,
method='knn',
tuneGrid=expand.grid(.k=1:25),
metric='Accuracy',
trControl=trainControl(
method='repeatedcv',
number=10,
repeats=15))
model
plot(model)
> confusionMatrix(model)
Cross-Validated (10 fold, repeated 15 times) Confusion Matrix
(entries are percentages of table totals)
Reference
Prediction setosa versicolor virginica
setosa 33.3 0.0 0.0
versicolor 0.0 31.9 1.2
virginica 0.0 1.4 32.1
So, by this criteria, I get an answer of 17, but it looks like the "true" value could lie anywhere between 5 and 20. You can substitute "Kappa" or some other metric if you want, and add more cv-folds as well. You can also try different methods of cross validation, such as leave-one-out, or bootstrap re-sampling.
/Edit: in response for your request for variety, I wrote this function to calculate a variety of metrics for multi-class problems:
#Multi-Class Summary Function
#Based on caret:::twoClassSummary
require(compiler)
multiClassSummary <- cmpfun(function (data, lev = NULL, model = NULL){
#Load Libraries
require(Metrics)
require(caret)
#Check data
if (!all(levels(data[, "pred"]) == levels(data[, "obs"])))
stop("levels of observed and predicted data do not match")
#Calculate custom one-vs-all stats for each class
prob_stats <- lapply(levels(data[, "pred"]), function(class){
#Grab one-vs-all data for the class
pred <- ifelse(data[, "pred"] == class, 1, 0)
obs <- ifelse(data[, "obs"] == class, 1, 0)
prob <- data[,class]
#Calculate one-vs-all AUC and logLoss and return
cap_prob <- pmin(pmax(prob, .000001), .999999)
prob_stats <- c(auc(obs, prob), logLoss(obs, cap_prob))
names(prob_stats) <- c('ROC', 'logLoss')
return(prob_stats)
})
prob_stats <- do.call(rbind, prob_stats)
rownames(prob_stats) <- paste('Class:', levels(data[, "pred"]))
#Calculate confusion matrix-based statistics
CM <- confusionMatrix(data[, "pred"], data[, "obs"])
#Aggregate and average class-wise stats
#Todo: add weights
class_stats <- cbind(CM$byClass, prob_stats)
class_stats <- colMeans(class_stats)
#Aggregate overall stats
overall_stats <- c(CM$overall)
#Combine overall with class-wise stats and remove some stats we don't want
stats <- c(overall_stats, class_stats)
stats <- stats[! names(stats) %in% c('AccuracyNull', 'Prevalence', 'Detection Prevalence')]
#Clean names and return
names(stats) <- gsub('[[:blank:]]+', '_', names(stats))
return(stats)
})
It's a doozy of a function, so it's going to slow down caret a bit, but I'd be very happy if you posted the results of your 1000 repeats of 10-fold CV (I have neither the time not the computational capacity to attempt this at present). Here's my code for 15 repeats of 10-fold CV. Note that you can easily modify this code to try other re-sampling methods, such as bootstrap sampling:
library(caret)
set.seed(19556)
model <- train(
Species~.,
data=iris,
method='knn',
tuneGrid=expand.grid(.k=1:30),
metric='Accuracy',
trControl=trainControl(
method='repeatedcv',
number=10,
repeats=15,
classProbs=TRUE,
summaryFunction=multiClassSummary))
Both ROC and LogLoss seem to peak around 8:
While sensitivity and specificity seem to peak around 15:
Here's some code to output all the plots as a pdf:
dev.off()
pdf('plots.pdf')
for(stat in c('Accuracy', 'Kappa', 'AccuracyLower', 'AccuracyUpper', 'AccuracyPValue',
'Sensitivity', 'Specificity', 'Pos_Pred_Value',
'Neg_Pred_Value', 'Detection_Rate', 'ROC', 'logLoss')) {
print(plot(model, metric=stat))
}
dev.off()
If you put a gun to my head, I'd probably say 8...
summaryFunction,
perhaps starting with the average of AUC, Sensitivity, Specificity, Precision, and Recall for each of the three classes. That will get you to a sample size of 35...
$\endgroup$
– Zach
Jul 6 '12 at 14:31
caret
in R.
$\endgroup$
– Shea Parkes
Jul 6 '12 at 14:32