What is the optimal $k$ for the $k$ nearest neighbour classifier on the Iris dataset? What is the optimal value of $k$, for an unweighted Euclidean  kNN classifier applied to the Iris data set?
Where optimal implies the value for $k$ which leads to the lowest generalisation error.
 A: Lets say you want to use Accuracy (or % correct) to evaluate "optimal," and you have time to look at 25 values for k.  The following R code will answer your question using 15 repeats of 10-fold cross-validation.  It will also take a long time to run.
library(caret)
model <- train(
    Species~., 
    data=iris, 
    method='knn',
    tuneGrid=expand.grid(.k=1:25),
    metric='Accuracy',
    trControl=trainControl(
        method='repeatedcv', 
        number=10, 
        repeats=15))

model
plot(model)
> confusionMatrix(model)
Cross-Validated (10 fold, repeated 15 times) Confusion Matrix 

(entries are percentages of table totals)

            Reference
Prediction   setosa versicolor virginica
  setosa       33.3        0.0       0.0
  versicolor    0.0       31.9       1.2
  virginica     0.0        1.4      32.1


So, by this criteria, I get an answer of 17, but it looks like the "true" value could lie anywhere between 5 and 20.  You can substitute "Kappa" or some other metric if you want, and add more cv-folds as well.  You can also try different methods of cross validation, such as leave-one-out, or bootstrap re-sampling.
/Edit: in response for your request for variety, I wrote this function to calculate a variety of metrics for multi-class problems:
#Multi-Class Summary Function
#Based on caret:::twoClassSummary
require(compiler)
multiClassSummary <- cmpfun(function (data, lev = NULL, model = NULL){

  #Load Libraries
  require(Metrics)
  require(caret)

  #Check data
  if (!all(levels(data[, "pred"]) == levels(data[, "obs"]))) 
    stop("levels of observed and predicted data do not match")

  #Calculate custom one-vs-all stats for each class
  prob_stats <- lapply(levels(data[, "pred"]), function(class){

    #Grab one-vs-all data for the class
    pred <- ifelse(data[, "pred"] == class, 1, 0)
    obs  <- ifelse(data[,  "obs"] == class, 1, 0)
    prob <- data[,class]

    #Calculate one-vs-all AUC and logLoss and return
    cap_prob <- pmin(pmax(prob, .000001), .999999)
    prob_stats <- c(auc(obs, prob), logLoss(obs, cap_prob))
    names(prob_stats) <- c('ROC', 'logLoss')
    return(prob_stats) 
  })
  prob_stats <- do.call(rbind, prob_stats)
  rownames(prob_stats) <- paste('Class:', levels(data[, "pred"]))

  #Calculate confusion matrix-based statistics
  CM <- confusionMatrix(data[, "pred"], data[, "obs"])

  #Aggregate and average class-wise stats
  #Todo: add weights
  class_stats <- cbind(CM$byClass, prob_stats)
  class_stats <- colMeans(class_stats)

  #Aggregate overall stats
  overall_stats <- c(CM$overall)

  #Combine overall with class-wise stats and remove some stats we don't want 
  stats <- c(overall_stats, class_stats)
  stats <- stats[! names(stats) %in% c('AccuracyNull', 'Prevalence', 'Detection Prevalence')]

  #Clean names and return
  names(stats) <- gsub('[[:blank:]]+', '_', names(stats))
  return(stats)
})

It's a doozy of a function, so it's going to slow down caret a bit, but I'd be very happy if you posted the results of your 1000 repeats of 10-fold CV (I have neither the time not the computational capacity to attempt this at present).  Here's my code for 15 repeats of 10-fold CV.  Note that you can easily modify this code to try other re-sampling methods, such as bootstrap sampling:
library(caret)
set.seed(19556)
model <- train(
  Species~., 
  data=iris, 
  method='knn',
  tuneGrid=expand.grid(.k=1:30),
  metric='Accuracy',
  trControl=trainControl(
    method='repeatedcv', 
    number=10, 
    repeats=15,
    classProbs=TRUE,
    summaryFunction=multiClassSummary))

Both ROC and LogLoss seem to peak around 8:


While sensitivity and specificity seem to peak around 15:


Here's some code to output all the plots as a pdf:
dev.off()
pdf('plots.pdf')
for(stat in c('Accuracy', 'Kappa', 'AccuracyLower', 'AccuracyUpper', 'AccuracyPValue', 
              'Sensitivity', 'Specificity', 'Pos_Pred_Value', 
              'Neg_Pred_Value', 'Detection_Rate', 'ROC', 'logLoss')) {

  print(plot(model, metric=stat))
}
dev.off()

If you put a gun to my head, I'd probably say 8...
