R Caret train / rfe optimize for positive predictive value instead of Accuracy or Kappa In train or rfe I can only set Accuracy or Kappa. Is there a way to edit the functions to define a scoring function? I am using Kappa at the moment but I need to optimize for positive predictive Value (= hit rate = fraction of positives recognized as positive). Of course it is not that simple as 16/20 correct is better than 8/10 correct. So I would say (#hits * ppv)
# for 16/20: 16* 0.8 = score of 12.8
# for 8/10: 8* 0.8 = score of 6.4

ctrl <- rfeControl(functions = rfFuncs,method = "repeatedcv",number = 10,repeats = 4, verbose = TRUE)

lmProfile <- rfe(x = dtrain[,predictors], y = dtrain[,target],
                     newdata = dtest,
                     sizes = c(1:15,20,25),
                     metric="Kappa",
                     ntree = 100,
                     rfeControl = ctrl)

 A: A custom summary function and metric can be supplied to caret's train() and trainControl() to optimize by a metric not included in the default. 
Caret includes an alternative summaryFunction, twoClassSummary(). However, it doesn't include positive predictive value (aka precision). You can copy and modify twoClassSummary() to also include the precision metric, using caret's posPredValue() function. In the below example, all I did was append the precision calculation to the out object.
twoClassSummaryCustom = function (data, lev = NULL, model = NULL) 
{
  lvls <- levels(data$obs)
  if (length(lvls) > 2)
    stop(paste("Your outcome has", length(lvls), "levels. The twoClassSummary() function isn't appropriate."))
  if (!all(levels(data[, "pred"]) == lvls)) 
    stop("levels of observed and predicted data do not match")
  rocAUC <- ModelMetrics::auc(ifelse(data$obs == lev[2], 0, 
                                     1), data[, lvls[1]])
  out <- c(rocAUC,
           sensitivity(data[, "pred"], data[, "obs"], lev[1]),
           specificity(data[, "pred"], data[, "obs"], lev[2]),
           posPredValue(data[, "pred"], data[, "obs"], lev[1]))
  names(out) <- c("ROC", "Sens", "Spec", "Prec")
  out
}

Then, supply this function to trainControl() and specify the metric name in train(). Note that you'll also have to set classProbs = TRUE. See here:
trctrl    = trainControl(method = "repeatedcv", number = 10, repeats = 1, summaryFunction = twoClassSummaryCustom, classProbs = T)
dtree_fit = train(target ~ ., data = dtrain, method = "rpart",
                  metric = "Prec", # Specifying the custom metric here
                  trControl=trctrl,
                  tuneLength = 10)

