0
$\begingroup$

I am attempting to compare a few methods for multi-class classification using caret: 'multinom' (logistic regression), 'nnet' (neural net), and 'svmPoly' and 'svmLinear' (two types of support vector machines). As expected, when I randomly generate data, 'multinom' and 'svmLinear' give me roughly chance 10-fold-validation accuracies (1/6 = .1666...). But 'nnet' and 'svmPoly' give above-chance cross-validation accuracies.

The code below generates random datasets 100 times, each time training a a 6-class classifier and collecting 10-fold cross-validation accuracies. I have set caret's train function to perform PCA (including centering and scaling) on the data, keeping only the top 36 of PCs (i.e., 10% of the number of observations/rows - a number low enough to keep the model full rank).

Also below is a series of histograms plotting the frequency of the model's cross-validation accuracies across 500 simulations for each of the 4 methods. Because the data are randomly generated, accuracies should form normal distributions around chance (i.e., 1/6 = .1666...). Only the multinom method does this, with mean accuracy = .1667. 'nnet' gives mean accuracy = .1923 (which is significantly higher than multinom: t(1577)=30.3, p<.0001); 'svmPoly' gives mean accuracy = (also higher than multinom, p<.0001); and 'svmLinear' gives mean accuracy =.1652 (not significantly different from multinom p=.23).

Needless to say, this shouldn't be possible. Can anyone help me understand what's going on? Thanks!!

### Packages
library('caret') # for machine learning functions
library('nnet')

### Hard code metadata
n.classes = 6 # 6 classes to decode
n.observations.per.class = 60 # number of rows per class
n.cols = 118 # number of features/columns
n.rows = n.observations.per.class * n.classes # total number of rows
n.pcs = floor(n.rows / 10) # keep this number of PCs
n.sims = 100 # number of simulations to run per method

### Classification methods to try:
methods = c('multinom','svmPoly','nnet','svmLinear')
### Initialize storage
accuracies <- data.frame(matrix(NA, nrow=n.sims, ncol=length(methods)))
colnames(accuracies) <- methods

### Classify
# 100 simulations (or whatever n.sims is set to)
for(i in 1:n.sims){
  for(method.loop in methods){
    # Create simulated data
    sim.data <- data.frame(matrix(data = rnorm(n.cols * n.rows),
                                  nrow = n.rows, ncol = n.cols))
    # Add Y labels (6 classes: A thru F)
    sim.data$Y <- rep(c("A","B","C","D","E","F"),
                      each = n.observations.per.class)
    
    # Classify
    current.classifier <- train(Y ~ .,
                                data = sim.data,
                                method = method.loop,
                                preProcess = "pca",
                                trControl = trainControl(preProcOptions = list(pcaComp = n.pcs),
                                                         method='cv', number=10,
                                                         allowParallel = FALSE),
                                verbose=FALSE)
    
    # Get accuracy
    best.result <- current.classifier$results[which(rownames(current.classifier$results) == rownames(current.classifier$bestTune)),]
accuracies[i, method.loop] <- best.result$Accuracy
    
    # Clean up workspace
    rm(sim.data, current.classifier, best.result)
  }
  # Status update
  message("Loop ",i," of ",n.sims," complete.")
}

# Print mean accuracies:
colMeans(accuracies)

# Statistically different?
t.test(accuracies$nnet, accuracies$multinom)

Histograms of cross-validation accuracies for each method; black dashed line = 1/6 (chance) and solid green line = mean accuracy over the 1000 simulations. Results of simulations

Edit: The same problem occurs for binary classification and without centering, scaling, and PCA. This means that the problem is not related to pre-processing or the fact that the classification problem is multi-class. (To create the image below I only minorly changed the code above: I created only 2 classes -- A and B, with 180 observations/rows each; I generated only 36 features/columns (rather than 118); and I got rid of the preprocessing options in the train() and trControl() functions.) I have also tried setting linout to TRUE for 'nnet' but the results are the same. Binary classification

$\endgroup$
5
  • 1
    $\begingroup$ This is a good example of how overfitting can happen. $\endgroup$
    – Sycorax
    Jul 3 '21 at 17:05
  • $\begingroup$ Try to predict some new data. That should show your predictions not to be better than chance (and perhaps far worse). $\endgroup$
    – Dave
    Jul 3 '21 at 20:15
  • $\begingroup$ @Dave so, as I understand it, this is the result of predicting new data. The "method='cv', number=10" bit tells the model to do 10-fold validation. From what the caret manual says, this subsets the data 10 times into 10 training sets (90% of the data) and 10 test sets (the complement 10%s). For each of the 10 train sets, it trains the model, and then predicts the data in the test set. The reported accuracy is the mean accuracy of the predictions from the 10 sets of test-set predictions. Am I understanding this wrong? $\endgroup$ Jul 3 '21 at 21:33
  • $\begingroup$ @Sycorax sorry, can you say more about how this could be caused by overfitting? overfitting should lead to lower accuracy in cross-validation, not higher, shouldn't it? and i think even that would require that there be structure in the data -- these data are just random numbers. $\endgroup$ Jul 3 '21 at 21:53
  • $\begingroup$ It looks like you're getting the accuracy specifically by picking the best result among the cross-validation tuning, which you would expect to be better than random, since it's the best one. I think you need to go back and apply that model with that parameterization to all the folds. I think you're showing that choosing the best result of a random classifier gives you above-random performance, rather than that the average performance of a random classifier is in fact random. $\endgroup$ Jul 6 '21 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.