0
$\begingroup$

I am implementing machine learning algorithm for multiclass classification problem in R programming. The problem is that when I predict the accuracy I am getting around 90% accuracy but when I calculate the AUC score I am getting more than 1. Will the AUC support multi-class classification?

Which is the better approach to consider the evaluation metric of machine learning model for multi class classification?

Below is my code

library(dplyr)
smp_size <- floor(0.80 * nrow(New_data))
set.seed(123)
train_ind <- sample(seq_len(nrow(New_data)), size = smp_size)
train <- New_data[train_ind, ]
test <- New_data[-train_ind, ]

train_features <- train[,-ncol(train)]
train_labels <- train[,ncol(train)]
test_features <- test[,-ncol(test)]
test_labels <- test[,ncol(test)]

library(e1071)
library(rpart)
library(mlbench)
svm.model <- svm(as.factor(train$labels) ~ ., data = train,cross = 10)
class(svm.model)
summary(svm.model)
print(svm.model)
svm_pred <- predict(svm.model,test_features)

length(svm_pred)
length(test_labels)
table(pred = svm_pred, true = t(test_labels))
conf_matrix <- table(svm_pred, test_labels)
conf_matrix 

library(MLmetrics)
Accuracy(svm_pred, test_labels)
AUC(svm_pred, test_labels)

Output:

> conf_matrix
        test_labels
svm_pred    0    1    2    3    4
       0  896   75   50   18   28
       1   71  919   28   13   16
       2    7    6  112    1    9
       3   31   44   12 1023    2
       4    1    5    9    0   55
> library(MLmetrics)
> Accuracy(svm_pred, test_labels)
[1] 0.8758379
> AUC(svm_pred, test_labels)
[1] 1.047074
$\endgroup$
  • $\begingroup$ Please refer to the documentation: cran.r-project.org/web/packages/MLmetrics/MLmetrics.pdf the AUC functions calculates AUC for binary problems, while you have multiple categories. It is a bug that the function does not throw an error. $\endgroup$ – Tim Sep 11 '17 at 19:02
  • $\begingroup$ @Tim How can this be off topic. If AUC has a bug I was looking for an alternative approach for considering the metric evaluation of my machine learning model which has a multi class problem. $\endgroup$ – vinaykva Sep 11 '17 at 19:07
  • $\begingroup$ Please check stats.stackexchange.com/help/on-topic Software or programming related questions are off-topic in here. $\endgroup$ – Tim Sep 11 '17 at 19:11
  • 2
    $\begingroup$ The question is still off-topic. The previous version was more relevant: you got AUC > 1 since you used your software incorrectly. This is not a statistical issue, but it is just about using your software. AUC can be implemented for multi-label problems, see stats.stackexchange.com/questions/21551/… $\endgroup$ – Tim Sep 11 '17 at 20:08
  • 1
    $\begingroup$ If you set aside the code related issues, is your question answered by How to plot ROC curves in multiclass classification? $\endgroup$ – gung - Reinstate Monica Sep 11 '17 at 20:24

Browse other questions tagged or ask your own question.