-1
$\begingroup$

I am very new to ML therefore my question might be primitive. I am working on a binary-class problem. The response (target) variable is occurrence : a factor variable with two levels : oui and non (french equivalents of yes and no). The trainset has 175 rows (of which 173 rows with non and 2 rows with oui target values) and looks like :

occurrence  QMAX    deve    tm  P15M  P30M  P1H  P4H  P6H  P12H  P24H   P48H
non         35.6    72.0    34.3 5.5 10.2  12.9 17.3  18.4 18.7  18.7   18.7
non        238.9   143.3    49.5 3.9 6.6   11.5 20.7  28.5 42.3  48.9   65.6
non         23.5    72.0    39.3 6.6 8.8   12.0 17.6  17.6 25.4  26.5   28.4
....
oui        2396.5   72.0   34.1  28.5 47.7 68.6 112.1 112.3 125.8 125.9 126.0

the testset has exactly the same structure but of 19 rows of which 18 rows with non and 1 rows with oui target values). I use rf (random forest) of caret package to predict testset based on trainset :

train.control <- trainControl(method = "none",
                              search = "grid",
                              classProbs = TRUE)
expandgrid  <- expand.grid(.mtry = 4)
model.train <- train(occurrence ~ .,
                     data = trainset, 
                     method = "rf", 
                     trControl = train.control,
                     tuneGrid = expandgrid)
predict.class <- predict(model.train, testset)

I need to get classe probabilities for each row in my trainset, so I do :

classprobs <- predict(model.train, testset, type = "prob")

What I get is as follows :

non oui
1   0
1   0
1   0
1   0
1   0
1   0
1   0
1   0
1   0
1   0
1   0
1   0
1   0
1   0
1   0
1   0
1   0
1   0
1   0

as probability, it gives either 1 or 0 values and no values in between. So, when I construct roc object using pROC package :

ROC <- roc(testset$occurrence, classprobs$oui, levels = c("oui", "non"), auc = TRUE) 
thresh <- coords(ROC, x = "best", best.method = "closest.topleft", ret = "threshold", transpose = FALSE)

the returned AUC value and best threshold are :

ROC$auc
Area under the curve: 0.5

thresh
[1] -Inf  Inf

When I use other data sets such as iris (after subsetting it to a data.frame with binary versicolor and virginica response variables) and use the same script, reported probabilities values are not exclusively 0 and 1, i.e. there are probability values such as 0.04, 0.96 and the corresponding roc has a non-Inf best.threshold value.

I would be thankful if you could help me find what is wrong with my case.

$\endgroup$
2
$\begingroup$

This is a very imbalanced dataset. As you have stated:

The trainset has 175 rows (of which 173 rows with non and 2 rows with oui target values)

When you run any ML model in caret, by default it is trained to improve accuracy, which means getting the labels correct in your training set. When most of your data set is "non", the model simply needs to assign everything to "non" and its accuracy will be 173/175.

When this happens, the AUC etc simply doesn't make sense anymore, because the other label is missing.

You need to have more of the other label before you go on...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.