1
$\begingroup$

I'm studying the behavior of machine failures in a production scenario. For this, I generated random data to form my imbalanced training set, consisting of categorical data, which indicate whether or not there was a failure in each subperiod.

The failures were generated according to an exponential distribution. I have 24 features (Period_1 to Period_24), each containing information about the historical failures for 448 subperiods. Furthermore, I have three more features consisting of Temperature, Moisture, and Pressure (generated using the Normal distribution). My intention is to predict the behavior of the failures for the next period based on these features.

I used the ROC metric and considered several strategies to deal with unbalanced data, such as oversampling, undersampling, ROSE, and ADASYN. Furthermore, I tried to use the ensemble to improve performance. I tested all the following models: gradient boosting algorithm, random forest, Classification and Regression Trees, neural networks, Bagged CART, SVM, C5.0, eXtreme Gradient Boosting, and k-Nearest Neighbors. I also tried to use regularized models but none of these strategies worked. The best result obtained was using the model "SVMRadial" considering resampling with the ROSE package. In this case, ROC = 0.7614, Sensitivity = 0.7639, and Specificity = 0.6065 for the training set and Sensitivity = 0.75, and Specificity = 0.6914 for the test set (the latter obtained through the Confusion Matrix). However, when making predictions, the trained model is resulting in high probabilities for wrong predictions. So, I would like to know if this is a problem of the training model. Also, would anyone have any idea how to improve these results?

Any help will be appreciated.

A sample of the data:

enter image description here

The code for the chosen model:

set.seed(123);
    
partition <- createDataPartition(data_failures$Period_24, p = 0.8, list = F)
trainingSet <- data_failures[partition,]
testingSet <- data_failures[-partition,]
    
train.control <- trainControl(method = "repeatedcv", number = 10, repeats = 3, sampling = "rose", classProbs = TRUE, summaryFunction = twoClassSummary)
    
model_24 <- train(Period_24 ~., data = trainingSet, method = "svmRadial", preProc = "zv", metric = "ROC", trControl = train.control)
    
print(model_24)
    
predictions <- predict(model_24, newdata = testingSet)
print(confusionMatrix(predictions, testingSet$Period_24))

I'm using R with caret package.

$\endgroup$
2
  • $\begingroup$ Do you have a situation like $100$ $1$s for every $0$, and there are $0$ predicted with high probability (say $>0.5$) of being $1$? $\endgroup$
    – Dave
    Commented Aug 12, 2020 at 18:14
  • $\begingroup$ @Dave The model is predicting more periods as "Normal" (0) than "Failure" (1), as expected. However, there are 0 predicted with high probability of being 1. $\endgroup$
    – Fernanda
    Commented Aug 12, 2020 at 18:58

1 Answer 1

0
$\begingroup$

From the comments:

However, there are 0 predicted with high probability of being 1.

This should be the case. If you have an event that occurs with probability $0.95$, then roughly one out of twenty attempts should result in the event not occurring. If you have a large amount of data, that could translate into a large number of these bad predictions.

You can check this by assesing your model calibration. You use R, and the rms package has two functions of interest.

rms::calibrate is the technically correct function to use. However, it only accepts models made within the rms functions, so you cannot apply this function to your models. rms::val.prob, however, just takes true values and predicted probabilities as inputs, which you will have from your model outputs. This function plots the predicted probabilities and the (estimated) true event probabilities. While this is technically incorrect and rms::calibrate should be preferred, it seems that using val.prob will do little damage in most cases.

In the simulation below, I show that events with low probability can happen once in a while. The number of event $0$ instances with predicted probability $P(y=1) = 0.8$ is about $2\%$. Across many thousands of observations, this could be a fair number of values that you observe exhibiting this behavior. However, the probability validation shows that the predicted probabilities align with true event occurrence probabilities.

library(rms)
set.seed(2023)
N <- 100000
p <- rbeta(N, 1, 1)
y <- rbinom(N, 1, p)
rms::val.prob(p, y)
length(p[y == 0 & p > 0.8])/length(p)*100 # I get 1.994%

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.