2
$\begingroup$

I have a dataset that is a high dimensional imbalanced dataset. The dataset is a categorical data set and I applied label encoder to transfer categorical values into numerical values. the dataset is a tabular dataset. I also use the mean imputation method to impute missing values. I use the oversampling technique on the training set and got the prediction recall around 0.800 for logistic regression. I used other classifiers like Naive bayes, random forest but did not get such a high prediction accuracy.

I used weka software for data training.

My question is why I got good accuracy for logistic regression and not for other classifiers?

Thank you.

Improve this question
$\endgroup$
7
  • $\begingroup$ Are you measuring your performance on the training data or a separate test set? $\endgroup$
    – Dave
    Commented Oct 29, 2022 at 23:52
  • $\begingroup$ I am measuring my performance on test data. $\endgroup$
    – Encipher
    Commented Oct 30, 2022 at 0:55
  • $\begingroup$ How does the performance compare on the training data? $\endgroup$
    – Dave
    Commented Oct 30, 2022 at 0:57
  • 1
    $\begingroup$ "Unbalanced" data is not a problem, and oversampling will not address a non-problem. Recall suffers from the exact same problems as accuracy. I would recommend that you first rethink your evaluation measures and ideally go with probabilistic predictions, assessing these using proper scoring rules and possibly calibration diagrams. I do not think you can trust recall on an oversampled dataset to tell you whether one model is better than another one. $\endgroup$
    – Stephan Kolassa
    Commented Oct 30, 2022 at 7:42
  • 1
    $\begingroup$ @Encipher Did you look at the linked material? // Recall is iffy for resampled data because you are giving the model an incorrect prior probability, so the posterior probabilities and derived quantities (like recall) are affected. $\endgroup$
    – Dave
    Commented Oct 31, 2022 at 5:09

1 Answer 1

Reset to default
0
$\begingroup$

This is speculation, since your comments mention that you do not have access to the training performance, but with poor out-of-sample performance by fancy models, this sounds like a classic case of overfitting.

Yes, a model like a random forest allows much more flexibility in the modeling than a vanilla logistic regression, but this runs the risk of fitting the model to coincidences in your data (noise) instead of genuine trends (signal). In the extreme, think about playing connect-the-dots.

When you go and apply this overfitted model to new data, the predictions have little to do with the true trend, and your predictive accuracy is poor.

Balancing the ability to fit complicated trends while guarding against overfitting is really the key to doing good machine learning work.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Can you please elaborate what you tried to mean? What should be my step wise approach for predicting the performance of the models like logistic regression or random forest? $\endgroup$
    – Encipher
    Commented Oct 30, 2022 at 1:18
  • $\begingroup$ What do you mean? You already have the performance. $\endgroup$
    – Dave
    Commented Oct 30, 2022 at 1:20
  • $\begingroup$ I have doubts. To evaluate the performance of a model - 1st do I need to find out the performance on the training set? Then I need to supply the test set and find the performance on the test set? Now, if I use only training set for performance evaluation do I need use k-fold cross validation? Also, if I think on pandas data frame we split the dataset using train test splitting and then go for the model training. At that time, we predict the result on the test set. am I right in this concept? $\endgroup$
    – Encipher
    Commented Oct 30, 2022 at 1:30
  • $\begingroup$ Please post that as a separate question. $\endgroup$
    – Dave
    Commented Oct 30, 2022 at 1:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.