0
$\begingroup$

I am trying to classify a data set X of 2000 examples (rows) and 20 features (Columns) . The class labels are 0 or 1. Out of 2000 examples, 98 are labelled as 1 and the rest are 0.

I am using MATLAB 2018 version.

I want to apply cross-validation using HoldOut method and using 60/40 of the data X.

Question:
I am getting all predicted classes as 1 and accuracy of 98.1%. Eventhough the accuracy is so high, the predicted class labels are mostly incorrect.

Where am I going wrong? Is it because out of 2000 examples, only 98 are positive examples (labelled 1) and the remaining 0?

Is SVM not the proper choice for this kind of data?

$\endgroup$
1
$\begingroup$

The most important question here is: why does the data have this imbalance.

  • If the imbalance reflects the base rate* of your classes, you should go with the imbalanced data.
  • Resampling to rebalance the data should IMHO be avoided unless you positively know the proper base rates and resample to get there. But applications with equal base rates are very rare in my experience.
    So if you know the base rates should be equal but your data doesn't reflect this, go ahead and rebalance.
  • Training and validating on equal base rates will often yield a model with seemingly nice performance - which however often breaks down if evaluated for the real (imbalanced) base rates.
  • As you already noticed, plain acuracy may not be the best figure of merit for unbalanced classes, and
  • the naive model to compare your performance to is not 50:50 random guessing but "always predict 0" with 98 % accuracy.
  • have a look at your misclassification costs: is missing a positive worse than a false positive? and
  • adjust your loss function accordingly.

  • Consider only figures of merit that properly take into account class base rates (or are independent of class base rate).


  • Classifiers that take class imbalance "naturally" are obviously better suited. However, this is often a question of implementation than of the general capability of the classifier in question.
  • If you cannot "tell" your SVM implementation about base rates and misclassification costs, but it can return a continuous score (e.g. predicted class membership probability) you can request that score and then choose your working point (cutoff score) yourself. E.g. you can choose the cutoff so that the number of false positives = number of false negatives (times factor for respective costs).

Seeing that you have only 98 positive examples, consider resampling validation (stratified cross valiation or out-of-bootstrap) rather than hold out.


* base rate or prior probability: the fraction of cases truly belonging to the class in question in the relevant population (which is not necessarily the same in the sample at hand).

"Balanced situation" typically refers to all classes having equal prior probability (or base rate) of $\frac{1}{nclasses}$, so 50:50 in case of 2 classes.

$\endgroup$
5
  • $\begingroup$ Thank you for your answer. I have few questions, could you please clarify?(1) what is base rate? (2) under what conditions do we say that the data is balanced? Does 50-50 examples of positives and negatives suggest a balanced case? (3) I will always have more number of 0 labeled classes which denote normal operating conditions and 1 denote a rare event or a rare operating condition. Therefore, removing the 0's may not be a good idea as you have correctly pointed out. Thus, can you please suggest classifiers that take imbalance class naturally? $\endgroup$ – Srishti M Jun 6 '18 at 18:19
  • $\begingroup$ terms: please see updated answer. One classifier that is almost always implemented with an argument for prior probabilites is LDA. Also, you can often tweak a class-wise cost parameter to adapt for unequal priors or priors/base rates that differ from those of the training data. E.g. libsvm allows this. And via the scores you can get the required working point from any classifier that is able to output scores in the way I described. This is also possible with libsvm (have a look at the -b option). See also section 6 of the libsvm guide. $\endgroup$ – cbeleites unhappy with SX Jun 6 '18 at 18:44
  • $\begingroup$ Thank you for the updated answer. I am using Matlab 2018 and it does not have libsvm. Would it be possible for you if time permits to suggest how I can apply stratified cross-validation or bootstrap with SVM or MLP methods? I can post a new question on this if needed. $\endgroup$ – Srishti M Jun 6 '18 at 19:12
  • $\begingroup$ "Matlab 2018 [...] does not have libsvm" But libsvm includes a Matlab/Octave interface, which is plainly stated on their web site. "Would it be possible for you if time permits to suggest" yes, I could help you with your stratified cross validation code. But a) such a question would be better suited at e.g. codereview.sx (where it would need to show your proper effort) and b) I earn my living by such consulting and training, so to get me put in significant amounts of time to teach you how to do this, you'd need to contact me professionally. $\endgroup$ – cbeleites unhappy with SX Jun 7 '18 at 8:13
  • $\begingroup$ Note that we have lots of answered questions on stratified cross validation. $\endgroup$ – cbeleites unhappy with SX Jun 7 '18 at 8:18
1
$\begingroup$

If yours is an imbalanced dataset where your class proportion ( no. of minority class samples to no. of majority class samples) is very low as compared to how it actually should be .

For example , in case of the famous Iris Dataset where the classes should be balanced(No. of positive class samples ≈ No. of negative class samples) , but , say you have a version of the Iris dataset where there are 120 positive samples and just 10 negative samples , you can try out synthetic undersampling or synthetic oversampling techniques.

In undersampling , you can take a subset of the majority class whose size is comparable to that of minority class.

While, in oversampling, you can replicate the minority class to make its size comparable to that of majority class.

The undersampling and oversampling techniques will not be viable when dealing with datasets such as Fraud Detection Dataset, where the no. of minority samples are supposed to be very less as compared to that of no. of majority samples , i.e. , the class proportion matches the base rate.

$\endgroup$
4
  • $\begingroup$ @cbeleites I am sorry , but what do you mean by 'base rate'? $\endgroup$ – Faraz Gerrard Jamal Jun 5 '18 at 20:17
  • $\begingroup$ Base rate = prior probability to encounter a case of each of the classes. In other words, which fraction of the relevant population (not necessarily the sample!) truly belongs to each class. $\endgroup$ – cbeleites unhappy with SX Jun 6 '18 at 18:20
  • 1
    $\begingroup$ @cbeleites edited. $\endgroup$ – Faraz Gerrard Jamal Jun 7 '18 at 7:31
  • $\begingroup$ @FarazGerrardJamal: Thank you for your answer. My dataset is naturally supposed to be imbalanced quite similar to the fraud detection dataset. So, does it still fall under imbalanced category? Therefore, SVM and Neural Networks are giving very high accuracy but the predicted labels are all wrong. Can I apply boosting methods such as RUSBoost in Matlab which is a variant of Adaboost? thank you for your help again. $\endgroup$ – Srishti M Jun 7 '18 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.