2
$\begingroup$

I have a data set of brain tumours, 700 malignant, and 225 benign. And I want to build a classification model using SVM, to classify the tumours types based on the data I have. My first question, is it considered an imbalanced dataset? if so, should I do undersampling of the malignant class?

Also, is it correct to use the below code to do cross-validation for my dataset? NOTE: groups = instances' labels vector (sorted malignant 0s then benign 1s) data = instances' data feature matrix

k=10;
cp = classperf(groups); 
cvFolds = crossvalind('Kfold', groups, k);   
for i = 1:k                                 
 testIdx = (cvFolds == i);                %# get indices of test instances
 trainIdx = ~testIdx;                     %# get indices training instances
 svmModel = fitcsvm(data(trainIdx,:), groups(trainIdx), 
'Standardize',true,'KernelFunction','RBF','KernelScale','auto');
 pred = predict(svmModel, meas(testIdx,:));
 cp = classperf(cp, pred, testIdx);

end

I still couldn't understand how crossvalind works? I mean does it guarantee that it takes instances from both classes at each fold?

Improve this question
$\endgroup$
6
  • $\begingroup$ No it is not consider imbalanced. No, there is little reason to under-sample your malignant class. $\endgroup$
    – usεr11852
    Commented Mar 20, 2019 at 16:18
  • $\begingroup$ so, no need to modify the cross-validation code above? I thought of separating the data into their classes, and in each fold, I take 90% train,10% test from first class, and from second class as well. Then combine them and randomize them then build the classification model. $\endgroup$
    – Atheer
    Commented Mar 20, 2019 at 18:01
  • $\begingroup$ You can stratify our sampling if you wish. I do not think it will make a huge difference but if might make it such that you have the same ratio across all folds. $\endgroup$
    – usεr11852
    Commented Mar 20, 2019 at 18:25
  • $\begingroup$ IMHO the more important question is not whether your data is balanced, but whether the relative frequencies of the classes are what you can expect to meet in the real world of the application? And what the relative costs of the various misclassifications are. $\endgroup$
    – cbeleites
    Commented Mar 21, 2019 at 0:45
  • $\begingroup$ You mean by relative frequencies and relative costs, which of the two classes are more significant and the prediction results of it affects more in the real world? $\endgroup$
    – Atheer
    Commented Mar 21, 2019 at 12:49

2 Answers 2

Reset to default
2
$\begingroup$

The fact that you are bringing up the issue of balance means that you have not considered the fact that proportion "classified" "correctly" is a discontinuous improper accuracy scoring rule. If you use a proper scoring rule (e.g., Brier score or pseudo $R^2$) the issue goes away. See this and this for more.

Improve this answer
$\endgroup$
0
$\begingroup$

My first question, is it considered an imbalanced dataset?

Yes - there are almost three times the number of malignant to benign so you can consider this to be unbalanced. Generally, an unbalanced dataset will result in a model biased towards the class with most data.

If so, should I do under-sampling of the malignant class?

There are different approaches each with advantages and disadvantages. The main problem with undersampling is that you can lose information from the samples left out. With oversampling you create additional samples from the benign class – lets say you randomly duplicate existing samples. However, the training and test data is no longer independent so the issue here is that you can end up overfitting the model and all that implies – eg overestimating the model’s performance.

Cross Validation

Whichever approach you decide on you can mitigate the effects somewhat by performing cross validation, where the under or oversampling is performed on each fold.

There is a lot of material online that can help – this link for example covers under and oversampling, as well as the option of ignoring the lack of balance and the implications of these options. It also covers SVM and Cross validation with example code in Python. This should help you understand how to properly use cross validation.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you, sir, but the link doesn't work. Also, what would you recommend if you face such an issue? $\endgroup$
    – Atheer
    Commented Mar 20, 2019 at 17:54
  • 2
    $\begingroup$ Most imbalanced data literature considers data to imbalanced after a 1:10 ratio between classes. Common highly cited papers (e.g. He & Garcia (2009) or Galar et al. (2012) typically examine use case with 1:100+ ratios.) A 1:3 ratio is pretty benign by almost any measure. I have never seen a case study with anything that was not at least 1:5; can you please provide some references to your answer? $\endgroup$
    – usεr11852
    Commented Mar 20, 2019 at 18:23
  • $\begingroup$ Sorry - link should be fine now $\endgroup$
    – martino
    Commented Mar 21, 2019 at 8:16
  • $\begingroup$ @usεr11852 Yes I agree 1:3 is probably pretty benign but I wouldn't rule out comparing how well a model works when using a balancing approach to modelling without balancing thr data. If nothing else it will give the OP a better understanding of his problem. $\endgroup$
    – martino
    Commented Mar 21, 2019 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.