1
$\begingroup$

I need a bit of help with interpretation of classification results.

I have unbalanced data set (80% = 0 20% = 1), fitting classifiers (SVM, GradientBoosting or kNN) on such data does not yield good results (even using weighting). I mean accuracy is very good but minority class is mostly misclassified - as should be expected.

So I decided to balance train data using undersampling (number of samples is sufficient to do so).

This way I introduce selection bias and I get good classification results on test data (not balanced).

Can I assume that those results are reliable ?

Improve this question
$\endgroup$
1
  • $\begingroup$ The rate of under sampling can be seen as a meta parameter. In that case such a parameter is best determined using a validation set or cross validation and a grid search. After determining the parameter you should be able to report the quality of the model based on a test set. Using evaluation on the test set directly is not good practice. $\endgroup$
    – spdrnl
    Commented Jun 2, 2015 at 13:51

1 Answer 1

Reset to default
1
$\begingroup$

Perhaps you have an validation data set to which you use your classifier algorithm if it gives directly class labels? Then you know how it behaves.

Other possibility is to have an probability estimator which itself classifies nothing but you can use predicted probabilities to form an threshold for classification. There are formulas which might change predicted probabilities from the training sample into population level probabilities.

Here is link to the latter subject:

http://support.sas.com/kb/22/601.html

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.