2
$\begingroup$

I'm working on a spam detection binary classification problem, but the dataset is very imbalanced (99% to 1%). I know there are techniques like over/under sampling, but I don't think it can be used in this scenario due to the extremely low raw number of positive class samples.

However, a large number of the samples in the dataset have a "NaN" class. I was wondering if it is sensible to assign these classes as belonging to the positive class, or if this is something you should never do in a binary classification problem (in case it's considered as "cheating").

I'm planning on using common algorithms such as logistic regression, KNN, SVM, etc. Perhaps instead of reassigning NaN samples, there is another way of modelling which is better suited to this problem?

Thanks

Improve this question
$\endgroup$
5
  • 3
    $\begingroup$ Do you have reason to believe that they’re really part of the class to which you’d be assigning them? $\endgroup$
    – Dave
    Commented Mar 5, 2022 at 20:54
  • 2
    $\begingroup$ Why are they NaN? $\endgroup$
    – Tim
    Commented Mar 5, 2022 at 21:07
  • $\begingroup$ @Dave I don't have reason to believe it - I tried clustering and PCA to see if there was any relationships, but the samples are all over the place irrespective of the class label. $\endgroup$
    – fx-85
    Commented Mar 5, 2022 at 22:46
  • $\begingroup$ @Tim The dataset was just provided in this manner (positive, negative, NaN). $\endgroup$
    – fx-85
    Commented Mar 5, 2022 at 22:47
  • $\begingroup$ You say "positive, negative", but are these really just +1 or -1 or gradual? Perhaps you should in fact tackle this as a regression problem, not as a classification one at all? $\endgroup$
    – leftaroundabout
    Commented Mar 6, 2022 at 12:55

2 Answers 2

Reset to default
5
$\begingroup$

You could treat the NaNs as unlabelled data and use a semi-supervised learning algorithm, but as @Tim (+1) suggests, labelling them as positive is likely to bias the results. Unless you know the mechanism generating the NaNs, the best you can do is likely to be to ignore the labels entirely and just use the attribute vector. There are semi-supervised versions of logistic regression and the SVM.

Note that imbalance generally is not an inherent problem. If ignoring the minority class is unacceptable, it probably means your application has unequal misclassification costs, and you just need to take them into account (e.g. for the SVM by having different slack penalties for positive and negative patterns). But considering misclassification costs is something we should be doing anyway, and the imbalance does not affect the solution to the cost-sensitive learning problem.

Just a thought, as it is a spam dataset, it could be that the negative examples are emails that were caught by the spam filter, but labelled by the user as ham, and the positive examples could be labelled as spam by the user. The NaN ones may be emails that were caught by a spam filter, but were not checked by the user?

Improve this answer
$\endgroup$
3
  • $\begingroup$ The problem I'm having with training with common classification algorithms is that the model just predicts everything as being negative, so it's a pretty useless model and I have no way of actually understanding if the model is good or not. In situations like this are you just supposed to accept the fact that your model could be bad? $\endgroup$
    – fx-85
    Commented Mar 6, 2022 at 11:33
  • $\begingroup$ @fx-85 It depends, if false-negative errors are equally bad as false-positive errors, then from a classification perspective, that is the optimal solution. However for things like spam, classifying ham as spam is probably worse than classifying spam as ham, so if you take that into account in your training criterion, it may be possible to have a meaningful classification. It is probably best to use a probabilistic classifier, such as kernel logistic regression, as then you can experiment with misclassification costs without having to re-fit the model. $\endgroup$
    – Dikran Marsupial
    Commented Mar 6, 2022 at 21:51
  • $\begingroup$ I tend to use KLR more often than SVM these days, if the dataset isn't too large it is a very good model, and tuning the hyper-parameters reliably is easier. $\endgroup$
    – Dikran Marsupial
    Commented Mar 6, 2022 at 21:52
3
$\begingroup$

No, you can't do that. If you are missing the labels, you don't know them, they can be either positive or negative. If you made the labels positive you are possibly adding incorrect labels to the data, so adding noise to it, biasing your results.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Would it be viable if I was to draw a decision boundary around the negative class and then check whether the NaN samples fit within this decision boundary (thus labelling them as negative, else positive)? Or is it just a bad idea to infer class labels in general? $\endgroup$
    – fx-85
    Commented Mar 6, 2022 at 0:00
  • $\begingroup$ @fx-85 what for? $\endgroup$
    – Tim
    Commented Mar 6, 2022 at 6:40
  • $\begingroup$ Otherwise my model will just predict everything as being negative as it barely has any positive samples to learn from. $\endgroup$
    – fx-85
    Commented Mar 6, 2022 at 12:49
  • $\begingroup$ @fx-85 if most of your labels are negative it seems to be reasonable result. Changing NaN’s to positive labels is as arbitrary as a solution as changing negative labels to positive. $\endgroup$
    – Tim
    Commented Mar 6, 2022 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.