2
$\begingroup$

I have a medical dataset which I am using select features to predict for 1 of 3 diseases. I have found that Random Forest works best for my dataset after testing out various combinations. The original data set was imbalanced towards one of the diseases, so I balanced the dataset with 33% of the dataset filled with each class. However, I know that in reality, roughly 70% of the population has disease 1, 20% disease 2, 10% disease 3.

I was wondering if I should be matching my dataset class frequencies to match the reality distribution of diseases or keep the classes balanced with Random Forest and why?

Improve this question
$\endgroup$
2

1 Answer 1

Reset to default
3
$\begingroup$

Yes, your classifier needs to be optimised for the operational class frequencies if you want it to perform well in operation. However, this can be done after fitting the model if it produces an estimate of the a-posteriori probability of class membership. Ideally, to make a decision, we need to know the probability that the input vector $x$ belongs to class $i$, using Bayes rule,

$p_t(C_i|x) = \frac{p_t(x|C_i)p_t(C_i)}{p_t(X)}$

where the $t$ subscript represents the conditions given in the training set. Now if the training set is representative of operational conditions, then the output of the classifier will be a good estimate of the probability of class membership in operational conditions as well,i.e. $P_t(C_i|x) \approx P_o(C_i|x)$.

But what if this is not the case. Say we have re-balanced the data set so that the classes are each represented by the same number of examples, but this was done in a way that did not affect the likelihoods, $P_t(x|C_i)$. In this case all we need to do is to multiply by the ratio of the operational and training set prior probabilities, to give un-normalised operational class probabilities,

$q_o(C_i|x) = p_t(x|C_i)p_t(C_i)\times\frac{p_o(C_i)}{p_t(C_i)} = p_t(x|C_i)p_o(C_i) \approx p_o(x|C_i)p_o(C_i)$

The $o$ subscript indicates the operational conditions. We can then just re-normalise these probabilities so we have the probabilities of class membership calibrated for operational conditions,

$p_o(C_i|x) = \frac{q_o(C_i|x)}{\sum_{j}q_o(C_j|x)}$

If you have information about misclassification costs, these can also be factored in in a similar manner.

I don't completely agree with @Frank_Harrell about discarding samples (although I would generally advise against it, I wouldn't say it invalidates the exercise). As long as $p_t(x|C_i)$ adequately characterises $p_o(x|C_i)$, it doesn't really matter which sub-sample of the data is used. If we are trying to estimate the mean of a univariate Gaussian from 1,000,000 samples, then the estimate from 999,999 won't be invalid (and won't necessarily ruin predictions for future observations), just marginally less reliable. There comes a point where the uncertainty is too small for our operational purposes, and it becomes reasonable to sub-sample the data for our model to be computationally feasible (or other such practical constraints). But this only applies near assymptotically when you have a superfluosity of data, however I hardly ever find myself in that situation!

There are also applications where we do have to make a hard decision, for instance whether to perform a more sensitive test based on the outcome of a medical screening test. So we do need to choose a threshold for some applications, and for those we need to consider class frequencies (in operation and in the training set) and often misclassification costs (it is generally worse for a screening test to give a false negative than a false positive).

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ Hi, Thanks for the response. I sort of understand what you mean (I am still new to statistics with ML). However, since I am using Random Forest instead of Naive Bayes classifier, would you recommend that I still match the class frequencies to match reality, even though it would unbalance the dataset? $\endgroup$
    – Gamer
    Jul 24, 2021 at 17:06
  • $\begingroup$ @Gamer Class imbalance is not a problem. Why do you want to avoid it? $\endgroup$
    – Dave
    Jul 25, 2021 at 17:30
  • 1
    $\begingroup$ @Dave it is not true that it is not a problem. It is a problem for some classifiers and there are applications where you have to make a hard decision (rather than a propensity) and need to set a threshold. It is much less of a problem that data science blogs will suggest, but it is an issue that needs to be attended to in some applications. $\endgroup$
    – Dikran Marsupial
    Jul 26, 2021 at 6:38
  • $\begingroup$ @Gamer, you can use the scheme I suggest for any classifier that outputs a probability (which can be done with RF). Bayes' rule is a law of probability that isn't specific to Bayesian methods. If your training set class frequencies are different to those in operational use, then this trick is essentially how to set a new classification threshold. $\endgroup$
    – Dikran Marsupial
    Jul 26, 2021 at 6:44
  • 1
    $\begingroup$ @paperskilltrees if you have an infinite dataset, the uncertainty in estimating the model parameters will be zero. That is indeed a good thing. In practice you often don't need zero uncertainty in estimating the parameters, so using a finite dataset would be fine, and would be computationally cheaper. The point I was making is that adding extra data is a matter of diminishing returns, so if you are already at the point where the parameter uncertainty is negligible, there is probably no real harm in downsampling the dataset. $\endgroup$
    – Dikran Marsupial
    Jan 29 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.