Imbalanced data to match reality with Random Forest? I have a medical dataset which I am using select features to predict for 1 of 3 diseases. I have found that Random Forest works best for my dataset after testing out various combinations. The original data set was imbalanced towards one of the diseases, so I balanced the dataset with 33% of the dataset filled with each class. However, I know that in reality, roughly 70% of the population has disease 1, 20% disease 2, 10% disease 3.
I was wondering if I should be matching my dataset class frequencies to match the reality distribution of diseases or keep the classes balanced with Random Forest and why?
 A: Yes, your classifier needs to be optimised for the operational class frequencies if you want it to perform well in operation.  However, this can be done after fitting the model if it produces an estimate of the a-posteriori probability of class membership.  Ideally, to make a decision, we need to know the probability that the input vector $x$ belongs to class $i$, using Bayes rule,
$p_t(C_i|x) = \frac{p_t(x|C_i)p_t(C_i)}{p_t(X)}$
where the $t$ subscript represents the conditions given in the training set.  Now if the training set is representative of operational conditions, then the output of the classifier will be a good estimate of the probability of class membership in operational conditions as well,i.e. $P_t(C_i|x) \approx P_o(C_i|x)$.
But what if this is not the case.  Say we have re-balanced the data set so that the classes are each represented by the same number of examples, but this was done in a way that did not affect the likelihoods, $P_t(x|C_i)$.  In this case all we need to do is to multiply by the ratio of the operational and training set prior probabilities, to give un-normalised operational class probabilities,
$q_o(C_i|x) = p_t(x|C_i)p_t(C_i)\times\frac{p_o(C_i)}{p_t(C_i)} = p_t(x|C_i)p_o(C_i) \approx p_o(x|C_i)p_o(C_i)$
The $o$ subscript indicates the operational conditions. We can then just re-normalise these probabilities so we have the probabilities of class membership calibrated for operational conditions,
$p_o(C_i|x) = \frac{q_o(C_i|x)}{\sum_{j}q_o(C_j|x)}$
If you have information about misclassification costs, these can also be factored in in a similar manner.
I don't completely agree with @Frank_Harrell about discarding samples (although I would generally advise against it, I wouldn't say it invalidates the exercise).  As long as $p_t(x|C_i)$ adequately characterises $p_o(x|C_i)$, it doesn't really matter which sub-sample of the data is used.  If we are trying to estimate the mean of a univariate Gaussian from 1,000,000 samples, then the estimate from 999,999 won't be invalid (and won't necessarily ruin predictions for future observations), just marginally less reliable.  There comes a point where the uncertainty is too small for our operational purposes, and it becomes reasonable to sub-sample the data for our model to be computationally feasible (or other such practical constraints).  But this only applies near assymptotically when you have a superfluosity of data, however I hardly ever find myself in that situation!
There are also applications where we do have to make a hard decision, for instance whether to perform a more sensitive test based on the outcome of a medical screening test.  So we do need to choose a threshold for some applications, and for those we need to consider class frequencies (in operation and in the training set) and often misclassification costs (it is generally worse for a screening test to give a false negative than a false positive).
