How do you know that your classifier is suffering from class imbalance? In cases where there is a substantial difference in relative class frequencies, it could be that the density of the minority class is never higher than the density of the majority class anywhere in the attribute space.  Here is a simple example using univariate Gaussian classes, with an imbalance ratio of 1:9.

In this case, if my classifier assigns all patterns to the majority class, it is doing exactly the right thing, and there is no problem to solve.
In this case, we know the true data generating process, so we know that the classifier is doing the right thing.  However in general we don't know the true distributions of positive and negative classes, so we don't know whether the classifier is doing the right thing or not.
So my question is:  In practical applications, how do we decide if we have a class imbalance problem, or whether the classifier is just giving the correct answer, to the question as posed?
Full disclosure:  My intuition is that in most cases, especially when the data is not unduly scarce, the classifier is doing exactly what it should do and there is no class imbalance problem.  I am primarily interested to hear how other practitioners and researchers diagnose class imbalance problems.
 A: I challenge that there is a problem. Let's go with the scenario you described in the comments where your red graph is shifted to the right a bit.

I will make up some (plausible) numbers and go through Bayes' theorem.
$$
P(\text{red}) = 0.2$$$$P(\text{blue}) = 0.8
$$
$$
P(X>3\vert \text{red}) = 0.6$$$$P(X>3\vert \text{blue}) = 0.05
$$
Now Bayes' theorem:
$$
P(\text{red}\vert X>3) = \dfrac{P(X>3\vert\text{red})P(\text{red})}{P(X>3)}
$$
$$P(X>3) = P(X>3\cap\text{red}) + P(X>3\cap\text{red}^C) $$$$= P(X>3\cap\text{red}) + P(X>3\cap\text{blue})$$
$$
= P(X>3\vert \text{red})P(\text{red}) + P(X>3\vert \text{blue})P(\text{blue})
$$$$=(0.6)(0.2) + (0.05)(0.8) = 0.16$$
Now let's put it all together in Bayes' theorem.
$$
P(\text{red}\vert X>3) = \dfrac{(0.6)(0.2)}{(0.16)} = 0.75
$$
That's a much larger probability of being red than the prior probability of $0.2$.
Varying the prior probability of being red reveals a similar story of consistently having a higher posterior probability of being red than prior probability of being red.
posterior <- function(x, y, z){
    return(x*y/(x*y + z*(1-x)))
}
prior <- seq(0, 1, 0.0001)
plot(prior, posterior(prior, 0.6, 0.05), xlab = "Prior of Red", ylab = "Posterior of Red", col = 'red')
lines(prior, prior)

The class imbalance does not overwhelm the posterior probability, and I have tried this with even smaller shifts of red to the right. A tiny shift results in a plot that is very close to the diagonal, but it still bends up a little bit.

A: Well, I think the lack of an answer that explains how to detect whether class imbalance is a problem in a particular application, even when a modest bounty of +50 reputation was on offer, suggests cause for concern about research on the topic of class imbalance.  I suspect practitioners are frequently re-balancing or re-weighting the datasets simply because they are imbalanced, rather than because the imbalance is actually causing a problem.  I further suspect that often this is just making matters worse by over-compensating (e.g. by fully balancing the dataset).
Class imbalance can cause a problem when there are two few examples of the minority class to adequately characterise it's statistical distribution.  When this happens, the decision boundary does tend to be unduly biased in favour of the majority class.  However, as you add more data, the problem goes away.  This shouldn't be a surprise.  If you have a large enough neural network, it will be a universal approximator, able to implement essentially any (one-to-one or many-to-one) mapping between the input and output spaces.  If it is fitted using a proper scoring rule then asymptotically it will output the true posterior probabilities of class membership.  So if you have enough data, it doesn't matter how imbalanced the problem is, a complex enough model will learn the optimal decision surface.
I think any means of detecting and dealing with class imbalance problems will be very tricky though.  Essentially if there is a bias, you will want to re-sample or re-weight the training sample just the right amount to compensate for the bias due to the "imbalance".  Exactly balancing the dataset is likely to way over-compensate and make accuracy (or expected loss) worse rather than better.  The trouble is, if you don't have enough data to describe the minority class, where are you going to get the data to choose the optimal degree of bias?  I suspect the best approach will be some Bayesian scheme that determines what the plausible true positive rate (for example) could be if the model were correct.
Essentially, I know from experience that class imbalance can cause estimation problems, in a small data setting, but I'm not convinced that there is a great deal we can do about it because we don't have enough independent data to tune the compensation applied.  I think we should be very wary of up/down sampling or reweighting simply because there is an imbalance, and if we do, we need to be able to determine whether it has worked or not.  This requires at least that we know what criterion is important for our application, and why it is important.  No application is primarily interested in the true positive rate, if that were true, we would just assign everything to the positive class and go home satisfied with having done the optimal job! ;o)
