Is there an optimal loss function for dealing with imbalanced classes? I'm aware that there are many ways of dealing with datasets where there is a strong class imbalance in the target variable: downsampling the more prevalent and less important class, over-weighting the valuable minority class, etc. Are there any loss functions that are ideally suited for these types of problems? I've heard of AUC, but AUC sounds like a means of evaluating a model once its training is complete (correct me if I'm wrong). Log loss and squared loss penalize a model's over confidence, but overconfidence and class imbalance seem unrelated. Is that true?
 A: Think for a moment about the phrase "optimal loss function". A loss function is what tells you what counts as a good prediction or a bad prediction. It is the basis on which you can assess whether a statistical method or model is optimal. Hence, there can be no such thing as an optimal loss function (unless perhaps you have committed to some kind of meta-loss-function or philosophical framework). Ultimately, you have to choose the loss function that represents what you care about for the problem in question.

AUC sounds like a means of evaluating a model once its training is complete (correct me if I'm wrong)

I'm not sure what you mean by that, but area under the receiver operating characteristic curve is not a particularly good measure of anything, really.

Log loss and squared loss penalize a model's over confidence, but overconfidence and class imbalance seem unrelated. Is that true?

Yes, they're distinct things. On the subject of overconfidence, if you're talking about the problem of probabilistic classification, log loss is much harsher than squared loss (i.e. the Brier score) on extreme predicted probabilities that are on the wrong side of 1/2. In particular, when the model predicts a probability of 0 for belonging to the class that a case ends up actually belonging to, that prediction is regarded as infinitely bad.
