47

There are many frameworks and approaches. This is a recurrent issue. Examples: Undersampling. Select a subsample of the sets of zeros such that it's size matches the set of ones. There is an obvious loss of information, unless you use a more complex framework (for a instance, I would split the first set on 9 smaller, mutually exclusive subsets, train a ...


40

Generally, scale_pos_weight is the ratio of number of negative class to the positive class. Suppose, the dataset has 90 observations of negative class and 10 observations of positive class, then ideal value of scale_pos_weight should be 9. See the doc: http://xgboost.readthedocs.io/en/latest/parameter.html


38

The intuitive reasoning has been explained in the blogpost: If our goal is Prediction, this will cause a definite bias. And worse, it will be a permanent bias, in the sense that we will not have consistent estimates as the sample size grows. So, arguably the problem of (artificially) balanced data is worse than the unbalanced case. Balanced ...


36

Not a direct answer, but it's worth noting that in the statistical literature, some of the prejudice against unbalanced data has historical roots. Many classical models simplify neatly under the assumption of balanced data, especially for methods like ANOVA that are closely related to experimental design—a traditional / original motivation for developing ...


33

Both hxd1011 and Frank are right (+1). Essentially resampling and/or cost-sensitive learning are the two main ways of getting around the problem of imbalanced data; third is to use kernel methods that sometimes might be less effected by the class imbalance. Let me stress that there is no silver-bullet solution. By definition you have one class that is ...


31

Down-sampling is equivalent to case–control designs in medical statistics—you're fixing the counts of responses & observing the covariate patterns (predictors). Perhaps the key reference is Prentice & Pyke (1979), "Logistic Disease Incidence Models and Case–Control Studies", Biometrika, 66, 3. They used Bayes' Theorem to rewrite ...


26

Mathematically, b_acc is the arithmetic mean of recall_P and recall_N and f1 is the harmonic mean of recall_P and precision_P. Both F1 and b_acc are metrics for classifier evaluation, that (to some extent) handle class imbalance. Depending of which of the two classes (N or P) outnumbers the other, each metric is outperforms the other. 1) If N >> P, f1 is ...


24

A few comments: The option (1) is a very bad idea. Copies of the same point may end up in both the training and test sets. This allows the classifier to cheat, because when trying to make predictions on the test set the classifier will already have seen identical points in the train set. The whole point of having a test set and a train set is that the test ...


23

This question evidently came from a study with an unbalanced two-way design, analyzed in R with the aov() function; this page provides a more recent and detailed example of this issue. The general answer to this question, as to so many, is: "It depends." Here it depends on whether the design is balanced and, if not, which flavor of ANOVA is chosen. First, ...


22

Unbalanced data is only a problem depending on your application. If for example your data indicates that A happens 99.99% of the time and 0.01% of the time B happens and you try to predict a certain result your algorithm will probably always say A. This is of course correct! It is unlikely for your method to get better prediction accuracy than 99.99%. ...


21

First, the evaluation metric for imbalanced data would not be accuracy. Suppose you are doing fraud detection, that 99.9% of your data is not fraud. We can easy make a dummy model that have 99.9% accuracy. (just predict all data non-fraud). You want to change your evaluation metric from accuracy to something else, such as F1 score or precision and recall. ...


19

I typically use a 15:1 rule (ratio of min(events, non-events) to number of candidate parameters in the model). More recent work found that for a more rigorous validation 20:1 is needed. More information may be found in my course handouts linked from http://hbiostat.org/rms, in particular an argument for a minimum sample size of 96 just to estimate the ...


19

For imbalanced data sets we typically change the misclassification penalty per class. This is called class-weighted SVM, which minimizes the following: $$ \begin{align} \min_{\mathbf{w},b,\xi} &\quad \sum_{i=1}^N\sum_{j=1}^N \alpha_i \alpha_j y_i y_j \kappa(\mathbf{x}_i,\mathbf{x}_j) + C_{pos}\sum_{i\in \mathcal{P}} \xi_i + C_{neg}\sum_{i\in \mathcal{N}...


19

Consistent with @kjetil-b-halvorsen's comment, the rapid adoption of machine learning has confused researchers about prediction vs. classification. As I described in more detail here, classification is only appropriate in a minority of cases. When the outcome is rare (or too common), probabilities are everything because in that case one can only ...


18

WLOG you can focus on imbalance in a single factor, rather than a more nuanced concept of "data sparsity", or small cell counts. In statistical analyses not focused on learning, we are faced with the issue of providing adequate inference while controlling for one or more effects through adjustment, matching, or weighting. All of these have similar power and ...


17

Many SVM implementations address this by assigning different weights to positive and negative instances. Essentially you weigh the samples so that the sum of the weights for the positives will be equal to that of the negatives. Of course, in your evaluation of the SVM you have to remember that if 95% of the data is negative, it is trivial to get 95% accuracy ...


17

ROC curves are insensitive to class balance. The straight line you obtain for a random classifier now is already the result of using different probabilities of yielding positive (0 brings you to (0, 0) and 1 brings you to (1, 1) with any range inbetween). Nothing changes in an imbalanced setting.


17

ROSE uses smoothed bootstrapping to draw artificial samples from the feature space neighbourhood around the minority class. SMOTE draws artificial samples by choosing points that lie on the line connecting the rare observation to one of its nearest neighbors in the feature space. Source: Training and assessing classification rules with unbalanced data My ...


16

I've found He and Garcia (2009) to be a helpful review of learning in imbalanced class problems. Here are a few definitely-not-comprehensive things to consider: Data-based approaches: One can undersample the majority class or oversample the minority class. (Breiman pointed out that this is formally the equivalent to assigning non-uniform misclassification ...


16

You don't have to get predicted categories from a logistic regression model. It can be fine stay with predicted probabilities. If you do get predicted categories, you should not use that information to do anything other than say 'this observation is best classified into this category'. For example, you should not use 'accuracy' / percent correct to select ...


16

It sounds like what you have is a powerfully predictive variable, and there is no reason to remove it. What you have to watch out in situations like this is what is called leakage. Leakage is when you have a predictor that is just some version of your response in disguise. For example, suppose that you have a system at your company that, when fraud is ...


15

Weighting is a procedure that weights the data to compensate for differences in sample and population (King 2001). For example, in rare events (such as fraud in credit risk, deaths in medical literature) we tend to sample all the 1’s (rare events) and a fraction of 0’s (non events). In such cases we have to weight the observations accordingly. Example: Let ...


15

The short answer is that logistic regression is for estimating probabilities, nothing more or less. You can estimate probabilities no matter how imbalanced $Y$ is. ROC curves and some of the other measures given in the discussion don't help. If you need to make a decision or take an action you apply the loss/utility/cost function to the predicted risk and ...


15

Class imbalance issues can be addressed with either cost-sensitive learning or resampling. See advantages and disadvantages of cost-sensitive learning vs. sampling, copypasted below: {1} gives a list of advantages and disadvantages of cost-sensitive learning vs. sampling: 2.2 Sampling Oversampling and undersampling can be used to alter the class ...


15

You could use the Shannon entropy as a measure of balance. On a data set of $n$ instances, if you have $k$ classes of size $c_i$ you can compute entropy as follows: $$ H = -\sum_{ i = 1}^k \frac{c_i}{n} \log{ \frac{c_i}{n}}. $$ This is equal to: $0$ when there is one single class. In other words, it tends to $0$ when your data set is very unbalanced $\log{...


14

The second (2) option is the right way of doing it. The synthetic samples you create with the oversampling techniques are not real examples but rather synthetic. These are not valid for testing purposes while they still ok for training. They are intended to modify the behavior of the classifier without modifying the algorithm.


14

First 18 isn't a lot of features at all and you should see if you can get more data. Google uses a ridiculous number of features in their ad targeting and takes a different online/game theoretical approach to choosing what ad to show to the audience Second, skewed class labels like this are a common problem. Search terms to look at include imbalanced or ...


14

Ching, You do not have to make your data set balanced in terms of 1’s and 0’s. All you need is sufficient number of 1’s for the maximum likelihood to converge. Looking at the distribution of 1’s (100,000) in your dataset, you should not have any problems. You can do a simple experiment here Sample 10 % of the 1’s and 10% of the 0’s and use a weight of 10 ...


14

Besides the AUC and Kohonen's kappa already discussed in the other answers, I'd also like to add a few metrics I've found useful for imbalanced data. They are both related to precision and recall. Because by averaging these you get a metric weighing $TP$s and both types of errors ($FP$ and $FN$): F1 score, which is the harmonic mean of precision and recall. ...


14

An entry from the Encyclopedia of Machine Learning (https://cling.csd.uwo.ca/papers/cost_sensitive.pdf) helpfully explains that what gets called "the class imbalance problem" is better understood as three separate problems: (1) assuming that an accuracy metric is appropriate when it is not (2) assuming that the test distribution matches the ...


Only top voted, non community-wiki answers of a minimum length are eligible