24

A few comments: The option (1) is a very bad idea. Copies of the same point may end up in both the training and test sets. This allows the classifier to cheat, because when trying to make predictions on the test set the classifier will already have seen identical points in the train set. The whole point of having a test set and a train set is that the test ...


20

{1} gives a list of advantages and disadvantages of cost-sensitive learning vs. sampling: 2.2 Sampling Oversampling and undersampling can be used to alter the class distribution of the training data and both methods have been used to deal with class imbalance [1, 2, 3, 6, 10, 11]. The reason that altering the class distribution of the training data aids ...


17

ROSE uses smoothed bootstrapping to draw artificial samples from the feature space neighbourhood around the minority class. SMOTE draws artificial samples by choosing points that lie on the line connecting the rare observation to one of its nearest neighbors in the feature space. Source: Training and assessing classification rules with unbalanced data My ...


14

The second (2) option is the right way of doing it. The synthetic samples you create with the oversampling techniques are not real examples but rather synthetic. These are not valid for testing purposes while they still ok for training. They are intended to modify the behavior of the classifier without modifying the algorithm.


13

I have encountered a similar problem, and I solved it by transferring the class values ("status" in your case) into factor type. After using data$status=factor(data$status), newData prints as follows: looking risk every status 7 0 0 0 1 2 0 0 0 1 7.1 0 0 0 1 12 0 0 0 1 4 ...


8

ROSE and SMOTE are designed to handle categorical variables, so, unless your categorical variables are expressed in a binary format, you shouldn't normally have to worry about synthetic observations being assigned mutually exclusive categorical features. If they are, you can always restructure them as factors. In your two-region example, you would create a ...


8

It doesn't play much of a difference but you should do most pre-processing steps (encoding, normalization/standardization, etc) before under/over-sampling the data. This is because many sampling techniques require a simple model to be trained (e.g. SMOTE uses a k-NN algorithm to generate samples, ClusteringCentroids under-sampling technique involves k-means ...


7

You are falling in the logical mistake of Baron Munchausen: http://www.lhup.edu/~dsimanek/museum/themes/BaronMunch.jpg You cannot create more information than that provided by the sample, unless you collect a larger sample. If you simulate more observations from the estimated model $f(\cdot;\hat{\theta})$, then, the more samples you simulate, the more ...


6

Original Answer: There is no benefit to under- or over-sampling either group New Conclusion, see edit at end: There can be a benefit of sampling, but not random sampling of failures and non-failures. You are forgetting the point of a Cox analysis. It is not to analyze who dies and who does not die; rather it is to study effects of covariates on the ...


6

It seems like you are oversampling (i.e. generating synthetic data instances) before splitting training and testing data. This causes over-fitting and hence your optimistic initial results. As pointed here, you should consider applying oversampling after splitting your data.


5

More than 100 classes shouldn't be a problem for most classification algorithms. However, if that number increases you should start thinking about new models for large-scale (in this case for the number of classes) classification. You can probably find some hint in this (a bit old) workshop about large-scale (hierarchical) text classification. About the ...


5

This does not answer why, but to get around this, one can duplicate the data for the rare class in the training data, and take a stratified sample of the result. Two drawbacks to this approach, compared with a "natural" oversampling: the out of bag estimates are no longer meaningful more resources are required to store the object and take random samples ...


4

I have the exact same question and found this in the changelog for randomForest: Changes in 4.1-0: In randomForest(), if sampsize is given, the sampling is now done without replacement, in addition to stratified by class. Therefore sampsize can not be larger than the class frequencies. Setting replace=TRUE manually also does not seem to override this.


4

Let 2 binary random variables $X$ and $Y$ e.g. exposure and disease status. Here is the logistic model: $logit(\pi(X=x)) = \alpha + \beta x $ where $\pi(X=x)=P(Y=1|X=x)$. Then, $\alpha = logit(\pi(X=0)) = \log (\frac{\pi(X=0)}{1-\pi(X=0)})$ so $\alpha$ is simply the log of odds for the unexposed, $X=0$. But why is the estimator of $\alpha$ biased when the ...


4

Your test set should be as close to a sample from the distribution on which you are actually going to apply your classifier as possible. I would definitely split your dataset first (in fact, that is usually the first thing I would do after obtaining a dataset), put away your test set, and then do everything you want to do on the training set. Otherwise, it ...


3

The estimation of sensitivity is not dependent on the proportion of positive cases or events in the sample. Consider the following table: Disease+ Disease- Test+ a b Test- c d a is the number of indivudals with a certain disease, condition, event or such, and who are classified as positive in the test (true positives). b is the ...


3

Yes, it does make sense and actually this is what is done in practice. Why does it makes sense? Random permutations might change the statistics of the dataset. By performing a new random permutation in each epoch, these effects tend to cancel out. Imagine performing a random rotation on an image with a range of $[-5\%, +5\%]$. By storing the rotated image,...


3

I don't think factoring or oversampling would be beneficial just to increase the number of data points for training or modeling. If you wanted to adjust your dataset because you believe it is not representative of the "true" dataset or the prospective dataset, then I think this sounds reasonable although very difficult in practice and difficult to support. ...


3

There is actually quite an expansive and rich literature on oversampling and linear regression. The general survey methods literature has discussed and explored many types of estimators. For starters, you might reference "Complex Surveys: A Guide To Analysis Using R" by Lumley. Duplicating data is not oversampling. The point of oversampling is to increase ...


2

Off the cuff, I presume one could proceed as in logistic regression: a generalisation to $K>2$ categories and base category $K$ would be to set the $i$-th correction term to be $$\log \frac{(r_i p_K)}{(r_K p_i)}$$ corresponding to the $i$ vs $K$ contrast. For $K=2$, $p_1$ is as before and $p_K = p_2 = 1-p_1$, so it reduces to $$\log \frac{r_1 (1-p_1)}{...


2

What sort of SVM do you use?. If you are using $\nu$-SVM, then you can find the answer in this paper. The idea is the following: "v is an upper bound on the fraction of margin errors, a lower bound on the fraction of support vectors, and both quantities approach v asymptotically". Additionally, this number cannot exceed the quantity 2*lmin/l, where l is the ...


2

You're confusing the labeled and unlabeled data. You have 1 million labeled samples: what percentage of the 1 million are good and what percentage are bad? If that ratio is high, to use your example numbers 99% good to 1% bad, you may need to consider over/under sampling before training. From the 1 million, you would hold out a test set, which you might ...


2

It doesn't work just to divide the probabilities. Basically you have to adjust the odds, not the probabilities. There's a nice description and some sample calculations here: https://yiminwu.wordpress.com/2013/12/03/how-to-undo-oversampling-explained/ (added in edit) There's a different derivation that gives the same results here: http://blog.data-miners....


2

you correctly adopted the train-and-test approach, instead of cross validation. In fact, you should test the model on non-resampled data, in order to maintain the same distribution as in the population and obtain realiable indices of performances. To do this, you need to apply a filtered classifier, then SMOTE and/or undersampling. Still, accuracy (correct ...


2

There is a lot going on in here and it's hard to say what your actual question is, but here are two pieces of general advice: First, you probably don't have enough data that it's safe do do train-test-validate splitting. Your results will probably vary depending on which samples end up in the test set. Unless you have many thousands of data points and a ...


2

The question is, why do you oversample? If you have different relative frequencies in your data than you expect in the real application and oversampling is to correct this - then oversampling should be done first (or, to put it differently, you calculated weighted mean and standard deviation, and train a classifier for the corrected prior probabilities). ...


2

I think one needs to take into account that changing the class proportions in the training sample will substantially change the final model that is going to be learned through training. This might not be something we necessarily want. The cost of misclassifying majority class samples might be not negligible (invasive medical treatments to rare diseases being ...


2

In general, more data is preferred, but not necessarily the raw full data, especially when it is imbalanced. The robustness of algorithms to imbalanced data varies from one to another, take naive Bayes as an example, the core formula of this algorithm is: $$\mathbb{P}(Y|X) = \mathbb{P}(Y)\mathbb{P}(X|Y)/\mathbb{P}(X)$$ We can see that the prediction ...


2

As far as I know, in spite of how widespread this idea is, there is no convincing argument that artificial oversampling makes sense or leads to a better model. See for instance this question, or this question, both about whether class imbalance is a real problem, both highly voted/often viewed, neither with any clear indication that this practice is a good ...


Only top voted, non community-wiki answers of a minimum length are eligible