Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [oversampling]

Sampling cases with differential probability, so that classes that occur rarely in the population occur more often in the training data. Does *not* address the problems in unbalanced classes.

0
votes
0answers
14 views

Is Oversampling beneficial in semi-supervised learning?

I have suggested a semi-supervised approach for the hierarchical multi-label classification task. I have included the MLSMOTE oversampling technique as a pre-processing step, and then evaluate the ...
0
votes
0answers
27 views

Semi-supervised multi-label learning

I have suggested a semi-supervised approach for the hierarchical multi-label classification task. I have included the MLSMOTE oversampling technique as a pre-processing step, and then evaluate the ...
0
votes
0answers
6 views

A single logit model to estimate the churn of more than 1 product: how can i deal with a different % of 1's? [closed]

I need to estimate 1 single logit model to predict the probability of churn for two different products (25% prod1-75% prod2). Each product have a different churn rate (7% vs 10%). I have not enough ...
0
votes
0answers
12 views

Balancing continuous covariates for oversampling

I'm currently looking into methods for restoring balance of a biased dataset with respect to a continuous variable. My problem is similar to this question, with the slight difference that I'm dealing ...
2
votes
0answers
37 views

Logistic Regression Class Imbalance and the use of weighting and undersampling

I have been working on a machine learning model using Spark (binomial) LogisticRegression. The dataset has what I think is a high degree of imbalance - roughly 1% of rows are labelled as events. The ...
0
votes
0answers
12 views

Analyzing imbalanced data using linear and logistic regression [duplicate]

According to Wikipedia page on Oversampling and undersampling in data analysis: The end-result of over-/under-sampling is the creation of a balanced dataset. Many machine-learning techniques, such ...
0
votes
0answers
12 views

oversampling data with subclass

Oversampling of under-represented data is a way to combat class imbalance. For example, if we have a training data set with 100 data points of class A and 1000 data points of class B, we can over ...
0
votes
1answer
23 views

Adjusting precision recall curve for oversampling

I built a model for a binary target using oversampled data. The population target prevalence is 0.25. I oversampled to 0.5 by keeping the entirety of the minority class and sampling a portion of the ...
0
votes
1answer
730 views

Running XGBoost with *highly* imbalanced data returns near 0% true positive rate. Tried SMOTE and it did not improve much. What else can I do?

I'm using XGBoost on a dataset of ~2.8M records of hard drive failures, where less than 200 are tagged as failures. After cleaning, there are 11 features in this dataset. Below is my ...
1
vote
0answers
8 views

Oversampling to Create Correlated Feature from Existing Feature

For demo purposes in Python, I would like to create an array or pandas df column that shares a high degree of correlation with an existing array. This is not as easy as it sounds and I don't think it ...
0
votes
0answers
16 views

Should oversampling/undersampling be applied only during CV or also for final model creation?

I am dealing with a highly class imbalanced dataset and am going to try oversampling and see how my nested CV is affected when comparing algorithms. When it comes to model finalization, should I ...
0
votes
0answers
10 views

For ADASYN, if the neighbourhood of a minority sample contains no other minority sample, do I double the sample?

In ADASYN, for the last step in the paper linked below, if there exists no other minority class in the k-NN other than the one minority example, do we simply just double the training example? Because ...
0
votes
0answers
37 views

Imbalanced class SVM prediction results using different validation data

I am trying to fit my data to a classifier using SVM. My data has 2 classes, the positive class which occurs with a probability of 0.002 and the negative class which is the dominant one. Suppose that ...
1
vote
0answers
36 views

While dealing with imbalanced classes, to what extent can we upsample a minority class? [duplicate]

I have my training data with the following approximate distribution: Negative events : 90,000 positive events : 5,000 Training a model would require to oversample the minority class (and might also ...
1
vote
0answers
241 views

How do we adjust for Oversampling in R? [closed]

How do we adjust for Oversampling in R?
0
votes
0answers
159 views

Oversampling using SMOTE leading to bad predictions on test set

I have a dataset with an imbalanced binary target. One class accounts for about 94 % of the target variable. I used SMOTE to oversample the minority class but after the oversampling step when I train ...
0
votes
0answers
190 views

Decision Tree with unbalanced dataset in SAS

I have a dataset with a binary target variable. This variable is highly imbalanced i.e. the # of True case is ~1% and # of False cases is ~99% The other limitation I have is that I can only use ...
0
votes
0answers
22 views

Resampling imbalanced data in a multi-view scenario

Assume a multi-view scenario, where multiple views of the same entity are available. If each data pair is assigned a label and the resulting scenario is highly imbalanced, what are proper ways of ...
5
votes
1answer
812 views

Normalization/standardization: Should one do this before oversampling/undersampling the data or after?

When working with imbalanced datasets, should one do one-hot encoding and data standardization before or after sampling techniques (such as oversampling or undersampling)?
0
votes
0answers
86 views

Oversampling for multi-class neural net

Does this make sense or do I have no idea what I'm doing? I want to train a model that takes a sentence and outputs a binary multi-class vector of size $K$ where each dimension is a question class. ...
1
vote
1answer
256 views

How to report the results of a cross-validation to a paper: Can I manually select the best results?

Good morning. With regard to the interpretation of the results of the N-fold cross-validation with over-sampling (SMOTE), I raise the question to write up the paper. Currently, I have performed 5-...
0
votes
1answer
191 views

Oversampling problems in prediction

I have a dataset that contains 284315 samples of class 0 and 492 of class 1. I know, that's huge. I heard about oversampling methods, so I did the following using the RandomOverSampler library: ...
15
votes
0answers
276 views

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? [duplicate]

TL;DR See title. Motivation I am hoping for a canonical answer along the lines of "(1) No, (2) Not applicable, because (1)", which we can use to close many wrong questions about unbalanced datasets ...
1
vote
1answer
189 views

Random Over Sampling to handle Data Imbalance

Was reading an article on imbalanced datasets where the event occurs and look at balancing the dataset. In that article, the event records were 2% of the total records. The author of the blog ...
1
vote
0answers
46 views

Bias-Variance Tradeoff when using Oversampling Technique

Oversampling techniques (e.g. SMOTE) are often used when target values are not approximately equally represented. How does this technique affect bias and variance of the predictive model that is ...
4
votes
1answer
626 views

Oversampling: whole set or training set

I have a rather small dataset of 4 000 points (140 features) to feed to a NN binary classifier. The problem is only ~700 of them represent the second class. Is it more common to resample the whole ...
0
votes
0answers
261 views

Comparing precision-recall curves for training and test data set

Referring to these links for some of my assumptions - https://classeval.wordpress.com/introduction/introduction-to-the-precision-recall-plot Does it make sense to plot train and test results on a ...
1
vote
0answers
411 views

random forest imbalanced data-over, under, Smote Sampling

I am using random forest model for an imbalanced dataset. The dependent variable is Yes=73, No=7100. I have 65 independent variables both factor and numeric. I have tried to develop models for ...
2
votes
2answers
65 views

Using patient characteristics to predict disease and confront with per hospital real cases

We have a large, nationwide prevalence study about the prevalence of healthcare-associated infections (HAI). We need to see if hospitals have more or less HAI than expected by their patients' ...
0
votes
1answer
178 views

Modeling Attrition: Imbalanced Training

First of all, let me say I am new to Machine Learning and am eager for any sort of feedback. I am attempting to create a predictive attrition model, and my training and test data each have ~ 17% ...
0
votes
1answer
100 views

oversampling in nested cross validation

Introduction I have a small mixed dataset consisting of continuous and categorical independent variables with a dichotomous dependant variable. I'm running various algorithms (neural networks, random ...
4
votes
1answer
326 views

During oversampling of rare events, why are the beta coefficients of the independent variables not affected, but only the intercept?

I have followed the King and Zeng paper and understand the consistency of the prior correction after oversampling in logistic regression. But I am trying to understand why the beta coefficients of the ...
1
vote
0answers
50 views

The problem between over-sampled, down-sampled data and original data using SVM with unbalanced data

I am working on a classifier that is supposed to do binary classification on a dataset includes 5996 examples. ~800 of this examples belongs to class 1 and the rest is class 0. Since there is a huge ...
0
votes
1answer
93 views

How to interpret results of a predictive model when an external factors leads to imbalanced target labels

I have a prediction task, in which I use DecisionTreeRegressor of scikit-learn to predict a target label, which is about a certain user behaviour in a web platform (...
1
vote
1answer
69 views

Oversampling - Does not contradict for the requirement for Independence of Observations?

A requirement for many statistical and machine learning prediction models is the independence of observations. However, when using oversampling we reintroduce the same observations in the dataset. ...
2
votes
0answers
736 views

Should SMOTE oversampling be done before or after holdout validation's training/testing split? [duplicate]

Originally, without SMOTE, my ML learning steps go like this: Feature vectorization split data into X_train, X_test, y_train, and y_test use X_train and y_train for machine learning predict/test ...
0
votes
1answer
334 views

Should I first oversample or standardize (when cross-validating on imbalanced data)?

I have an imbalanced (two-class) classification dataset, based on which I am trying to train and cross-validate a classifier. During the process of the k-fold cross-validation, I set aside the test ...
2
votes
2answers
605 views

Balance classes when sampling

I have a large dataset describing numerous customers' behaviour and I am trying to solve a binary classification problem with a null accuracy on 90% (90/10 distribution amongst the two classes). ...
2
votes
0answers
60 views

What are the recommended over-sampling or under-sampling techniques to handle multi-class imbalanced datasets for machine learning? [duplicate]

Many techniques are for two-class classification. What is the recommended techniques to over- or under-sample multi-class datasets?
0
votes
0answers
62 views

Why over/under-sampling could not help my model fitting?

I fit random forest to my imbalanced dataset with minority class 1. I found that the AUC under the imbalanced data was better than that of re-sampled dataset (over/under sampling). Can someone help to ...
1
vote
0answers
288 views

Correcting Bias

I have a data set that includes locations of where certain rocks were observed on the Earth. Populated areas have a higher number of observations in general. Remote areas have less observations. I'm ...
2
votes
1answer
227 views

Over sampling for minority classes

I'm a bit confused with the idea of over/under sampling. People have mentioned that rotations are a good technique. By rotation, do they literally mean rotate the image by a few degrees and then add ...
4
votes
0answers
910 views

After oversampling/undersampling is it always appropriate to adjust probabilities using the odds ratio regardless of the sampling method used?

I have an imbalanced dataset where the target class is <1% of sample. I apply oversampling or undersampling e.g. https://github.com/scikit-learn-contrib/imbalanced-learn. I run random forest on ...
2
votes
1answer
440 views

Does oversampling affect sensitivity

I learned that sensitivity is not affected by oversampling since it identifies the proportions of correct classification for positive events. However, I also learned that by doing oversampling of ...
1
vote
1answer
320 views

Assessing classification error with synthetic oversampling (SMOTE, ADASYN, et)

Consider a situation, where there are two unbalanced classes (n1 < n2). Some standard statistical methods advise to use SMOTE (or similar) oversampling methods to balance classes and train a ...
17
votes
1answer
11k views

Opinions about Oversampling in general, and the SMOTE algorithm in particular [closed]

What is your opinion about oversampling in classification in general, and the SMOTE algorithm in particular? Why would we not just apply a cost/penalty to adjust for imbalance in class data and any ...
0
votes
1answer
141 views

Does under/over sampling lead to data leakage? [closed]

I could not imagine how can we applied over/under sampling in practice. Let's say, a client gave me 1 million of labeled samples, and 1000 of unlabeled samples to classify. There are two classes <...
0
votes
1answer
1k views

How to exact prediction from over sampled data(Undoing oversampling)?

We are oversampling the data to use in logistic regression. Aim is to predict CTR(click probability) which is rare event scenario. I have predicted the probabilities of click but CTR results are ...
0
votes
1answer
639 views

Applying SMOTE and PCA to high dimensional data giving low accuracy

I have a high dimensional datasets of around 2300+ columns. The dataset consist of two class labels of which one is extremely biased and occurs less than 10%. I looked at the various algorithms and ...
5
votes
1answer
53 views

Does it makes sense to perturb images on-the-run while training CNNs?

I'm training a convolutional neural network by adding small perturbations (like rotation and shifting) to the images each time it gets a batch data. I think a better way of doing this could be to ...