Skip to main content

Questions tagged [oversampling]

Sampling cases with differential probability, so that classes that occur rarely in the population occur more often in the training data. Does *not* address the problems in unbalanced classes.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

How can I apply multiple sampling techniques to a single dataset?

Related: How should I split my dataset if I am applying oversampling? Suppose I have a dataset called my_dataset.dat with three classes. This dataset has a class ...
user366312's user avatar
  • 2,195
0 votes
1 answer
116 views

How to find the correct sample size for a non parametric hypotheis test

I have two independent population data with each having 500 K rows. The goal is to perform a hypothesis test and test the claim that the new machine takes less time than the old machine. This 500 K ...
AKK's user avatar
  • 3
0 votes
0 answers
32 views

Unexpected distribution of scores after using class-weighted loss, when data is highly imbalanced (2%), low N and high p

I won't go into the way the data is built because I want to keep the discussion general. Relative to balancing, I couldn't find a lot of materials online about the results of cost-sensitive learning. ...
David Harar's user avatar
2 votes
1 answer
259 views

How to address bias in AI Image Recognition Model: Oversampling, Undersampling, and Ensemble Techniques Not Working

I am currently working on an image recognition project using AI, but I am facing challenges with bias in my model's predictions. The model seems to be biased toward the majority classes in my dataset. ...
Amsyar Nifail's user avatar
0 votes
0 answers
31 views

Can I use SMOTE technique to increase the number of observations in forecasting problem?

I have done my thesis on forecasting problem in advertisment industry. However, it resulted that the prediction accuracy for traditional econometric models (Lasso, Ridge) showed the better results ...
Amirgiano's user avatar
1 vote
0 answers
27 views

Is there actually a right and wrong way to deal with major imbalance in logistic regression (or other models, really)? [duplicate]

I have seen a lot of different advice on how to deal with imbalance, and I get that it can be case-specific. But I learned in school that SMOTE oversampling or undersampling were basically the ways ...
Siri C's user avatar
  • 11
1 vote
1 answer
86 views

Poor classification even after oversampling minority class

I want to predict mortality so minority class (dead=1) is important for me but my XGBoost model performing poorly for this class. In other words, the model performed the opposite of what I wanted. the ...
Nima Yousefi's user avatar
1 vote
2 answers
67 views

why does a model with a larger val loss produce higher accuracy than a model with a smaller val loss?

I did ANN classification on training data with oversampling and without oversampling. For each data, the smallest validation loss is sought with trial and error of 18 models. In the data without ...
andryan86's user avatar
  • 127
0 votes
0 answers
38 views

Accuracy on NN model decrease after random oversampling using library ROSE

I did random oversampling to handle unbalanced positive and negative data. When I didn't do random oversampling, the accuracy I got was 88%, when I oversampled the train data, it got 87% accuracy and ...
andryan86's user avatar
  • 127
0 votes
0 answers
39 views

ANN uses python smote random oversampling

I did ANN classification using SMOTE random sampling in python but I found strange plot loss and accuracy results. This is my code: ...
Andryan's user avatar
  • 47
3 votes
1 answer
81 views

What are the differences and common points, if any, between oversampling as a survey design method and oversampling in a machine learning context?

I've seen the term "oversampling" used in a survey design methodology context and in a machine learning context (e.g. methods like SMOTE). I'm intrigued by the differences between the two. ...
Kap's user avatar
  • 33
0 votes
1 answer
78 views

oversampling/undersampling

if you have a confusion matrix : actual 0 1 0 2658 204 1 24 110 y-axis = predicted x-axis = actual You ...
shawn's user avatar
  • 21
1 vote
0 answers
130 views

How to deal with probabiliy estimation error after fitting a model via Oversampling/undersampling?

I am confronted with the following issue. I fit a classification model for an unbalanced dataset with a binary target. The minority class is very less frequent than the majority class, therefore I ...
davide cassinari's user avatar
0 votes
2 answers
84 views

Data is unbalanced, but the train and test set don't represent it [duplicate]

I'm training a classifier with a binary target variable. My data is unbalanced. The problem is that the training data (split into train, val, and test sets) is more balanced than the real data (the ...
Amit S's user avatar
  • 57
2 votes
1 answer
507 views

Oversampling for Continuous Values

I am trying to predict the processing time of a process by using xgboost regression algorithm in python. However I realised that my samples data is skewed to left and my algorithm struggles to predict ...
Barış Oruç's user avatar
2 votes
0 answers
39 views

Problems due to sampling data from the same case patients multiple times?

Please imagine the following scenario: In a predictive modeling task for the probability that a hospital patient will encounter a certain medical event, all available data is split into cases and ...
Dohn Joe's user avatar
0 votes
0 answers
48 views

How to deal with oversampling and undersampling of countries to be able to compare them?

I have a dataset that looks like this: cc samples population % sample US 100 1,000 1% CA 20 100 20% BR 9 10 90% AU 600 300 200% I have samples from a certain population and the percentage of ...
Rebe's user avatar
  • 1
1 vote
0 answers
46 views

Undersampling and Oversampling

I've an unbalanced dataset. I need do to perform feature selection and then I'm going to fit my model. Is it conceptually wrong doing undersampling and perform feature selection and then once ...
lorenzlorg's user avatar
1 vote
0 answers
126 views

At what point in the ML pipeline should I under/over sample?

I have an imbalanced multi-class dataset, and am under/over sampling to balance it out. My questions is when should I do this resampling? Should it occur before creating the test set, before creating ...
jda5's user avatar
  • 175
1 vote
0 answers
1k views

Neural network for imbalanced data

I have an imbalanced data (n = 600, about 97% majority and 3% minority) with 20 features and a binary outcome. The data has been split into a training set and a test set (80%/20%). I used H2o autoML ...
user145331's user avatar
0 votes
0 answers
67 views

Statistical test when comparing oversampling to no oversampling on ANN

I use 70% of the dataset for training and 30% for testing. I use oversampling on the training dataset with an ANN. I use the test dataset on my ANN and look at the performance of oversampling against ...
Don_twice's user avatar
6 votes
1 answer
2k views

Uniformly sample from Gaussian distributed data

I have data that is roughly Gaussian distributed, bounded on a range of [x0, x1], w/ mean m and standard deviation ...
DankMasterDan's user avatar
2 votes
0 answers
88 views

Does threshold on the model probability depend upon the spread in the dataset among positive and negative classes (binary classification)?

I think that the threshold on model probability through which one discern positive (y=0) and negative(y=1) class depends on the spread in the training dataset b/w y=0 and y=1. This question came when ...
Mins's user avatar
  • 31
0 votes
0 answers
271 views

Oversampling in Longitudinal/Panel Data

Would make sense to apply any oversampling (e.g. SMOTE et similia) techniques in order to balance the outcome classes in the context of longitudinal/panel data? Wouldn't such procedures ignore the ...
joosefupas's user avatar
0 votes
0 answers
2k views

How to use SMOTE effectively in below case?

I have an imbalanced multiclass data set, where I am trying to apply SMOTE to synthesis data for minority class. The problem is, for many classes, I have only 1 sample. Even when I try to tackle it ...
Ahasanul Haque's user avatar
1 vote
0 answers
87 views

Improving synthetic oversampling with unlabelled data

I am working on a classification problem with a small amount of labelled data (~200 instances) and a larger sample of unlabelled data (~500 instances). To increase the size of the training data I am ...
A. Bollans's user avatar
0 votes
0 answers
100 views

ROSE acceptable dispersion/shrinkage

To solve imbalanced data, I used oversampling strategy using ROSE algorithm in Python. As you may know, ROSE is a smoothed bootstrapping method and we can control the dispersion of the augmented data. ...
Darren Christopher's user avatar
0 votes
0 answers
520 views

Threshold / Ratio to consider undersampling / oversampling

I have a classification task (predicting DNA methylation) with a somewhat unbalanced dataset - 38% of values are in the minority class, and the other 62% in the majority class. I have read that one ...
charelf's user avatar
  • 243
1 vote
0 answers
144 views

How to oversample when data needs to be labeled?

I am working on an NLP problem where we obtain labels through a Mechanical Turk–like system. We started with a random sample, experimented with models and determined that rebalancing the minority ...
Ken's user avatar
  • 83
0 votes
2 answers
89 views

How to choose between 2 strategies to train a Deep Learning model on an unbalanced Dataset?

I have a Training Set of respiratory disease sounds, so there are 2 classes: 0 for respiratory sounds of healthy patients. 1 for breathing sounds of patients with a disease. The Training Set is ...
Francesco Ladogana's user avatar
4 votes
1 answer
327 views

Getting different results when running SMOTE

I have this code which runs SMOTE and then getting roc_auc_score. The issue is that every I run the code on the same dataset, I get different results. How can I fix this? I need the same sample when ...
Eliza's user avatar
  • 41
0 votes
1 answer
276 views

Callibration after oversampling

We have build a credit scoring model with OptBinning library in python. In the process we oversampled the minority class and now we want to callibrate it back to ...
PalimPalim's user avatar
0 votes
0 answers
68 views

Random sampling and over sampling

I have binary $\{0,1\}$ classification data that I will use for statistical analysis. The data provider said that the data are randomly drawn from the population and at the same time he said samples ...
user1292919's user avatar
0 votes
0 answers
34 views

Is balancing class data for imbalanced problems helpful or just folklore when considering thresholds?

(In the context of predictive models) Caveat: I'm aware that imbalanced data questions are a dead horse, but I haven't found an answer to this flavor of it directly. When working with highly ...
Josh's user avatar
  • 308
3 votes
0 answers
347 views

How to determine the correct amount of oversampling (in regression)

For my regression problem I splitted my dataset into a stratified train- and testset (70:30) and I now want to train my models (random forest, gbm, logistic regression) using the trainset. The dataset ...
XeeD's user avatar
  • 31
0 votes
0 answers
298 views

Can I inverse the standardscaler after using SMOTE?

As it is written here, you should standardize the data before applying SMOTE. If I inverse the standardscaler action with inverse_transform after using SMOTE, will ...
Amit S's user avatar
  • 57
0 votes
0 answers
51 views

Is upsampling a tiny class before cross-validation valid?

I'm working with a dataset containing several classes. The largest class has over 500 samples, and the smallest classes have fewer than 10 samples. I know that you should perform upsampling inside the ...
Avelina's user avatar
  • 1,148
0 votes
0 answers
220 views

How to tune an weighted voting ensemble method?

I am working with a data with 5 unbalanced labels. These codes are contained of Normalization, Oversampling on Feature Engineering part. A list of 9 ordinary Machine Learning methods is provided ...
amin's user avatar
  • 176
0 votes
1 answer
272 views

Training with oversampling

I'm building a Random Forest model over an unbalaced 4 class dataset. So far I understood how to use oversampling and train my model. My doubt was about when to perform Oversampling. I've already seen ...
Mattia Surricchio's user avatar
0 votes
0 answers
599 views

How H2O perform class balancing?

I wanto to perform class balancing using h2o autoML. I know there is a parameter class_sampling_factors that allow to specify the under/over sampling factor for ...
A1010's user avatar
  • 213
0 votes
0 answers
221 views

Limits of oversampling

I have a dataset with an event rate of less than 0.3 percent. To improve the modeling results, I did some oversampling using SMOTE. I initially oversampled so that the event rate increases 10 times to ...
Clock Slave's user avatar
  • 1,087
1 vote
0 answers
128 views

Will oversampling help with generalization (small imbalanced dataset)?

I have an imbalanced dataset (2:1 ratio) with about 60 patients and 80 features. I performed RFE + stratified cross validation to reduce the features to 15 and I get an AUC of 0.9 with Logistic ...
Luis Pinto's user avatar
0 votes
0 answers
358 views

Classification on rare events (~%3) and only categorical variables

I need to build a model based on about 10 independent variables, all categorical (only two of which are potentially ordinal), to predict a dichotomous output ('1': 3%; '0': 97%). To overcome the ...
fx7sts's user avatar
  • 21
4 votes
1 answer
267 views

When is oversampling preferable to undersampling and vice versa?

When data is unbalanced, that is, when the distribution of classes being predicted is very uneven (e.g. 90%/10% for two classes or 10%/15%/75% for three classes), many machine learning models have ...
Tripartio's user avatar
  • 2,216
0 votes
2 answers
2k views

SMOTE in unbalanced dataset with binary features

after reading different posts about unbalanced datasets I didn't make my mind clear about my specific problem so that's why I'm posting a new question. In my case, I have a dataset with around 20K ...
Jose LHS's user avatar
3 votes
0 answers
3k views

Oversampling methods for numerical data (regression)

There are many oversampling methods for categorical labels (for example SMOTE and Rose, etc.). But, are there oversampling method for numerical labels (the thing that I want to predict with my ...
jennifer ruurs's user avatar
1 vote
1 answer
372 views

Is doing oversampling on train set and undersampling on test set correct?

I have an imbalanced dataset (95% in class 0 and 5% in class 1) and I am using machine learning for classification. The AUC(Area under ROC curve) was high (about 0.86) but AUPRC(Area under precision-...
user229019's user avatar
5 votes
3 answers
8k views

Does oversampling/undersampling change the distribution of the data?

I have an imbalanced dataset (10000 positives and 300 negatives) and have divided this into train and test sets. I perform oversampling/undersampling only on the train set since doing this on the test ...
Anuj's user avatar
  • 59
1 vote
1 answer
123 views

How do i re-train a final model after using oversampling?

I am a bit puzzled about the process of experimenting with a model and oversampling and then translating it to the final version of the model that will be used: I oversample the data (only the ...
Emil Filipov's user avatar
1 vote
0 answers
174 views

SMOTE and Lagged Observations

I'm doing a project about the effect of synthetic oversampling in a machine learning context (more precise SMOTE for the oversampling of the minority class of a highly imbalanced target variable). The ...
mabreitling's user avatar