Questions tagged [oversampling]

Sampling cases with differential probability, so that classes that occur rarely in the population occur more often in the training data. Does *not* address the problems in unbalanced classes.

Filter by
Sorted by
Tagged with
1
vote
0answers
40 views

Will oversampling help with generalization (small imbalanced dataset)?

I have an imbalanced dataset (2:1 ratio) with about 60 patients and 80 features. I performed RFE + stratified cross validation to reduce the features to 15 and I get an AUC of 0.9 with Logistic ...
0
votes
0answers
33 views

Classification on rare events (~%3) and only categorical variables

I need to build a model based on about 10 independent variables, all categorical (only two of which are potentially ordinal), to predict a dichotomous output ('1': 3%; '0': 97%). To overcome the ...
0
votes
0answers
20 views

Oversampling for imbalanced time series classification

I'm doing multivariate time series classification (two classes) with GRU/LSTM models. Each observation is a multivariate time series with one label (0 or 1). But the two classes are highly imbalanced. ...
0
votes
0answers
17 views

Is it possible to have overfitting within the first epoch of training?

Usually after training a few epochs we have overfitting and stop the training. But, is there any circumstances or is it possible that overfitting happens within the first epoch of training? Maybe ...
1
vote
0answers
18 views

When is oversampling preferable to undersampling and vice versa?

When data is unbalanced, that is, when the distribution of classes being predicted is very uneven (e.g. 90%/10% for two classes or 10%/15%/75% for three classes), many machine learning models have ...
0
votes
1answer
68 views

SMOTE in unbalanced dataset with binary features

after reading different posts about unbalanced datasets I didn't make my mind clear about my specific problem so that's why I'm posting a new question. In my case, I have a dataset with around 20K ...
1
vote
1answer
62 views

Oversampling methods for numerical data (regression)

There are many oversampling methods for categorical labels (for example SMOTE and Rose, etc.). But, are there oversampling method for numerical labels (the thing that I want to predict with my ...
0
votes
0answers
32 views

Is doing oversampling on train set and undersampling on test set correct?

I have an imbalanced dataset (95% in class 0 and 5% in class 1) and I am using machine learning for classification. The AUC(Area under ROC curve) was high (about 0.86) but AUPRC(Area under precision-...
1
vote
3answers
377 views

Does oversampling/undersampling change the distribution of the data?

I have an imbalanced dataset (10000 positives and 300 negatives) and have divided this into train and test sets. I perform oversampling/undersampling only on the train set since doing this on the test ...
1
vote
1answer
22 views

How do i re-train a final model after using oversampling?

I am a bit puzzled about the process of experimenting with a model and oversampling and then translating it to the final version of the model that will be used: I oversample the data (only the ...
1
vote
0answers
32 views

SMOTE and Lagged Observations

I'm doing a project about the effect of synthetic oversampling in a machine learning context (more precise SMOTE for the oversampling of the minority class of a highly imbalanced target variable). The ...
0
votes
1answer
126 views

Oversampling a multi-labeled data set

Given a data set where each individual data point can be assigned to more than 1 class (a multi-class, multi-label data set), are there any guidelines for calculating oversampling weights, i.e., the ...
1
vote
3answers
400 views

Binary Classification in Imbalanced Data; Oversampling and Imputation

Together with two friends I participate in a university course about data mining in R and we chose the topic of bankruptcy prediction. We started with some "clean" data found on an "In class" kaggle ...
0
votes
0answers
76 views

Run time of SMOTE function in package DMwR

I have a dataframe with 930 000 rows and 220 variables. The objective is a binary classification but my response classes are imbalanced. (88% - 12%) I want to use SMOTE to artificially create ...
1
vote
1answer
44 views

Running SMOTE on unbalanced data

I've read a few answers to similar questions that advise the completion of SMOTE after splitting the data set into Train and Test sets however, the documentation and other examples I've seen run SMOTE ...
0
votes
0answers
28 views

Resampling methods for curves and time series

In the case of imbalanced datasets, different oversampling/downsampling methods exist such as SMOTE, ADASYN, etc. However, this methods mostly simply interpolate in the feature space, treating the ...
0
votes
0answers
24 views

How to do post-stratification weighting in sampled groups with low or zero cell populations?

Let's say you received data from a sample. You don't have information on the sample design, but you have population data so you want to employ post-stratification weights to better align the sample ...
1
vote
1answer
198 views

Effects of Oversampling on Linear Regression

We often see that oversampling is a useful technique to combat skewed classes. But there is not much discussion about linear regression. If we just duplicate the original data, and train the linear ...
1
vote
0answers
86 views

influence of oversampling on Semi-supervised multi-label learning

I have suggested a semi-supervised approach for the hierarchical multi-label classification task. I have included the MLSMOTE oversampling technique as a pre-processing step, and then evaluate the ...
0
votes
0answers
21 views

Balancing continuous covariates for oversampling

I'm currently looking into methods for restoring balance of a biased dataset with respect to a continuous variable. My problem is similar to this question, with the slight difference that I'm dealing ...
2
votes
0answers
236 views

Logistic Regression Class Imbalance and the use of weighting and undersampling

I have been working on a machine learning model using Spark (binomial) LogisticRegression. The dataset has what I think is a high degree of imbalance - roughly 1% of rows are labelled as events. The ...
0
votes
0answers
13 views

Analyzing imbalanced data using linear and logistic regression [duplicate]

According to Wikipedia page on Oversampling and undersampling in data analysis: The end-result of over-/under-sampling is the creation of a balanced dataset. Many machine-learning techniques, such ...
0
votes
0answers
14 views

oversampling data with subclass

Oversampling of under-represented data is a way to combat class imbalance. For example, if we have a training data set with 100 data points of class A and 1000 data points of class B, we can over ...
0
votes
1answer
82 views

Adjusting precision recall curve for oversampling

I built a model for a binary target using oversampled data. The population target prevalence is 0.25. I oversampled to 0.5 by keeping the entirety of the minority class and sampling a portion of the ...
1
vote
3answers
4k views

Running XGBoost with *highly* imbalanced data returns near 0% true positive rate. Tried SMOTE and it did not improve much. What else can I do?

I'm using XGBoost on a dataset of ~2.8M records of hard drive failures, where less than 200 are tagged as failures. After cleaning, there are 11 features in this dataset. Below is my ...
1
vote
0answers
9 views

Oversampling to Create Correlated Feature from Existing Feature

For demo purposes in Python, I would like to create an array or pandas df column that shares a high degree of correlation with an existing array. This is not as easy as it sounds and I don't think it ...
0
votes
0answers
66 views

Imbalanced class SVM prediction results using different validation data

I am trying to fit my data to a classifier using SVM. My data has 2 classes, the positive class which occurs with a probability of 0.002 and the negative class which is the dominant one. Suppose that ...
1
vote
0answers
67 views

While dealing with imbalanced classes, to what extent can we upsample a minority class? [duplicate]

I have my training data with the following approximate distribution: Negative events : 90,000 positive events : 5,000 Training a model would require to oversample the minority class (and might also ...
0
votes
0answers
296 views

Oversampling using SMOTE leading to bad predictions on test set

I have a dataset with an imbalanced binary target. One class accounts for about 94 % of the target variable. I used SMOTE to oversample the minority class but after the oversampling step when I train ...
6
votes
1answer
2k views

Normalization/standardization: Should one do this before oversampling/undersampling the data or after?

When working with imbalanced datasets, should one do one-hot encoding and data standardization before or after sampling techniques (such as oversampling or undersampling)?
1
vote
1answer
597 views

How to report the results of a cross-validation to a paper: Can I manually select the best results?

Good morning. With regard to the interpretation of the results of the N-fold cross-validation with over-sampling (SMOTE), I raise the question to write up the paper. Currently, I have performed 5-...
0
votes
1answer
302 views

Oversampling problems in prediction

I have a dataset that contains 284315 samples of class 0 and 492 of class 1. I know, that's huge. I heard about oversampling methods, so I did the following using the RandomOverSampler library: ...
22
votes
0answers
459 views

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? [duplicate]

TL;DR See title. Motivation I am hoping for a canonical answer along the lines of "(1) No, (2) Not applicable, because (1)", which we can use to close many wrong questions about unbalanced datasets ...
1
vote
1answer
254 views

Random Over Sampling to handle Data Imbalance

Was reading an article on imbalanced datasets where the event occurs and look at balancing the dataset. In that article, the event records were 2% of the total records. The author of the blog ...
1
vote
0answers
83 views

Bias-Variance Tradeoff when using Oversampling Technique

Oversampling techniques (e.g. SMOTE) are often used when target values are not approximately equally represented. How does this technique affect bias and variance of the predictive model that is ...
5
votes
1answer
1k views

Oversampling: whole set or training set

I have a rather small dataset of 4 000 points (140 features) to feed to a NN binary classifier. The problem is only ~700 of them represent the second class. Is it more common to resample the whole ...
1
vote
0answers
498 views

random forest imbalanced data-over, under, Smote Sampling

I am using random forest model for an imbalanced dataset. The dependent variable is Yes=73, No=7100. I have 65 independent variables both factor and numeric. I have tried to develop models for ...
3
votes
2answers
79 views

Using patient characteristics to predict disease and confront with per hospital real cases

We have a large, nationwide prevalence study about the prevalence of healthcare-associated infections (HAI). We need to see if hospitals have more or less HAI than expected by their patients' ...
0
votes
1answer
238 views

Modeling Attrition: Imbalanced Training

First of all, let me say I am new to Machine Learning and am eager for any sort of feedback. I am attempting to create a predictive attrition model, and my training and test data each have ~ 17% ...
0
votes
1answer
127 views

oversampling in nested cross validation

Introduction I have a small mixed dataset consisting of continuous and categorical independent variables with a dichotomous dependant variable. I'm running various algorithms (neural networks, random ...
4
votes
1answer
389 views

During oversampling of rare events, why are the beta coefficients of the independent variables not affected, but only the intercept?

I have followed the King and Zeng paper and understand the consistency of the prior correction after oversampling in logistic regression. But I am trying to understand why the beta coefficients of the ...
1
vote
0answers
54 views

The problem between over-sampled, down-sampled data and original data using SVM with unbalanced data

I am working on a classifier that is supposed to do binary classification on a dataset includes 5996 examples. ~800 of this examples belongs to class 1 and the rest is class 0. Since there is a huge ...
0
votes
1answer
95 views

How to interpret results of a predictive model when an external factors leads to imbalanced target labels

I have a prediction task, in which I use DecisionTreeRegressor of scikit-learn to predict a target label, which is about a certain user behaviour in a web platform (...
1
vote
1answer
91 views

Oversampling - Does not contradict for the requirement for Independence of Observations?

A requirement for many statistical and machine learning prediction models is the independence of observations. However, when using oversampling we reintroduce the same observations in the dataset. ...
3
votes
0answers
1k views

Should SMOTE oversampling be done before or after holdout validation's training/testing split? [duplicate]

Originally, without SMOTE, my ML learning steps go like this: Feature vectorization split data into X_train, X_test, y_train, and y_test use X_train and y_train for machine learning predict/test ...
0
votes
1answer
435 views

Should I first oversample or standardize (when cross-validating on imbalanced data)?

I have an imbalanced (two-class) classification dataset, based on which I am trying to train and cross-validate a classifier. During the process of the k-fold cross-validation, I set aside the test ...
2
votes
2answers
689 views

Balance classes when sampling

I have a large dataset describing numerous customers' behaviour and I am trying to solve a binary classification problem with a null accuracy on 90% (90/10 distribution amongst the two classes). ...
2
votes
0answers
73 views

What are the recommended over-sampling or under-sampling techniques to handle multi-class imbalanced datasets for machine learning? [duplicate]

Many techniques are for two-class classification. What is the recommended techniques to over- or under-sample multi-class datasets?
1
vote
0answers
67 views

Why over/under-sampling could not help my model fitting?

I fit random forest to my imbalanced dataset with minority class 1. I found that the AUC under the imbalanced data was better than that of re-sampled dataset (over/under sampling). Can someone help to ...
1
vote
0answers
318 views

Correcting Bias

I have a data set that includes locations of where certain rocks were observed on the Earth. Populated areas have a higher number of observations in general. Remote areas have less observations. I'm ...