Questions tagged [down-sample]
Using aggregate data (e.g. monthly) when data on a finer scale (e.g. daily) is available.
24
questions
1
vote
0
answers
43
views
Reducing Sample Size from Data Set
I have a large data set where the number of unit is around 300,000 (given that the original data is in a panel setting, the number of observations is larger than 300,000). For the efficiency of ...
- 151
0
votes
1
answer
101
views
Should models built using under-sampled data be evaluated against the population
I have a dataset of 11 mil. rows with a 1:10 ratio between minority and majority classes.
To train a model, I have selected all the minority class members and 1/3 of the majority class.
The ratio is ...
- 1
0
votes
0
answers
34
views
Is balancing class data for imbalanced problems helpful or just folklore when considering thresholds?
(In the context of predictive models) Caveat: I'm aware that imbalanced data questions are a dead horse, but I haven't found an answer to this flavor of it directly.
When working with highly ...
- 278
0
votes
0
answers
9
views
Reduce size of sample with very high data content
I am analysing cells at 10 different timepoints (T0-T9).
...
- 1
1
vote
1
answer
50
views
Deterministic time-aggregation of principal component factors. Is it wrong?
I have estimated the first factor/score using PCA on a set of 190 monthly timeseries. For my analysis I also need the quarterly factor. Two choices come to mind:
Take the 3-month average of the ...
- 31
4
votes
1
answer
123
views
Importance of a Data Point in Regression
I am doing a gaussian process regression. This regression doesnt scale well as it grows $\mathcal{O}^3$. I would like to know if there are any methods that can be used to determine the importance of ...
- 175
4
votes
1
answer
443
views
correcting for extremely downsampled data: keras class_weight is hurting my model
I have an extremely imbalanced dataset (millions of times more negatives) for a binary classification NN model. I am aggressively downsampling solely for the purpose of making training time manageable,...
- 256
0
votes
2
answers
122
views
Downscaling climate data with a random forest model
I want to downscale climate data from CHELSA, which pixels are around 800 m long, to a 500x500 m or 50x50 m grid. I was thinking to perform a random forest model for each variable, in which I could ...
- 11
2
votes
0
answers
126
views
Forecasting monthly stock returns with daily data and down-sampling concerns
I have daily stock return data (log returns). I want to forecast returns for the next two months. I am creating forecasts with both univariate ARIMA and GARCH models with regressors.
What are the ...
- 33
0
votes
2
answers
2k
views
Sampling highly imbalance multi-class response variable
I have a dataset (11000 x 117) with response variable having multiple classes.
Here is a plot of class distribution:
Some of the classes have only 1 sample in the entire dataset and some have 2, 3 ...
- 123
0
votes
1
answer
199
views
Do ANOVA & MANOVA require balanced levels?
I have one independent variable with two levels, or categories, and four dependent variables.
When doing a MANOVA or ANOVA test, how important is it for the number of observations in each category to ...
- 315
1
vote
0
answers
162
views
While dealing with imbalanced classes, to what extent can we upsample a minority class? [duplicate]
I have my training data with the following approximate distribution:
Negative events : 90,000
positive events : 5,000
Training a model would require to oversample the minority class (and might also ...
- 85
1
vote
1
answer
158
views
Does downsampling decrease the entropy of the data?
Suppose we have an $n-dim$ time-series $X={x_1, x_2, \cdots, x_n}$ and we resample it to $m-dim$, $\hat{X}={\hat{x}_1, \hat{x}_2, \cdots, \hat{x}_m}$, where $m < n$.
Can we say this downsampling ...
- 486
1
vote
0
answers
91
views
should I resample/downsample the patients in the control arms?
I have retrospectively collected clinical data of two sets of patients, one set with the diagnosis of tumor A (group A) and the other with tumor B (group B). There're 90 patients in group A, and 1100 ...
- 11
1
vote
1
answer
1k
views
Is Tomek Link undersampling the same as Edited Nearest Neighbours with 1 neighbour?
From what I've read I've understood that undersampling the majority class with Tomek Links or Edited Nearest Neighbours with 1 neighbour should yield the same result. However, I've tried it on this ...
0
votes
1
answer
719
views
Statistics of bootstrapped subsampled data
Scenario
I have a treatment (T) and control (C) sample.
In my T sample, I have 600 observations, and in my C sample, I have 250 observations. Because the number of observations might be important ...
- 53
6
votes
1
answer
4k
views
Convert predicted probabilities after downsampling to actual probabilities in classification
If I use undersampling in case of an unbalanced binary target variable to train a model, the prediction method calculates probabilities under the assumption of a balanced data set. I discovered two ...
- 173
1
vote
1
answer
333
views
Kappa and downsampling, selection of data set
I have a unbalanced data set and use Cohen's kappa and AUC as performance measure.
Without down sampling the Kappa value is around 0.85, with random down sampling it is 0.95. and with a house-made ...
- 343
0
votes
1
answer
158
views
Correlation between vectors with non-matching y values without using interpolation
Is there a way to calculate the correlation between two time series that have been adaptively downsampled and thus (may) have different y values? This is easiest to explain with an example, so ...
1
vote
0
answers
323
views
Does downsampling affect regression results?
How is linear regression affected by downsampling the explanatory variable?
To be more precise, I would sort all the values of $x$, and then split into a a number bins with equal number of points in ...
- 1,584
2
votes
0
answers
751
views
How to use cross validation when you have missing data & rare events?
I am trying to use repeated cross validation to test my classifier. Moreover, I want to use imputation due to missing values and downsampling due to unbalanced data (I have 88% of my data in the ...
- 623
1
vote
0
answers
290
views
Bias from stratified sampling
Due to a lack of significance and the large size of the dataset (which had binomial responses with 20,000 responses out of a sample of 15,000,000) my peer has used random sampling to reduce the amount ...
- 11
1
vote
0
answers
2k
views
How to determine the correlation between data sets with the same period but different sample rates?
I am trying to determine the correlation between two sets of data points which span the same time period (20 minutes) but have different resolutions. The first set was recorded at 1-minute intervals, ...
- 11
3
votes
1
answer
236
views
Numerical differentiation (derivative) and downsampling
I have some time course data which I would like obtain the first derivative of. As it seems quite difficult to model, I do not intend to fit a function to it, but rather compute the first derivative ...
- 944