Questions tagged [resampling]

Resampling is taking a sample from a sample. Common uses are jackknifing (taking a subsample, eg all values but 1) & bootstrapping (sampling w/ replacement). These techniques can provide a robust estimate of a sampling distribution when it would be difficult or impossible to derive analytically.

Filter by
Sorted by
Tagged with
5
votes
1answer
179 views

Why are all the permutations of i.i.d. samples from a continuous distribution equally likely?

Suppose $X$ is i.i.d from a continuous distribution Why is$$P(X_{i_1}<X_{i_2}<\cdots<X_{i_3})=P(X_{j_1}<X_{j_2}<\cdots<X_{j_3})=\frac{1}{n!}$$for all $i,j$? I think we can reason ...
1
vote
1answer
22 views

Running SMOTE on unbalanced data

I've read a few answers to similar questions that advise the completion of SMOTE after splitting the data set into Train and Test sets however, the documentation and other examples I've seen run SMOTE ...
0
votes
0answers
39 views

R-Caret, Regression, different number of PCA components for finalModel and resampling with PCA?

In my project I train models with the "Timeslice" method from the caret package but this question also fits in with other methods, such as cross-validation. Imagine you have 2584 records and split ...
0
votes
0answers
24 views

Which statistics elective courses should I take? [closed]

I don't know much about these topics. Which are most important? (for either academia or industry/data science) I can choose 2-3 courses from here for my undergrad degree. Introduction to Re-sampling ...
3
votes
1answer
324 views

How to choose resample size when drawing without replacement?

Say I have some second-order statistic $m(x)$ where $x$ is a data vector of length $n$. Let's also assume that the limiting distribution of $x$ is gaussian-ish, but generally unknown, so that the ...
3
votes
1answer
40 views

How to compute confidence intervals from *weighted* samples?

Imagine we have a webserver, which serves a total of N static URLS. There are users visiting the URLs every day. At the end of each day, we have data like this: ...
1
vote
2answers
256 views

Reduce 'oversampling' when using bootstrapping

I'm wanting to use bootstrapping to estimate the sampling distribution of values over a geographical area. However, the samples I'm using are more heavily sampled in some areas that in others, for ...
1
vote
0answers
11 views

Downsampling, AUROC and accuracy equal

I am using downsampling to create perfectly balanced classes in my target feature. I have found that accuracy is exactly equal to the AUROC score. I was thinking that this is because I've used ...
0
votes
0answers
14 views

Nested cross-validation with resampling

I am aware that sckit-learn has a provision to perform nested cross-validation. I wanted to perform the same with resampling. My confusion stems from whether their example listed here performs ...
0
votes
0answers
12 views

When is the bias of a statistic of the form a/n + b/n^2 + c/n^3 +

In many books the bias-correction of the Jackknife resampling method is being prooved under the assumption, that the bias has a special form, namely a/n + b/n^2 + c/n^3 * ... Sometimes it's written "...
0
votes
0answers
26 views

KDE that better-preserves percentile distributions

My understanding is that a Gaussian KDE, because the kernel is symmetric, preserves the statistical mean of a distribution. For my particular case, I'd really prefer a KDE that preserved the median, ...
0
votes
0answers
17 views

Correct approach to define a null distribution by resampling?

I am trying to devise a test for significance by resampling from my data. The data consists of 60 individuals, grouped in 6 populations (10 per population). There are about 9,000 genes as ...
2
votes
0answers
29 views

Central limit theorem for resampling

What is the analog of the central limit theorem or concentration theorem for resampling, say, an i.i.d. samples? Are there any references for this topic? As a simple example, suppose there are $n$ i....
7
votes
2answers
11k views
20
votes
2answers
7k views

what are the assumptions of permutation test?

It's often stated that permutation tests have no assumptions, however this is certainly not true. For example if my samples are somehow correlated, I can imagine that permuting their labels would not ...
0
votes
0answers
25 views

Is there any way to resample in order to change the distribution of a sample?

I ran an experiment that involved an independent variable which is known to have a right skewed distribution. However, in the experiment, the independent variable was distributed differently between ...
0
votes
0answers
9 views

Generating new samples from dataset to expand dataset

I want to choose one dataset and then expand the dimensions/number of samples to show how a dimension reduction method(not yet decided) reacts to changes in dimensions/number of samples. My plan was ...
0
votes
0answers
18 views

Can one construct “original” data from a function of jackknifed data?

Say I have original, uncorrelated data, $x_i$, with $i = 1,2 \ldots N$. I can jackknife this data set (a simple delete-one) $$ \bar{x}_{i} = \frac{1}{N-1}\sum_{j \neq i}x_{j} \quad\quad (1) $$ to ...
0
votes
1answer
10 views

Resample dataframe by random subset of years

I have a dataset of 59 years worth of daily rainfall data. I'd like to resample the data by 6 randomly selected years 1,000 times, using the entire year of data for each of the 6 years (so 2190 ...
1
vote
0answers
42 views

How to calculate p-value comparing bootstrap-based predicted probabilities and observed probabilities

I posted this question on stackoverflow first but I have got no answer so far, so I decided to post it here in the hope that here I might get an answer. I hope my procedure is acceptable. ...
1
vote
0answers
34 views

When to use ordinary, balanced, antithetic, or permutation resampling for bootstrap?

I am using boot and would appreciate any explanation as to when using each of these resampling methods would be recommended in practice. I have come across Do who says that: Simulation results ...
1
vote
0answers
27 views

How does one estimate error of extrapolated value with only 2 data points?

I am trying to do a simple linear regression. I have only two data points with errors. How do I estimate the y-intercept with errors? Currently I have: ...
2
votes
0answers
25 views

Correlating Two Time Series with Gaps in Data and of Different Lengths

I am attempting to correlate the time series from 4 separate tilt monitors that sample every 5 minutes. The time series have slightly different base times and end times, and the resulting arrays are ...
1
vote
1answer
1k views

Resampling correlated data using bootstrap

I have a dataset of $n×m$ numbers, where the $m$ variables ($m$ is in the order of 5–10) exhibit various degrees of correlation with one another. The format of the data is not necessarily a timeseries,...
2
votes
1answer
117 views

Bootstrap, Rubin's rules, and uncertainty of sub-estimates?

Can someone provide an intuition for why, when using bootstrap to calculate the variability of an estimate (say a regression coefficient $\beta$) we don't need to incorporate the uncertainty of each ...
3
votes
1answer
199 views

Selecting uncorrelated samples from a set of bulk data that contains correlated and dependent samples

i have a set of data that is generated by expensive computational model evaluations, on a total data set of 10000 samples in 40 dimensions. This sample data set is composed of different data sets, ...
0
votes
0answers
14 views

Appropriate visualization and statistical testing using resampling stats

I'm using a linear SVM to perform classification on a dataset. It seems to me that there are many ways to visualize my results and report their statistical significance, and I'm unsure of best ...
0
votes
0answers
30 views

Adjusting predicted probabilities after resampling

Suppose I've got highly imbalanced data and I want to train a model, for binary classification. So I upsample the minority class or downsample the majority class or whatever. My question is whether ...
0
votes
0answers
44 views

Permutation tests for paired data with several variables

I am trying to answer my own previous question, with a permutation method. I have a reference method $M_{ref}$ that I would like to compare with other methods $M_1$, $M_2$, $M_3$ when taking ...
1
vote
0answers
47 views

Calibration curve for mixed model (logistic) [closed]

I did some searching using the CV search function and was unable to locate any information on R packages or general approach to creating a calibration curve with a bootstrapped curve overlay (similar ...
1
vote
2answers
111 views

How should I resample the training and testing set with imbalanced data whilst having meaningful performance metrics?

I have an imbalanced dataset of approx. 200 positive and 800 negative examples. I run nested cross-validation where i=5 and j=5; (i is inner and j is outer). The cross-validation procedure isn't the ...
1
vote
0answers
21 views

Bootstrap test of means using pooled groups - is there a problem?

In his book "Resampling: the New Statistics" (available for free online), Julian Simon presents several examples where he Bootstraps the differences in group means, assuming that the observations of ...
1
vote
0answers
173 views

Monte Carlo testing: number of required permutations

I want to perform a statistical hypothesis test, however I don't know the exact distribution of my test statistic under $H_0$. Therefore, I need to calculate a Monte Carlo estimate $\hat{p}$ of the ...
0
votes
0answers
12 views

Adding Bad Events from the past to the risk default model to avoid Down/Up sampling techniques

We have been trying to build a classification model for credit default prediction using two different models one being Random forest and another being the Logistic regression based scorecard model. ...
0
votes
0answers
93 views

how to use boot package to do stratified bootstrapping?

Here's a toy data set that replicates my problem. I am interested in knowing the confidence intervals of an empirical distribution that is composed of the scores of each school at the proportion that ...
0
votes
0answers
9 views

Hypothetical sample size

I have a data set of n=98 of values that is normal distributed. This data set has a certain spread and hence a certain standard error of the mean. I now want to test how many samples (ns) I would need ...
1
vote
0answers
26 views

Practical questions about cluster bootstrap confidence intervals

I want to estimate the accuracy of a machine learning model. I have an independent test set with a vector of trusted labels and a corresponding vector of model-based predictions. If I assume the test ...
0
votes
0answers
25 views

How to approximate continuous function from discrete uneven interval value?

To clarify, I have a discrete value of an unknown distribution. The value of this distribution domain is from 0 to 1. But the value which the discrete value is sampled has an unknown interval. For ...
1
vote
1answer
470 views

Is Tomek Link undersampling the same as Edited Nearest Neighbours with 1 neighbour?

From what I've read I've understood that undersampling the majority class with Tomek Links or Edited Nearest Neighbours with 1 neighbour should yield the same result. However, I've tried it on this ...
1
vote
2answers
1k views

What are RMSE SD and Rsquared SD metrics in resampling results using R package:caret?

I've been doing predictive modelling with R package caret. When resampling regression models, I get the traditional RMSE and Rsquared metrics, but also RMSE SD and ...
0
votes
0answers
29 views

General way to construct a confidence interval for a unknown constant to which a sample estimator converges

Assuming that a sample estimator converges to some unknown constant (a wild assumption to be sure) and without assuming the distribution of either the sample estimator or the variables from which it ...
0
votes
1answer
69 views

ROC and PR curves after over/under sampling in Unbalanced datasets

As I understood till now, ROC curves are not a good presentation of unbalanced datasets and PR curves are preferred because ROC curves are not sensitive to false positives. If we now use resample ...
3
votes
1answer
370 views

How can we explain the fact that “Bagging reduces the variance while retaining the bias” mathematically?

I am able to understand the intution behind saying that "Bagging reduces the variance while retaining the bias". What is the mathematically principle behind this intution? I checked with few experts ...
2
votes
0answers
59 views

Application of Permutation Tests

I don't know how to post the question more formally. Therefore, let me introduce an example. Suppose you want to estimate the following regression: $n_i = f(x_i)+\beta \cdot 1[x_i = j]$, $n_i$ is ...
0
votes
0answers
12 views

Why with probability close to 1 bootstrap and classical tests provide the same decision? Is it caused by loss of pivotality? How?

Here is an example on which my question is based can please someone tell me what is the relation between the probability close to one and the fact that we have same decisions as classical tests and ...
2
votes
1answer
85 views

Sample size calculation for Kendall's Tau in reproducibility study

Can anyone help with justifying (or rubbishing) a couple of aspects of a sample size calculation for a grant proposal. A pilot study gave significant results for the correlation of a continuous (non-...
0
votes
0answers
32 views

How to interpolate/resample both dense and sparse points?

Suppose I have data like red points below I would like to interpolate/resample these points at black ticks. At right the points are sparse and it is obvious to interpolate them linearly or with ...
0
votes
1answer
32 views

Dealing with dataset imbalance: test if adjusting is necessary

I'm currently working on a project which uses a imbalanced dataset (two classes) for training, and I'm not sure if I should do a resampling procedure or not. Is there a way to actually test if it's ...
1
vote
0answers
44 views

How to bootstrap data with unknown correlation structure?

I am interested in how to validly bootstrap data with an unknown correlation structure. Let's say I am bootstrapping in order to obtain inference for some smooth function of the data similar to a ...
1
vote
2answers
217 views

Mean of the sampling distribution of the OLS estimator

Suppose b represents the OLS estimator, and B the true coefficient in the regression model without intercept y = Bx + u. Under certain assumptions b is unbiased ...