I'm investigating the possibility of creating a non parametric bootstrap null distribution for hypothesis testing in multivariable regression analysis.

The null hypothesis is based on the absence of relationship between predictors and outcome. That is, the $\beta$ coefficient of all the predictors is 0. To do this I need to destroy the association between the predictors and the outcome by reshuffling the values in the variables.

First I need to choose a method to reshuffle the variables. I thought of several methods to achieve this and I would like your opinion on them (NB: names invented by me):

  • Global permutation: Permute the variable BEFORE bootstrap resampling of observations.
  • Local permutation: Permute the variable AFTER bootstrap resampling of observations.
  • Global resample: Resample with replacement the variable BEFORE apply bootstrap resampling to the data set.
  • Local resample: Like the previous but resampling is performed AFTER bootstrap resampling.
  • Uniform randomization: The variable is resampled giving each unique value an equal probability.

The global methods assures that the permutation/resampling will be applied on all values present in the variable; with the local methods the permutation/resampling will be applied only to the subset of unique values still present after the bootstrap resample of your dataset (~63.2% of the original values if I'm not wrong). With the permutation methods we will end up with the same values' distribution as in the original variable, while with resampling with replacement method the reshuffled variable will have a slightly different probability distribution. The uniform method instead will eliminate the original distribution of values in the variable allowing each modality of the variable to be present with the same probability.

For example, let's take a categorical bimodal variable, say SEX, with a M:20%, F:80% distribution:

  • with permutation methods you will still get M:20%, F:80% in each iteration;
  • with the resample methods you will get values around the original ones e.g M:23%, F:77%;
  • with the uniform method you will get values around M:50%, F:50%.

I created an example on a dataset of us with ~380 observations regarding the association between some mutated genes and a syndrome, adjusted for sex. Reshuffling was applied on the outcome.

Here's the R code:

if (permutation == 'permute.global') {
       data[,1] <- sample(data[,1])
       data <- data[i,]
} else if (permutation == 'permute.local') {
        data <- data[i,]
        data[,1] <- sample(data[,1])
} else if (permutation == 'replace.global') {
        data[,1] <- sample(data[,1], replace = T)
        data <- data[i,]
} else if (permutation == 'replace.local') {
        data <- data[i,]
        data[,1] <- sample(data[,1], replace = T)
} else if (permutation == 'uniform') {
        data[,1] <- sample(unique(data[,1]), length(data[,1]), replace = T)
        data <- data[i,]
else  data <- data[i,]

The i variable contains the indexes for resampling the original dataset, generated by the bootstrap function (not shown).

Then I plotted the distributions: enter image description here

The red curve is the density curve of the original (not null) distribution of the regression coefficient. The yellow and the green ones are the permutation based density curves (global and local), and the cyan and blue ones are the resampling based ones. The violet arise from the uniform distribution method. It's clear that there's not much difference between permutation and resampling method, while instead doing the reshuffling prior or after (global/local methods) bootstrap resampling has an effect. The violet (uniform distribution) density curve stands apart, being almost normal in shape and with 0 mean.

Given this data, which is the theoretical meaning of each method? which should be used for null hypothesis testing?

Second, I need to understand which variable is to be reshuffled: the outcome, each predictor singularly, all predictors together, both predictors and outcome? By reshuffling just the outcome you keep the correlation structure between predictor unaltered and I don't know whether it's the right thing to do or not. Reshuffling each predictor separately could be computationally cumbersome, since I would need to run the bootstrap analysis for each predictor.

What are your opinions in this regard?

  • $\begingroup$ No contributions? $\endgroup$ – Bakaburg Nov 10 '15 at 16:42
  • $\begingroup$ The question has been completely ignored. Is there a way I can improve it? $\endgroup$ – Bakaburg Nov 23 '15 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.