I am doing two independent bootstrap shuffles on my data set, and any result (from my original dataset) that is greater than 95% of each of my shuffles (independently) is considered to be significant.

Is my final significance rating 0.05^2?

  • $\begingroup$ Do you mean that you are bootstrapping a null distribution twice from the same data? $\endgroup$ – Ruben van Bergen Aug 3 '17 at 13:04
  • $\begingroup$ Same data, but both times I shuffle different features. Ex: If my data has feature A and B, both are independent, and describe my observations: first time I am shuffling feature A keeping feature B unshuffled; second time I shuffle feature B keeping feature A unshuffled. $\endgroup$ – poultrynews Aug 3 '17 at 13:07
  • $\begingroup$ What is the null hypothesis you are testing? $\endgroup$ – Stephan Kolassa Aug 3 '17 at 13:56
  • $\begingroup$ I have sequential observations, and I want to check if my sequential observations are significantly different from random. My null hypothesis is that the sequence observed in my observation is random. The sequence of my observation depends on two features: A and B, both of which are independent. My two bootstraps shuffle my observations according to A and B (independently). Observations that are more sequential than 95% of shuffles A and B are considered significant. $\endgroup$ – poultrynews Aug 3 '17 at 14:19

By the definition of independence you would have a significance of $0.025$ only if you have independence between the two significance tests. Depending on the hypothesis testing scenario this may or may not be true.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.