If I have understood correctly, Null Hypothesis Significance Testing is a device, which eats data and a conservative guess, and outputs a 'probability' that the data was generated by the hypothesized process.

In that, H0 can produce any p-value, while alternative hypotheses are probable only at low p-values. That's why we are encouraged to repeat an experiment in which we think we have found an effect.

Now for the question. Suppose we have 100 measurements and no way to obtain more. Let approach A be to input the in a NHST device and compare the p-value with the significance level. Let approach B be to split the sample into two parts (with random selection, bootstrapping or other algorithm), use two NHST devices and check if their results are in agreement. Of course power will be reduced.

Are approaches A and B somehow equivalent or is one of them 'better' - always or in certain situations?


Your understanding of hypothesis testing (NHST) is in error, and you really need to do some study! What you describe is closer to bayesian thinking. You could search this site or better, read an introductory book.

As for your approaches A and B, you need to explain us why you want to do B, it is not very clear. For a better answer we need to know more context, describe your data, what does it measure? and what is your research hypothesis, in simple english.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.