I tried to ask this on a more general site first, but it was deleted, with a recommendation of asking it here. I'm from completely different field and know little about how statistics is taught and how social studies are performed, so please feel free to correct my understanding of how things work. My wife studied social science some time ago, so I got this info from her. Yet, she was unable to answer my question below.

My question is about experimental social studies where researchers, for example, look for differences between >=2 groups of people. They develop some sort of a test (or task), look how people in two groups respond, collect answers, transform answers into sets of numbers representing parameters of interest for them, then do the statistical analysis of those answers and draw conclusions from the outcome of this analysis. There might be other steps in the research process, like obtaining a permission from the ethics committee etc. but unless they are relevant for my question, lets ignore them for now.

So, I heard it is very common and sometimes even highly advisable to formulate a research hypothesis (for example, ppl from group A perform better on a test then those from group B) before actually doing their experiment. Then, the eventual statistical analysis is aimed at verifying or falsifying this particular hypothesis. If, so whatever reason, the researcher prefers to change the hypothesis on the fly, a new experiment needs to be devised, performed and analysed.

Now, imagine two researchers come up with mutually exclusive hypotheses on a particular research question and, having decided to collaborate, they carry out an experiment and obtain the same dataset. Then they split and each of them analyses the same data from the point of view of their own hypothesis. In the simplest case, researcher R1 verifies if A performs better than B, and R2 verifies if B performs better than A (I know this is a very primitive example, but just to illustrate mutual exclusiveness). They work at their analyses independently until each of them can reach some conclusions.

So, the question is: is there a strict mathematical proof that, with available and commonly used statistical techniques, it is impossible that R1 and R2 come to mutually exclusive conclusions (in the sense of veryfying exclusive hypotheses) based on the same dataset? I know common sense tells us it is impossible, but I am asking specifically about the mathematical proof.

A follow-up question would be: is this proof (if it exists) taught to social science students (in your country/university/etc)?

  • 2
    $\begingroup$ Short answer: no. Reasons: (1) this is not a matter for mathematical proof and (2) when the researchers bring different assumptions to their analyses they may objectively and legitimately arrive at opposite conclusions. This has been well established by empirical studies (where, for example, various researchers are given the same dataset and the same instructions for investigating it but still arrive at greatly varying conclusions). $\endgroup$
    – whuber
    Mar 26, 2021 at 13:58
  • 1
    $\begingroup$ What if we narrow the question down to statistical tests? Assuming same data as input, are there examples where two mutually exclusive H1- Hypothesis could both be chosen based on the same p-value threshold? That seems to me easier to tackle mathematically and likely to be true. $\endgroup$
    – nope
    Mar 26, 2021 at 14:12
  • 1
    $\begingroup$ One does not choose hypotheses based on p-values: that would be reversing the standard procedure of establishing hypotheses based on research objectives and statistical principles and only then testing the hypotheses. You seem to be looking--in vain--for mathematical certainty in a setting where none is going to be possible because the overriding considerations are those of model assumptions, loss functions, and study objectives. $\endgroup$
    – whuber
    Mar 26, 2021 at 16:06


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.