0
$\begingroup$

I'm new to hypothesis testing in statistics.

Say if we were to run an experimental trial on two control groups (split of 50% each), and after a large enough sample size, conclude by looking at just "mean" success rate in both trials that one group was better than the other.

This method seems very intuitive (since we've allowed the mean to settle over a long time), and seems accurate.

However, statistical formuale usually include a standard deviation component when computing the "p" value - (x - u)/(σ / sqrt(n)). Are there cases where would it be wrong to conclude the result of an experiment by just looking at the mean (due to the fact that we neglected the σ factor which would reverse the conclusion) ?

$\endgroup$
1
$\begingroup$

Coming from your description, you can regard the standard deviation in parametric tests as a means to determine, whether you have reached "a large enough sample size". If you have real world samples of millions and billions and large differences between groups, then statistics tests are usually meaningless. Statistical tests help you decide where results are not obvious, that is if you either have little data or large variance or both. The standard deviation will not "reverse the conclusion". It merely helps with the question, on how stable grounds you can draw the conclusion.

$\endgroup$
  • $\begingroup$ So what this means is for very large samples, the denominator should approach 1 ? $\endgroup$ – Shubham Kanodia Jan 12 '18 at 8:56
  • $\begingroup$ No. For large values of $n$ the term $sqrt(n)$ gets larger towards $\infty$ and for finite values of $\sigma$ the denominator $\sigma/sqrt(n)$ approaches 0. Thus the t-value gets very large for large $n$ and the p-value becomes small. $\endgroup$ – Bernhard Jan 12 '18 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.