I am very confused about the effect of sample size on statistical and practical significance levels. I have two questions about them: 1. Can somebody explain the p value variation with sample size intuitively and mathematically. 2. Lets say I increase my practical significance from 2% to 3%. How would that effect the sample size requirement for the same power.

I have browsed internet for them but have gotten more confused. Can somebody please explain these in intuitively and mathematically ?

  • 1
    $\begingroup$ Practical significance depends on what an important difference is for your problem. It is not something that you should change. Statistical significance is a difference that you can choose. Often practical significance can be obtained with a smaller sample size. $\endgroup$ – Michael R. Chernick Apr 1 '17 at 19:11

In your question you write that you are confused, so I will try to keep this answer as close to a general understanding as possible. My definitions are made to give you an intuition of what is going. From a strict statistical perspective they are actually very inaccurate and maybe you will see in a few months why. But let's start now:

Practical significance is a matter of what significance means in a general meaning. For example, you could conduct an experiment with 100.000 participants. Half of them gets a chewing gum under their shoes and you compare their walking speed with the control group, who don't have chewing gum under their shoes. You can observe that the average walking speed for the chewing gum group is 5.00000 km/h and the control group has an average of 5.00001 km/h (which might be statistically significant). Ask your personal reasoning: would this be a sufficient result to forbid chewing gums? That's the question practical significance answers you. Mostly this may be measured using so called effect sizes.

Statistical significance is a calculation you do to check wether your results are really due to chance or a systematic effect (this is not exactly accurate but in a very very loose definition, this is what it's all about). For example you may ask two women and two men for their IQ. The average for men is 95 and for women 105 (fictional example). How strongly would you claim that there is a difference. Maybe you just got two women with above average intelligence?! Now consider you sample 100'000 women and men and get the same result. Which survey would you trust more? Probably the latter one. This is what statistical significance is about. Now imagine you have to predict the next elections. Party A and Party B both have 45% in the polls. You will probably not be able to say that very easily who will win. On the other hand consider party A has 70% in the polls and party B 10%. It is fairly easy to see that Party A will probably win, isn't it? This difference between the two things you compare is the effect. The more they differ the easier to show that they are different. Now let us consider Party A has 55% and Party B 40%. Now I ask you three questions. 1) what is your guess who will win? 2) can you tell me who will most likely win? 3) can you tell me with absolute certainty who will win? You may answer 1) with A might win and 2) with A will probably win. At least their odds are better at the moment than for Party B. Question 3) you would not be able to answer at all. Do you see that with the increasing level of required certainty or confidence it become harder to give a sound answer / result?

Now imagine a pharmaceutical company which has been in the press because their product has been suspected to cause cancer for 1 in 20 patients. Three days later the PR department issues a statement "we conducted a study among 3 participants and haven't found any negative side effect!" Would you say that 3 people is enough to really find an effect? Or might it be that you actually have a low chance of finding an effect for 1 in 20 people if you only sample 3 persons? That is what power tells you: the probability of finding an effect if it really exists. In this example, you could certainly say: they just didn't try enough!

The summary: General significant depends on the effect but not sample size.

Statistical significance gets better ( p value smaller) with more people ( larger sample), a greater difference (effect) between the groups and a lower required level of certainty ( lower confidence level).

Power increases with larger sample size, larger effect and smaller confidence.

The general discussion depends on understanding null hypothesis, alternate hypothesis and the decision which hypothesis is better supported by the data. You must know what type I and type II errors are. Then you will understand that significance is broadly about controlling type I error rates and power the opposite to type II errors. However, to explain all that is beyond the scope of this post.

For the mathematical calculation I find it difficult to show you what's going on since I do not know which distributions you know, and if you know about normalisation. Do you know about limits? That would certainly help your understanding of the mathematical part of all questions about what influences significance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.