1
$\begingroup$

Say that I have a dataset of 100 observations, I build a model to predict a dependent variable given independent variables, and the p-value is such that the null hypothesis cannot be rejected.

Is there a way to determine whether a model requires additional data to reject the null hypothesis or increasing the sample size will not lower the p-value?

My initial thought is to graph p-value as a function of sample size p(n), and if it appears that the model approaches statistical significance as the sample size approaches 100 and the slope suggests the model will achieve statistical significance at some point where n>100, then it would be worth collecting the additional data.

$\endgroup$

1 Answer 1

1
$\begingroup$

You can't keep examining p-values as you add observations without affecting the behavior of the test procedure. For example, if you work this way (check for significance, decide whether to stop or to add some data) you change the effective significance level -- often dramatically.

Instead you must modify the procedure to deal with the impact of such "peeking" if you don't want an inflated type I error rate.

See sequential analysis which can give you a way to formalize such a process so that you have proper control over significance level.


You might consider using the initial set of data as a pilot sample to compute a sample size with a given power for a new sample. However, it's necessary when doing so to consider that you have sample statistics, not population quantities, on which the usual power calculations rely; basing the calculations on estimates would require larger samples to be confident of achieving a given power.

$\endgroup$
2
  • $\begingroup$ just to add what to glen_b said ( what he said is crucial, this is less ). Eventually, if you add enough observations ( which depends on the particular problem you're dealing with ) , the sd estimate will become so small that you will end up rejecting the null. This is because statistical tests really aren't designed for absurd sample sizes. So, the answer is yes but it doesn't mean that the rejection really means anything because of the flaw in using a HUGE number of observations. It then becomes a question of effect size and all that sort of stuff. Is the actual difference meaningful ? $\endgroup$
    – mlofton
    Dec 2, 2019 at 0:46
  • $\begingroup$ Definitely not kosher. Chapter 11 of Doing Bayesian Data Analysis by John Kruschke has a very lucid explanation of why it is bad to do this and how it can lead to false-positives. $\endgroup$
    – llewmills
    Dec 6, 2019 at 5:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.