Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Stopping rules affect the relationship between P-values and the error rates associated with decisions. A recent paper by Simmons et al. 2011 coins the term researcher degrees of freedom to describe a collection of behaviours that they consider to be responsible for many of the reports in the psychology literature that have been found to be not reproducible.

Of those behaviours, optional stopping rules or undeclared interim analyses are what I am currently interested in. I describe their effect on error rates to my students, but they do not seem to be described in the textbooks that my students use (or don't use!). In the main bookshop at my university there are fourteen statistics textbooks aimed at introductory-level students in various disciplines such as biosciences, business, engineering etc. Only one of those texts contained an index item "sequential testing' and none had an index item 'stopping rule'.

Is there an introductory level statistics textbook that explains the issue of optional stopping rules?

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. doi:10.1177/0956797611417632

share|improve this question
Doesn't the problem go away if you ditch frequentist stats and go with IT or Bayesian methods? (Or even pure machine learning, depending on the size of your dataset) That's not being flippant--the incompatible mashup of Fisher and N-P causes nothing but problems, even when done "correctly." In the future, there will be no more frequentists. – thedude Jan 19 '13 at 18:40
Yes, the problem would be gone if there were no use of methods that should comply with the frequency principle. However, such a future may not be coming in this world. What is IT? – Michael Lew Jan 19 '13 at 20:39
@Michael: Almost undoubtedly it (i.e., IT) stands for "information theoretic". – cardinal Jan 20 '13 at 3:02
On a related topic:… – Fr. Apr 7 '13 at 11:50
@thedude using a different theoretical framework introduces other problems. The problem here is that you are all treating math as something other than just a description of the world. Frequentist stats is one very useful way to describe the world, bayesian is another. Neither will provide you with an Oracle of truth. – Indolering Jul 7 '13 at 6:35

You can't have a stopping rule without some idea of your distribution and your effect size - which you don't know a priori.

Also yes, we need to focus on effect size - and it has never been regarded as correct to consider only p-values, and we should certainly not be showing tables or graphs that show p-values or F-values rather than effect size.

There are problems with traditional Statistical Hypothesis Inference Testing (which Cohen says is worthy of its acronym, and Fisher and Pearson would both turn over in the graves if they saw all that is being done in their violently opposed names today).

To determine N, you need to have already determined a target significance and power threshold, as well as making lots of assumptions about distribution, and in particular you also need to have determined the effect size that you want to establish. Indolering is exactly right that this should be the starting point - what minimum effect size would be cost effective!

The "New Statistics" is advocating showing the effect sizes (as paired difference where appropriate), along with the associated standard deviations or variance (because we need to understand the distribution), and the standard deviations or confidence intervals (but the latter is already locking in a p-value and a decision about whether you are predicting a direction or an each way bet). But setting a minimum effect of specified sign with a scientific prediction, makes this clear - although the pre-scientific default is to do trial and error and just look for differences. But again you have made assumptions about normality if you go this way.

Another approach is to use box-plots as a non-parametric approach, but the conventions about whiskers and outliers vary widely and even then themselves originate in distributional assumptions.

The stopping problem is indeed not a problem of an individual researcher setting or not setting N, but that we have a whole community of thousands of researchers, where 1000 is much more than 1/alpha for the traditional 0.05 level. The answer is currently proposed to be to provide the summary statistics (mean, stddev, stderr - or corresponding "non-parametric versions - median etc. as with boxplot) to facilitate meta-analysis, and present combined results from all experiments whether they happen to have reached a particular alpha level or not.

Closely related is the multiple testing problem, which is just as fraught with difficulty, and where experiments are kept oversimplistic in the name of preserving power, whilst overcomplex methodologies are proposed to analyze the results.

I don't think there can be a text book chapter dealing with this definitively yet, as we still have little idea what we are doing...

For the moment, the best approach is probably to continue to use the traditional statistics most appropriate to the problem, combined with displaying the summary statistics - the effect and standard error and N being the most important. The use of confidence intervals is basically equivalent to the corresponding T-test, but allows comparing new results to published ones more meaningully, as well as allowing an ethos encouraging reproducibility, and publication of reproduced experiments and meta-analyses.

In terms of Information Theoretic or Bayesian approaches, they use different tools and make different assumptions, but still don't have all the answers either, and in the end face the same problems, or worse ones because Bayesian inference steps back from making a definitive answer and just adduces evidence relative assumed or absent priors.

Machine Learning in the end also has results which it needs to consider for significance - often with CIs or T-Test, often with graphs, hopefully pairing rather than just comparing, and using appropriately compensated versions when the distributions don't match. It also has its controversies about bootstrapping and cross-validation, and bias and variance. Worst of all, it has the propensity to generate and test myriads of alternative models just by parameterizing thoroughly all the algorithms in one of the many toolboxes, applied to the datasets thoughtfully archived to allow unbridled multiple testing. Worst still it is still in the dark ages using accuracy, or worse still F-measure, for evaluation - rather than chance-correct methods.

I have read dozens of papers on these issues, but have failed to find anything totally convincing - except the negative survey or meta-analysis papers that seem to indicate that most researchers don't handle and interpret the statistics properly with respect to any "standard", old or new. Power, multiple testing, sizing and early stopping, interpretation of standard errors and confidence intervals, ... these are just some of the issues.

Please shoot me down - I'd like to be proven wrong! In my view there's lots of bathwater, but we haven't found the baby yet! At this stage none of the extreme views or name-brand approaches looks promising as being the answer, and those that want to throw out everything else have probably lost the baby.

share|improve this answer
It isn't a matter of shooting you down, I don't think that there CAN be a solution to these problems. We are humans recognizing patterns in the world, we have to settle with convergent validity. After his poor attempt at proving a god exists, Descartes bunted to convergent validity. Sometimes it's there, sometimes it isn't, but we mostly just run up against our infinitesimally small powers of cognitive computation. – Indolering Jul 7 '13 at 6:43

I do not believe that optional "stopping rules" is a technical term in regards to optimal stopping. However, I doubt that you will find much in-depth discussion on the topic in intro-level psychology statistics textbooks.

The cynical rationale for this is that all social-science students have weak math skills. The better answer, IMHO, is that simple t-Tests are not appropriate for most social science experiments. One has to look at the effect strength and figure out if that resolves the differences between groups. The former can indicate that the latter is possible but that's all it can do.

Measures of welfare spending, state regulation, and urbanization all have statistically significant relationships with measures of religious behavior. However, just stating the p-value is framing the test in an all-or-nothing causal relationship. See the following:

enter image description here

Results from both welfare spending and urbanization have statistically significant p-values but welfare spending is much more strongly correlated. That welfare spending shows such a strong relationship to other measures of religiosity (non-religious rate as well as comfort in religion) for which urbanization doesn't even attain a p-value of < .10, suggesting that urbanization doesn't impact general religious beliefs. Note, however, that even welfare spending doesn't explain Ireland or the Philippines, showing that some other effect(s) are comparatively stronger than that of welfare spending.

Relying on "stopping rules" can lead to false positives, especially in the small sample sizes of psychology. Psychology as a field is really being held back by these kind of statistical shenanigans. However, placing all of our faith on an arbitrary p-value is pretty stupid as well. Even if we all sent our sample sizes and hypothesis statements to a journal before conducting the experiment, we would still run into false positives as academia is collectively trolling for statistical significance.

The right thing to do isn't to stop data mining, the right thing to do is to describe the results in relation to their effect. Theories are judged not just by the accuracy of their predictions but also by the utility of those predictions. No matter how good the research methodology, a drug that provides a 1% improvement in cold symptoms isn't worth the cost to pack into a capsule.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.