# Optional stopping rules not in textbooks

Stopping rules affect the relationship between P-values and the error rates associated with decisions. A recent paper by Simmons et al. 2011 coins the term researcher degrees of freedom to describe a collection of behaviours that they consider to be responsible for many of the reports in the psychology literature that have been found to be not reproducible.

Of those behaviours, optional stopping rules or undeclared interim analyses are what I am currently interested in. I describe their effect on error rates to my students, but they do not seem to be described in the textbooks that my students use (or don't use!). In the main bookshop at my university there are fourteen statistics textbooks aimed at introductory-level students in various disciplines such as biosciences, business, engineering etc. Only one of those texts contained an index item "sequential testing' and none had an index item 'stopping rule'.

Is there an introductory level statistics textbook that explains the issue of optional stopping rules?

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. doi:10.1177/0956797611417632

• Doesn't the problem go away if you ditch frequentist stats and go with IT or Bayesian methods? (Or even pure machine learning, depending on the size of your dataset) That's not being flippant--the incompatible mashup of Fisher and N-P causes nothing but problems, even when done "correctly." In the future, there will be no more frequentists. – thedude Jan 19 '13 at 18:40
• Yes, the problem would be gone if there were no use of methods that should comply with the frequency principle. However, such a future may not be coming in this world. What is IT? – Michael Lew Jan 19 '13 at 20:39
• @Michael: Almost undoubtedly it (i.e., IT) stands for "information theoretic". – cardinal Jan 20 '13 at 3:02
• On a related topic: errorstatistics.com/2013/04/06/… – Fr. Apr 7 '13 at 11:50
• @thedude using a different theoretical framework introduces other problems. The problem here is that you are all treating math as something other than just a description of the world. Frequentist stats is one very useful way to describe the world, bayesian is another. Neither will provide you with an Oracle of truth. – Indolering Jul 7 '13 at 6:35

You can't have a stopping rule without some idea of your distribution and your effect size - which you don't know a priori.

Also yes, we need to focus on effect size - and it has never been regarded as correct to consider only p-values, and we should certainly not be showing tables or graphs that show p-values or F-values rather than effect size.

There are problems with traditional Statistical Hypothesis Inference Testing (which Cohen says is worthy of its acronym, and Fisher and Pearson would both turn over in the graves if they saw all that is being done in their violently opposed names today).

To determine N, you need to have already determined a target significance and power threshold, as well as making lots of assumptions about distribution, and in particular you also need to have determined the effect size that you want to establish. Indolering is exactly right that this should be the starting point - what minimum effect size would be cost effective!

The "New Statistics" is advocating showing the effect sizes (as paired difference where appropriate), along with the associated standard deviations or variance (because we need to understand the distribution), and the standard deviations or confidence intervals (but the latter is already locking in a p-value and a decision about whether you are predicting a direction or an each way bet). But setting a minimum effect of specified sign with a scientific prediction, makes this clear - although the pre-scientific default is to do trial and error and just look for differences. But again you have made assumptions about normality if you go this way.

Another approach is to use box-plots as a non-parametric approach, but the conventions about whiskers and outliers vary widely and even then themselves originate in distributional assumptions.

The stopping problem is indeed not a problem of an individual researcher setting or not setting N, but that we have a whole community of thousands of researchers, where 1000 is much more than 1/alpha for the traditional 0.05 level. The answer is currently proposed to be to provide the summary statistics (mean, stddev, stderr - or corresponding "non-parametric versions - median etc. as with boxplot) to facilitate meta-analysis, and present combined results from all experiments whether they happen to have reached a particular alpha level or not.

Closely related is the multiple testing problem, which is just as fraught with difficulty, and where experiments are kept oversimplistic in the name of preserving power, whilst overcomplex methodologies are proposed to analyze the results.

I don't think there can be a text book chapter dealing with this definitively yet, as we still have little idea what we are doing...

For the moment, the best approach is probably to continue to use the traditional statistics most appropriate to the problem, combined with displaying the summary statistics - the effect and standard error and N being the most important. The use of confidence intervals is basically equivalent to the corresponding T-test, but allows comparing new results to published ones more meaningully, as well as allowing an ethos encouraging reproducibility, and publication of reproduced experiments and meta-analyses.

In terms of Information Theoretic or Bayesian approaches, they use different tools and make different assumptions, but still don't have all the answers either, and in the end face the same problems, or worse ones because Bayesian inference steps back from making a definitive answer and just adduces evidence relative assumed or absent priors.

Machine Learning in the end also has results which it needs to consider for significance - often with CIs or T-Test, often with graphs, hopefully pairing rather than just comparing, and using appropriately compensated versions when the distributions don't match. It also has its controversies about bootstrapping and cross-validation, and bias and variance. Worst of all, it has the propensity to generate and test myriads of alternative models just by parameterizing thoroughly all the algorithms in one of the many toolboxes, applied to the datasets thoughtfully archived to allow unbridled multiple testing. Worst still it is still in the dark ages using accuracy, or worse still F-measure, for evaluation - rather than chance-correct methods.

I have read dozens of papers on these issues, but have failed to find anything totally convincing - except the negative survey or meta-analysis papers that seem to indicate that most researchers don't handle and interpret the statistics properly with respect to any "standard", old or new. Power, multiple testing, sizing and early stopping, interpretation of standard errors and confidence intervals, ... these are just some of the issues.

Please shoot me down - I'd like to be proven wrong! In my view there's lots of bathwater, but we haven't found the baby yet! At this stage none of the extreme views or name-brand approaches looks promising as being the answer, and those that want to throw out everything else have probably lost the baby.

• It isn't a matter of shooting you down, I don't think that there CAN be a solution to these problems. We are humans recognizing patterns in the world, we have to settle with convergent validity. After his poor attempt at proving a god exists, Descartes bunted to convergent validity. Sometimes it's there, sometimes it isn't, but we mostly just run up against our infinitesimally small powers of cognitive computation. – Indolering Jul 7 '13 at 6:43

I do not believe that optional "stopping rules" is a technical term in regards to optimal stopping. However, I doubt that you will find much in-depth discussion on the topic in intro-level psychology statistics textbooks.

The cynical rationale for this is that all social-science students have weak math skills. The better answer, IMHO, is that simple t-Tests are not appropriate for most social science experiments. One has to look at the effect strength and figure out if that resolves the differences between groups. The former can indicate that the latter is possible but that's all it can do.

Measures of welfare spending, state regulation, and urbanization all have statistically significant relationships with measures of religious behavior. However, just stating the p-value is framing the test in an all-or-nothing causal relationship. See the following:

Results from both welfare spending and urbanization have statistically significant p-values but welfare spending is much more strongly correlated. That welfare spending shows such a strong relationship to other measures of religiosity (non-religious rate as well as comfort in religion) for which urbanization doesn't even attain a p-value of < .10, suggesting that urbanization doesn't impact general religious beliefs. Note, however, that even welfare spending doesn't explain Ireland or the Philippines, showing that some other effect(s) are comparatively stronger than that of welfare spending.

Relying on "stopping rules" can lead to false positives, especially in the small sample sizes of psychology. Psychology as a field is really being held back by these kind of statistical shenanigans. However, placing all of our faith on an arbitrary p-value is pretty stupid as well. Even if we all sent our sample sizes and hypothesis statements to a journal before conducting the experiment, we would still run into false positives as academia is collectively trolling for statistical significance.

The right thing to do isn't to stop data mining, the right thing to do is to describe the results in relation to their effect. Theories are judged not just by the accuracy of their predictions but also by the utility of those predictions. No matter how good the research methodology, a drug that provides a 1% improvement in cold symptoms isn't worth the cost to pack into a capsule.

Update To be clear, I totally agree that social scientists should be held to a higher standard: we need to improve education, give social scientists better tools, and up the significance levels to 3-sigma. I'm trying to emphasize an under represented point: the vast majority of psychology studies are worthless because the effect size is so small.

But with Amazon Turk, I can properly compensate for running 10 parralel studies and maintain >3-sigma confidence level very cheaply. But if the effect strength is small, then there are significant threats to external validity. The effect of the manipulation might be due to a news story, or the ordering of the questions, or ....

I don't have time for an essay, but the quality issues within the social sciences go far beyond crappy statistical methods.

• I understand there is some conflation of sociologic (usually non-experimental studies) and clinical trials here. However, your first sentence doesn't make sense: stopping rules are a huge area of research in clinical trials. The rationale for this is that multiple, correlated hypotheses tested sequentially are part of a prespecified analysis plan. The link in OPs question, however, is not an issue of bad math, it is one of bad science. Conducting multiple statistical tests to "feel out" the right analysis and stopping when one finds significance is bad science no matter how you cut it. – AdamO Feb 13 '17 at 17:40
• @AdamO I agree! When I wrote this, I was an undergrad trying to use data-mining methods and when I went to make sure I was doing everything correctly (which I was) the initial reactions I got from professors and statisticians were ... naive. Ironically, the standard operating procedure for social science labs is to run pilot studies until they find something interesting. I was doing the same thing, but actually trying to compensate for it : p – Indolering Mar 18 '17 at 23:10

The article you cite makes no mention of stopping rules and seems to be of little bearing to the problem at hand. Their only, very slight relation is that of multiple testing which is a statistical concept, not a scientific one.

In the literature of clinical trials, you will find that stopping rules are made rigorous with explicit information about the conditions in which a study will "look": based on calendar year, or person-years enrollment, the setting of an alpha level, and also bounds on effects for "effective" versus "harmful" treatments. Indeed, we should look to the rigorous conduct of such studies as an example of science done well. The FDA will even go so far as to say, following a significant finding of efficacy other than that prespecified, a second trial must be conducted to validate these findings. This remains an issue so much so that Thomas Flemming recommends that all clinical studies need to validated with a completely independent second confirmatory trial, conducted by separate entities. So bad is the problem of false-positive errors when considering life and medical care.

With seemingly innocuous oversight, other fields of science have perpetuated bad ethics in research. Indeed, social sciences don't affect the treatments people receive, they deal in abstracts, and conceptual models which only enhance our understanding of the interplay of theory and observation. However, any consumer of social science, lay or scientific, is frequently presented with conflicting findings: chocolate is good for you, chocolate is bad for you (chocolate is good for you, by the way, the sugar and fat in chocolate is bad for you), sex is good for you, marriage makes you sad/marriage makes you happy. The field is remiss with bad science. Even I am guilty of working on analyses where I was unhappy with the strongly causal language which was then tied to strong recommendations about policy and federal support, totally unjustified and yet it was publised.

Simmons' article describes effectively, how disclosure would assist in making explicit the kinds of "shortcuts" researchers make in social studies. Simmons gives in Table 1 an example of how data dredging dramatically increases false positive error rates in a manner typical of unethical scientist "fishing for findings". The summary of findings in Table 2 describes the frequently omitted aspects of articles which would serve to greatly improve an understanding of how possibly more than one analysis was conducted.

To summarize, stopping rules would only be appropriate with a pre-specified hypothesis: these are ethically sound and require statistical methods. Simmons' article admits that much of research does not even grant that, and it is ethically unsound but the statistical language is compelling for why exactly it is wrong.

• I don't understand why you would say that the cited paper has little bearing on the problem at hand. It contains a headlines section called "A closer look at flexibility in sample size" that is all about optional stopping. Have another look. – Michael Lew Feb 13 '17 at 19:55
• @MichaelLew to summarize the answer: stopping rules has to do with clinical trials, recruitment, and follow-up but testing a single prespecified hypothesis this is an acceptable practice in research for FDA devices and therapeutic agents. Simmons' paper addresses research ethics, criteria, and p-hacking in social medicine research and academics. Can you describe more precisely how you see the relation? Perhaps you can edit your post to define terms and provide references to other literature, specifically regarding "stopping rules" which AFAIK do not exist outside clinical trials. – AdamO Feb 13 '17 at 19:58
• I also do not think that your characterisation "other fields of science have perpetuated bad ethics in research" is fair or helpful. The point of my original question is that there seems to be no reason why part time statistics users would even be aware of the potential problems arising from undeclared interim analyses. It is unfair to call ignorance unethical. – Michael Lew Feb 13 '17 at 19:58
• @MichaelLew what are you defining to be an "interim analysis"? – AdamO Feb 13 '17 at 20:01
• Adam, I do not think that you are correct in saying that "stopping rules do not exist outside of clinical trials". They may not often be mentioned outside clinical trials (see my original question), but they exist for every experiment. Even a fixed sample size trial has the stopping rule 'continue until the sample size is achieved'. – Michael Lew Feb 13 '17 at 20:02