# Source for inter-ocular trauma test for significance

TL;DR: I am looking for the paper that proposed the following "inter-ocular trauma test for statistical significance".

## Longer version

The idea of the proposed informal "test" is as follows. Assume you have observations and a null hypothesis, and assume that there is some quantity(ies) which can be derived either from your observations or by simulation under the null hypothesis. Assume further that this quantity can be graphed easily.

For instance, the quantity we are interested in could be a regression parameter estimate. Or in the example above, which is about examining uniformity of distributions, it could be histograms of the counts in the five fullest and the five emptiest bins.

Now, simulate the quantity of interest under the null hypothesis, say, 19 times. Arrange the graphical representations in a $$4\times5$$ grid, including the representation of the actual observations at a random spot.

Does the panel for your actual observation stand out sufficiently that it is obvious? (I.e., does it "hit you right between the eyes", which is sometimes called an "inter-ocular trauma test"?) If so, then there is something there.

For additional social ineptness, accost random strangers, show them the plot and ask them to identify which panel "doesn't fit". If 95% of your victims correctly identify the panel corresponding to the actual observations, we can informally say that $$p=0.05$$.

I read about this proposal in a paper, which I believe dates from the 2000s, by a well known statistician, on the order of Tibshirani or Breiman, but no matter how much I dig through my literature database, I can't locate the original paper. It may not even have been published (it doesn't seem to be among the papers I have read from the Journal of Computational and Graphical Statistics).

Can anyone identify the paper in which this was proposed?

## R code for the graphic above

set.seed(1)
n_items <- 5000
n_bins <- 1000

actual_distribution <- factor(sample(1:n_bins,n_items,replace=TRUE,prob=0.996^(1:n_bins)),levels=1:n_bins)

y_max <- 30 # set through trial and error

n_plots <- 20
(where_to_insert <- sample(1:n_plots,1))

opar <- par(mfrow=c(4,5),las=2,mai=c(.1,.5,.1,.1))
for ( ii in 1:n_sims ) {
if ( ii == where_to_insert ) {
sim <- actual_distribution
} else {
sim <- factor(sample(1:n_bins,n_items,replace=TRUE),levels=1:n_bins)
}
barplot(c(sort(table(sim),decreasing=TRUE)[1:5],
NA,NA,
rev(sort(table(sim),decreasing=FALSE)[1:5])),
xaxt="n",lwd=2,col="gray",ylim=c(0,y_max))
text(7.2,1,"...",cex=2,font=2)
}
par(opar)

• Here is a preprint discussing the use of these "lineups". Another paper by Hadley Wickham is this one. The R package nullabor by Hadley Wickham and others facilitates this procedures. Maybe you'll find more reference therein. – COOLSerdash Apr 2 at 12:20
• @COOLSerdash: perfect, thanks a lot! That preprint led me to Buja et al., "Statistical Inference for Exploratory Data Analysis and Model Diagnostics" (2009, Philosophical Transactions: Mathematical, Physical and Engineering Sciences), which my notes tell me I read in May 2017. Do you want to turn this into an answer I can upvote and accept? – Stephan Kolassa Apr 2 at 12:25
• I have been using the Cook et al. papers given by @COOL when referencing this method (which I have frequently used). – whuber Apr 2 at 13:17
• I first encountered the concept of this sort of grid of plots more than three decades ago (to my recollection, in a conversation with a professor; I seem to recall that he didn't think of it as a new idea even then), I used a version of that here; the only reference I had then was to the 2009 Buja et al paper you just mentioned, but as I mention in my answer there I have been using this idea (informally) since the '80s, and was talking about it on a number of occasions - usually in one-on-one conversations - in the 90s and 2000s – Glen_b -Reinstate Monica Apr 3 at 2:02