Datasets constructed for a purpose similar to that of Anscombe's quartet

Question

I've just come across Anscombe's quartet (four datasets that have almost indistinguishable descriptive statistics but look very different when plotted) and I am curious if there are other more or less well-known datasets that have been created to demonstrate the importance of certain aspects of statistical analyses.

Answer 1

Data sets that act as counterexamples to popular misunderstandings* do exist - I've constructed many myself under various circumstances, but most of them wouldn't be interesting to you, I'm sure.

*(which is what the Anscombe data does, since it's a response to people operating under the misunderstanding that the quality of a model can be discerned from the identical statistics you mentioned)

I'll include a few here that might be of greater interest than most of the ones I generate:

1) One example (of quite a few) are some example discrete distributions (and thereby data sets) I constructed to counter the common assertion that zero third-moment skewness implies symmetry. (Kendall and Stuart's Advanced Theory of Statistics offers a more impressive continuous family.)

Here's one of those discrete distribution examples:

\begin{array}{cccc} \\ x&-4&1&5\\ \hline P(X=x)&2/6&3/6&1/6 \\ \end{array}

(A data set for a counterexample in the sample case is thereby obvious: $-4, -4, 1, 1, 1, 5$)

As you can see, this distribution isn't symmetric, yet its third moment skewness is zero. Similarly, one can readily construct counterexamples to a similar assertion with respect to the second most common skewness measure, the second Pearson skewness coefficient ($3(\frac{mean-median}{\sigma})$).

Indeed I have also come up with distributions and/or data sets for which the two measures are opposite in sign - which suffices to counter the idea that skewness is a single, easily understood concept, rather than a somewhat slippery idea we don't really know how to suitably measure in many cases.

2) There's a set of data constructed in this answer Box-and-whisker plot for multimodal distribution, following the approach of Choonpradub & McNeil (2005), which shows four very different-looking data sets with the same boxplot.

In particular, the distinctly skewed distribution with the symmetric boxplot tends to surprise people.

3) There are another couple of collections of counterexample data sets I constructed in response to people's over-reliance on histograms, especially with only a few bins and only at one bin-width and bin-origin; which leads to mistakenly confident assertions about distributional shape. These data sets and example displays can be found here

Here's one of the examples from there. This is the data:

  1.03, 1.24, 1.47, 1.52, 1.92, 1.93, 1.94, 1.95, 1.96, 1.97, 1.98, 
  1.99, 2.72, 2.75, 2.78, 2.81, 2.84, 2.87, 2.90, 2.93, 2.96, 2.99, 3.60, 
  3.64, 3.66, 3.72, 3.77, 3.88, 3.91, 4.14, 4.54, 4.77, 4.81, 5.62

And here are two histograms:

That's the the 34 observations above in both cases, just with different breakpoints, one with binwidth $1$ and the other with binwidth $0.8$. The plots were generated in R as follows:

x <- c(1.03, 1.24, 1.47, 1.52, 1.92, 1.93, 1.94, 1.95, 1.96, 1.97, 1.98, 
  1.99, 2.72, 2.75, 2.78, 2.81, 2.84, 2.87, 2.9, 2.93, 2.96, 2.99, 3.6, 
  3.64, 3.66, 3.72, 3.77, 3.88, 3.91, 4.14, 4.54, 4.77, 4.81, 5.62)
hist(x,breaks=seq(0.3,6.7,by=0.8),xlim=c(0,6.7),col="green3",freq=FALSE)
hist(x,breaks=0:8,col="aquamarine",freq=FALSE)

4) I recently constructed some data sets to demonstrate the intransitivity of the Wilcoxon-Mann-Whitney test - that is, to show that one might reject a one tailed alternative for each of three or four pairs of data sets, A, B, and C, (and D in the four sample case) such that one concluded that $P(B>A)>\frac{1}{2}$ (i.e. conclude that B tends to be bigger than A), and similarly for C against B, and A against C (or D against C and A against D for the 4 sample case); each tends to be larger (in the sense that it has more than even chance of being larger) than the previous one in the cycle.

Here's one such data set, with 30 observations in each sample, labelled A to D:

       1     2     3     4     5     6     7     8     9    10    11    12
 A  1.58  2.10 16.64 17.34 18.74 19.90  1.53  2.78 16.48 17.53 18.57 19.05
 B  3.35  4.62  5.03 20.97 21.25 22.92  3.12  4.83  5.29 20.82 21.64 22.06
 C  6.63  7.92  8.15  9.97 23.34 24.70  6.40  7.54  8.24  9.37 23.33 24.26
 D 10.21 11.19 12.99 13.22 14.17 15.99 10.32 11.33 12.65 13.24 14.90 15.50

      13    14    15    16    17    18    19    20    21    22    23    24
 A  1.64  2.01 16.79 17.10 18.14 19.70  1.25  2.73 16.19 17.76 18.82 19.08
 B  3.39  4.67  5.34 20.52 21.10 22.29  3.38  4.96  5.70 20.45 21.67 22.89
 C  6.18  7.74  8.63  9.62 23.07 24.80  6.54  7.37  8.37  9.09 23.22 24.16
 D 10.20 11.47 12.54 13.08 14.45 15.38 10.87 11.56 12.98 13.99 14.82 15.65

      25    26    27    28    29    30
 A  1.42  2.56 16.73 17.01 18.86 19.98
 B  3.44  4.13  6.00 20.85 21.82 22.05
 C  6.57  7.58  8.81  9.08 23.43 24.45
 D 10.29 11.48 12.19 13.09 14.68 15.36

Here's an example test:

> wilcox.test(adf$A,adf$B,alt="less",conf.int=TRUE)

    Wilcoxon rank sum test

data:  adf$A and adf$B
W = 300, p-value = 0.01317
alternative hypothesis: true location shift is less than 0
95 percent confidence interval:
      -Inf -1.336372
sample estimates:
difference in location 
             -2.500199 

As you see, the one-sided test rejects the null; values from A tend to be smaller than values from B. The same conclusion (at the same p-value) applies to B vs C, C vs D and D vs A. This cycle of rejections, of itself, is not automatically a problem, if we don't interpret it to mean something it doesn't. (It's a simple matter to obtain much smaller p-values with similar, but larger, samples.)

The larger "paradox" here comes when you compute the (one-sided in this case) intervals for a location shift -- in every case 0 is excluded (the intervals aren't identical in each case). This leads us to the conclusion that as we move across the data columns from A to B to C to D, the location moves to the right, and yet the same happens again when we move back to A.

With a larger versions of these data sets (similar distribution of values, but more of them), we can get significance (one or two tailed) at substantially smaller significance levels, so that one might use Bonferroni adjustments for example, and still conclude each group came from a distribution which was shifted up from the next one.

This shows us, among other things, that a rejection in the Wilcoxon-Mann-Whitney doesn't of itself automatically justify a claim of a location shift.

(While it's not the case for these data, it's also possible to construct sets where the sample means are constant, while results like the above apply.)

Added in later edit: A very informative and educational reference on this is

Brown BM, and Hettmansperger TP. (2002)
Kruskal-Wallis, multiple comaprisons and Efron dice.
Aust&N.Z. J. Stat., 44, 427–438.

5) Another couple of related counterexamples come up here - where an ANOVA may be significant, but all pairwise comparisons aren't (interpreted two different ways there, yielding different counterexamples).

---

So there's several counterexample data sets that contradict misunderstandings one might encounter.

As you might guess, I construct such counterexamples reasonably often (as do many other people), usually as the need arises. For some of these common misunderstandings, you can characterize the counterexamples in such a way that new ones may be generated at will (though more often, a certain level of work is involved).

If there are particular kinds of things you might be interested in, I might be able to locate more such sets (mine or those of other people), or perhaps even construct some.

---

One useful trick for generating random regression data that has coefficients that you want is as follows (the part in parentheses is an outline of R code):

a) set up the coefficients you want with no noise (y = b0 + b1 * x1 + b2 * x2)

b) generate error term with desired characteristics (n = rnorm(length(y),s=0.4)

c) set up a regression of noise on the same x's (nfit = lm(n~x1+x2))

d) add the residuals from that to the y variable (y = y + nfit$residuals)

Done. (the whole thing can actually be done in a couple of lines of R)

Answer 2

With regard to generating (e.g., your own) datasets for similar purposes, you might be interested in:

• Chatterjee, S. & Firat, A. (2007). Generating data with identical statistics but dissimilar graphics: A follow up to the Anscombe dataset. The American Statistician, 61, 3, pp. 248–254.

As far as datasets that are simply used to demonstrate tricky / counter-intuitive phenomena in statistics, there a lot, but you need to specify what phenomena you want to demonstrate. For example, with respect to demonstrating Simpson's paradox, the Berkeley gender bias case dataset is very famous.

For a great discussion of the most famous dataset of all, see: What aspects of the "Iris" data set make it so successful as an example/teaching/test data set.

Answer 3

In the paper "Let's Put the Garbage-Can Regressions and Garbage-Can Probits Where They Belong" (C. Achen, 2004) the author creates a synthetic data set with a non-linearity that is meant to reflect real-life cases when data might have suffered a coding error during measurement (e.g. a distortion in assigning data to categorical values, or incorrect quantization procedures).

The synthetic data is created from a perfect linear relationship with two positive coefficients, but once you apply the non-linear coding error, standard regression techniques will produce a coefficient that is of the wrong sign and also statistically significant (and would become more so if you bootstrapped a larger synthetic data set).

Though it is just a small synthetic data set, the paper presents a great refutation of naive "dump everything I can think of on the right hand side" sorts of regression, showing that with even tiny / subtle non-linearities (which actually are quite common in things like coding errors or quantization errors), you can get wildly misleading results if you just trust the output of standard regression push-button analysis.