32
$\begingroup$

I've just come across Anscombe's quartet (four datasets that have almost indistinguishable descriptive statistics but look very different when plotted) and I am curious if there are other more or less well-known datasets that have been created to demonstrate the importance of certain aspects of statistical analyses.

$\endgroup$
30
$\begingroup$

Data sets that act as counterexamples to popular misunderstandings* do exist - I've constructed many myself under various circumstances, but most of them wouldn't be interesting to you, I'm sure.

*(which is what the Anscombe data does, since it's a response to people operating under the misunderstanding that the quality of a model can be discerned from the identical statistics you mentioned)

I'll include a few here that might be of greater interest than most of the ones I generate:

1) One example (of quite a few) are some example discrete distributions (and thereby data sets) I constructed to counter the common assertion that zero third-moment skewness implies symmetry. (Kendall and Stuart's Advanced Theory of Statistics offers a more impressive continuous family.)

Here's one of those discrete distribution examples:

\begin{array}{cccc} \\ x&-4&1&5\\ \hline P(X=x)&2/6&3/6&1/6 \\ \end{array}

(A data set for a counterexample in the sample case is thereby obvious: $-4, -4, 1, 1, 1, 5$)

As you can see, this distribution isn't symmetric, yet its third moment skewness is zero. Similarly, one can readily construct counterexamples to a similar assertion with respect to the second most common skewness measure, the second Pearson skewness coefficient ($3(\frac{mean-median}{\sigma})$).

Indeed I have also come up with distributions and/or data sets for which the two measures are opposite in sign - which suffices to counter the idea that skewness is a single, easily understood concept, rather than a somewhat slippery idea we don't really know how to suitably measure in many cases.

2) There's a set of data constructed in this answer Box-and-whisker plot for multimodal distribution, following the approach of Choonpradub & McNeil (2005), which shows four very different-looking data sets with the same boxplot.

enter image description here

In particular, the distinctly skewed distribution with the symmetric boxplot tends to surprise people.

3) There are another couple of collections of counterexample data sets I constructed in response to people's over-reliance on histograms, especially with only a few bins and only at one bin-width and bin-origin; which leads to mistakenly confident assertions about distributional shape. These data sets and example displays can be found here

Here's one of the examples from there. This is the data:

  1.03, 1.24, 1.47, 1.52, 1.92, 1.93, 1.94, 1.95, 1.96, 1.97, 1.98, 
  1.99, 2.72, 2.75, 2.78, 2.81, 2.84, 2.87, 2.90, 2.93, 2.96, 2.99, 3.60, 
  3.64, 3.66, 3.72, 3.77, 3.88, 3.91, 4.14, 4.54, 4.77, 4.81, 5.62

And here are two histograms:

Skew vs bell

That's the the 34 observations above in both cases, just with different breakpoints, one with binwidth $1$ and the other with binwidth $0.8$. The plots were generated in R as follows:

x <- c(1.03, 1.24, 1.47, 1.52, 1.92, 1.93, 1.94, 1.95, 1.96, 1.97, 1.98, 
  1.99, 2.72, 2.75, 2.78, 2.81, 2.84, 2.87, 2.9, 2.93, 2.96, 2.99, 3.6, 
  3.64, 3.66, 3.72, 3.77, 3.88, 3.91, 4.14, 4.54, 4.77, 4.81, 5.62)
hist(x,breaks=seq(0.3,6.7,by=0.8),xlim=c(0,6.7),col="green3",freq=FALSE)
hist(x,breaks=0:8,col="aquamarine",freq=FALSE)

4) I recently constructed some data sets to demonstrate the intransitivity of the Wilcoxon-Mann-Whitney test - that is, to show that one might reject a one tailed alternative for each of three or four pairs of data sets, A, B, and C, (and D in the four sample case) such that one concluded that $P(B>A)>\frac{1}{2}$ (i.e. conclude that B tends to be bigger than A), and similarly for C against B, and A against C (or D against C and A against D for the 4 sample case); each tends to be larger (in the sense that it has more than even chance of being larger) than the previous one in the cycle.

Here's one such data set, with 30 observations in each sample, labelled A to D:

       1     2     3     4     5     6     7     8     9    10    11    12
 A  1.58  2.10 16.64 17.34 18.74 19.90  1.53  2.78 16.48 17.53 18.57 19.05
 B  3.35  4.62  5.03 20.97 21.25 22.92  3.12  4.83  5.29 20.82 21.64 22.06
 C  6.63  7.92  8.15  9.97 23.34 24.70  6.40  7.54  8.24  9.37 23.33 24.26
 D 10.21 11.19 12.99 13.22 14.17 15.99 10.32 11.33 12.65 13.24 14.90 15.50

      13    14    15    16    17    18    19    20    21    22    23    24
 A  1.64  2.01 16.79 17.10 18.14 19.70  1.25  2.73 16.19 17.76 18.82 19.08
 B  3.39  4.67  5.34 20.52 21.10 22.29  3.38  4.96  5.70 20.45 21.67 22.89
 C  6.18  7.74  8.63  9.62 23.07 24.80  6.54  7.37  8.37  9.09 23.22 24.16
 D 10.20 11.47 12.54 13.08 14.45 15.38 10.87 11.56 12.98 13.99 14.82 15.65

      25    26    27    28    29    30
 A  1.42  2.56 16.73 17.01 18.86 19.98
 B  3.44  4.13  6.00 20.85 21.82 22.05
 C  6.57  7.58  8.81  9.08 23.43 24.45
 D 10.29 11.48 12.19 13.09 14.68 15.36

Here's an example test:

> wilcox.test(adf$A,adf$B,alt="less",conf.int=TRUE)

    Wilcoxon rank sum test

data:  adf$A and adf$B
W = 300, p-value = 0.01317
alternative hypothesis: true location shift is less than 0
95 percent confidence interval:
      -Inf -1.336372
sample estimates:
difference in location 
             -2.500199 

As you see, the one-sided test rejects the null; values from A tend to be smaller than values from B. The same conclusion (at the same p-value) applies to B vs C, C vs D and D vs A. This cycle of rejections, of itself, is not automatically a problem, if we don't interpret it to mean something it doesn't. (It's a simple matter to obtain much smaller p-values with similar, but larger, samples.)

The larger "paradox" here comes when you compute the (one-sided in this case) intervals for a location shift -- in every case 0 is excluded (the intervals aren't identical in each case). This leads us to the conclusion that as we move across the data columns from A to B to C to D, the location moves to the right, and yet the same happens again when we move back to A.

With a larger versions of these data sets (similar distribution of values, but more of them), we can get significance (one or two tailed) at substantially smaller significance levels, so that one might use Bonferroni adjustments for example, and still conclude each group came from a distribution which was shifted up from the next one.

This shows us, among other things, that a rejection in the Wilcoxon-Mann-Whitney doesn't of itself automatically justify a claim of a location shift.

(While it's not the case for these data, it's also possible to construct sets where the sample means are constant, while results like the above apply.)

Added in later edit: A very informative and educational reference on this is

Brown BM, and Hettmansperger TP. (2002)
Kruskal-Wallis, multiple comaprisons and Efron dice.
Aust&N.Z. J. Stat., 44, 427–438.

5) Another couple of related counterexamples come up here - where an ANOVA may be significant, but all pairwise comparisons aren't (interpreted two different ways there, yielding different counterexamples).


So there's several counterexample data sets that contradict misunderstandings one might encounter.

As you might guess, I construct such counterexamples reasonably often (as do many other people), usually as the need arises. For some of these common misunderstandings, you can characterize the counterexamples in such a way that new ones may be generated at will (though more often, a certain level of work is involved).

If there are particular kinds of things you might be interested in, I might be able to locate more such sets (mine or those of other people), or perhaps even construct some.


One useful trick for generating random regression data that has coefficients that you want is as follows (the part in parentheses is an outline of R code):

a) set up the coefficients you want with no noise (y = b0 + b1 * x1 + b2 * x2)

b) generate error term with desired characteristics (n = rnorm(length(y),s=0.4)

c) set up a regression of noise on the same x's (nfit = lm(n~x1+x2))

d) add the residuals from that to the y variable (y = y + nfit$residuals)

Done. (the whole thing can actually be done in a couple of lines of R)

$\endgroup$
  • 6
    $\begingroup$ 0,0,1,1,1,1,3 is a counter-example to the common assertion that mean, median and mode coinciding implies a symmetric distribution, although a binomial such as ${10 \choose k} 0.1^k 0.9^{10-k}, k = 0, \dots, 10$ is likely to seem a better one. $\endgroup$ – Nick Cox Dec 20 '13 at 12:55
  • $\begingroup$ @Glen_b thanks. On the contrary, very, very interesting. For example, I've just saved a file named sturges.R with your data x and the following lines of code: hist(x,col="green3",freq=FALSE); hist(x,breaks="Scott",col="aquamarine",freq=FALSE); hist(x,breaks="FD",col="darkgreen",freq=FALSE) I know some people failed to convince the R community to not use Sturges' rule as a default for the number of cells - your example is perhaps a more convincing argument than that unpublished theoretical note by Rob Hyndman. $\endgroup$ – Hibernating Dec 22 '13 at 16:24
  • $\begingroup$ @Hibernating My apologies for lack of clarity - I chose the interesting ones that occurred to me. As I said, generating counterexamples arises regularly, but most of them would not be interesting (outside their direct audience). Occasionally some are, so I mentioned all the ones I could think of. If I was going to construct an example to show the problems with Sturges' rule, I'd make the example different to that. (I think that example's main value is in clearly demonstrating that you shouldn't rely on a single rule at all, and should generally lean toward more bins than the common rules.) $\endgroup$ – Glen_b Dec 22 '13 at 19:40
  • 2
    $\begingroup$ @NickCox A smaller counterexample to "mean = median = mode implies symmetry" is -2, -1, 0, 0, 3 which I made up for this question. I suspect $n=5$ is the smallest possible, as we use two data points to form the mode, a third distinct point would spoil mean = median, and I think a fourth point could restore mean = median = mode only by being placed symmetrically. At any rate, your binomial example is more satisfying as it seems less contrived! $\endgroup$ – Silverfish Nov 23 '14 at 0:57
16
$\begingroup$

With regard to generating (e.g., your own) datasets for similar purposes, you might be interested in:

As far as datasets that are simply used to demonstrate tricky / counter-intuitive phenomena in statistics, there a lot, but you need to specify what phenomena you want to demonstrate. For example, with respect to demonstrating Simpson's paradox, the Berkeley gender bias case dataset is very famous.

For a great discussion of the most famous dataset of all, see: What aspects of the "Iris" data set make it so successful as an example/teaching/test data set.

$\endgroup$
1
$\begingroup$

In the paper "Let's Put the Garbage-Can Regressions and Garbage-Can Probits Where They Belong" (C. Achen, 2004) the author creates a synthetic data set with a non-linearity that is meant to reflect real-life cases when data might have suffered a coding error during measurement (e.g. a distortion in assigning data to categorical values, or incorrect quantization procedures).

The synthetic data is created from a perfect linear relationship with two positive coefficients, but once you apply the non-linear coding error, standard regression techniques will produce a coefficient that is of the wrong sign and also statistically significant (and would become more so if you bootstrapped a larger synthetic data set).

Though it is just a small synthetic data set, the paper presents a great refutation of naive "dump everything I can think of on the right hand side" sorts of regression, showing that with even tiny / subtle non-linearities (which actually are quite common in things like coding errors or quantization errors), you can get wildly misleading results if you just trust the output of standard regression push-button analysis.

$\endgroup$

protected by Glen_b Aug 10 '17 at 11:15

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.