# How to fix the problem in this XKCD comic?

Slope Hypothesis Testing (Randall Munroe, xkcd)

The problem with this comic is obviously that the "measurements" are not independent, violating a key assumption for computing valid p-values. But I see also a possible problem with the fact that the dependent variable "grade" is constant for each student.

What is the correct way to analyze this data?

I've scraped the data and I've tried a random effects model but it wasn't clear how to specify the model structure. I got the best results so far by using the lm.cluster function from the miceadds package, and specifying each student as its own cluster. Still, I get a p-value of 0.12 which is far from the original p-value of 0.586.

Reproducible example:

pane3 <-
structure(list(loudness = c(86.23265220830784, 86.525433974655584,
86.70530770034479, 86.885181426033995, 88.689627893786295, 88.892405630102843,
89.049912682657435, 89.207285400838856, 92.147394659462378, 92.417541083929137,
76.148932041733218, 76.064749167873202, 75.980566294013173, 93.944878128871437,
94.130035673239107, 94.134737376300436, 94.049882830574518, 89.122721912921492,
89.220338224099606, 89.314596175948196, 89.139513709569087),
student = c("A", "A", "A", "A", "B", "B", "B", "B", "C",
"C", "C", "C")), row.names = c(NA, -12L), class = c("spec_tbl_df",
"tbl_df", "tbl", "data.frame"), spec = structure(list(cols = list(
loudness = structure(list(), class = c("collector_double",
"collector")), grade = structure(list(), class = c("collector_double",
"collector"))), default = structure(list(), class = c("collector_guess",
"collector")), delim = ","), class = "col_spec"))

ggplot(pane3, aes(x = loudness, y = grade)) +
geom_point() +
geom_smooth(method = "lm")

library(lme4)
summary(lmer(grade ~ loudness + (1 | student), pane3))

summary(lm.cluster(data = pane3, formula = grade ~ loudness, cluster = "student"))

• You should not get the original value of 0.586 (even assuming that is correctly calculated). The new points are not independent but they do contain some information.
– mkt
Commented Feb 24, 2023 at 11:05
• A further issue here is that collecting further observations conditionally on not observing significance violates standard test assumptions anyway. One would need a sequential protocol pre data for doing this, and the actual computations for testing would be quite different. So even better looking test results from better models should not be trusted as long as the data were collected in the shown way and the better model only adopted after drawing further data as consequence of insignificance. Commented Feb 24, 2023 at 14:17
• Do you analyze these data? Commented Feb 24, 2023 at 19:30
• The model in the comic is clear: loudness, conditional on grade, is what varies. Use loudness, then, as the conditional response.
– whuber
Commented Feb 25, 2023 at 22:12
• A "correct way" to analyze their data with their setup is with a measurement error model. The regression coefficients and their standard error can be consistently estimated when there are independent replicates of the covariate available, which is the case here. In contrast to @whubers interesting approach, the measurement error model permits measurement error in both the covariate and the response and also follows the same setup as on the left-most pane of the comic. (See section 3.4.2 in Carroll et al. Measurement Error...)
– Ben
Commented Feb 27, 2023 at 19:44

"What is the correct way to analyze this data?"

There have been good answer suggesting analyses already. I'd like to add that there is no unique "correct" analysis of the data. For example, does it make sense to assume linearity given that the grade scale is limited? Obviously a linear model will be wrong, but then "all models are wrong but some are useful" - this one may or may not be useful, and an effective sample size of three (beyond which you don't go repeating observations for the three students) for sure is not enough to assess how good or harmful the linearity assumption actually is. One could do something like a logit transform of the exam (percentage) grades, and based on the fact that this at least produces the correct value range (unless people score 0 or 100, but this is easy to repair, if not in a uniquely correct manner) arguably this should be preferred, but once more the data do not allow to assess this in any detail. Furthermore, if we're talking about students from the same class, there may be dependence even between the students, for which there is no way to assess this from the data alone - and collecting more students from the same class won't help with that either.

So the baseline is that any analysis will require subjective decisions for or against which there may be arguments, in some way or another taking background knowledge into account that may go further than what we actually know (like grade distributions from earlier courses), but there is no formal "objective" way to decide between them (other than pointing out obvious flaws in certain analyses as the ignorance of dependence in the cartoon), and so there is no single "correct way" to analyse these (or any) data.

• These are valid points, but for this toy example I was only wondering what was the correct approach for estimating the fixed effects of a linear model when the points are repeated, almost identical measures. Commented Feb 25, 2023 at 7:31

Your question is composed of multiple questions/issues. I can answer the first one, the second is still unclear to me.

summary(lmer(grade ~ loudness + (1 | student), pane3))

What happened here is that the mixed effects model is effectively fitting three lines that have different (random) intercepts, instead of a single line. This results in the following output for fixed effects

Fixed effects:
Estimate Std. Error t value
(Intercept)   76.521     11.866   6.449
loudness       0.111      0.119   0.933


the slope of $$\beta = 0.111$$ instead of $$\beta = 1.94$$ is more like an effect within subjects rather than between subjects. It could be possible that the effect of loudness has an effect within subjects, and by measuring the subjects many times, you could observe a significant effect.

summary(lm.cluster(data = pane3, formula = grade ~ loudness, cluster = "student"))

to be continued, it is not so clear what this method does, but giving a p-value of 0.12 is indeed weird

In addition you have a third question

How to fix the problem in this XKCD comic?

You fix this by sampling more people.

Alternatively, you can sample the same people multiple times*. This should reduce the noise in the case that there is not only an error from person to person, but also within the persons. If you sample the same person multiple times then you get more accurate information about that average performance of that person, and you reduce the influence of the noise due to the same person potentially having different exam results.

In the first pane of the XKCD image, the results where highly variable. In the third pane it was still the same.

The XKCD comic, stages the situation where the exam grade is not much variable within the same person. (Or it is even the same because they seem to have only repeated the loudness measurements and the variations in exam grade are due to your data scraping).

That is not necessarily the case when some resampling is done. Consider the following

Of course, it might be still doubtful whether inference with only three data points is a good idea. And possibly one should use a model that considers the error in the x-variable (like Deming regression). But the point is that repetitions within subjects can improve the accuracy of the estimates.

*In fact, this way might be even better, because it allows you to observe the within subject error and assess the goodness of fit. This is why scientists in quantitative fields like to perform duplicate or triplicate measurements.

• Note that keeping the number of students at three and just collecting more data for every single student won't get your "effective sample size" for assessing the slope larger than three. In any analysis that pretends to get beyond this, this can only be achieved as a consequence of model assumptions that are untestable and should not be trusted. Commented Feb 24, 2023 at 14:07
• Chances are for the model with a single random effect one can derive a lower bound of the variance of the slope estimator for three students with any number of replicates per student that should really at best be marginally lower than the variance based on a single observation, if at all, although I don't have the time to derive this. Commented Feb 24, 2023 at 14:08
• Note in particular that the random effects model will not imply that if you observe infinitely many observations per student, the mean of those will be on the regression line. There is an irreducible random error of every student's mean that cannot be reduced by replicating observations from the same student. Commented Feb 24, 2023 at 14:11
• @ChristianHennig stats.stackexchange.com/questions/606535/… Commented Feb 24, 2023 at 15:35
• The grade is not constant due to measurement errors in scraping the Web graphic. The original comic and its text make it clear that the grades are just repeated: that's the entire point of the comic!
– whuber
Commented Feb 26, 2023 at 14:34

This can be done with the nlme or lme4 packaes easily, all you have to add is a random effect for the students (and slope).

> library(nlme)
...
Random effects:
Formula: ~1 | student
(Intercept)
StdDev:    9.131586

Formula: ~1 | loudness %in% student
(Intercept)   Residual
StdDev:  0.04725001 0.07510526

Value Std.Error DF  t-value p-value
(Intercept) 76.52081 11.866096  8 6.448693  0.0002
loudness     0.11097  0.118994  8 0.932552  0.3783
...


loudness not significant.

• Too few groups for a random effect to be a good idea, I would think.
– mkt
Commented Feb 24, 2023 at 11:05
• Can you explain why the coefficient from the mixed model is so much lower than the coefficient from the linear model with 3 points? Commented Feb 24, 2023 at 11:28
• @GeorgeSavva This is just some toy data, made up, it is not the original data from the comic and we do not even know if the results reported there are real, so take these values with 100% salt. Commented Feb 24, 2023 at 11:37
• @GeorgeSavva I think it is, OP is asking how to correctly analyse these results using a mixed model, as I understand. Commented Feb 24, 2023 at 12:06
• @mkt Fair enough as idea, even though 3 clusters isn't exactly a recommended number either. Commented Feb 24, 2023 at 16:18