10
$\begingroup$

Let's suppose you have been given some data from a randomized block design with 4 repetitions and 23 treatments. After an initial inspection of the data, you notice that for 8 treatments all repetitions are identical, which is obviously wrong. After reporting the problem, you are told that it was due to a confusion from the person responsible for the data, who will later send you the "correct version" of the data. The corrected version of the data would be something like this:

Treatment  Rep  Value 
   A        1   5727.000
   A        2   5400.000
   A        3   5800.000
   A        4   5473.000
   B        1   4618.000
   B        2   4844.000
   B        3   4966.000
   B        4   4496.000 
...
   Z        1   4329.345
   Z        2   4597.275
   Z        3   4833.246
   Z        4   4199.098 

The first thing that would grab my attention in such data would be the fact that only the eight treatments for which the problem was reported did not have any decimal fraction (all remaining treatments being fine). So I would decide to give them a closer look and subtract each observation from its sample mean within treatments, finding something like

Treatment  Rep  Value    Delta
   A        1   5727.000 +127
   A        2   5400.000 -200
   A        3   5800.000 +200
   A        4   5473.000 -127
   B        1   4618.000 -113
   B        2   4844.000 +113
   B        3   4966.000 +235
   B        4   4496.000 -235
...
   Z        1   4329.345  ...
   Z        2   4597.275  ...
   Z        3   4833.246  ...
   Z        4   4199.098  ...

After seeing that the differences are symmetrical around the mean, I would immediately call the person responsible for the project, and report the problem. Of course, I would also give up working on that project.

Although the evidence is quite compelling, it would be good to attach a probability to the report, just to give an idea of how bad that data looks. So I would have thought of something like the following and I would like to know if there would be any flaw in my reasoning:

Let's say that if the data was legitimate, it would be reasonable to assume normality for that kind of data, based on experience from previous analysis of that kind of data.

So, let's define 4 i.i.d. normal random variables for each of the four repetitions within each treatment: $$X_i\sim N(\mu_X,~\sigma_X); ~~~i = 1, 2, 3, 4$$ The symmetry (with a tolerance of .5) observed above could be expressed as the event: $$A: -0.5 < X_1 + X_2 - X_3 - X_4 < 0.5$$ There are still other equivalent ways in which the samples could be arranged to satisfy the symmetry (X1 + X3 - X4 - X2; X1 + X4 - X2 - X3) so the probability of the symmetry (S) would be: $$P(S) \leq 3 * P(A)$$

The inequality is simply because I don't want to remove the intersection.

If we define the random variable Y as follows: $$Y = X_1 + X_2 - X_3 - X_4$$

It follows from that: $$Y\sim N(0,~2*\sigma_X)$$

Let's say that from the residuals of the model with that data, I estimated the $\sigma_X$ as 350. From that, I would use the CDF of y to calculate the probability of Y falling between -0.5 and 0.5, which would be P(A) = 0.0005699175

pnorm(0.5, sd = 700, lower = TRUE) - pnorm(-0.5, sd = 700, lower = TRUE)

As such, the probability of S would be:

$$P(S) \leq 0.001709752$$

Because there would be no clear block effect and the data would come from a randomized experiment, it would be reasonable to assume statistical independence. Let's suppose that out of 8 suspect treatments, 3 had this symmetry. Then, assuming independence, we could calculate the probability of such event (let's call it D) from a binomial distribution:

D: 3 out of 8 treatments have symmetry of observations around the sample mean.

$$ P(D) \leq {8\choose3} \cdot p^3(1-p)^5$$

$$ P(D) \leq 2.7 \cdot 10^{-7}$$

I am not a statistician, so I would like to know if there is any flaw in that reasoning and whether you would also report the data as fraudulent.

$\endgroup$
4
  • 10
    $\begingroup$ The ethical issue is paramount here. I'd agree on this information in regarding the data as highly suspect and report it as such; I would never call "fraudulence" because that's for others to determine after investigation and in some countries could lay you open to legal action or its threat. As for attaching a probability estimate, I would back off from that. The case is clear(er) without it and just about every assumption or approximation is open to challenge. $\endgroup$
    – Nick Cox
    Jan 23, 2018 at 13:06
  • 6
    $\begingroup$ Agree fully with Nick Cox. Note that ANY specific data pattern is highly unlikely, so the probability estimate is just going to confuse what otherwise is a completely clear and compelling case of unusable data (I'd use that term or something similar rather than fraudulent). $\endgroup$
    – zbicyclist
    Jan 23, 2018 at 13:29
  • 2
    $\begingroup$ Any specific pattern is highly unlikely to occur randomly, but not all are equally likely to be produced conciously by man. As there is no universal model of how a man would 'create' such data, there is not much point in probability calculation. You might consider a graphical depiction of the symmetry of the data, though. A for demonstration purposes B to give people a feeling of how much this fish stinks. $\endgroup$
    – Bernhard
    Jan 23, 2018 at 14:13
  • 1
    $\begingroup$ Very helpful answers! In fact, I couldn't figure out their intention from this data only. The only thing that would be beyond reasonable doubt is that this data is unusable (I liked that term). $\endgroup$
    – Teo
    Jan 24, 2018 at 18:21

1 Answer 1

6
$\begingroup$

The ethical issue is paramount here. I'd agree on this information in regarding the data as highly suspect and report it as such; I would never call "fraudulence" because that's for others to determine after investigation and in some countries could lay you open to legal action or its threat. As for attaching a probability estimate, I would back off from that. The case is clear(er) without it and just about every assumption or approximation is open to challenge.

$\endgroup$
1
  • 3
    $\begingroup$ I've copied this comment by @NickCox as a community wiki answer because the comment is, more or less, an answer to this question. We have a dramatic gap between answers and questions. At least part of the problem is that some questions are answered in comments: if comments which answered the question were answers instead, we would have fewer unanswered questions. $\endgroup$
    – mkt
    Jan 16, 2019 at 10:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.