# When do missing data indicate a data collection bug?

I have collected this data, counting how many users engage in a certain activity how many times:

Activity    Users           Percent
1           2,005,752       62.57%
2           1,005,669       31.37%
3           616,328         19.23%
4           408,571         12.75%
5           285,103         8.89%
6           207,127         6.46%
7           154,861         4.83%
8           117,353         3.66%
9           90,143          2.81%
10          70,174          2.19%
11          55,196          1.72%
12          44,050          1.37%
13          34,883          1.09%
14          28,027          0.87%
15          22,948          0.72%
17          18,629          0.58%
18          15,173          0.47%
19          12,376          0.39%
20          10,422          0.33%
21          8,435           0.26%


(I omitted further rows)

Observe that

1. No user engaged in the activity 16 times. One would expect about 20k users (~0.65%) there.

2. log-log plot is convex up, which indicates that the number of users decays faster than polynomially (i.e., this is not a power-law distribution, as I, perhaps naively, expected).

What are the chances that the absence of the users engaging in the activity 16 times is a statistical fluke as opposed to a bug in data collection?

E.g., is the following approach reasonable?

Assume that the 20k individuals with 16 activities are missing. Add them to the table and compute chi-squared=20,076 which is far too large for the 20 degrees of freedom, so this is not a statistical fluke and there must be a data collection bug.

PS. there is no underlying reason which makes 16 special.

I'd say it is extremely unlikely that this absence is a statistical fluke. Especially looking at your large sample size.

Though I don't know how to give a quantitative estimate as to how unlikely this is.

A more insidious mistake might be if the column is shifted down. i.e. Where he says 17 he means 16 and so on. Could this be what has happened here?

OTOH, be certain there's no underlying feature of your activity that makes 16 a special number.

Hypothetical example: Say, this activity was ordering an online pay-per-view movie. But the seller had a promotion going that gave you one free movie on every 16 movies. You might expect to see an abnormally low number of users that have watched 16 movies. Though probably not zero still.

A derivative question: (a) Has someone forgotten the row corresponding to n=16? or (b) Has n=16 been accidentally labelled n=17 and so on. To me that's a question that might have a nice statistical answer.

• please see an edit – sds Mar 13 '13 at 20:00
• Good answer. With such a large sample size and a basically regular distribution it is implausible that 16 has just ended up at zero at random with nothing special about it. A bayesian approach could help quantify this but the answer would certainly be "no chance". – Peter Ellis Mar 14 '13 at 9:31