# Are observations consistent with each other?

I have 60 apps. For each app, I look at 10 review of that app, and count how many of the reviews say the app is unreliable. Let $X_i$ denote the number of such reviews, for the $i$th app. I want to check whether the reviews seem to be pretty consistent for each app. For instance, if one review says the app is unreliable, does this make it more likely that the other reviews for that app also mention unreliability?

So, one model for the situation is that the reviews don't reflect anything about the apps: there is a parameter $p$, and each of the 600 reviews is an iid Bernoulli($p$) random variable. In this model, $X_i \sim \text{Binomial}(10, p)$ for each $i$. The value of $p$ is not known a priori.

An alternative model is that each app has some inherent degree of reliability, modeled by a parameter $p_i$ for the $i$th app, and each of the 10 reviews of the $i$th app is an iid Bernoulli($p_i$) random variable. In this model, $X_i \sim \text{Binomial}(10, p_i)$ for each $i$. The value of the $p_i$'s is not known a priori.

1. If I observe the values of $X_1,\dots,X_{60}$, is there any way I can determine which model seems to better reflect the data? Is there a good way to construct a hypothesis test to compute a $p$-value for the null hypothesis, that $X_i \sim \text{Binomial}(10, p)$ for all $i$?

2. Let's say that I reject the null hypothesis. Can you suggest a meaningful way to measure the degree of internal consistency across reviews of the same app? What would be a good measure of this sort of consistency/correlation? Ideally, something that is amenable to a relatively natural interpretation?

(For instance, maybe I compute the sample standard deviation $\hat\sigma$ of $X_1,\dots,X_{60}$; I compute the expected standard deviation $\sigma_0$ under the null hypothesis, $\sigma_0 = \sqrt{10 \hat{p} (1 - \hat{p})}$, where $\hat{p} = (X_1+\dots+X_{60})/600$; and I look at the ratio $\hat{\sigma}/\sigma_0$?)

• Thanks! If I understand correctly, the GLM model will give us a way to infer the parameter of the Binomial (e.g., $p_i$ or $p$); this might be done, e.g., by logit or probit regression on the equation $x \sim 1$ (no independent vars). Now how do I get a $p$-value for the null hypothesis? And how do I characterize, quantitatively, the degree of internal consistency among the reviews? Can you elaborate on that at all? Also, how is logit/probit regression preferable to simply taking $p_i=X_i/10$ or $p=(X_1+\dots+X_{60})/600$ as our estimate of the parameter of the Binomial distributions? – D.W. Apr 11 '12 at 9:04