2
$\begingroup$

I have 60 apps. For each app, I look at 10 review of that app, and count how many of the reviews say the app is unreliable. Let $X_i$ denote the number of such reviews, for the $i$th app. I want to check whether the reviews seem to be pretty consistent for each app. For instance, if one review says the app is unreliable, does this make it more likely that the other reviews for that app also mention unreliability?

So, one model for the situation is that the reviews don't reflect anything about the apps: there is a parameter $p$, and each of the 600 reviews is an iid Bernoulli($p$) random variable. In this model, $X_i \sim \text{Binomial}(10, p)$ for each $i$. The value of $p$ is not known a priori.

An alternative model is that each app has some inherent degree of reliability, modeled by a parameter $p_i$ for the $i$th app, and each of the 10 reviews of the $i$th app is an iid Bernoulli($p_i$) random variable. In this model, $X_i \sim \text{Binomial}(10, p_i)$ for each $i$. The value of the $p_i$'s is not known a priori.

  1. If I observe the values of $X_1,\dots,X_{60}$, is there any way I can determine which model seems to better reflect the data? Is there a good way to construct a hypothesis test to compute a $p$-value for the null hypothesis, that $X_i \sim \text{Binomial}(10, p)$ for all $i$?

  2. Let's say that I reject the null hypothesis. Can you suggest a meaningful way to measure the degree of internal consistency across reviews of the same app? What would be a good measure of this sort of consistency/correlation? Ideally, something that is amenable to a relatively natural interpretation?

    (For instance, maybe I compute the sample standard deviation $\hat\sigma$ of $X_1,\dots,X_{60}$; I compute the expected standard deviation $\sigma_0$ under the null hypothesis, $\sigma_0 = \sqrt{10 \hat{p} (1 - \hat{p})}$, where $\hat{p} = (X_1+\dots+X_{60})/600$; and I look at the ratio $\hat{\sigma}/\sigma_0$?)

Improve this question
$\endgroup$
1

3 Answers 3

Reset to default
1
$\begingroup$

Generalized linear mixed effects model with a binomial response. This would allow you directly to compare a model with "App" as a group in the randomness to your null hypothesis in q1. Then q2 might also flow from the way the model has been fit.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! If I understand correctly, the GLM model will give us a way to infer the parameter of the Binomial (e.g., $p_i$ or $p$); this might be done, e.g., by logit or probit regression on the equation $x \sim 1$ (no independent vars). Now how do I get a $p$-value for the null hypothesis? And how do I characterize, quantitatively, the degree of internal consistency among the reviews? Can you elaborate on that at all? Also, how is logit/probit regression preferable to simply taking $p_i=X_i/10$ or $p=(X_1+\dots+X_{60})/600$ as our estimate of the parameter of the Binomial distributions? $\endgroup$
    – D.W.
    Commented Apr 11, 2012 at 9:04
0
$\begingroup$

Fleiss' kappa looks like a possible way of quantitatively characterizing the consistency/reliability of reviews (q2).

Improve this answer
$\endgroup$
0
$\begingroup$

It looks like this may fall into the category of statistical methods for rater agreement, for dichotomous (categorical) ratings and multiple raters. Apparently there are many methods for assessing rater agreement.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.