Thing one: feel free to RTFM me: I'm definitely looking for search-able terms or background reading.

Our situation is this: we have a set of 140 reviewers and 20 elements. Each reviewer reviews each element, and describes the element in a free-response field. Here's the question. Which terms of the reviewers' responses are likely to refer to real characteristics of the elements?

Our approach: consider each term in isolation. Try to reject the null hypothesis that each reviewer uses the given term with an individual base rate in a way that is uncorrelated with the given element.

In order to reject this null hypothesis, we infer a base rate of usage for the term for each individual reviewer, based on that reviewers responses. Then, using this base rate, compute the expected distribution of "votes"; that is, the likelihood that a given element would receive one use of the term, two uses of the term, etc. From this distribution, we can determine the expected variance of a set of samples, and, more importantly, the distribution of expected variances of samples.

Finally, we compute the variance of the set of the numbers of votes for the full set of elements, and determine the likelihood that this variance occurred by chance.

Does this sound reasonable? More important: is there a standard name for this process?


The null hypothesis that your test reject here contains an additional component: it also supposes a particular distribution of word usage.

For instance, suppose you're testing this hypothesis for the word "the". Suppose that each reviewer uses "the" once every 100 words (and suppose for simplicity that every item has 1000 words of reviews). Then under your null hypothesis, the variance of the counts of "the" would be about 10 ($ = np(1-p) = 1000 * 0.01 * (1 - 0.01)$). But this variance calculation assumes that each word is independently likely to be the word "the". In reality there are many confounding effects that almost certainly violate this assumption.

  • Words near an instance of "the" are less likely to be the word "the". For instance, even if your reviewer wrote one million words, they would probably not write the string "the the the". This will reduce the variance relative to the null variance since it will cause occurrences of "the" to be spread out more evenly.

  • Someone's probability of using a word may vary from review to review for reasons unrelated to the item. For instance, today I am unusually likely to use "Lagrange" as an example of a mathematician, because I just answered a CV question using Lagrange multipliers. Tomorrow I will be in a hurry and write answers in broken English without using articles. This will increase the variance relative to the null variance since it clusters word usages more.

  • More problematically, using any word will bring it to the "tip of one's tongue" and make it more likely to be used again, even if it's not relevant to the topic (as happened to "Lagrange" above). This will increase variance by causing terms to be more clustered.

Because of confounders like these, I think that rejecting the null hypothesis for a word will not be very informative about the thing you actually want to know.

  • $\begingroup$ Many thanks for your detailed analysis! In my case, the open responses are short, and (I know from my corpus that) no word is used more than once in a given review. It looks to me like this addresses your first concern, but perhaps not the others. More generally: is there a name for this field, or a good place to read more about this problem? Many thanks again! $\endgroup$ Dec 8 '14 at 23:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.