Thing one: feel free to RTFM me: I'm definitely looking for search-able terms or background reading.
Our situation is this: we have a set of 140 reviewers and 20 elements. Each reviewer reviews each element, and describes the element in a free-response field. Here's the question. Which terms of the reviewers' responses are likely to refer to real characteristics of the elements?
Our approach: consider each term in isolation. Try to reject the null hypothesis that each reviewer uses the given term with an individual base rate in a way that is uncorrelated with the given element.
In order to reject this null hypothesis, we infer a base rate of usage for the term for each individual reviewer, based on that reviewers responses. Then, using this base rate, compute the expected distribution of "votes"; that is, the likelihood that a given element would receive one use of the term, two uses of the term, etc. From this distribution, we can determine the expected variance of a set of samples, and, more importantly, the distribution of expected variances of samples.
Finally, we compute the variance of the set of the numbers of votes for the full set of elements, and determine the likelihood that this variance occurred by chance.
Does this sound reasonable? More important: is there a standard name for this process?