1
$\begingroup$

I just read: http://www.r-statistics.com/2010/02/post-hoc-analysis-for-friedmans-test-r-code/

Here is the example from the blog post:

Let’s make up a little story: let’s say we have three types of wine (A, B and C), and we would like to know which one is the best one (in a scale of 1 to 7). We asked 22 friends to taste each of the three wines (in a blind fold fashion), and then to give a grade of 1 till 7 (for example sake, let’s say we asked them to rate the wines 5 times each, and then averaged their results to give a number for a persons preference for each wine. This number which is now an average of several numbers, will not necessarily be an integer).

Why let them rate the wine "5 times each"? This is just an arbitrary number. More importantly, how do you know that "5" is enough? How should you define "enough"? Is "4" or "2" also enough? Are there methods to quantify how good a sample size is?

For my personal problem, I have to test with datapoints, where the mean is around 100 and the standard deviation is 300. This is averaged over 500 samples, but is this enough, for such a huge variance?

$\endgroup$

3 Answers 3

3
$\begingroup$

Having them rate each wine five times is useful especially when they are tasted blind, i.e.: if they don't know which wine they are drinking. This may help avoid bias due to the order of the tasting (by randomizing the 15 tastings) or different circumstances at the time of each tasting (maybe the first wine is tasted on a sunny day, the other on a day where the test person just got divorced).

Whether 5 is/was 'enough' can only truly be assessed when there is an estimate of the variance in grades given to each wine, and even then it wil depend upon your goal.

I don't understand how your own problem relates to the wine example ('the' mean and 'the' standard deviation? You already know this? Then why/what do you need to test?). If you clarify, I'll edit my answer.

$\endgroup$
1
  • $\begingroup$ Picking "5" is arbitrary. Why not "4" or "6"? $\endgroup$
    – beza1e1
    Commented Jun 17, 2011 at 10:29
3
$\begingroup$

Yes the 5 was arbitrary, but you can look in textbooks on survey sample design and they have formulas and algorithms for determining sample sizes within clusters and numbers of clusters. These take into account things like the within cluster/person variability, between cluster/person variability (in the example you have an upper bound on the variability since the data has to be in the 1-7 range), cost per cluster/person, cost per measurement within cluster/person, and sometimes more. If the above example had gone into detail on exactly how they came up with 5 using the above then it would have been much longer and would probably have distracted from their main point.

$\endgroup$
0
$\begingroup$

There are a bundle of distinct sampling issues, which are intertwined with the validity of the rating procedure:

  1. The first relates to reliability. You need to be sure the raters are are self-consistent across trials ("intra-rater reliability") and the (self-consistent) raters consistent with on another ("inter-rater reliability") in their ratings. Otherwise, you will not be able to draw any inferences about whether their ratings are replicable. Since individual ratings are noisy, the greater the number of trials you perform with each individual rater, the more confidence you'll have that he or she is reliable. However, even if each rater performs on only one trial (ranks A, B, & C based on one taste each), you'll be able to assess inter-rater reliability--w/ more precision as the number of raters increases. You won't be able to say much w/ confidence in that case, though, about how reliable any one of your raters is (or how much he or she differs from the others), given how noisy the individual ratings are likely to be-- but you might not care about that (you will if you want to find a team of competent raters to use repeatedly; won't if you are doing a marketing test, etc.). These are all measurement error issues.

  2. You also have sampling error issues to consider. Are you trying to estimate how members of the general population are likely to rate the quality of the wines? How a collection of "experts" would rate the wines? How your "friends" -- people, say, you might invite to your next party -- would feel? Whatever the answer, you need to draw a sufficient number of raters from the relevant population.

There are formulae for assessing these maters & heuristics that go along with them. Applying them will likely require judgment given what the goal of the rating task is (e.g., to market a new product generally, to estimate how some subgroup will respond, to come up w/ "expert ratings" etc.). Likely someone else will be able to reel them off-- I can't. I can tell you, however, that these are standard issues in marketing research & if you look for a text on Rasch measurement, you'll be able to see how people in that field tend to think about this (including how they feel about use of a likert item for this purpose).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.