0
$\begingroup$

I have a number of data such as the following. Each represents a trial where two options (V1 and V2) were given to a participant ("judge") and they were asked to pick one over the other. V1 and V2 can fall into one of 5 categories, labeled below as 1, 2, 3, 4 or 5. All participants will see the same set of pairings.

The category of V1 and V2 is not systematic -- a participant might be asked to compare 2 to 5, 3 to 1 or even 4 to 4. They also do not occur an equal number of times. For the purposes of this post, I am looking at the data from 2 judges, who gave preferences for ~144 pairings (~288 items which could be of a category 1 through 5).

The wrinkle that I'm struggling with is that the categories appear an irregular number of times, i.e. the judges might see 16 cat5s, 3 cat4s, 37 cat3s, 8 cat2s and 4 cat1s. As such, reporting that 5 was chosen 30 times and 4 only twice in data from two participants is not useful. Rather, it's that 16 in 32 chances (16 occurrences of 5 times 2 participants) went to 5, while 3 in 8 chances went to 4. But I'd like an extensible, normalized way to represent this.

| P    | C   | V1  | V2  | Pref |
|------|-----|-----|-----|------|
| A-09 | 6   | 3   | 2   | V1   |
| B-02 | 12  | 5   | 1   | V2   |
| A-32 | 7   | 3   | 4   | V1   |
| B-93 | 11  | 4   | 5   | V2   |
| A-44 | 8   | 3   | 2   | V1   |
| ...  | ... | ... | ... | ...  |

P and C are arbitrary identifiers.

I want to know to what extent participants chose, e.g., cat5 as compared to the other options, normalized by the number of occurrences of that category in the set of pairings shown to the judges. For now, I don’t need anything fancy, just a descriptive measure. In pursuit of this, I have done the following.

First, I recast the data into the following format:

| P    | C   | Preferred V |
|------|-----|-------------|
| A-09 | 6   | 2           |
| B-02 | 12  | 5           |
| A-32 | 7   | 4           |
| B-93 | 11  | 5           |
| A-44 | 8   | 3           |
| ...  | ... | ...         |

I then calculated various totals:

| term     | definition                                                                     
|----------|----------------------------------------------------------------------------------------------
| vFreq    | Total number of times each version (1-5) appears in the set of pairings given to the judges. 
| vPref    | Total number of times each version (1-5) was chosen by one judge or the other.              
| trials   | Total number of measures (~288, actually 274 with missing data accounted for).         

I now want to in some way weight the vPref values by the vFreq values, relative to vFreq/total.

My initial thought was something like:

For each possibile version (1-5)

Weighted Frequency (wFreq) = vPref * (vFreq/trials)

This gives me something like:

| v score | weightedPref | prefFreq | totalFreq | weight  | prefPct |
|---------|--------------|----------|-----------|---------|---------|
| 1       | 0.47         | 8        | 16        | 5.84%   | 2.93%   |
| 2       | 4.53         | 31       | 40        | 14.60%  | 11.36%  |
| 3       | 17.12        | 67       | 70        | 25.55%  | 24.54%  |
| 4       | 1.07         | 14       | 21        | 7.66%   | 5.13%   |
| 5       | 70.92        | 153      | 127       | 46.35%  | 56.04%  |
| SUM     | 94.10        | 273      | 274       | 100.00% | 100.00% |

Some notes: vPref represents the choices of two judges combined, i.e. how many times a category was chosen across all trials of two judges. totalFreq represents the number of times a given category appears in the set of pairings presented to judges. It happens that total of the vPref column ~= the some of the totalFreq colum (the total number of items), but this is coincidence (it's because there are two judges who looked at every pairing, and two items per pairing). The vPref numbers could go up (with more judges) without vFreq being effected.

I’m honestly not sure I understand what the wFreq measure is telling me. Is there a better way to accomplish something like this? Is there a way to make sense of my wFreq? How can I describe what wFreq means to lay people?

$\endgroup$
  • $\begingroup$ I don't understand (a) why you are doing all these calculations and (b) how it's possible you could propose them, perform them, and then claim you don't understand what they mean! As to (a), since your question asks "to what extent participants chose, e.g., 5 as compared to the other options," what is the matter with reporting the relative frequencies of each of the five options? $\endgroup$ – whuber Oct 6 '17 at 19:13
  • $\begingroup$ Thanks for your response. I think what I've failed to communicate is that the number of occurrences of each of the versions is different. That is, 5 shows up as an option many more times than 4, and 3 also occurs much more often as a choice than 4. So while 5 was chosen 155 times, it was available as a choice 122 times. 3 was chosen 67 times, but it only was only available to choose 70 times. I want a way to represent the relationship between a version's times being chosen, and its number of occurrences as an option. $\endgroup$ – Tyler Peckenpaugh Oct 6 '17 at 20:38
  • $\begingroup$ I don't follow, because I don't see any plausible way an option could be chosen more times (e.g., 155) than it was available as a choice (e.g., 122 times). $\endgroup$ – whuber Oct 6 '17 at 20:42
  • $\begingroup$ This is because the 155 represents data from two judges, e.g. one person chose 5 in 75/122 cases, and another chose it in 80/122 cases. $\endgroup$ – Tyler Peckenpaugh Oct 6 '17 at 20:49
  • $\begingroup$ The role of the judges is obscure. It would help immensely to explain the data more clearly, right at the outset of the question. $\endgroup$ – whuber Oct 6 '17 at 20:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.