1
$\begingroup$

I have data looking like this:

group refusal proportion (judge A) refusal proportion (judge B) refusal proportion (judge C)
group 1 0.2 0.3 0.05
group 2 0.1 0.5 0.6
group 3 0.6 0.9 0.7
group 4 0.09 0.12 0.1
group 5 0.55 0.5 0.3
group 6 0.2 0.15 0.15
... ... ... ...
group 500 0.55 0.5 0.3

Context (data and study design)

Below is all the information I have about this data, which hopefully is relevant to answer my question (see the next sections for my question and the hypothesis I want to test).

People in groups 1 to 500 (first column) can request something from judge A, B, and/or C, who can deny or accept the request. The table shows the proportion of refusal for each judge (each cell = total number of refusals by judge X for group N, divided by the total number of requests to judge X from group N).

If needed I also have the count of total requests and refusals in each cell.

An individual can belong to one group only, can make a single request to each judge, but can make one request simultaneously to multiple judges - so the same individual can possibly be counted multiple times in the same row, at most 3 times.

There is no particular order among the judges or groups, the labels "A, B, C" and "1,2,3,4...500" are arbitrary and are just used as identifiers.

I don't know the exact number of people in each group, however logically it cannot be inferior to the largest cell count of the considered row.

The number of total requests can be different for each cell (e.g. group 1 can make 50 requests to judge A, 200 requests to judge B, and 1000 requests to judge C). The number of requests can vary a lot for each cell (from 10 to 50,000), but generally there are a few thousand requests in any given cell, and low numbers of requests (< 30) are a pretty rare occurrence (< 2% of all cells).

A potential problem with the data is that I can't identify what requests were done by the same individual (for example I can't say if the same individual made a request simultaneously to judge A, B and C, or made a request to judge B and C only, or to judge A only).

Hypothesis to test

My hypothesis is that judge A, judge B, and judge C have positively "correlated" refusal rates (I don't know if "correlated" is the right term here, glad to be corrected), that is to say when the rate is relatively high for A, it will be relatively high for B and C, and when it is relatively small for A, it will be also relatively small for B and C.

Another way of putting it is that if a judge denies a lot of requests from a given group, other judges will be more likely to deny requests from the same group.

Additional information

It's secondary data from an observational study conducted years ago, so unfortunaly it's very difficult or impossible to collect additional data to correct possible flaws or missing information in the data.

My question

Is there a way to test my hypothesis with this data, or is it a lost cause?

$\endgroup$

2 Answers 2

1
+50
$\begingroup$

This question is often referred in social sciences (especially psychometrics) as "rater agreement" or "interrater reliability", and there's a lot of literature about it, with ready-to-use tests and everything. See for example here, here, here and here. The only problem is that usually rater agreement relies on having the whole data. In your case, I believe that if you are able to transfer the proportions into integers (multiplying by 100 or some other factor) then you're able to go, as your question regards some "global" correlation of the judges and not a specific group.

PS. The wording "Another way of putting it is that if a judge denies a lot of requests from a given group, other judges will be more likely to deny requests from the same group" is wrong. The "other way" (if one rejects, the others reject too) reflects conditional probability, which is not a part of this question.

$\endgroup$
0
$\begingroup$

If you want to see if some things are correlated, why not just run correlations?

You can, if you wish, then test whether they are significantly different from 0.

$\endgroup$
1
  • $\begingroup$ Thanks. I was wondering if the nature of the data (proportions, partial dependence between certain cells) was precluding using "vanilla" correlation. But given your answer, it sounds like I'm being overcautious here. $\endgroup$
    – Daniela
    Commented Oct 29, 2023 at 11:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.