Apologies for the ambigious title - I wasn't sure how a problem like this is expressed.

I have a small sample of 20 individuals, of which the dependent variable (of interest) is binary with 5 individuals classified as 1 (and therefore 15 classified as 0). I am looking to explore the affect of productivity on the probability of an individual being classified as 1.

I have two measures (of which I will likely use one, due to the limited number of individuals classified as 1):

  1. The number of pieces of work that individual did which, when reviewed, did not require any major changes.
  2. The percentage of all their work which, when reviewed, did not require any major changes (so this is effectively the first measure divided by the number of pieces of work they did in total).

A piece of work is reviewed by an reviewer from a group of 'manager-level' staff. Each individual has had a piece of work reviewed by at least 1 reviewer. Each reviewer may have reviewed multiple pieces of work for an individual. In some cases, there has been up to 12 reviewers across 12 or more pieces of work for one individual. While the work produced is technical in nature, I have no doubt there will be variation amongst the 'manager-level' staff as to what is considered work that doesn't require any major changes and what is considered work that does require major changes.

Assuming I do NOT have any idea of which reviewers reviewed which pieces of work and for what individuals, what is the risk of running some basic analysis? I am presuming any output will almost certainly be biased? Can I run some sort of simulation to show just how biased they could be (by assuming a distribution in the 'strictness' of reviewers)?

Assuming I can obtain the following:

  1. Who the reviewers are

  2. How many pieces of work they reviewed for each individual and whether they considered the work to need a major change or not.

Is there anything I can do to incorporate the 'manager-level' effect with such a limited sample size?


This is connected to rater effect and more generally to Generalizability Theory. Check out the paper by Brennan (1992) to find out more on it.

Generally, it would be helpful if you could identify the reviewers, because then you could separate the reviewers influence and individuals "ability" using ANOVA, linear mixed models (e.g. Raudenbush, 1993, Leckie & Baird, 2011) or IRT-based methods (e.g Patz et al. 2002).

If you cannot identify the reviewers then you cannot use those models (however, maybe they'll inspire you to some other approach). The problem is that I know several examples where it was hard to find variables that enabled to distinguish "strict" raters from less "strict" ones, so it could be hard for you to control this effect without information identifying the reviewers.

As I understand, the reviewers were not randomly assigned. If this is true that is problematic, because with random assignment you could limit the impact of rater bias.

You could try to run some simulations to check what are the possible scenarios.

Were all the pieces of work that individuals did the same for all individuals? If yes, you have another "constant" term in your data. In this case you could use some variation of IRT model to find out more on two latent variables you have: individuals ability and the difficulty of the tasks - this would leave you with an unknown "strictness" of different reviewers rating different tasks.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.