Apologies for the ambigious title - I wasn't sure how a problem like this is expressed.
I have a small sample of 20 individuals, of which the dependent variable (of interest) is binary with 5 individuals classified as 1 (and therefore 15 classified as 0). I am looking to explore the affect of productivity on the probability of an individual being classified as 1.
I have two measures (of which I will likely use one, due to the limited number of individuals classified as 1):
- The number of pieces of work that individual did which, when reviewed, did not require any major changes.
- The percentage of all their work which, when reviewed, did not require any major changes (so this is effectively the first measure divided by the number of pieces of work they did in total).
A piece of work is reviewed by an reviewer from a group of 'manager-level' staff. Each individual has had a piece of work reviewed by at least 1 reviewer. Each reviewer may have reviewed multiple pieces of work for an individual. In some cases, there has been up to 12 reviewers across 12 or more pieces of work for one individual. While the work produced is technical in nature, I have no doubt there will be variation amongst the 'manager-level' staff as to what is considered work that doesn't require any major changes and what is considered work that does require major changes.
Assuming I do NOT have any idea of which reviewers reviewed which pieces of work and for what individuals, what is the risk of running some basic analysis? I am presuming any output will almost certainly be biased? Can I run some sort of simulation to show just how biased they could be (by assuming a distribution in the 'strictness' of reviewers)?
Assuming I can obtain the following:
Who the reviewers are
How many pieces of work they reviewed for each individual and whether they considered the work to need a major change or not.
Is there anything I can do to incorporate the 'manager-level' effect with such a limited sample size?