I am doing a systematic literature review on theories for measuring user satisfaction. I have to rate the quality of these theories, and one of the criteria should be the number of participants in the initial validation. So, we assume that each published theory was tested at least once for the initial publication (not always true, but the ones who didn't bother get a zero rating in my evaluation), and the ones who used too little participants get a rating decrease.
I don't have to be very precise in this. I will be rating overall quality on a five-point ordinal scale, and I think it will be sufficient to really have only three ratings of participants, "too little", "enough" and "a whole lot", and adjust the quality rating one step up for "a whole lot" and one step down for "too little".
But there is still the tripping point: when do I set the cutoff point between the three categories?
Note that I am dealing with very large variations in theory and experiment design here, and with generally bad reporting. I cannot create a cutoff based on common metrics for effect size such as Cohen's d, because most of these study don't report such a metric at all.
Is there any way to create rough categrization criteria across studies based on just participant number and the complexity of the theory? For the number of variables, I think that dividing the number of participants by the number of variable relations proposed by the theory will be good enough for me. (So, if study A studies the relation of one variable to satisfaction, and study B studies the relation of three variables to satisfaction, study B needs three times as many participants to be considered the same quality). But what do I do from there? I thought of making a relative comparison by using quantiles from the distributions of all studies I find, and call, say, the worst 20% of studies bad and the best 20% good. But I am not sure this will be a good strategy, as I expect a skewed distribution, and software engineering empirical experiments are known to have lower standards than those in other disciplines.
I am aware that I can't do a really good comparison here, as it requires data I don't have. But is there a way of generalizing based on number of participants which is not worse than useless?