I am doing a systematic literature review on theories for measuring user satisfaction. I have to rate the quality of these theories, and one of the criteria should be the number of participants in the initial validation. So, we assume that each published theory was tested at least once for the initial publication (not always true, but the ones who didn't bother get a zero rating in my evaluation), and the ones who used too little participants get a rating decrease.

I don't have to be very precise in this. I will be rating overall quality on a five-point ordinal scale, and I think it will be sufficient to really have only three ratings of participants, "too little", "enough" and "a whole lot", and adjust the quality rating one step up for "a whole lot" and one step down for "too little".

But there is still the tripping point: when do I set the cutoff point between the three categories?

Note that I am dealing with very large variations in theory and experiment design here, and with generally bad reporting. I cannot create a cutoff based on common metrics for effect size such as Cohen's d, because most of these study don't report such a metric at all.

Is there any way to create rough categrization criteria across studies based on just participant number and the complexity of the theory? For the number of variables, I think that dividing the number of participants by the number of variable relations proposed by the theory will be good enough for me. (So, if study A studies the relation of one variable to satisfaction, and study B studies the relation of three variables to satisfaction, study B needs three times as many participants to be considered the same quality). But what do I do from there? I thought of making a relative comparison by using quantiles from the distributions of all studies I find, and call, say, the worst 20% of studies bad and the best 20% good. But I am not sure this will be a good strategy, as I expect a skewed distribution, and software engineering empirical experiments are known to have lower standards than those in other disciplines.

I am aware that I can't do a really good comparison here, as it requires data I don't have. But is there a way of generalizing based on number of participants which is not worse than useless?

  • $\begingroup$ I am not sure of the correct terminology here, there are probably better ways to tag the question. $\endgroup$ – rumtscho May 29 '14 at 18:57
  • $\begingroup$ You can usually calculate Cohen's d from information available in the study. $\endgroup$ – jona May 30 '14 at 7:42

There's no way to estimate what large or small Ns are unless you have some estimate of effects. So, the first thing you need to do is work out the range of plausible effect sizes. That will vary across models and model parameters. That's OK. The observed effects will also vary across studies with significant findings with the observed effects being higher in low N studies. You already recognize that findings of low N studies are less plausible so you should get a weighted average effect placing more value on the large N findings. Essentially, you need to first do a meta-analysis before critiquing the Ns.

After that you could use power analysis to work out cutoffs for adequate sample sizes. If you have a reasonable estimate of the effect then it's pretty simple. For example, if you have an effect size of 0.6 using t-tests then anything under 30 subjects per group is going to be less than 60% of finding the effect. With 45/group you get a power of 0.8 but you have to go all the way up to 60 to get 0.9. I'm not suggesting those levels per se but you should be able to come up with a reasonable argument to pick some values.

Even better would be to just report the post-hoc power of the studies using the aggregate effect size (or range of power from plausible range of effect sizes). This could be used as a way to generate a critique of the subject numbers in studies without having to generate some arbitrary cutoffs.

  • $\begingroup$ Thank you for this, but I don't think you understood the whole problem. I don't have a single effect I am looking at; I am looking at dozens of effects. Unlike a medical review which asks "how strong is the effect of treatment A on disease B", I am asking "which concepts have ever been considered to have any effect on satisfaction", and the effect size is obviously never the same. And also, the studies employ very different methods, so I am afraid that a size which will be a good cutoff for a t-test will be bad for a study using semantic equation modeling. $\endgroup$ – rumtscho May 29 '14 at 20:00
  • 1
    $\begingroup$ No, I'm not missing the problem. The effects have a size. You can utilize them to estimate power. I've simplified the example I gave. Feel free to complicate it for your purposes. BTW, are you thinking structural equation modelling or semantic modelling? I've never heard of the semantic equation modelling terminology you're using. $\endgroup$ – John May 29 '14 at 22:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.