2
$\begingroup$

I do a community survey on a website once each year, I have data from the past two years and this year I'd like to remove questions which are getting similar answers each year.

The survey questions are all categorical data so for simplicity let's stick to the idea of each answer being a Bernoulli trial.

I understand that in null hypothesis testing we can prove to a certain significance that two samples have different population means but we cannot prove that two samples have the same mean. Even if sample means are very close they could have different population means which are very similar so it is impossible to test.

I'm aware that we can look at the effect size of the difference between means, for example Cohen's d. If d is very small it is an indication that means are the same.

My question is this: Is there any standard level of effect size below which we assume that the means are equal, similar to how p=0.05 is a standard (but arbitrary) value for rejecting that the means are equal. If I was in industry and was paying for this survey to be done is there a cut-off point where I would say "The means aren't changing, we are going to stop funding this survey"?

$\endgroup$
0
$\begingroup$

let's stick to the idea of each answer being a Bernoulli trial.

Then you generally shouldn't use Cohen's $d$ -- especially not with small samples or proportions near 0 or 1. If you're interested in effect size with proportions, there's measures for those.

I'll answer as if Cohen's $d$ was a reasonable thing to use (let's say you're dealing with some numeric variable where variance and mean are not related).

If d is very small it is an indication that means are the same

No, it isn't, for exactly the same reason that failure to reject in the hypothesis test doesn't tell you they're the same. The premise of your question is simply wrong.

Is there any standard level of effect size below which we assume that the means are equal,

Aside from the fact that you can pretty safely assume that the means do actually differ (can the population means be exactly equal? In the sort of situation where you would calculate Cohen's $d$, essentially never), let's consider the slightly more useful question of whether there's a standard effect size which you can act as if the means were equal.

The answer should still usually be no:

  1. What makes for a large difference should depend on what you're doing

  2. What is accepted as making for a large difference can vary from area to area. A high energy physicist won't see things at all the same as an economist and both will (and should) see things differently from a psychologist (and not all of them will necessarily think in terms of Cohen's $d$. (there are suggestions by Cohen about what constitutes large or small, but those won't make sense far outside the area he was writing for -- and are overly prescriptive within it, or at least are overly prescriptive the way they're used in practice)

  3. What's accepted as a small $d$ may not be the same thing as what's reasonable to treat as identical. Cohen has suggested what might be regarded as small in terms of effect size but what it's reasonable to treat as equal is not quite the same thing.

  4. What you decide you could treat as equal should also depend on the consequences of the two choices.

If I was in industry and was paying for this survey to be done is there a cut-off point where I would say "The means aren't changing, we are going to stop funding this survey"?

No - and there should not be (Which industry? What are you measuring? What are the costs/benefits of the two choices -- treating the means as equal or unequal?)

Within some specific application areas (which you don't mention) there may well be a tendency to treat a particular Cohen's $d$ the way you suggest, but it would depend on the area, and even then should depend on more than just that.

You may want to look at some of the work on equivalence and noninferiority testing. (There are numerous posts on site relating to this topic, some offer references.)

$\endgroup$
4
  • $\begingroup$ Thanks this cleared things up, what I thought was really interesting was point 2 in your list that even within the field which Cohen was writing for the large/small values are overly prescriptive. I guess the big difference between standard p values and standard effect sizes is that although what a particle physicist and a psychologist consider to be small p values are different, we are still able to apply typical p values like 0.95 to many fields because it is a probability which has a meaningful value whereas the meaning of a given effect size depends on the situation a lot more. $\endgroup$
    – Hugh
    Jan 16 '16 at 0:28
  • $\begingroup$ When I stated that "If d is very small it is an indication that means are the same" I meant an "indication" not in any formal manner. I thought this would be clear since I made a little explanation of why failure to reject equal means doesn't mean that means are equal. I was asking the question for an informal setting. I guess I should have made that more clear. $\endgroup$
    – Hugh
    Jan 16 '16 at 0:35
  • $\begingroup$ I know you had that word there -- but I responded to what you wrote. It does not indicate anything of the sort. What you actually said -- your actual premise with the word indication in there -- was still wrong. I didn't say the situation was the same as hypothesis testing, I said the reason why it was wrong was the same (that they're not very different doesn't suggest they're the same). Consider sampling a group of Italian and a group of Swedish males we find an tiny average difference in heights. Is that in any sense an "indication" that the population means are actually the same? (No!) $\endgroup$
    – Glen_b
    Jan 16 '16 at 0:59
  • $\begingroup$ You should probably not accept my answer so quickly; it may discourage other (possibly better) answers. Give it a day or maybe even two. (It's a good thing to accept an answer when there's one that's acceptable, but you don't want to flag your problem as solved too early.) $\endgroup$
    – Glen_b
    Jan 16 '16 at 1:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.