5
$\begingroup$

I am working with a product that turns survey data into useful statistics. Reviewing their code, has made me somewhat nervous, and I'm not a statistician, so I hope I can ask for clarity of the following problem:

Out of a survey S, for a product P. Respondents where asked if they

  1. liked the product
  2. were indifferent
  3. hated the product

The group of respondents were separated into men and women. The chart supplied by the software when crunching some survey data, says that "Men are significantly more likely to be likers." OR "Men and Women..." Or "Women..."

For me this already raises issues:

  1. Men are significantly more likely to be likers than what?
  2. Men and Women are more signicantly likely to be likers than what?
  3. How are these things measured?
  4. What test is being used?... etc.

When I had a look at the code, I noticed they were using a chi-test(!). I had to ask what exactly the null hypothesis was, because this was making less and less sense. Apparently the null hypothesis is that "the chance that men and women are likers is the same" ...ok, fine. But wait.

So, we have the following table:

                Men       Women      Total
 likers         54        46         100

 indifferent    23        26         49

 non-likers     22        31         53

 Total          99        103        202

We can populate the expected distributions for all three rows:

                Men          Women      Total
 likers         54-49        46-50      100

 indifferent    23-24        26-24      49

 non-likers     22-25        31-27      53

 Total          99           103        202

The code then populates a matrix with chi values based on the above. The programmer decided that the degrees of freedom when doing these calculations was (m-1)(n-1) = 2, which at this point made me think the null hypothesis was rather that if you are a liker, indifferent or a non-liker, there is equal probability that you are a man or a woman.

We're using a 90% confidence level, so all I imagined we needed to do was to sum over all the 6 chi values, and compare that with a critical value given by the degrees of freedom and the confidence interval. From that point we could say with 90% certainty that men and women were equally as likely to be a liker, etc... or reject the N.H.

This is what the code does instead:

  1. It uses 1 degree of freedom instead of 2 (still at 90%), so we have a new critical value 2.706
  2. For each row (liker, etc...) of the chi value matrix, if an element is greater than the critical value reject the null hypothesis, and add the element to a 'significance' list.

To illustrate, it looks at [likers;men] > cv i.e. chi_value[0][0] > cv, if that is true, reject N.H., and add 'men' to the list.

On the chart this result is reflected as: men are more likely to be likers. For me this single evaluation of men and women for each row seems wrong. It doesn't make sense to make pronouncements about two variables when you're only looking at one...

I am not nearly as smart as my boss, but I feel like something has gone wrong here and I would appreciate it if someone could help clarify this.

Lastly the client has asked to know the % more likely men are to be likers than women -- I think this is an erroneous request, as a chi square test does not address questions of which is greater or smaller, but only serves to confirm that a set of variables are independent. Am I right?

I just want to add, that I used the following statement to guide my thinking:

Cautionary Note It is important to keep in mind that the chi-square test only tests whether two variables are independent. It cannot address questions of which is greater or less. Using the chi-square test, we cannot evaluate directly the hypothesis that men are likers more than girls; rather, the test (strictly speaking) can only test whether the two variables Like and Gender, are independent or not.

$\endgroup$
1
$\begingroup$

It appears that you are first doing an omnibus test (Chi square test for independence) with 2 df to determine if the "like status" and "gender" are independent or not. And then you are doing post-hoc tests on the individual rows (Chi square goodness of fit tests) to see if the males/females are equally likely under each row. According to This Link under the section "Post Hoc Follow-up Tests", these post-hoc tests are allowable. Each row would generate a Chi square test with 1 df. They would test, for instance "Ho: men and women 'are likers' at the same rate", for each row.

However, I am leery that no adjustment was made for multiple comparisons. Since it appears you are doing three of these 1 df tests, you should adjust your $\alpha$ to correct the familywise error rate (Bonferroni correction for instance).

If your client wants to know how much more likely men are to be a "liker", etc. you could (a), provide a point estimate based on your data as Peter Flom suggested, or (b) you could construct a CI for the difference between the two proportions if you want an interval estimate. Along with the statement that the difference is significant (or not significant), my guess is that a point estimate would suffice for your clients.

Other than the problem with not controlling the familywise error rate, the analysis seems adequate to me. I hope this helps.

$\endgroup$
  • $\begingroup$ So I do an omnibus test to test that "like, ind', non-liker" are independent with gender. If they are not independent with gender, I do the Post Hoc Follow-up Tests for each row, and use Bonferroni correction. This will tell me what exactly? I didn't quite understand. But this is very helpful. $\endgroup$ – dominic Dec 24 '13 at 10:43
  • $\begingroup$ That is correct. The post hoc tests will individually test whether "liker" is equally seen in males and females; same for "ind" and "nonliker". It's basically drilling into the data to find out which rows are showing significance difference between genders. $\endgroup$ – Underminer Dec 24 '13 at 15:26
  • $\begingroup$ Thanks for your help. So I did the omnibus as an independence test, the post hoc tests allow me to test for goodness of fit for each row. I used k = r!/2!(r-1)! * c!/2!(c-1)! and my new alpha become a = a/k. If I rejected Ho, I reported that sex influences the response, and gave a point estimate, but did not state that men were more likely than women (or visa versa). In the case of accepting the null hypothesis, should I inspect the p value, just to make sure the test statistic is acceptable for say 0.05 ? $\endgroup$ – dominic Dec 27 '13 at 12:47
2
$\begingroup$

The portion after "this is what the code does instead" seems off, although it is hard to tell.

The client's request is reasonable. It isn't answered by chi-square, but it still a reasonable request. The proportion of men who liked it is 54/99 = about 54%, of women it is 46/103 = about 46% (you can calculate the exact values) so the difference is about 8%.

The chi-square reported here is about two variables: Liking and sex. Specifically, it looks at whether they are associated. Given that one variable is ordinal, there are more powerful tests that regular chi-square.

$\endgroup$
  • $\begingroup$ Hi, the point estimate you are talking about does not give you that "men are x% more likely than women to like Product P". It gives you "Men liked this product 8% more than Women". If there is a significance between Gender and Liking, the chi test will only address this, and only this - i.e. they the variable are not independent. As to which gender is significantly more likely to to be Liker, that is something that may require other tests? Am I right? $\endgroup$ – dominic Dec 27 '13 at 7:14
  • $\begingroup$ The language around % differences gets confusing. $\endgroup$ – Peter Flom Dec 27 '13 at 12:29
  • 2
    $\begingroup$ Although somebody is repeatedly flagging the preceding comment, I see nothing in the least offensive or incorrect about it and therefore have been dismissing the flags, which I will continue to do if they recur. $\endgroup$ – whuber Dec 31 '13 at 15:17
  • $\begingroup$ Thanks for your help Peter, could I ask you to explain what you mean with a little more detail? $\endgroup$ – dominic Jan 13 '14 at 13:00
  • 1
    $\begingroup$ Sure. Let's say, for instance, that in 2012 20% of respondents say "Yes" to some question. In 2013, 25% say "yes" to the same question. Is that a 5% improvement (25-20)? Or is it a 25% improvement ((25-20)/20)? Or possibly it's 6.25% ((25-20)/80)? $\endgroup$ – Peter Flom Jan 13 '14 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.