Chi Square test for survey data

Question

I am working with a product that turns survey data into useful statistics. Reviewing their code, has made me somewhat nervous, and I'm not a statistician, so I hope I can ask for clarity of the following problem:

Out of a survey S, for a product P. Respondents where asked if they

1. liked the product
2. were indifferent
3. hated the product

The group of respondents were separated into men and women. The chart supplied by the software when crunching some survey data, says that "Men are significantly more likely to be likers." OR "Men and Women..." Or "Women..."

For me this already raises issues:

1. Men are significantly more likely to be likers than what?
2. Men and Women are more signicantly likely to be likers than what?
3. How are these things measured?
4. What test is being used?... etc.

When I had a look at the code, I noticed they were using a chi-test(!). I had to ask what exactly the null hypothesis was, because this was making less and less sense. Apparently the null hypothesis is that "the chance that men and women are likers is the same" ...ok, fine. But wait.

So, we have the following table:

                Men       Women      Total
 likers         54        46         100

 indifferent    23        26         49

 non-likers     22        31         53

 Total          99        103        202

We can populate the expected distributions for all three rows:

                Men          Women      Total
 likers         54-49        46-50      100

 indifferent    23-24        26-24      49

 non-likers     22-25        31-27      53

 Total          99           103        202

The code then populates a matrix with chi values based on the above. The programmer decided that the degrees of freedom when doing these calculations was (m-1)(n-1) = 2, which at this point made me think the null hypothesis was rather that if you are a liker, indifferent or a non-liker, there is equal probability that you are a man or a woman.

We're using a 90% confidence level, so all I imagined we needed to do was to sum over all the 6 chi values, and compare that with a critical value given by the degrees of freedom and the confidence interval. From that point we could say with 90% certainty that men and women were equally as likely to be a liker, etc... or reject the N.H.

This is what the code does instead:

1. It uses 1 degree of freedom instead of 2 (still at 90%), so we have a new critical value 2.706
2. For each row (liker, etc...) of the chi value matrix, if an element is greater than the critical value reject the null hypothesis, and add the element to a 'significance' list.

To illustrate, it looks at [likers;men] > cv i.e. chi_value[0][0] > cv, if that is true, reject N.H., and add 'men' to the list.

On the chart this result is reflected as: men are more likely to be likers. For me this single evaluation of men and women for each row seems wrong. It doesn't make sense to make pronouncements about two variables when you're only looking at one...

I am not nearly as smart as my boss, but I feel like something has gone wrong here and I would appreciate it if someone could help clarify this.

Lastly the client has asked to know the % more likely men are to be likers than women -- I think this is an erroneous request, as a chi square test does not address questions of which is greater or smaller, but only serves to confirm that a set of variables are independent. Am I right?

I just want to add, that I used the following statement to guide my thinking:

Cautionary Note It is important to keep in mind that the chi-square test only tests whether two variables are independent. It cannot address questions of which is greater or less. Using the chi-square test, we cannot evaluate directly the hypothesis that men are likers more than girls; rather, the test (strictly speaking) can only test whether the two variables Like and Gender, are independent or not.

Answer 1

It appears that you are first doing an omnibus test (Chi square test for independence) with 2 df to determine if the "like status" and "gender" are independent or not. And then you are doing post-hoc tests on the individual rows (Chi square goodness of fit tests) to see if the males/females are equally likely under each row. According to This Link under the section "Post Hoc Follow-up Tests", these post-hoc tests are allowable. Each row would generate a Chi square test with 1 df. They would test, for instance "Ho: men and women 'are likers' at the same rate", for each row.

However, I am leery that no adjustment was made for multiple comparisons. Since it appears you are doing three of these 1 df tests, you should adjust your $\alpha$ to correct the familywise error rate (Bonferroni correction for instance).

If your client wants to know how much more likely men are to be a "liker", etc. you could (a), provide a point estimate based on your data as Peter Flom suggested, or (b) you could construct a CI for the difference between the two proportions if you want an interval estimate. Along with the statement that the difference is significant (or not significant), my guess is that a point estimate would suffice for your clients.

Other than the problem with not controlling the familywise error rate, the analysis seems adequate to me. I hope this helps.

Answer 2

The portion after "this is what the code does instead" seems off, although it is hard to tell.

The client's request is reasonable. It isn't answered by chi-square, but it still a reasonable request. The proportion of men who liked it is 54/99 = about 54%, of women it is 46/103 = about 46% (you can calculate the exact values) so the difference is about 8%.

The chi-square reported here is about two variables: Liking and sex. Specifically, it looks at whether they are associated. Given that one variable is ordinal, there are more powerful tests that regular chi-square.