I am working with a product that turns survey data into useful statistics. Reviewing their code, has made me somewhat nervous, and I'm not a statistician, so I hope I can ask for clarity of the following problem:
Out of a survey S, for a product P. Respondents where asked if they
- liked the product
- were indifferent
- hated the product
The group of respondents were separated into men and women. The chart supplied by the software when crunching some survey data, says that "Men are significantly more likely to be likers." OR "Men and Women..." Or "Women..."
For me this already raises issues:
- Men are significantly more likely to be likers than what?
- Men and Women are more signicantly likely to be likers than what?
- How are these things measured?
- What test is being used?... etc.
When I had a look at the code, I noticed they were using a chi-test(!). I had to ask what exactly the null hypothesis was, because this was making less and less sense. Apparently the null hypothesis is that "the chance that men and women are likers is the same" ...ok, fine. But wait.
So, we have the following table:
Men Women Total likers 54 46 100 indifferent 23 26 49 non-likers 22 31 53 Total 99 103 202
We can populate the expected distributions for all three rows:
Men Women Total likers 54-49 46-50 100 indifferent 23-24 26-24 49 non-likers 22-25 31-27 53 Total 99 103 202
The code then populates a matrix with chi values based on the above. The programmer decided that the degrees of freedom when doing these calculations was (m-1)(n-1) = 2, which at this point made me think the null hypothesis was rather that if you are a liker, indifferent or a non-liker, there is equal probability that you are a man or a woman.
We're using a 90% confidence level, so all I imagined we needed to do was to sum over all the 6 chi values, and compare that with a critical value given by the degrees of freedom and the confidence interval. From that point we could say with 90% certainty that men and women were equally as likely to be a liker, etc... or reject the N.H.
This is what the code does instead:
- It uses 1 degree of freedom instead of 2 (still at 90%), so we have a new critical value 2.706
- For each row (liker, etc...) of the chi value matrix, if an element is greater than the critical value reject the null hypothesis, and add the element to a 'significance' list.
To illustrate, it looks at [likers;men] > cv i.e. chi_value > cv, if that is true, reject N.H., and add 'men' to the list.
On the chart this result is reflected as: men are more likely to be likers. For me this single evaluation of men and women for each row seems wrong. It doesn't make sense to make pronouncements about two variables when you're only looking at one...
I am not nearly as smart as my boss, but I feel like something has gone wrong here and I would appreciate it if someone could help clarify this.
Lastly the client has asked to know the % more likely men are to be likers than women -- I think this is an erroneous request, as a chi square test does not address questions of which is greater or smaller, but only serves to confirm that a set of variables are independent. Am I right?
I just want to add, that I used the following statement to guide my thinking:
Cautionary Note It is important to keep in mind that the chi-square test only tests whether two variables are independent. It cannot address questions of which is greater or less. Using the chi-square test, we cannot evaluate directly the hypothesis that men are likers more than girls; rather, the test (strictly speaking) can only test whether the two variables Like and Gender, are independent or not.