3
$\begingroup$

I have a survey asking a person's gender (M/F), geographic region (West, East, Midwest, South) and a few other demographic variables along with two dozen questions in which they can disagree / slightly disagree / slightly agree / agree. The data was given to me and whoever worked on the data has refactored the responses to a simple disagree / agree.

I am interested in whether people of different gender / region / other demographic variable have differences in responses. For example, Q1 could be something similar to 'You are stressed often' and the responses are either agree / disagree. I would want to know if gender is related to agreeing/disagreeing with Q1.

The analysis suggested to me was to build a set of contingency tables for every combination of demographic vars and questions. For example: Gender vs. Q1, Region vs. Q1, Gender vs Q2 and so on. Then I would use Fisher's exact test or the chi-squared test of independence on each of the contingency tables.

Example table:

    Q1: Agree Disagree
Male       10       30
Female     15       32

I have two dozen questions and at least 4 demographic variables. If I were to do all combinations I would have to run at least 96 chi-squared tests. Obviously this also brings in a multiple testing issue.

As an alternative route, a colleague of mine suggested logistic regression with the following model:

Question Response ~ gender + region + other demographic vars + interactions.

This would mean fitting slightly fewer models but will still have the same issue with multiple testing. I would still need to fit a logistic regression to each question. There's also the issue of interpretability: it's a lot easier to explain the results of a chi-square test than to explain what a logistic regression is.

Which technique is preferable? Is there a better way to handle this data? In either case, how should I handle the multiple testing?

Improve this question
$\endgroup$
1
  • $\begingroup$ The logistic regression model your colleague has suggested is likely problematic, if you have a sample size as low as 87, as suggested by the example table you give. See stats.stackexchange.com/questions/26016/… $\endgroup$
    – J-J-J
    Commented Nov 19 at 6:48

2 Answers 2

Reset to default
1
$\begingroup$

Is there any rhyme or reason to the two dozen questions that were asked? Are they all trying to measure the same thing in different ways? If so, you might be able to create one composite score and see if that differs by gender or region, perhaps using a two-way ANOVA.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Unfortunately, the questions are not related enough to aggregate them into a composite score. $\endgroup$
    – stats_noob
    Commented Jun 10, 2014 at 13:04
  • 1
    $\begingroup$ Two further thoughts. First, if you still have the original data, you might want to use the 4 point scale rather than the 2 point, derived scale. Second, if you use the multiple chi-square approach, how much you worry about the inflated alpha level depends on what you will do with the findings. If you want to publish, then an experiment-wise alpha is probably in order. If you want to decide which variables to consider for future research, then you might look at both effect size and level of significance for the individual tests as you decide what to look at more in the future. $\endgroup$
    – Joel W.
    Commented Jun 11, 2014 at 21:51
  • $\begingroup$ Thanks for the suggestion. I will see if I can get the original scale; I suspect the reason the 4 point scale was refactored into simple disagree/agree is to be able to run the chi-squared test. I suspect that there will be cells in the contingency table <5 and refactoring was a hack to get around this. $\endgroup$
    – stats_noob
    Commented Jun 12, 2014 at 13:37
  • 1
    $\begingroup$ If you have ordinal data (as in your 4 point rating scale), you lose power by using a statistical test that is designed for categorical data. $\endgroup$
    – Joel W.
    Commented Jun 12, 2014 at 19:28
  • 1
    $\begingroup$ Perhaps a Kruskal-Wallis one way analysis of variance or a Jonckheere test, depending on the nature of the independent variable. $\endgroup$
    – Joel W.
    Commented Jun 13, 2014 at 19:23
0
$\begingroup$

Not sure which approach is better but I think if we have binary responses and two categories male-female, you can do something simple like a difference in proportions z-test. This will basically answer the question if there is a difference in proportion of agree between females and males for a question.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.