2
$\begingroup$

A study has been asking the same ratings questions repeatedly year over year. Currently, the study is using a Column Means test in IBM SPSS Survey Reporter to compare the number of people responding with the top responses in each period, testing the current period against the previous, and then the current period against two periods ago. (So, say, 2015 is being compared against 2014, and then separately against 2013).

In my gut, something is telling me that this is a big no-no. The column means test seems extremely similar to a t-test, for which you aren't able to do multiple comparisons due to an increase in the chance of a type 1 error.

Is my hunch correct that we can't be comparing the data this way, and should be doing another test? What test should we be using, if that's the case?

Similarly, it looks like this sort of data violates the assumptions used for these tests, in that it's seemingly hierarchical data.

$\endgroup$

2 Answers 2

0
$\begingroup$

A t-test strikes me as the most sensible choice here to compare response rates. To correct for multiple hypothesis testing, a natural choice would be to use Bonferroni correction by dividing your significance threshold by the number of tests performed.

For example, comparing to guarantee at least 5% significance when comparing 2015 statistics to the past 3 years, do the tests on 2015 vs 2014, 2015 vs 2013, and 2015 vs 2012 individually at a (5%)/3 significance level.

$\endgroup$
0
$\begingroup$

If you use the SPSS Statistics Ctables procedure, the Bonferroni adjustment is automatically applied unless you choose to suppress it. Check the Survey Reporter doc to see if this is also true there.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.