1
$\begingroup$

I have aggregated survey data for various groups, but do not have access to the raw responses. Is there an appropriate test for statistical significance between group means with only access to the means and the number of responses for each group? I found that a Z-test for proportions is similar, but only deals with binary categories. The survey data I have involves interval data 1-10.

I have included two groups within the data. The columns include all possible information I have access to, the "total people in group" column may not be useful. Survey respondents are also segmented into groups by job function, geography, and ethnicity; however, I believe this simple example will help me with the remaining groups.

Survey Results

Improve this question
$\endgroup$
6
  • $\begingroup$ Welcome to CV, Jared Ucherek. $\endgroup$
    – Alexis
    Commented Oct 4, 2022 at 18:58
  • 2
    $\begingroup$ Do you have any other statistics? Standard deviation, min, max, anything? $\endgroup$
    – whuber ♦
    Commented Oct 4, 2022 at 19:09
  • $\begingroup$ @whuber I have min and max based on the interval used. Min=1, Max=10. $\endgroup$
    – Jared Ucherek
    Commented Oct 4, 2022 at 19:47
  • 1
    $\begingroup$ Too bad--that's useless. You cannot conclude anything either way: these results could be consistent with statistically significant differences or not. Except where dramatic differences are apparent, you will have to settle for reporting the effect sizes. The nonresponse rates are high enough to call into question the results of formal tests, anyway. $\endgroup$
    – whuber ♦
    Commented Oct 4, 2022 at 19:54
  • 1
    $\begingroup$ Yes, that's correct: given some statistics, there will be mathematical bounds on the possible SD and perhaps they would suffice for an answer. Nonresponse, on the face of it, means your samples are not random and therefore no standard statistical test applies. But if you have evidence to show the nonresponders might not differ from the responders in their answers (or if their possible answers couldn't change your conclusions no matter what those answers might be), you can proceed as if your sample were random. $\endgroup$
    – whuber ♦
    Commented Oct 5, 2022 at 13:58

1 Answer 1

Reset to default
1
$\begingroup$

There is not a reliable test for difference in means between your two groups without stringent assumptions about variances or data varability.

Consider: if the variance in average response is, say, about 36 in both groups, then a t test would find no evidence of difference in means at the $\alpha=0.05$ level. However, if the variance in both groups is, say, about .036, then you would find evidence of a difference in means at the $\alpha = 0.05$ level.

Why is this? Because hypothesis tests (confidence intervals also) rely on estimates of the standard error, which depends on the estimated variance to calculate the test statistic (there's a simpler form if the variances in both groups can be assumed equal, but this works whether or not they are equal):

$$ t = \frac{\overline{x}_1 - \overline{x}_2}{\sqrt{\frac{s^{2}_1}{n_1} + \frac{s^2_{2}}{n_2}}}$$

The $s^2$s in the above equation is what you are missing, and why you cannot perform a test for group differences in means (or group inferiority/superiority of means, or group equivalence of means).

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ I figured as much. It seems there may be advanced papers that discuss this topic with those stringent assumptions you have made. Given the use case for this testing, it probably makes more sense to recover the SD or raw data instead. $\endgroup$
    – Jared Ucherek
    Commented Oct 4, 2022 at 19:49
  • $\begingroup$ @JaredUcherek If you find my answer useful I would appreciate an upvote (top left of answer). If you feel it is a satisfactory answer to your question you can also 'accept' the answer by clicking on the checkmark at top left. $\endgroup$
    – Alexis
    Commented Oct 4, 2022 at 19:58
  • 1
    $\begingroup$ Thanks again, I accepted the answer and upvoted. It might not show up on your end given my new account status. $\endgroup$
    – Jared Ucherek
    Commented Oct 4, 2022 at 20:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.