2
$\begingroup$

I have patient mental and physical health scores and looking for differences pre and post injury. The injuries are categorical by anatomic site and severity. I was told to run a GEE with Walds test and everything came back significant, which doesn't make sense to me. Why should an ankle injury be as significant as a spinal injury? Since this data is collected every few years, I doubt a twisted ankle would leave significant decreased mental health scores after a few years. Something is not right.

Doing some research, I'm finding a ANCOVA or repeated measures ANOVA to be appropriate. However, when I run those tests I'm not seeing which classes are significant, just overall significance of the injury site / severity.

Can someone guide me to which is the appropriate test and if I'm using SAS, how to see significance by class?

Information about the data: injury group = 21,400 no injury group = 5900 dependent variables are continuous Quality of Life score(can be pre and post or can calculate change in pre/post, either way continuous) predictor variables are categorical (can be 2 variables, anatomic site + severity, or combined as 1 variable), also have some covariates anatomic site is 8 levels (1-8) severity is 5 levels (A-E) so they can be combined as 1A, 1B, ... 8E

Most important information is which anatomic site / severity class level are significant and which are not.

I ran Cohens d and got little to moderate effect size. I got higher effect sizes by stratifying by anatomic site + severity (1A, 1B, ... 8E) but still highest is 0.4.

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ Regardless of whether you expect a certain result or not, we cannot tell without the data whether there's anything wrong. You collect data to learn something, so results shouldn't be dismissed solely on the basis that they differ from what you expect. In fact, re-analysing data because you don't like the results invalidates testing. (It will obviously lead to a final result that confirms your expectations with a too high probability.) $\endgroup$
    – Christian Hennig
    Feb 17, 2022 at 9:57
  • 1
    $\begingroup$ Note also that significance doesn't tell you anything about effect sizes, and injury A being "as significant" as injury B does not mean that the effects of the two injuries are the same. For such comparisons, you better interpret the effect sizes. $\endgroup$
    – Christian Hennig
    Feb 17, 2022 at 9:58
  • $\begingroup$ Following up on the comments by @ChristianHennig please provide more information about the nature of your data and the results you have obtained. With a large enough study almost everything can appear "significant" statistically; the question is whether the differences are "significant" practically. That's why you will see emphasis here on "effect sizes" rather than "statistical significance." Please provide that information by editing the question, as comments are easy to overlook and can be deleted. $\endgroup$
    – EdM
    Feb 17, 2022 at 15:47
  • $\begingroup$ I added more information. Is there anything else I can tell you? Although tests shouldn't be dismissed on the basis that they are not what we expect, I still get suspicious when everything comes back significant. It's a red flag. $\endgroup$
    – wisamb
    Feb 18, 2022 at 5:37
  • $\begingroup$ Your sample sizes are quite large, and this means that even small effect sizes can come out significant. $\endgroup$
    – Christian Hennig
    Feb 20, 2022 at 13:40

1 Answer 1

Reset to default
0
$\begingroup$

First, your results show the distinction between statistical and practical significance. As comments indicate, with a large study like this it's quite possible for almost everything to be "statistically significant." The ratio of a difference to its standard error is used to evaluate statistical significance. A standard error tends to decrease with the square root of the number of observations, so even a small absolute difference can be "statistically significant" in a large study.

The question is whether a differences is large enough to matter in practice. Cohen's d tries to get to that issue by evaluating the ratio of the difference to its standard deviation. That doesn't necessarily indicate how much a difference in outcome will mean to any individual, however. I'd suggest that you pay attention to the actual estimated difference values and use your knowledge of the subject matter and the literature to evaluate how important an individual would think that differences of those absolute sizes in her own outcomes are.

What you seem to want for comparing various classes are the estimated marginal means, sometimes called LS-means. Those are model-based estimates that use all the information from the data and model, and thus are superior to stratification into subsets. Such analysis is available in SAS.

With respect to the best way to evaluate repeated measures with continuous outcomes, there are several approaches beyond GEE and repeated-measures ANOVA. Chapter 7 of Frank Harrell's course notes has an interesting summary table of those 2 methods plus 5 more, illustrating their strengths and weaknesses. The expected "effect size" is not a major reason for choosing among them.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.