2
$\begingroup$

I'm analyzing experimental data and the model shows a significant treatment effect, but the raw data and graph of the effect don't seem to match it. I want to understand why. I've been looking at this for too long so I may be missing something obvious.

The df, "distress," is a count variable without zero-inflation. Participants (N = 60) were randomized to receive a treatment designed to reduce distress or a control. They completed a baseline measure, received the experimental treatment, and then were assessed at posttreatment and a follow-up.

The model I'm running is as follows:

glmer(distress ~ condition*time + (1|id), family="poisson", data = df)

It gives the following output, showing a significant treatment effect in reducing distress:

enter image description here

However, the plotted interaction with model-predicted values makes it look like there's no difference, or a negligible difference (grey shading is 95% CI):

enter image description here

The raw means show a slightly larger decrease for the intervention group from pre to post, but the control group decreases more from pre to follow-up.

enter image description here

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ The plot certainly looks like it has negative interaction to me (I wonder if you're thinking 'curves look parallel, therefore no interaction' -- is that the case? If not, please indicate what is leading you to conclude there's no interaction). You might find it easier to see on the scale of the linear predictor (i.e. log scale) $\endgroup$
    – Glen_b
    Jul 14, 2019 at 1:03
  • $\begingroup$ I am not sure the Poisson model is the correct one... try a linear model as well (possibly after transformation)... in addition, make sure you account for within-subject clustering... $\endgroup$
    – Giuseppe Biondi-Zoccai
    Jul 16, 2019 at 8:30

2 Answers 2

Reset to default
5
$\begingroup$

A couple of points:

  • Statistical significance does not always translate to practical significance. This one of the critiques of only looking at p-values and not also at the effect size and its practical importance.
  • It was not clear whether the plot was based on the model or the observed data alone, like the table you show. In the latter case, and if you have missing data in post-treatment and follow-up, then note that the model gives you the expected counts if subjects did not dropout from the study. If the missing data are of the missing at random type, the observed data are not a representative sample of the target population, and may show you misleading results.
  • Yet another point is that the confidence intervals in the figure are for each of the two treatment groups. However, the interaction term above denotes the differences between the two groups. When the confidence interval for the difference does not include zero, this does not necessarily translate to non-overlapping confidence intervals for each group.
Improve this answer
$\endgroup$
0
1
$\begingroup$

I don't have the time to do some research and make sure, but what I believe is going on is the following. In a Poisson model the variance is equal to the mean, and the link function of a Poisson glm is the log. What this means is that under a Poisson model without interaction differences between lower means should be smaller (the differences between their logs are relatively bigger) than differences between larger means, where the variance is large as well. Your plot seems to indicate that the difference between experimental and control is the same in absolute terms between the earlier larger means and the later lower means. I don't know what the grey areas are but chances are they indicate variation, in which case there doesn't seem to be a significant difference between experiment and control pre-treatment (because the mean difference is within what is expected from random variation), but post-treatment the difference (which in absolute terms is still about the same) is clear. This can in the framework of a Poisson glm only be explained by the presence of interaction.

I should stress once more that I'm not 100% sure of this because I just see what I see and say what I think rather than checking anything.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ The gray areas are 95% CIs- added that to the question text. $\endgroup$
    – Emily
    Jul 13, 2019 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.