Basic statistical interpretation - first time quantitative researcher - silly question time!

Question

Please help. I am writing up a research report on my findings and whilst I've performed the analysis with my supervisors' guidance, I have to write up the results over Christmas period with no help. I am a qualitative researcher usually so this is all new and my brain hurts.

I have 2 groups. Both have similar baseline characteristics and both complete the same measure two times (pre and post-intervention) online (using Likert scales). Only one group receives the intervention and I want to see if the intervention makes a difference to particular constructs.

Results show differences between groups are not significant but 3 of the 5 construct results are significantly different within the intervention group.

I wanted to look at whether the intervention improved the 5 constructs I am interested in. I am confused with how to write this as there is no significant difference between groups but using Mean scores, all constructs in the intervention group improved after intervention participation (suggesting the intervention worked, but only significantly so for 3 of the 5 constructs measured). Is that right?

The real problem for my poor brain comes from the control group - of the 3 significant differences observed within the intervention group, the control group also showed an improvement one of them, but this was not significant. There were other changes in the control group but none were significant and some were negative (worse/poorer/lower measures than at T1) where all constructs for the intervention group showed improvement.

I'm so confused about what this means. I know it's basic but I feel like I'm totally overthinking it. Any help is appreciated. Just a hint will do.

Lional

Answer 1

I wanted to look at whether the intervention improved the 5 constructs I am interested in. I am confused with how to write this as there is no significant difference between groups but using Mean scores, all constructs in the intervention group improved after intervention participation (suggesting the intervention worked, but only significantly so for 3 of the 5 constructs measured). Is that right?

It sounds like you've run a randomized experiment of sorts. In these sorts of experiments, the intended contrast is between the control and treatment group. We typically don't look for improvement within groups. Frank Harrell writes about this a lot. Quoting from his blog

The purpose of a parallel-group randomized clinical trial is to compare the parallel groups, not to compare a patient with herself at baseline. The central question is for two patients with the same pre measurement value of x, one given treatment A and the other treatment B, will the patients tend to have different post-treatment values? This is exactly what analysis of covariance assesses. Within-patient change is affected strongly by regression to the mean and measurement error. When the baseline value is one of the patient inclusion/exclusion criteria, the only meaningful change score requires one to have a second baseline measurement post patient qualification to cancel out much of the regression to the mean effect. It is he second baseline that would be subtracted from the follow-up measurement.

I'm not sure how exactly you're analyzing your data, but given the information you've presented I would suggest using some kind of ordinal regression. You should include the baseline measure as a covariate in said regression, and should include a group indicator (for treatment/control).

You've mentioned you're new to quantitative analysis, and I'm sorry to say you've not chosen the easiest design to start your quantitative career.