2
$\begingroup$

I am running a multiple regression analyses on a sample of 1800 respondents. The dependent variable is the mean of a 5-point likert scale and I have 6 predictors (antecedents) also using mean of 5-point likert scales.

I have three different groups in my dataset and I am mainly interested in HOW they differ from oneanother rather than IF. Research question: How do the effects of antecedents of Organizational Identification differ among different stakeholder groups?

To compare the groups I have made dummy variables for the three groups. One has N=1000 where the others have N=400 (more or less)

I want to compare the importance of the predictors/antecedents per stakeholder group. For this I have made interaction variables (e.g. Predictor1 * Group1) I have put all this (6 antecedents, 3 groups and 18 interactions and the control variables) in the regression analyses. The biggest group is used as reference so in the model there’s 2 groups and 12 interactions visible.

Now if I understand correctly, the b-value of one of the interaction predictors is the extra effect relative to the reference dummy interaction. So if b-value for a predictor moderated by group 2 is .34 and group 1 is the reference then I could state that the effect is stronger in group 2 than in group 1. The p-value shows the significance.

However, as I have a large dataset it is quite sensitive and although the interactions show fewer significant results, the main effects are significant with p-values like .0000000000000453

This causes me concern about the interpretation and I would like to focus on effect sizes independent of sample size. I have read several papers stressing the importance of effect size and the overratedness of p values.

So in this case, what would be the best approach to describe the real significance (practical importance) of the differences between the groups? I have been reading a lot and so far I have come across:

  • Cohen’s B – effect size in differences between means – but can you use them for differences in b-values in multiple regression?
  • Calculate a r value from the t-value belonging to the b-value and it’s SE. Using sqrt(t^2 / t^2 + df)
  • Cohen's F2 - but this seems some sort of invert of R2 and seems to apply to the model, not differences between groups. For example, I have an R2 for the overall model of .51 but what does this say about the size of the effect of the differences?

Extra confusion about applicability is caused by that I believe to have learned that the interpretation of dummies in regression is limited. Also from trying to figure it out from similar questions on here I have heard some say you can't assess effect sizes of interactions individually - so how else would I best go about it regarding my reseach question?

I am still reading up but the deadline of my thesis is due soon and I have been struggling with this for days so I have come here to seek help. Any help is welcome, Thank you.

$\endgroup$
1
$\begingroup$

1) Please tell me what software you are using.

2) If the interaction is insignificant you should remove it from your model and re-run the analysis.

3) If there is still a significant interaction, then you don't need to interpret the main effects.

4) I usually don't dummy code, and just put variables in as factors. So if you end up having a significant interaction between a categorical variable and continuous variable, then you should plot the groups at different levels of the continuous variable.

5) I found this article to be helpful regarding effect sizes: http://journal.frontiersin.org/article/10.3389/fpsyg.2013.00863/abstract

And download the Excel sheet that is talked about within the article, it's a handy resource.

$\endgroup$
  • $\begingroup$ Hi. I am using SPSS. I don't know about removing insignificant interactions, I am interested in exploring similarities and differences of the effects of the antecedents between the groups. So insignificant results are of interest as well. All in all your answer (and the article) seems to relate to ANOVA and not regression or am I mistaken? $\endgroup$ – Martin G Jun 25 '15 at 13:47
  • $\begingroup$ Simplify it all by having your groups as factors, for example Gender with 2 levels (M/F), and not dummy coding it. Then use anova and posthoc tests to compare groups. $\endgroup$ – CoryB Jun 25 '15 at 13:54
  • $\begingroup$ I was looking into using Anova to test differences and I can see it being of use when only comparing the mean score on the dependent variable of for example gender and the stakeholder groups. But I mainly need it for the difference in groups on the antecedents measured on a 5-point likert scale. How would I use a summated scale of multiple likert items in Anova? If I input it in Anova it sees it as 25 different groups. So I see how to use it for measuring difference in means directly on one of the likert scales but not for the effect of one likert scale on the other $\endgroup$ – Martin G Jun 25 '15 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.