Reporting linear regression with Post-hoc comparisons

Question

I have a quantitative variable (Dim: which is actually factor scores) and 2 categorical variables (Category + Region) in my dataset. I am interested in knowing if the variance explained by each categorical variable (and their interaction) is significant. Additionally I also want to know how much overall variance is explained by the model (i.e. the R squared value). After consulting materials about this topic I came to know that following 2 commands in R can fulfill my requirements.

my_lm <- lm(Dim4 ~ Category*Region, data=df)
summary(my_lm)
Anova(my_lm, type = "III")

I'm also interested in reporting whether group comparisons within each category are significantly different from each other. For this purpose, TukeyHSD has been mentioned in the reference material.

TukeyHSD(aov(my_lm), ordered=TRUE)

This command outputs the whole range of group comparisons for example:

$Category
                            diff          lwr      upr     p adj
Tweets-Conversations   0.6843803 -0.279052730 1.647813 0.2961601
Comments-Conversations 1.6552203  0.665253423 2.645187 0.0000540
$Region
          diff       lwr      upr p adj
US-PK 1.146865 0.7036846 1.590046 5e-07

$`Category:Region`
                                        diff         lwr      upr     p adj
Conversations:US-Tweets:PK        0.04138269 -1.46974284 1.552508 1.0000000
Conversations:PK-Tweets:PK        0.30818926 -1.31628692 1.932665 0.9998612
Comments:PK-Tweets:PK             1.20670619 -0.35032424 2.763737 0.2931398
Tweets:US-Tweets:PK               1.78097509  0.24027735 3.321673 0.0097448

While I am interested in all comparisons of Category and Region individually, in the interaction subgroups I am only interested in certain comparisons, e.g. Tweets:US-Tweets-PK, Conversations:US-Conversations-PK. I have 2 questions regarding reporting the results of these tests. Firstly,How should I report the results of linear regression model (e.g. in case of Anova there would be F-statistic, p value and degrees of freedom like this One-way ANOVA: F(9, 3085) = 70.23, p < .001;). But I am looking at my data in terms of linear regression and want to report r squared + significance of each variable and their interaction as well. Does that involve reporting degrees of freedom and F statistic? Secondly, as mentioned above I am only interested in certain group comparisons when it comes to interaction effects, can I just report the selected group comparisons? Another relevant question is if I'm not interested in all comparisons, can I limit the output of post hoc test to my desired group comparisons only?

Answer 1

You could take a look at the emmeans (estimated marginal means) package in R. It provides a lot of flexibility for these kinds of purposes. Since it appears you do have an interaction of consequence, I recommend against doing main-effects comparisons of either factor, averaged over the other. Instead, get a good display of how the interaction plays out, e.g. an interaction plot:

require("emmeans")
emmip(my_lm, Category ~ Region)

and display the EMMs and comparisons of one factor's levels separately by the other:

emm = emmeans(my_lm, ~ Category * Region)
emm
pairs(emm, by = "Category")
pairs(emm, by = "Region")

If you do not want all pairwise comparisons even with a by variable, there are several other built-in contrast types, or you can specify your own. For example, to compare consecutive levels of Category for each Region, do

contrast(emm, "consec", by = "Region")

There are a lot of examples and discussion in the several vignettes in the package. Start with vignette("basics")

Answer 2

For post-hoc-like comparisons in multiple regression, I recommend using Structure Coefficients (Corville & Thompson, 2001). You can use the yhat package in R. For example:

require(yhat)
calc.yhat(my_lm)

You can interpret the structure coefficients, $r_s^2$ as an effect size. It is a variance partitioning method.