Comparing goodness of fit in linear regression

Question

I would like to compare the goodness of fit for a regression model fitted to two separate groups (patients and controls).

I want to compare goodness of fit (as opposed to difference in slope). I thought one way of doing this would be comparing the R value of the model in patients vs the value in controls in a Fisher R-to-z comparison.

My question is whether it is more appropriate to use the 'Multiple R-squared' or the 'Adjusted R-squared'. Additionally if the Adjusted R squared is preferred then how would one deal with negative values?

Or please do let me know if the whole approach is wrong headed.

Answer 1

Looking at goodness of fit is not a, um, good fit for your actual goal, "to determine whether a relationship appears to be present in one group, but not in the other, and ascribe a degree of statistical significance to this". If there were a relationship in one group but not the other, then the relationship would differ by group. That is an interaction. To test this, fit a model with $X$ and $Y$, an indicator variable for group membership, and an interaction between $X$ and group. The test of that interaction is the first thing you need. In R, the model might look like this:

lm(y~x*group)

You may want to follow that up by testing if the relationship is significant in each of the groups. Since your group variable is a dummy code (0, 1), the 'main effect' of x is a test of the relationship between X and Y in the reference group. The simplest way to get the test of the relationship in the other group is to change the reference level of group, refit the model and then examine the main effect of X again. Here is a simple example, coded in R:

set.seed(1)                                                  # making this reproducible
x     = runif(n=20, min=0, max=10)                           # making x values
group = rep(c("A", "B"), each=10)                            # making the groups
group = factor(group)                                        #  & making it a factor
y     = 5 + 1*x[1:10] + rnorm(10, mean=0, sd=1)              # the y values for A
y     = c(y, 5 + rnorm(10, mean=0, sd=1))                    # adding the y values for B
coef(summary(lm(y~x*group)))                                 # model w/ A as reference level
#               Estimate Std. Error    t value     Pr(>|t|)
# (Intercept)  5.9802513 0.51218827 11.6758849 3.056083e-09
# x            0.7980242 0.08161474  9.7779420 3.750885e-08
# groupB      -0.2107658 0.77806193 -0.2708856 7.899425e-01
# x:groupB    -0.9141179 0.12543254 -7.2877257 1.819064e-06
group = relevel(group, ref="B")                              # changing reference to B
coef(summary(lm(y~x*group)))                                 # the second model
#               Estimate Std. Error    t value     Pr(>|t|)
# (Intercept)  5.7694855 0.58569919  9.8505950 3.385481e-08
# x           -0.1160937 0.09524891 -1.2188456 2.405689e-01
# groupA       0.2107658 0.77806193  0.2708856 7.899425e-01
# x:groupA     0.9141179 0.12543254  7.2877257 1.819064e-06

The p-value for the test of the interaction is the last column, last row in both models. Note that the p-value is the same. The 'main effect' of x is different between the two models, though. That is for the main effect of x in the reference level, which is changed between the two models. It is significant in group A, but not in group B. Be careful, though! You cannot conclude that there is 'no relationship' for group B just because the relationship is not significant.

Answer 2

If you have exactly the same number of estimates, I believe both $R^2$ and adjusted $R^2$ would produce similar results. To my knowledge, the adjusted $R^2$ penalizes the inflation of the $R^2$ that comes from increasing the number of predictors. Given the two equations are equivalent, you should be fine by going with the regular $R^2$.

**** OLDER RESPONSE **** Sorry, I misunderstood the question. Disregard the response below:

The question is a bit vague. I'm assuming you are using R. Then,

    mod1 <- lm(y ~ x1 + x2 + x3, data=patients)
    mod2 <- lm(y ~ x1 + x2 + x3, data=controls)
    anova(mod1, mod2)

I hope this helps.