9
$\begingroup$

I have a regression done on two groups of the sample based on a moderating variable (say gender). I'm doing a simple test for the moderating effect by checking whether the significance of the regression is lost on one set while remains in the other.

Q1: The above method is valid, isn't it?

Q2: The level of confidence of my research is set at 95%. For one group, the regression is significant at .000. For the other, it is significant at 0.038 So, I believe I have to accept both regressions as significant and that there's no moderating effect. By accepting the regression is significant at while it's proven to be not at 0.01 am I causing a Type I error (accepting the falsy argument)?

$\endgroup$
12
$\begingroup$

Your method does not appear to address the question, assuming that a "moderating effect" is a change in one or more regression coefficients between the two groups. Significance tests in regression assess whether the coefficients are nonzero. Comparing p-values in two regressions tells you little (if anything) about differences in those coefficients between the two samples.

Instead, introduce gender as a dummy variable and interact it with all the coefficients of interest. Then test for significance of the associated coefficients.

For example, in the simplest case (of one independent variable) your data can be expressed as a list of $(x_i, y_i, g_i)$ tuples where $g_i$ are the genders, coded as $0$ and $1$. The model for gender $0$ is

$$y_i = \alpha_0 + \beta_0 x_i + \varepsilon_i$$

(where $i$ indexes the data for which $g_i = 0$) and the model for gender $1$ is

$$y_i = \alpha_1 + \beta_1 x_i + \varepsilon_i$$

(where $i$ indexes the data for which $g_i = 1$). The parameters are $\alpha_0$, $\alpha_1$, $\beta_0$, and $\beta_1$. The errors are the $\varepsilon_i$. Let's assume they are independent and identically distributed with zero means. A combined model to test for a difference in slopes (the $\beta$'s) can be written as

$$y_i = \alpha + \beta_0 x_i + (\beta_1 - \beta_0) (x_i g_i) + \varepsilon_i$$

(where $i$ ranges over all the data) because when you set $g_i=0$ the last term drops out, giving the first model with $\alpha = \alpha_0$, and when you set $g_i=1$ the two multiples of $x_i$ combine to give $\beta_1$, yielding the second model with $\alpha = \alpha_1$. Therefore, you can test whether the slopes are the same (the "moderating effect") by fitting the model

$$y_i = \alpha + \beta x_i + \gamma (x_i g_i) + \varepsilon_i$$

and testing whether the estimated moderating effect size, $\hat{\gamma}$, is zero. If you're not sure the intercepts will be the same, include a fourth term:

$$y_i = \alpha + \delta g_i + \beta x_i + \gamma (x_i g_i) + \varepsilon_i.$$

You don't necessarily have to test whether $\hat{\delta}$ is zero, if that is not of any interest: it's included to allow separate linear fits to the two genders without forcing them to have the same intercept.

The main limitation of this approach is the assumption that the variances of the errors $\varepsilon_i$ are the same for both genders. If not, you need to incorporate that possibility and that requires a little more work with the software to fit the model and deeper thought about how to test the significance of the coefficients.

$\endgroup$
  • $\begingroup$ Thanks I can understand how this works. Does this method work if I have multiple moderating variables? Say for example, region (rural/urban), education level (high school educated/not)? Can I add additional dummy variables and test the effect? $\endgroup$ – scorpion Jul 8 '11 at 16:01
  • 1
    $\begingroup$ @whuber, I occasionally come across functionally similar situations in which the analyst simply splits the sample into the two groups, uses the same set of independent variables for both groups, and just qualitatively compare the coefficients. Is there any advantages of that situation I just described over this formulation of using interaction effects? $\endgroup$ – Andy W Jul 8 '11 at 16:54
  • 3
    $\begingroup$ @Andy Without any intention of sounding critical or deprecating, the only advantage I can think of for the qualitative method is that it makes no demands on the analyst's understanding or competence: this makes it accessible to more people. The qualitative approach is fraught with difficulties. For instance, there can be large apparent differences between both the slopes and the intercepts by chance alone. A qualitative assessment of just the coefficients won't be able to distinguish this situation from real effects. $\endgroup$ – whuber Jul 8 '11 at 17:08
  • 1
    $\begingroup$ @whuber , my initial thought was the same, and I recently gave the same suggestion to a colleague who ignored the suggestion for the sake of simplicity (as you alluded to). I thought perhaps the comment about the assumption of the error variances being the same for both genders may make the two model approach more appropriate given that assumption is violated. $\endgroup$ – Andy W Jul 8 '11 at 18:06
  • 1
    $\begingroup$ @Andy Yes, but the possibility of different variances does not enhance the value of a non-qualitative comparison. Rather, it would call for a more nuanced quantitative comparison of the parameter estimates. For instance, as a crude (but informative) approximation, one could perform a variant of a CABF or Satterthwaite t-test based on the estimated error variances and their degrees of freedom. Even visual examination of a well-constructed scatterplot would be easy to do and far more informative than simply comparing the regression coefficients. $\endgroup$ – whuber Jul 8 '11 at 18:39
-1
$\begingroup$

I guess moderating a grouping variable would work equally well when comparing regression coefficients across independent waves of cross-sectional data (e.g, year1, year2 and year3 as group1 group2 and group3)?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.