15
$\begingroup$

I am running an OLS model with a continuous asset index variable as the DV. My data is aggregated from three similar communities in close geographic proximity to one another. Despite this, I thought it important to use community as a controlling variable. As it turns out, community is significant at the 1% level (t-score of -4.52). Community is a nominal/categorical variable coded as 1,2,3 for 1 of 3 different communities.

My question is if this high degree of significance means I should be doing regressions on the communities individually rather than as an aggregation. Otherwise, is using community as a controlling variable essentially doing that?

$\endgroup$
1
  • 1
    $\begingroup$ Would it make sense to use a hierarchical model with community as a random effect? The communities are not your primary concern, are they? By using a hierarchical model, you share strength. $\endgroup$
    – Wayne
    Commented Dec 23, 2015 at 13:00

1 Answer 1

19
$\begingroup$

The question suggests a comparison of three related models. To make the comparison clear, let $Y$ be the dependent variable, let $X \in \{1,2,3\}$ be the current community code, and define $X_1$ and $X_2$ to be indicators of communities 1 and 2, respectively. (This means that $X_1=1$ for community 1 and $X_1=0$ for communities 2 and 3; $X_2=1$ for community 2 and $X_2=0$ for communities 1 and 3.)

The current analysis may be one of the following: either

$$Y = \alpha + \beta X + \varepsilon\quad\text{(first model)}$$

or

$$Y = \alpha + \beta_1 X_1 + \beta_2 X_2 + \varepsilon\quad\text{(second model)}.$$

In both cases $\varepsilon$ represents a set of identically distributed independent random variables with zero expectation. The second model likely is the one intended, but the first model is the one that will be fit with the coding that is described in the question.

The output of the OLS regression is a set of fitted parameters (indicated with "hats" on their symbols) together with an estimate of the common variance of the errors. In the first model there is one t-test to compare $\hat{\beta}$ to $0$. In the second model there are two t-tests: one to compare $\hat{\beta_1}$ to $0$ and another to compare $\hat{\beta_2}$ to $0$. Because the question reports only one t-test, let's start by examining the first model.

Having concluded that $\hat{\beta}$ is significantly different from $0$, we can make an estimate of $Y$ = $\mathbb{E}[\alpha + \beta X + \varepsilon]$ = $\alpha + \beta X$ for any community:

for community 1, $X=1$ and the estimate equals $\alpha+\beta$;

for community 2, $X=2$ and the estimate equals $\alpha+2\beta$; and

for community 3, $X=3$ and the estimate equals $\alpha+3\beta$.

In particular, the first model forces the community effects to be in arithmetic progression. If the community coding is intended as just an arbitrary way of differentiating among communities, this built-in restriction is equally arbitrary and likely wrong.

It is instructive to perform the same detailed analysis of the second model's predictions:

For community 1, where $X_1=1$ and $X_2=0$, the predicted value of $Y$ equals $\alpha + \beta_1$. Specifically,

$$Y(\text{community 1}) = \alpha + \beta_1 + \varepsilon.$$

For community 2, where $X_1=0$ and $X_2=1$, the predicted value of $Y$ equals $\alpha+\beta_2$. Specifically,

$$Y(\text{community 2}) = \alpha + \beta_2 + \varepsilon.$$

For community 3, where $X_1=X_2=0$, the predicted value of $Y$ equals $\alpha$. Specifically,

$$Y(\text{community 3}) = \alpha + \varepsilon.$$

The three parameters effectively give the second model full freedom to estimate the three expected values of $Y$ separately. The t-tests assess whether (1) $\beta_1=0$; that is, whether there is a difference between communities 1 and 3; and (2) $\beta_2=0$; that is, whether there is a difference between communities 2 and 3. In addition, one can test the "contrast" $\beta_2-\beta_1$ with a t-test to see whether communities 2 and 1 differ: this works because their difference is $(\alpha + \beta_2) - (\alpha + \beta_1)$ = $\beta_2-\beta_1$.

Now we can assess the effect of three separate regressions. They would be

$$Y(\text{community 1}) = \alpha_1 + \varepsilon_1,$$

$$Y(\text{community 2}) = \alpha_2 + \varepsilon_2,$$

$$Y(\text{community 3}) = \alpha_3 + \varepsilon_3.$$

Comparing this to the second model, we see that $\alpha_1$ should agree with $\alpha+\beta_1$, $\alpha_2$ should agree with $\alpha+\beta_2$, and $\alpha_3$ should agree with $\alpha$. So, in terms of flexibility of fitting parameters, both models are equally good. However, the assumptions in this model about the error terms are weaker. All the $\varepsilon_1$ must be independent and identically distributed (iid); all the $\varepsilon_2$ must be iid, and all the $\varepsilon_3$ must be iid, but nothing is assumed about statistical relations among the separate regressions. Separate regressions therefore allow for additional flexibility:

  • Most importantly, the distribution of the $\varepsilon_1$ can differ from that of the $\varepsilon_2$ which can differ from that of the $\varepsilon_3$.

  • In some situations, the $\varepsilon_i$ may be correlated with the $\varepsilon_j$. None of these models explicitly handles this, but the third model (separate regressions) at least won't be adversely affected by it.

This additional flexibility means that the t-test results for the parameters will likely differ between the second and third model. (It should not result in different parameter estimates, though.)

To see whether separate regressions are needed, do the following:

Fit the second model. Plot the residuals against community, for example as a set of side-by-side boxplots or a trio of histograms or even as three probability plots. Look for evidence of different distributional shapes and especially of appreciably different variances. If that evidence is absent, the second model should be ok. If it's present, separate regressions are warranted.

When the models are multivariate--that is, they include other factors--a similar analysis is possible, with similar (but more complicated) conclusions. In general, performing separate regressions is tantamount to including all possible two-way interactions with the community variable (coded as in the second model, not the first) and allowing for different error distributions for each community.

$\endgroup$
2
  • 1
    $\begingroup$ When the variances are different, an alternative to separate regressions is to use a gamlss model, which also can model the variance. $\endgroup$ Commented May 31, 2022 at 2:28
  • $\begingroup$ @Kjetil Excellent point, thank you. $\endgroup$
    – whuber
    Commented May 31, 2022 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.