Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.

I have been running a linear regression where my dependent variable is a composite. By this I mean that it is built up of components that are added and multiplied together. Specifically, for the composite variable A:

A = (B*C + D*E + F*G + H*I + J*K + L*M)*(1 - N)*(1 + O*P)

None of the component variables are used as independent variables (the only independent variables are dummy variables). The component variables are mostly (though not completely) independent of one another.

Currently I just run a regression with A as the DV, to estimate each dummy variable's effect on A. But I would also like to estimate each dummy variable's effect on the separate components of A (and in the future I hope to try applying separate priors for each component). To do this I have been running several separate regressions, each with a different one of the components as the DV (and using the same IVs for all the regressions). If I do this, should I expect that for a given dummy IV, I could recombine the coefficient estimates from all the separate regressions (using the formula listed above) and get the same value as I get for that IV when I run the composite A regression? Am I magnifying the coefficient standard errors by running all these separate regressions and then trying to recombine the values (there is a lot of multicollinearity in the dummy variables)? Is there some other structure than linear regression that would be better for a case like this?

share|improve this question

1 Answer

up vote 3 down vote accepted

(1) Should I expect to obtain the same fits using the two models? No. Let's look at what's going on.

(a) In the regression of $A$ directly--I'll call it the "monolithic model," the model is

$$A_j = \sum{\beta_i X_{ij}} + \epsilon_j,$$

with the cases indexed by $j$, the variables (including a constant, if any) indexed by $i$, and with the $\epsilon_j$ random variables of zero mean.

(b) In the "composite model" you have a series of regressions. To be systematic, let's rename the variables $B$ though $P$ as $B_1, B_2, \ldots, B_{15}$ and write

$$A = (B C + \cdots + L M)(1 - N)(1 + O P) = f(B_1, B_2, \ldots, B_{15}) = f(\mathbf{B}).$$

Each component model in the composite is

$$B_{kj} = \sum_{i}{\gamma_{ki} X_{ij}} + \delta_{kj}, \quad k=1,2,\ldots,15.$$

Therefore

$$A_j = f(\mathbf{B}_j) = f(\sum_{i}{\gamma_{ki} X_{ij}} + \delta_{kj}).$$

To see how this differs from the monolithic model, let's consider the simpler case where $A = B_1 B_2 = f(B_1,B_2)$, which gives

$$\eqalign{ A_j &= f(B_{1j}, B_{2j}) \cr &=\left(\sum_{i}{\gamma_{1i} X_{ij}} + \delta_{1j}\right)\left(\sum_{l}{\gamma_{2l} X_{lj}} + \delta_{2j}\right) \cr &=\sum_{il}{\gamma_{1i}\gamma_{2l}X_{ij}X_{lj}} + \sum_{i}{\left(\gamma_{1i}\delta_{2j} + \gamma_{2i}\delta_{1j}\right) X_{ij}} + \delta_{1j}\delta_{2j}. }$$

Note the differences:

  • If the error terms $\delta_{ij}$ are not independent, the expectation of $\delta_{1j}\delta_{2j}$ will be nonzero, introducing a bias.
  • If the error terms are independent, the expectation of $\delta_{1j}\delta_{2j}$ is zero, which is good, but the expectation of $A$ equals $\sum_{il}{\gamma_{1i}\gamma_{1l}X_{ij}X_{lj}}$. In this model there are only interaction terms!
  • If you include all interaction terms in the monolithic model and the $\delta$s are independent, then the interaction coefficients for $X_iX_l$ can be compared to the coefficients $\gamma_{1i}\gamma_{2l}+\gamma_{2i}\gamma_{1l}$ that appear in the composite model. However, we cannot expect equality, because there are fewer $\gamma$ parameters in the composite model than there would be interaction parameters in the monolithic model. (In other words, the structure of the composite model introduces algebraic relationships among the coefficients that the monolithic model cannot enforce.)
  • The distribution of the random part of this model, $\sum_{i}{\left(\gamma_{1i}\delta_{2j} + \gamma_{2i}\delta_{1j}\right) X_{ij}} + \delta_{1j}\delta_{2j}$, depends on the data $X_{ij}$, on the parameters $\gamma_{ki}$, and on the products of $\delta_{kj}$. As such it will likely be heteroscedastic--that is, depending on the values of the variables--and have a different (and complex) distributional shape even when the $\delta_{kj}$ have a "nice" shape (such as normal).
  • The analog of $\beta_j$ in the composite model, $\gamma_{1i}\delta_{2j} + \gamma_{2i}\delta_{1j}$, depends on the residuals in the component models of the composite. Therefore we would not expect these to have equal values. In fact, they would only tend (probabilistically) to be equal when the $\beta_j$ are zero, because the expectation of the analog is zero.

Whence the monolithic and composite models are almost entirely different. There seems to be no valid way even to compare coefficients and where they can be compared we cannot expect them to be equal.

(2) Am I magnifying the coefficient standard errors? By the same reasoning, the standard errors cannot even be compared.

There is nothing overtly wrong with using either model, but they say two different things about the behavior of $A$. The choice between them should not be made on the basis of coefficient standard errors but on the basis of which (if either) appears to be more appropriate and useful for the intended purpose.

(3) Are there alternatives to linear regression? Of course. You could extend the models in many nonlinear ways. But normally we begin with the simplest models that can reasonably be expected to serve our analytical purposes. Note that the monolithic model uses far fewer parameters than the composite model and therefore is a fortiori simpler.

share|improve this answer
+1, excelent answer! – mpiktas Jan 28 '11 at 19:22

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.