5
$\begingroup$

I have a regression where I am trying to predict Y, using several variables. My variables are A, B, C.

enter image description here

I'm using R for my analysis. My equation is as follows:

mod <-lm(  log(Y/B) ~ log(A/B) + log(A/C), data = dataframe)

and a summary of the model is as follows:

Coefficients:
                      Estimate Std. Error t value Pr(>|t|)    
(Intercept)          -1.326698   0.080683  -16.44   <2e-16 ***
log(A/B)              0.756779   0.007507  100.81   <2e-16 ***
log(A/C)             -0.292373   0.012762  -22.91   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.6925 on 17063 degrees of freedom
Multiple R-squared: 0.4043, Adjusted R-squared: 0.4042 
F-statistic:  5790 on 2 and 17063 DF,  p-value: < 2.2e-16 

I understand that log(j/k) = log(j) - log(k); so log(Y/B) can be written similarly. Is there a way that I can include the B variable in the dependent part of my regression equation?

Also, it is acceptable to use a variable for normalizing, on both dependent and independent variables? I have seen other examples where the same variable is used in this way, but I was not certain if this makes sense.

$\endgroup$
2
  • $\begingroup$ As it is, $−\log(B)$ appears on both sides of the regression equation and so is redundant - whatever coefficient appeared in front of $\log(B)$ would actually be 1 unit too small. Also $\log(A)$ appears twice on the right hand side. Is this what you intended? $\endgroup$
    – Macro
    Commented Aug 17, 2011 at 22:23
  • $\begingroup$ If I write the LHS as log(Y) - log(B), the R-sq remains the same. If I write the RHS as (log(A) - log(B)) + log(A/C) the R-sq changes (though I don't understand why). I intended to include log(A) twice. $\endgroup$
    – celenius
    Commented Aug 17, 2011 at 22:32

2 Answers 2

8
$\begingroup$

By virtue of properties of logarithms, the original model,

$$\log(Y/B) = \beta_0 + \beta_1\log(A/B) + \beta_2\log(A/C) + \varepsilon$$

($\varepsilon$ is a zero - mean random variable) can be written

$$\log(Y) - \log(B) = \beta_0 + \beta_1 (\log(A)-\log(B)) + \beta_2(\log(A)-\log(C)) + \varepsilon,$$

which algebraically is identical to

$$\log(Y) = \beta_0 + (\beta_1+\beta_2)\log(A) + (1-\beta_1)\log(B) - \beta_2\log(C) + \varepsilon.$$

Note particularly that nothing funny has happened to the random term $\varepsilon$, so that fitting one (via least squares or maximum likelihood) will give the same results as fitting the other.

Conversely, if you start with the general model

$$\log(Y) = \gamma_0 + \gamma_1\log(A) + \gamma_2\log(B) + \gamma_3\log(C) + \delta,$$

comparing coefficients shows that $\gamma_0 = \beta_0$, $\delta = \varepsilon$, and

$$\gamma_1 + \gamma_2 + \gamma_3 = (\beta_1+\beta_2) + (1-\beta_1) + (-\beta_2) = 1.$$

These are the only relations among the coefficients, as evidenced by the fact you can recover the betas from the gammas in many ways such as

$$\beta_1 = 1 - \gamma_2, \quad \beta_2 = -\gamma_3 \quad \text{or}$$

$$\beta_1 = \gamma_1 + \gamma_3, \quad \beta_2 = \gamma_1 + \gamma_2 -1,$$

for example.

Thus the original model is the general model with a single linear constraint. Either method will result in equivalent fits and the same parameter estimates. Therefore,

  1. Yes, there is a way to include $B$ as an independent variable, as shown; and

  2. The original model makes sense.

$\endgroup$
14
  • $\begingroup$ I am considering the $R^2$ in both models, and in the first case the $R^2$ is much higher (0.4 compared to 0.07). I understand your clear explanation, but I'm struggling to understand why the $R^2$ is so different, if the model structures are similar. Is it a sign that my variables are not heteroskedastic? $\endgroup$
    – celenius
    Commented Aug 18, 2011 at 10:42
  • $\begingroup$ While you are correct, Macro's example with independent variables is important to consider. While the constrained model is predictive, it is difficult to tell whether it is only due to the trivial/artificial effect of having $-log(B)$ on both sides, or there is a deeper relationship. $\endgroup$
    – Aniko
    Commented Aug 18, 2011 at 14:50
  • $\begingroup$ @Aniko I don't understand your remark, because both Celenius' model and the linearly constrained model provide exactly the same inferences; they are related merely by a (linear) change of parameterization. Perhaps you could expand on your point? $\endgroup$
    – whuber
    Commented Aug 18, 2011 at 15:19
  • $\begingroup$ @celenius In this context (where one model is nested in the other with a common set of data), $R^2$ is directly (although negatively) related to the mean squared residual. The algebra shows the residuals will be the same, whence $R^2$ must be the same in either formulation (that is, your original one or the linearly constrained one). Am I misunderstanding what you mean by "both models" and "the first case"? $\endgroup$
    – whuber
    Commented Aug 18, 2011 at 15:22
  • $\begingroup$ @whuber, the $R^2$ is different for each model. When I use the original model it is $0.4042$; when I use the general model it is 0.07. (to clarify, by first case I meant original model) $\endgroup$
    – celenius
    Commented Aug 18, 2011 at 19:43
1
$\begingroup$

Just doing

$$ \log(Y) \sim \log(A) + \log(B) + \log(C) $$

would be equivalent to the model you're fitting, under the constraint that the coefficients sum to 1 (thanks whuber for making this clear). The coefficients would be changed to compensate for the redundancies in your current formulation, but the fit would be the same.

Edit: If you ignore the constraint and fit the model above then note that the $R^2$ value will change (and typically be much lower) when compared to your original formulation. This is because $B$ appears on both sides of the regression function, so there may appear to be an impressive $R^2$ regardless of whether $Y$ relates to $A,B,C$. See

A=runif(100)
B=runif(100)
C=runif(100)
Y=runif(100)

g1 = lm( log(Y/B) ~ log(A/B) + log(A/C))
g2 = lm( log(Y) ~ log(A)+log(B)+log(C))

as an example. Clearly none of the variables are related to each other, but model g1 shows a pretty impressive $R^2$ (around $45\%$ in my simulation).

$\endgroup$
13
  • $\begingroup$ I think this is part of my confusion - when I used the model you suggested, the fit is not the same; the R-sq value is now 0.07. I'm going to update my question showing a plot of all of the variables. $\endgroup$
    – celenius
    Commented Aug 17, 2011 at 22:37
  • $\begingroup$ What you have now is $\log(Y/B) \sim \log(A/B) + \log(A/C)$. This is the same as $\log(Y)-\log(B) \sim \log(A)-\log(B)+\log(A)-\log(C)$. So, $\log(A),\log(B),\log(C)$ are all still in the model. The coefficient of $\log(B)$ should just be shifted by 1, that for $\log(A)$ should be doubled, and that for $\log(C)$ should be negative what it was before. $\endgroup$
    – Macro
    Commented Aug 17, 2011 at 22:39
  • $\begingroup$ The $R^2$ will change. This is because $B$ appears on both the left and right side of the regression equation originally. I was only saying the relationship between $Y$ and the rest of the variables will not change. This is important enough to add as an edit to the main post. $\endgroup$
    – Macro
    Commented Aug 17, 2011 at 22:42
  • $\begingroup$ I think I understand - so is it incorrect to use $B$ on both sides of the equation? $\endgroup$
    – celenius
    Commented Aug 17, 2011 at 22:45
  • $\begingroup$ It's incorrect in the sense that it may lead one to believe something is going on when there truly isn't if $R^2$ is used as a guide. $\endgroup$
    – Macro
    Commented Aug 17, 2011 at 22:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.