3
$\begingroup$

I want to use the variability of the residual as a measure M and then test whether M is higher or lower after some event. However, I estimate separate regression before and after the event to obtain the residuals and the variability of the residuals.

Now I am wondering what the effect of including/excluding an intercept in the first regression to obtain the variability of the residuals is? I'm asking because it seems that this has a strong effect on the pre-post results in my design.

In formulas:

$Y = b_0 + b_1 X + e$ - estimated separately for pre/post; measure $M = \text{var}(e)$

$M = b_0 + b_1 \text{POST} + e$

|cite|improve this question
$\endgroup$
  • 3
    $\begingroup$ Do you have any good reason TO EXCLUDE THE INTERCEPT? A model without intercept is usually not sensible! Without a very good reason to exclude the intercept, you should include it! $\endgroup$ – kjetil b halvorsen Aug 19 '14 at 17:35
  • 1
    $\begingroup$ With the intercept, the residuals sum to zero. That property does not hold if you exclude the intercept. $\endgroup$ – kjetil b halvorsen Aug 19 '14 at 17:35
7
$\begingroup$

As mentioned before it is sort of hard to justify not using an intercept unless there is strong knowledge that the linear regression line passes through the origin.

However, how fitting the model with and without the intercept affects the residuals is kind of case by case. For example, if the true model that generated the data did have an intercept far from the origin, you would definitely being doing a poor job of modelling the data and your residuals would be much larger for the model without intercept.

To make this point more concrete, consider the following example. Imagine you have noisy data coming from the following model:

$$y=x+5$$

then fitting this model with an without intercept would result in the following model fits:

enter image description here

Clearly the red line (with intercept) fits the data better and the blue line (without intercept) does a much poorer job. Thus, the residuals for the blue line should be much more dispersed around 0 than the red lines residuals (see blow).

enter image description here

Now on the other hand, imagine that the true model that generated the data does not have an intercept (so the linear regression passes through zero). Let's say for example $$y=x$$ with a little bit of noise. Fitting a model with or without intercept should return very similar results (depending on how much data you have): enter image description here

Resulting in residuals that are basically the same.

enter image description here

So at the end of the day what it really boils down to is the quality of the data and whether or not you really believe the true linear model passes through the origin (i.e. no intercept) or not. Unless there is strong evidence of the latter I would recommend including an intercept as a practical safeguard against my first example above.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Thx! I will keep the intercept as I have no specific theory of why it should go through zero (except that literature sometimes does it without explanation). But I still have troubles interpreting the effect on the second model then, where variablity in residuals should be compared before and after an event. Some liteature computes the variability of residuals seperately for the period before the event and after and some even compute the variabilty of the resiuals over the whole period and then compare pre-post - which violates OLS assumptions right? $\endgroup$ – user51972 Aug 19 '14 at 18:27
  • 1
    $\begingroup$ To be honest, I have no idea why you want to predict the variance of your residuals in the second model so the whole idea is kind of convoluted to me. $\endgroup$ – Dan Aug 19 '14 at 18:33
  • $\begingroup$ The concept is from the finance lit.: A part of the reported earnings (Y) of a firm is determined by the environment, a part by the management. To extract the part which is due to management, Y is regressed on variables that reflect changes in the environment (X) and the residual should capture positive/negative earnings stemming from management decisions (relative to the conditional average). To capture the overall influence of the management on earnings (Y), lit. uses the absolute value (or also var of e) as measure (M). I want to compare M before and after an event. Does that help? $\endgroup$ – user51972 Aug 19 '14 at 18:58
  • $\begingroup$ @user51972: Wow, that explanation sounds crazy. If your list of X was perfect and all-inclusive and if your model was of the correct form, this method would work I guess. Is this really widely used? $\endgroup$ – Wayne Feb 3 '16 at 13:05
2
$\begingroup$

My answer got too long to fit in the comments so I have put in a new one. Apologies for the delay I have been out of town.

There is a more fundamental question: have I captured the environment accurately? if not the before after makes no sense because I am missing out the variance as a result of not understanding the environment (I work in stochastic biology where in general we have very little idea of how the environment affects the growth rate).

Even when we ignore that issue a subtle problem remains: Suppose Y varies quadratically with the environment. The environment is strongly determining Y but a linear model is not capturing it. Hence, changes of variance as you are modeling them may not be the solution.

As a rule of thumb, you cannot infer causal results from regression models which is what you seem to be trying to do.

If you do insist on doing it in any case, you can proceed as follows : Calculate the residuals from the two fits call them $b_{1},b_{2}...$ and $f_{1},f_{2}...$. We know that the means of these residuals are 0. We are asking the question is the variance the same? There are standard methods in statistics to approach these sorts of problems for example see

http://stat.ethz.ch/R-manual/R-patched/library/stats/html/var.test.html

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thx! Agree, it's a problem to capture the effect of env(X) on Y – but since rules limit the extent to which X can be reflected in Y, it’s maybe less a problem. However, lit does not only control for the mean effect of X on Y (and then take e), but also for the effect of env. (X2) again in the 2nd stage: M(Varofe)=b0+b1POST+b2X2+e. My question was more about the residual – should I estimate the residual in 1st stage jointly over pre- and post-period or separately for each group and then use the variances as M? What difference does it make if I suppress the intercept in the 2nd approach? $\endgroup$ – user51972 Aug 23 '14 at 4:35
  • $\begingroup$ +1. Omitted variables, misspecified models, and a whole host of issues make it hard to believe that the residual will meaningfully reflect the "manager" effect. $\endgroup$ – Wayne Feb 3 '16 at 13:32
1
$\begingroup$

I think the simplest way to visualize this is to plot your points. For a first pass don't do any calculations. Make a guess of what the best fit line will look like. Now make a guess of what the best fit will look like when the line you are constructing is forced to pass through the origin. If you feel the first case will provide a better fit then the intercept does matter.

That being said as @ kjetil mentions, you generally keep the intercept in unless you have a strong reason to believe otherwise.

I do not know the details of your model but the second regression you are proposing is kind of funky. Remember that in linear regression the base case assumption is that the variance is the same for all values of X. I am not sure I get what POST is but the estimate of M will be the same for all values of $X$ and hence when you test $b_{1} = 0$ in the second equation you are sure to get 0

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Ok, so I keep the intercept and estimate: Y=b0+b1X+e, M=Var(e) before and after the event. Then I compare M pre v.s post using the second regression (POST is the pre/post split). I think in this case, the literature assumes that there is heteroskedasticy that can then be explained by certain variables (e.g. POST) - an alternative way that is used is to run the first model over both periods (pre-post) and then explain differences in the variability of e using the second regression (POST explains difference in var(e)) - or maybe I misunderstand something here.. $\endgroup$ – user51972 Aug 19 '14 at 18:20
  • $\begingroup$ It will help if you specify some background. Are the input and output required to be linear maps? Are you performing a transformation that "linearizes" a noisy input? Are you interested in variance before and after? $\endgroup$ – Sid Aug 19 '14 at 18:32
  • $\begingroup$ The concept is from the finance lit.: A part of the reported earnings (Y) of a firm is determined by the environment, a part by the management. To extract the part which is due to management, Y is regressed on variables that reflect changes in the environment (X) and the residual should capture positive/negative earnings stemming from management decisions (relative to the conditional average). To capture the overall influence of the management on earnings (Y), lit. uses the absolute value (or also var of e) as measure (M). I want to compare M before and after an event. Does that help? $\endgroup$ – user51972 Aug 19 '14 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.