Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a regression model to estimate the completion time of a process, based on various factors. I have 200 trials of these processes, where the 9 factors being measured vary widely. When I perform a linear regression of the 9 factors (and all 2 and 3 factor interactions), with no explicit intercept, I get an adjusted R${^2}$ of 0.915, if I force the intercept to 0 I get an adjusted R${^2}$ of 0.953.

My intention of forcing the intercept to 0 was to ensure that trials that completed in very short amounts of time (< 1 second) did not result in predictions of < 0. Setting the intercept to 0 did not help with this.

So my question is threefold. 1) When is it acceptable/advisable to force an intercept? 2) Does the improved R${^2}$ actually mean the model is a better fit (the plot of fitted vs measured did look better)? 3) Is there a way of ensuring the fitted values are all > 0?

share|improve this question
1  
possible duplicate of When is it ok to remove the intercept in lm()? –  gung Jun 9 at 23:20
3  
1  
There are several votes to close this thread as a duplicate of one of those referenced in the preceding comments. Those votes are valid based on (1) and even (2), but (3) looks new. I would therefore like to suggest that respondents focus on the third question. –  whuber Jun 10 at 5:38
    
My apologies for the cross-over with other questions, however I found the questions I looked for, including the one linked, did not address specifics, such as where you expect an always positive value, I am happy for the focus to be on the third point though, as that is the most important. –  Zack Newsham Jun 10 at 18:52
add comment

6 Answers 6

It's unusual to not fit an intercept and generally inadvisable - one should only do so if you know it's 0, but I think that (and the fact that you can't compare the $R^2$ for fits with and without intercept) is well and truly covered already (if possibly a little overstated in the case of the 0 intercept); I want to focus on your main issue which is that you need the fitted function to be positive, though I do return to the 0-intercept issue in part of my answer.

The best way to get an always positive fit is to fit something that will always be positive; in part that depends on what functions you need to fit.

If your linear model was largely one of convenience (rather than coming from a known functional relationship that might stem from a physical model, say), then you might instead work with log-time; the fitted model is then guaranteed to be positive in $t$. As an alternative, you might work with speed rather than time - but then with linear fits you may get a problem with small speeds (long times) instead.

If you know your response is linear in the predictors, you can attempt to fit a constrained regression, but with multiple regression the exact form you need will depend on your particular x's (there's no one linear constraint that will work for all $x's$), so it's a bit ad-hoc.

You can also look at GLMs which can be used to fit models which have non-negative fitted values and can (if required) even have $E(Y)=X\beta$.

For example, one can fit a gamma GLM with identity link. You should not end up with a negative fitted value for any of your x's (but you might perhaps have convergence issues in some cases if you force the identity link where it really won't fit).

Here's an example: the cars data set in R, which records speed and stopping distances (the response).

enter image description here

One might say "oh, but the distance for speed 0 is guaranteed to be 0, so we should omit the intercept" but the problem with that reasoning is that the model is misspecified in several ways, and that argument only works well enough when the model is not misspecified - a linear model with 0 intercept doesn't fit at all well in this case, while one with an intercept is actually a half-decent approximation even though it's not actually "correct".

The problem is, if you fit an ordinary linear regression, the fitted intercept is quite a way negative, which causes the fitted values to be negative.

The blue line is the OLS fit; the fitted value for the smallest x-values in the data set are negative. The red line is the gamma GLM with identity link -- while having a negative intercept, it only has positive fitted values. This model has variance proportional to mean, so if you find your data are more spread as the expected time grows, it may be especially suitable.

So that's one possible alternative approach that may be worth a try. It's almost as easy as fitting a regression in R.

If you don't need the identity link, you might consider other link functions, like the log-link and the inverse link, which relate to the transformations already discussed, but without the need for actual transformation.


Since people usually ask for it, here's the code for my plot:

plot(dist~speed,data=cars,xlim=c(0,30),ylim=c(-5,120))
abline(h=0,v=0,col=8)
abline(glm(dist~speed,data=cars,family=Gamma(link=identity)),col=2,lty=2)
abline(lm(dist~speed,data=cars),col=4,lty=2)

(The ellipse was added by hand afterward, though it's easy enough to do in R as well)

share|improve this answer
    
Thanks for the answer, I'm a little confused about the purpose of the Gamma and abline functions, when I try to run the same command on my data I get: "only using the first two of 130 regression coefficients". I should not that my model is more complicated than "dist~speed", there are 9 factors and I am currently using all 3 factor interactions. When I try to plot the abline function, I just get a vertical line at 0. I tried simplifying the model (and using the log link) and got a near horizontal line too, how do I interpret this data using the above method? –  Zack Newsham Jun 10 at 18:45
    
abline draws a line when you specify slope (a) and intercept (b). If you pass it a fitted simple linear model it will extract them from that. When you have multiple predictors you can't draw a line (how would that work? You didn't fit a line). I was illustrating my suggestion to use a GLM so you could see what it did, not giving a recipe for you to follow. I don't know how your data are arranged, so it's hard to give advice about what went wrong with your fit. Fitting a GLM will be almost as easy as fitting a regression in almost any stats package, if you're used to something else, use it. –  Glen_b Jun 10 at 20:37
add comment

Short answer to question in title: (almost) NEVER. In the linear regression model $$ y = \alpha + \beta x + \epsilon $$, if you set $\alpha=0$, then you say that you KNOW that the expected value of $y$ given $x=0$ is zero. You almost never know that.

$R^2$ becomes higher without intercept, not because the model is better, but because the definition of $R^2$ used is another one! $R^2$ is an expression of a comparision of the estimated model with some standard model, expressed as reduction in sum of squares compared to sum of squares with the standard model. In the model with intercept, the comparison sum of squares is around the mean. Without intercept, it is around zero! The last one is usually much higher, so it easier to get a large reduction in sum of squares.

Coclusion: DO NOT LEAVE THE INTERCEPT OUT OF THE MODEL (unless you really, really know what you are doing).

share|improve this answer
    
But surely having data stating that when the 9 factors are thus, the actual time is 0 (or close to it) means that I am fairly sure that when X = 0, Y = 0? –  Zack Newsham Jun 9 at 19:33
4  
In your case, you know that the completion times cannot be negative, yes. So a linear model is maybe not appropriate? Try some other model, survival analysis maybe, or some GLM with a distribution on the positive real line (Gamma distribution, weibull distribution ...) –  kjetil b halvorsen Jun 9 at 19:37
    
One exception is mentioned elsewhere in the comments (but that is only seemeingly an exception, the constamt vector 1 is in the columnn space of the regressor matrix $X$. Otherwise, such as physical relationships $s=v t$ where there are no constant. But even then, if the model is only approximate (speed is not really constant), it might be better to leave in a constant even if it cannot be interpreted. With non-linear models this becomes more of an issue. –  kjetil b halvorsen Jun 10 at 10:24
add comment

1) It is never acceptable to suppress an intercept except in very rare types of DiD models where the outcome and predictors are actually computed differences between groups (this isn't the case for you).

2). Heck no it doesn't. What it means is that you may have a higher degree of internal validity (e.g. the model fits the data) but probably a low degree of external validity (e.g. the model would be poor in fitting experimental data obtained under similar conditions). This is generally a bad thing.

3) Suppressing the intercept will not necessarily do that, but I assume the predictor was continuous valued. In many situations, process completion times are analyzed using an inverse transform, e.g. $x = 1/t$ where $t$ is the time taken to complete a process. The inverse of the mean of inverse transformed data is called a harmonic mean and represents the average complete time for a task.

$$\mbox{HM} = \frac{1}{\mathbb{E}(x)} = \frac{1}{\mathbb{E}(1/t)} $$

You can also use a parametric exponential or gamma or weibull time-to-event models which are types of models built specifically for predicting completion times. These will give results very similar to the inverse transformed outcomes.

share|improve this answer
1  
I think it's a bit strong to state that you can never suppress an intercept. What if you want/need to code $k$ dummy groups instead of the more usual $k-1$? –  user777 Jun 9 at 19:29
    
You still estimate $k$ effects in the example you mention. OP's question is a matter of 2 versus 1 effects (avec intercept versus sans intercept, continuous predictor). –  AdamO Jun 9 at 19:33
1  
user777: Yes, but that works only in very specific instances. For instance, if you have cross-classifications into two grouos, youir trick does not work. –  kjetil b halvorsen Jun 9 at 19:33
1  
@AdamO Are you really saying that the one exception you list in (1) is the only possible exception to 'never', or do you mean that it's the only one you're aware of? –  Glen_b Jun 9 at 21:13
2  
@Glen_b I can really think of no good example when a specification of intercept-through-origin makes sense otherwise. Even in very practical situations, such as analysis of stopping distance, or chemical compositions after applying a catalyst to some substrate, intercepts that fit through non-zero origins can help to handle measurement error, calibration issues, timing problems, etc. etc. In the years I've done analyses, I've always seen reason to fit intercepts even when the values they take on don't make sense. –  AdamO Jun 10 at 0:28
show 3 more comments

1) Forcing $0$ intercept is advisable if you know for a fact that it is 0. Anything you know a priori, you should use in your model.

One example is the Hubble model for expansion of the Universe (used in Statistical Sleuth):

$$\mbox{Galaxy Speed} = k (\mbox{Distance from Earth}) $$

This model is rather crude, but uses 0 intercept as the consequence of the Big Bang Theory: at time $0$ all the matter is in one place.

On the other hand, the model you're describing will likely need an intercept term.

2) You might or might not get better $R^2_{adj}$, or you may accept null hypothesis for the test for intercept being 0, but both of these are not reasons to remove the intercept term.

3) To ensure positivity of answers, you can sometimes transform the response variable. Log or sqrt might work depending on your data, of course you will need to check the residuals.

share|improve this answer
add comment

The only way that I know to constrain all fitted values to be greater than zero is to use a linear programming approach and specify that as a constraint.

share|improve this answer
    
Not so. Poisson regression and any generalised linear model with log link will return all positive predicted values. @Glen_b already made this point in his answer. –  Nick Cox Jun 10 at 9:57
    
@Nick ... For sure ... Since Poisson variables are restricted by 0 this is so. Thanks ... –  IrishStat Jun 10 at 13:22
add comment

It does make pretty much sense in models with categorical covariate. In this case the removal of the intercept results in an equivalent model with just different parametrization:

> data(mtcars)
> mtcars$cyl_factor <- as.factor(mtcars$cyl)
> summary(lm(mpg ~ cyl_factor, data = mtcars))

Call:
lm(formula = mpg ~ cyl_factor, data = mtcars)

Residuals:
    Min      1Q  Median      3Q     Max 
-5.2636 -1.8357  0.0286  1.3893  7.2364 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  26.6636     0.9718  27.437  < 2e-16 ***
cyl_factor6  -6.9208     1.5583  -4.441 0.000119 ***
cyl_factor8 -11.5636     1.2986  -8.905 8.57e-10 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.223 on 29 degrees of freedom
Multiple R-squared:  0.7325,    Adjusted R-squared:  0.714 
F-statistic:  39.7 on 2 and 29 DF,  p-value: 4.979e-09

> summary(lm(mpg ~ 0 + cyl_factor, data = mtcars))

Call:
lm(formula = mpg ~ 0 + cyl_factor, data = mtcars)

Residuals:
    Min      1Q  Median      3Q     Max 
-5.2636 -1.8357  0.0286  1.3893  7.2364 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
cyl_factor4  26.6636     0.9718   27.44  < 2e-16 ***
cyl_factor6  19.7429     1.2182   16.21 4.49e-16 ***
cyl_factor8  15.1000     0.8614   17.53  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 3.223 on 29 degrees of freedom
Multiple R-squared:  0.9785,    Adjusted R-squared:  0.9763 
F-statistic: 440.9 on 3 and 29 DF,  p-value: < 2.2e-16

The second example in fact results in the categorical variable being a category-specific intercept, so in reality the intercept isn't actually removed, it just seems so.

share|improve this answer
    
Does this really relate to the question? You're not simply forcing an intercept to zero in the second model, but prompting R to use sum-to-zero rather than reference-level coding for a categorical predictor. –  Scortchi Jun 10 at 12:54
    
In fact it's backwards: "Nor does it often make sense in models with categorical covariates: if you try code you'd think would force the intercept to zero, R will assume you merely want to reparametrize the model." would be right. –  Scortchi Jun 10 at 13:17
    
You both just play with words. The usual understanding of 0 + or -1 in lm is removing global intercept, which it actually does. BTW, I say exactly the same as you both do in the last sentence of my answer, so I don't really get why someone downvoted. –  Curious Jun 10 at 13:38
    
(I'm one person, & the down-voter.) +0 can do two completely different things. If you manually code $x_1$ & $x_2$ dummies for cyl as 0 or 1, then using it will force the intercept to zero & fit the two-parameter model $\operatorname{E}Y= \beta_1 x_1 + \beta_2 x_2$. But if R's been told that cyl_factor's categorical it gets clever & fits the three-parameter model you describe. So "removing the intercept" is ambiguous. Now I know all this & you know all this, but IMO the answer's unclear - potentially misleading - for someone who doesn't. –  Scortchi Jun 10 at 15:02
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.