# Tag Info

37

The Elements of Statistical Learning by Hastie et al. define ridge regression as follows (Section 3.4.1, equation 3.41): $$\hat \beta{}^\mathrm{ridge} = \underset{\beta}{\mathrm{argmin}}\left\{\sum_{i=1}^N(y_i - \beta_0 - \sum_{j=1}^p x_{ij}\beta_j)^2 + \lambda \sum_{j=1}^p \beta_j^2\right\},$$ i.e. explicitly exclude the intercept term $\beta_0$ from the ...

27

$\beta_0$ is not the odds of the event when $x_1 = x_2 = 0$, it is the log of the odds. In addition, it is the log odds only when $x_1 = x_2 = 0$, not when they are at their lowest non-zero values.

27

It will almost never be meaningful to use the no intercept model in logistic regression. The intercept parameter $\beta_0$ is modelling the marginal distribution of the response $Y$, so using $\beta_0=0$ is tantamont to assuming that $P(Y=1)=0.5$, marginally. Do you really know that? If that is untrue, you cannot trust any inference from the no intercept ...

21

It's unusual to not fit an intercept and generally inadvisable - one should only do so if you know it's 0, but I think that (and the fact that you can't compare the $R^2$ for fits with and without intercept) is well and truly covered already (if possibly a little overstated in the case of the 0 intercept); I want to focus on your main issue which is that you ...

18

Adding +0 (or -1) to a model formula (e.g., in lm()) in R suppresses the intercept. This is generally considered a bad thing to do; see: When is it OK to remove the intercept in lm()? When forcing intercept of 0 in linear regression is acceptable/advisable The estimated slope is calculated differently depending on whether the intercept is estimated ...

16

It is logical, once you consider the matrix notation that your formula will be translated into internally. In the matrix, the non-constant predictors will be translated into (one or more) columns, and the intercept will be translated into a column consisting entirely of ones. For instance, in R you would write a very simple OLS as: lm(z~1+x+y) In matrix ...

14

Here is an illustration that simulates $y$ and $x$ independently of each other so that the true slope is zero. The mean of $y$ is nonzero, such that the true intercept is also nonzero. The LS line without intercept must start at $(0,0)$ without intercept, and will try to "catch up" with the data points as quickly as possible if $y$ has nonzero mean, which ...

13

The Ordinary Least Squares estimate of the slope when the intercept is suppressed is: $$\hat{\beta}=\frac{\sum_{i=1}^N x_iy_i}{\sum_{i=1}^N x_i^2}$$

13

The coefficients of each predictor are almost always going to change when you add more predictors. This is an example of the answer changing when you ask a different question. Your software should let you fit a regression with no predictor at all. For example, if I try to predict people's weights with a regression with no predictors, then I will get the mean ...

12

Short answer to question in title: (almost) NEVER. In the linear regression model $$y = \alpha + \beta x + \epsilon$$, if you set $\alpha=0$, then you say that you KNOW that the expected value of $y$ given $x=0$ is zero. You almost never know that. $R^2$ becomes higher without intercept, not because the model is better, but because the definition of $... 11 If you write out the fitted model for the log odds of smoking $$\log \frac{\Pr(Y=1)}{\Pr(Y=0)} = -4.380\,1 + -0.324\,56\ I_\mathrm{teen} + 1.451\,19 \ I_\mathrm{mature} + -0.989\,1\ I_\mathrm{old}$$ where the dummies are $$I_\mathrm{teen}=\left\{ \begin{array}{l l} 0 & X\neq\mathrm{teenager}\\ 1& X=\mathrm{teenager}\\ \end{array}\right.$$ &c., ... 11 This an example of linear regression fit. The intercept of this fit is negative and it fits well. 10 You need to drop the intercept (the vector of 1's) from the mm matrix since the glmnet package automatically demeans the data and reports the intercept term by default. Alternatively, you can use the intercept parameter to glmnet (TRUE by default). 10 @gung has given the OLS estimate. That's what you were seeking. However, when dealing with physical quantities where the line must go through the origin, it's common for the scale of the error to vary with the x-values (to have, roughly, constant relative error). In that situation, ordinary unweighted least squares would be inappropriate. In that situation,... 10 Start with a simple logistic regression:$\,\,\text{logit}(\mu) \,= \beta_0 + \beta_1 x\quad\quad$(original)$\quad\quad\quad\quad= \beta_0^* + \beta_1^* (x-\bar{x})/s_x\quad\quad$(standardized x)$\quad\quad\quad\quad= (\beta_0^* -\beta_1^*\bar{x}/s_x)+ (\beta_1^*/s_x) x$So$\beta_1=\beta_1^*/s_x$and$\beta_0=\beta_0^* -\beta_1^*\bar{x}/s_x$More ... 10 The formula lm(formula = y ~ x1 + x2) will include an intercept by default. The formula lm(formula = y ~ x1 + x2 -1) or lm(formula = y ~ x1 + x2 +0) is how R estimates an OLS model without an intercept. The formula lm(formula = y-1 ~ x1 + x2) estimates a model against a dependent variable y with 1 subtracted from it. Centering all terms at their ... 10 In addition to @DaveT's helpful answer, here are a few more clarifications regarding the estimated intercepts in your models. Model 1 The (true) intercept in your first model lm(mpg ~ 1, data=mtcars) represents the mean value of mpg for all cars represented by the ones included in this data set, regardless of their displacement (disp) or horse power (hp)... 9 The intercept in a linear regression model may represent two totally different things: A) Your theoretical model may lead you to a specification with a constant term. A basic example from Economics is when one wants to estimate the parameters of a production function (a statistical relationship that links output produced with production factors used)$$... 9 The intercept should generally only be omitted if all the predictors and the response have mean=0 (in which case the intercept must necessarily be 0). Setting standardize=TRUE, which is the default option for glmnet::glmnet, only standardizes the predictors. The function has another parameter to standardize the response, but by default this is set to ... 9 When dealing with categorical variables in LASSO regression, it is usual to use a grouped LASSO that keeps the dummy variables corresponding to a particular categorical variable together (i.e., you cannot exclude only some of the dummy variables from the model). A useful method is the Modified Group LASSO (MGL) described in Choi, Park and Seo (2012). In ... 8 The intercept has a meaning here, as in any regression. But the meaning is neither interesting nor useful. As calendar year is the predictor, the intercept here is the value predicted for year 0. Set aside the fact that there was no year 0 and years are reckoned in retrospect, in the calendar you are using, to have run ..., 1 BC (or BCE), 1 (AD), etc. ... 8 Something like this should do it: fit <- lm( I(y-9.81) ~ 0 + x1 + x2 + I(x3^2) + x4 + x5 + x6 , data=data[i:(i+k),]) Something similar should be possible in many packages. An alternative: interc <- rep(9.81,k+1) fit <- lm(y ~ 0 + x1 + x2 + I(x3^2) + x4 + x5 + x6 + offset(interc),data=data[i:(i+k),]) While the coefficients and standard errors ... 8 The formulas are the same as always, so let's focus on understanding what's going on. Here is a small cloud of points. Its slope is uncertain. (Indeed, the coordinates of these points were drawn independently from a standard Normal distribution and then moved a little to the side, as shown in subsequent plots.) Here is the OLS fit. The intercept is ... 8 Nick Cox provided an excellent response and I wanted to add a more intuitive answer. Model 1 Model 1 investigates the relationship between IQ and Brain size among subjects represented by the ones in the study, regardless of those subjects' Gender, Height and Weight. In other words, if you imagine the target population of subjects from which the subjects in ... 7 Most multiple regression models include a constant term (i.e., the intercept), since this ensures that the model will be unbiased--i.e., the mean of the residuals will be exactly zero. (The coefficients in a regression model are estimated by least squares--i.e., minimizing the mean squared error. Now, the mean squared error is equal to the variance of the ... 7 As mentioned before it is sort of hard to justify not using an intercept unless there is strong knowledge that the linear regression line passes through the origin. However, how fitting the model with and without the intercept affects the residuals is kind of case by case. For example, if the true model that generated the data did have an intercept far ... 7 This question touches on a number of existing posts but I didn't find one that related to all of it. You should not include the constraint as if it were an ordinary data point, with the same uncertainty (but see point 4.) ordinary regression through the origin is entirely straightforward. Consider$y_i = \beta x_i + \epsilon_iS = \sum_i (y_i - \beta ...

7

It does not penalize the intercept. But it does penalize the covariates, which are correlated with the intercept. Thus, changing the estimates of the coefficients for the non-constant variables changes the estimates of the intercept. To help see that, note that in your dataset, all your covariates are in the interval $[-100, -99]$, making the estimate of ...

7

It's not true that adding predictors should generally cause the estimate of the intercet $\alpha$ to decrease. The intercept is the predicted $y$ value when all the $x$ predictors are equal to 0. So adding new predictors can cause the intercept to increase or decrease, by pretty much any amount, based on the mean of the $x$ predictor you're adding and the ...

6

It does not make sense to have such a design matrix because the columns are linearly dependent (specifically, column 1 = column 2 + column 3) so you cannot compute the OLS estimator, which requires inversion of $X'X$, where $X$ is said design matrix. Your proposed design matrix falls under what is sometimes called the "dummy variable trap". What you can do, ...

Only top voted, non community-wiki answers of a minimum length are eligible