Questions tagged [linear-model]

Refers to any model where a random variable is related to one or more random variables by a function that is linear in a finite number of parameters.

Filter by
Sorted by
Tagged with
122
votes
3answers
258k views

What is the difference between linear regression and logistic regression?

What is the difference between linear regression and logistic regression? When would you use each?
118
votes
9answers
107k views

When is it ok to remove the intercept in a linear regression model?

I am running linear regression models and wondering what the conditions are for removing the intercept term. In comparing results from two different regressions where one has the intercept and the ...
101
votes
2answers
47k views

Removal of statistically significant intercept term increases $R^2$ in linear model

In a simple linear model with a single explanatory variable, $\alpha_i = \beta_0 + \beta_1 \delta_i + \epsilon_i$ I find that removing the intercept term improves the fit greatly (value of $R^2$ ...
97
votes
9answers
173k views

What is the difference between linear regression on y with x and x with y?

The Pearson correlation coefficient of x and y is the same, whether you compute pearson(x, y) or pearson(y, x). This suggests that doing a linear regression of y given x or x given y should be the ...
90
votes
4answers
106k views

PCA and proportion of variance explained

In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a ...
69
votes
2answers
25k views

Shape of confidence interval for predicted values in linear regression

I have noticed that the confidence interval for predicted values in an linear regression tends to be narrow around the mean of the predictor and fat around the minimum and maximum values of the ...
55
votes
4answers
32k views

Choosing between LM and GLM for a log-transformed response variable

I'm trying to understand the philosophy behind using a Generalized Linear Model (GLM) vs a Linear Model (LM). I've created an example data set below where: $$\log(y) = x + \varepsilon $$ The ...
50
votes
4answers
10k views

Fast linear regression robust to outliers

I am dealing with linear data with outliers, some of which are at more the 5 standard deviations away from the estimated regression line. I'm looking for a linear regression technique that reduces the ...
45
votes
3answers
3k views

Where does the misconception that Y must be normally distributed come from?

Seemingly reputable sources claim that the dependent variable must be normally distributed: Model assumptions: $Y$ is normally distributed, errors are normally distributed, $e_i \sim N(0,\sigma^2)...
45
votes
3answers
88k views

What is the effect of having correlated predictors in a multiple regression model?

I learned in my linear models class that if two predictors are correlated and both are included in a model, one will be insignificant. For example, assume the size of a house and the number of ...
38
votes
3answers
21k views

Why is polynomial regression considered a special case of multiple linear regression?

If polynomial regression models nonlinear relationships, how can it be considered a special case of multiple linear regression? Wikipedia notes that "Although polynomial regression fits a nonlinear ...
36
votes
2answers
33k views

How do I know which method of cross validation is best?

I am trying to figure out which cross validation method is best for my situation. The following data are just an example for working through the issue (in R), but my real ...
36
votes
3answers
64k views

Derive Variance of regression coefficient in simple linear regression

In simple linear regression, we have $y = \beta_0 + \beta_1 x + u$, where $u \sim iid\;\mathcal N(0,\sigma^2)$. I derived the estimator: $$ \hat{\beta_1} = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}...
34
votes
5answers
3k views

What if my linear regression data contains several co-mingled linear relationships?

Let's say I am studying how daffodils respond to various soil conditions. I have collected data on the pH of the soil versus the mature height of the daffodil. I'm expecting a linear relationship, ...
33
votes
4answers
16k views

(Why) do overfitted models tend to have large coefficients?

I imagine that the larger a coefficient on a variable is, the more ability the model has to "swing" in that dimension, providing an increased opportunity to fit noise. Although I think I've got a ...
31
votes
2answers
8k views

Do we need gradient descent to find the coefficients of a linear regression model?

I was trying to learn machine learning using the Coursera material. In this lecture, Andrew Ng uses gradient descent algorithm to find the coefficients of the linear regression model that will ...
31
votes
3answers
95k views

How does R handle missing values in lm?

I'd like to regress a vector B against each of the columns in a matrix A. This is trivial if there are no missing data, but if matrix A contains missing values, then my regression against A is ...
29
votes
5answers
76k views

How to derive the least square estimator for multiple linear regression?

In the simple linear regression case $y=\beta_0+\beta_1x$, you can derive the least square estimator $\hat\beta_1=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}$ such that you don't have to ...
29
votes
1answer
16k views

Proof that the coefficients in an OLS model follow a t-distribution with (n-k) degrees of freedom

Background Suppose we have an Ordinary Least Squares model where we have $k$ coefficients in our regression model, $$\mathbf{y}=\mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}$$ where $\mathbf{\beta}$ ...
26
votes
7answers
27k views

Testing for linear dependence among the columns of a matrix

I have a correlation matrix of security returns whose determinant is zero. (This is a bit surprising since the sample correlation matrix and the corresponding covariance matrix should theoretically be ...
25
votes
2answers
26k views

General Linear Model vs. Generalized Linear Model (with an identity link function?)

This is my first post, so please take it easy on me if I am not following some standards! I did a search for my question and nothing came up. My question relates mostly around the practical ...
25
votes
0answers
631 views

Bound for Arithmetic Harmonic mean inequality for matrices?

NOTE: This question has originally been posted in MSE, but it did not generate any interest. It was first posted there, because the question itself is a pure matrix-algebra question. Nevertheless, ...
24
votes
2answers
8k views

Bayesian lasso vs ordinary lasso

Different implementation software are available for lasso. I know a lot discussed about bayesian approach vs frequentist approach in different forums. My question is very specific to lasso - What are ...
22
votes
2answers
5k views

Least Squares Regression Step-By-Step Linear Algebra Computation

As a prequel to a question about linear-mixed models in R, and to share as a reference for beginner/intermediate statistics aficionados, I decided to post as an independent "Q&A-style" the steps ...
22
votes
3answers
13k views

Regression modelling with unequal variance

I would like to fit a linear model (lm) where the residuals variance is clearly dependent on the explanatory variable. The way I know to do this is by using glm with the Gamma family to model the ...
21
votes
1answer
87k views

How can I predict values from new inputs of a linear model in R?

I've created a linear model in R: mod = lm(train_y ~ train_x). I want to pass it a list of X's and get its predicted/estimateed/forecasted Y. I looked at ...
21
votes
1answer
945 views

Common statistical tests as linear models

(UPDATE: I dived deeper into this and and posted the results here) The list of named statistical tests is huge. Many of the common tests rely on inference from simple linear models, e.g. a one-sample ...
20
votes
5answers
77k views

Assumptions of linear models and what to do if the residuals are not normally distributed

I am a little bit confused on what the assumptions of linear regression are. So far I checked whether: all of the explanatory variables correlated linearly with the response variable. (This was the ...
20
votes
2answers
1k views

Casting a multivariate linear model as a multiple regression

Is recasting a multivariate linear regression model as a multiple linear regression entirely equivalent? I'm not referring to simply running $t$ separate regressions. I have read this in a few ...
18
votes
1answer
5k views

Goodness of fit and which model to choose linear regression or Poisson

I need some advice regarding two main dilemmas in my research, which is a case study of 3 big pharmaceuticals and innovation. Number of patents per year is the dependent variable. My questions are ...
17
votes
3answers
1k views

How is it possible to obtain a good linear regression model when there is no substantial correlation between the output and the predictors?

I have trained a linear regression model, using a set of variables/features. And the model has a good performance. However, I have realized that there is no variable with a good correlation with the ...
16
votes
1answer
2k views

Where do the assumptions for linear regression come from? [duplicate]

I'v already known that there are several assumpations when using linear regression model. But I cannot understand why some of them exists. They are: independent errors normal distribution of errors ...
16
votes
1answer
2k views

Conditional expectation of R-squared

Consider the simple linear model: $$\pmb{y}=X'\pmb{\beta}+\epsilon$$ where $\epsilon_i\sim\mathrm{i.i.d.}\;\mathcal{N}(0,\sigma^2)$ and $X\in\mathbb{R}^{n\times p}$, $p\geq2$ and $X$ contains a ...
15
votes
3answers
9k views

For linear classifiers, do larger coefficients imply more important features?

I'm a software engineer working on machine learning. From my understanding, linear regression (such as OLS) and linear classification (such as logistic regression and SVM) make a prediction based on ...
15
votes
5answers
9k views

Can I ignore coefficients for non-significant levels of factors in a linear model?

After seeking clarification about linear model coefficients over here I have a follow up question concerning non-signficant (high p value) for coefficients of factor levels. Example: If my linear ...
15
votes
2answers
5k views

Why is GLM different than an LM with transformed variable

As explained in this course handout (page 1), a linear model can be written in the form: $$ y = \beta_1 x_{1} + \cdots + \beta_p x_{p} + \varepsilon_i,$$ where $y$ is the response variable and $x_{...
15
votes
4answers
2k views

Classic linear model - model selection

I have a classic linear model, with 5 possible regressors. They are uncorrelated with one another, and have quite low correlation with the response. I have arrived at a model where 3 of the regressors ...
15
votes
2answers
7k views

Why is a T distribution used for hypothesis testing a linear regression coefficient?

In practice, using a standard T-test to check the significance of a linear regression coefficient is common practice. The mechanics of the calculation make sense to me. Why is it that the T-...
15
votes
1answer
8k views

Understanding QR Decomposition

I've got a worked example (in R), that I'm trying to understand further. I'm using Limma to create a linear model and I'm trying to understand what's happening step by step in the fold change ...
15
votes
3answers
44k views

When can we speak of collinearity

In linear models we need to check if a relationship exists among the explanatory variables. If they correlate too much then there is collinearity (i.e., the variables partly explain each other). I am ...
15
votes
2answers
27k views

VIF, condition Index and eigenvalues

I am currently assessing multicollinearity in my datasets. What threshold values of VIF and condition index below/above suggest a problem? VIF: I have heard that VIF $\geq 10$ is a problem. After ...
14
votes
2answers
8k views

Matrix notation for logistic regression

In linear regression (squared loss), using matrix we have a very concise notation for the objective $$\text{minimize}~~ \|Ax-b\|^2$$ Where $A$ is the data matrix, $x$ is the coefficients, and $b$ ...
14
votes
2answers
20k views

Mixing continuous and binary data with linear SVM?

So I've been playing around with SVMs and I wonder if this is a good thing to do: I have a set of continuous features (0 to 1) and a set of categorical features that I converted to dummy variables. ...
14
votes
2answers
8k views

If I repeat every sample observation in a linear regression model and rerun the regression how would the result be affected?

Say I have N observations, possibly multiple factors and I repeat each observation twice (or M times) how would a regression on this new set of size NM compare to a regression on just the original ...
14
votes
4answers
5k views

Updating linear regression efficiently when adding observations and/or predictors in R

I would be interested in finding ways in R for efficiently updating a linear model when an observation or a predictor is added. biglm has an updating capability when adding observations, but my data ...
14
votes
1answer
2k views

Confidence bands for QQ line

This question doesn't specifically pertain to R, but I chose to use R to illustrate it. Consider the code for producing ...
14
votes
1answer
2k views

Recovering raw coefficients and variances from orthogonal polynomial regression

It seems that if I have a regression model such as $y_i \sim \beta_0 + \beta_1 x_i+\beta_2 x_i^2 +\beta_3 x_i^3$ I can either fit a raw polynomial and get unreliable results or fit an orthogonal ...
14
votes
1answer
394 views

Restricted maximum likelihood with less than full column rank of $X$

This question deals with restricted maximum likelihood (REML) estimation in a particular version of the linear model, namely: $$ Y = X(\alpha)\beta + \epsilon, \\ \epsilon\sim N_n(0, \Sigma(\alpha)),...
13
votes
2answers
990 views

Linear vs. nonlinear regression

I have a set of values $x$ and $y$ which are theoretically related exponentially: $y = ax^b$ One way to obtain the coefficients is by applying natural logarithms in both sides and fitting a linear ...
13
votes
2answers
4k views

How can I use the value of $R^2$ to test the linearity assumption in multiple regression analysis?

The below graphs are residual scatter plots of a regression test for which "normality", "homoscedasticity" and "independence" assumptions have already been met for sure! For testing the "linearity" ...