Questions tagged [linear-model]

Refers to any model where a random variable is related to one or more random variables by a function that is linear in a finite number of parameters.

Filter by
Sorted by
Tagged with
118
votes
9answers
107k views

When is it ok to remove the intercept in a linear regression model?

I am running linear regression models and wondering what the conditions are for removing the intercept term. In comparing results from two different regressions where one has the intercept and the ...
18
votes
1answer
5k views

Goodness of fit and which model to choose linear regression or Poisson

I need some advice regarding two main dilemmas in my research, which is a case study of 3 big pharmaceuticals and innovation. Number of patents per year is the dependent variable. My questions are ...
97
votes
9answers
173k views

What is the difference between linear regression on y with x and x with y?

The Pearson correlation coefficient of x and y is the same, whether you compute pearson(x, y) or pearson(y, x). This suggests that doing a linear regression of y given x or x given y should be the ...
101
votes
2answers
47k views

Removal of statistically significant intercept term increases $R^2$ in linear model

In a simple linear model with a single explanatory variable, $\alpha_i = \beta_0 + \beta_1 \delta_i + \epsilon_i$ I find that removing the intercept term improves the fit greatly (value of $R^2$ ...
38
votes
3answers
21k views

Why is polynomial regression considered a special case of multiple linear regression?

If polynomial regression models nonlinear relationships, how can it be considered a special case of multiple linear regression? Wikipedia notes that "Although polynomial regression fits a nonlinear ...
122
votes
3answers
258k views

What is the difference between linear regression and logistic regression?

What is the difference between linear regression and logistic regression? When would you use each?
90
votes
4answers
106k views

PCA and proportion of variance explained

In general, what is meant by saying that the fraction $x$ of the variance in an analysis like PCA is explained by the first principal component? Can someone explain this intuitively but also give a ...
15
votes
5answers
9k views

Can I ignore coefficients for non-significant levels of factors in a linear model?

After seeking clarification about linear model coefficients over here I have a follow up question concerning non-signficant (high p value) for coefficients of factor levels. Example: If my linear ...
13
votes
3answers
1k views

Definition and delimitation of regression model

An embarrassingly simple question -- but it seems it has not been asked on Cross Validated before: What is the definition of a regression model? Also a support question, What is not a regression ...
26
votes
7answers
27k views

Testing for linear dependence among the columns of a matrix

I have a correlation matrix of security returns whose determinant is zero. (This is a bit surprising since the sample correlation matrix and the corresponding covariance matrix should theoretically be ...
31
votes
2answers
8k views

Do we need gradient descent to find the coefficients of a linear regression model?

I was trying to learn machine learning using the Coursera material. In this lecture, Andrew Ng uses gradient descent algorithm to find the coefficients of the linear regression model that will ...
50
votes
4answers
10k views

Fast linear regression robust to outliers

I am dealing with linear data with outliers, some of which are at more the 5 standard deviations away from the estimated regression line. I'm looking for a linear regression technique that reduces the ...
45
votes
3answers
88k views

What is the effect of having correlated predictors in a multiple regression model?

I learned in my linear models class that if two predictors are correlated and both are included in a model, one will be insignificant. For example, assume the size of a house and the number of ...
69
votes
2answers
25k views

Shape of confidence interval for predicted values in linear regression

I have noticed that the confidence interval for predicted values in an linear regression tends to be narrow around the mean of the predictor and fat around the minimum and maximum values of the ...
36
votes
3answers
64k views

Derive Variance of regression coefficient in simple linear regression

In simple linear regression, we have $y = \beta_0 + \beta_1 x + u$, where $u \sim iid\;\mathcal N(0,\sigma^2)$. I derived the estimator: $$ \hat{\beta_1} = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}...
29
votes
5answers
77k views

How to derive the least square estimator for multiple linear regression?

In the simple linear regression case $y=\beta_0+\beta_1x$, you can derive the least square estimator $\hat\beta_1=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}$ such that you don't have to ...
55
votes
4answers
32k views

Choosing between LM and GLM for a log-transformed response variable

I'm trying to understand the philosophy behind using a Generalized Linear Model (GLM) vs a Linear Model (LM). I've created an example data set below where: $$\log(y) = x + \varepsilon $$ The ...
29
votes
1answer
16k views

Proof that the coefficients in an OLS model follow a t-distribution with (n-k) degrees of freedom

Background Suppose we have an Ordinary Least Squares model where we have $k$ coefficients in our regression model, $$\mathbf{y}=\mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}$$ where $\mathbf{\beta}$ ...
16
votes
1answer
2k views

Conditional expectation of R-squared

Consider the simple linear model: $$\pmb{y}=X'\pmb{\beta}+\epsilon$$ where $\epsilon_i\sim\mathrm{i.i.d.}\;\mathcal{N}(0,\sigma^2)$ and $X\in\mathbb{R}^{n\times p}$, $p\geq2$ and $X$ contains a ...
22
votes
2answers
5k views

Least Squares Regression Step-By-Step Linear Algebra Computation

As a prequel to a question about linear-mixed models in R, and to share as a reference for beginner/intermediate statistics aficionados, I decided to post as an independent "Q&A-style" the steps ...
9
votes
3answers
7k views

How to apply coefficient term for factors and interactive terms in a linear equation?

Using R, I have fitted a linear model for a single response variable from a mix of continuous and discrete predictors. This is uber-basic, but I'm having trouble grasping how a coefficient for a ...
9
votes
2answers
636 views

Is there an elegant/insightful way to understand this linear regression identity for multiple $R^2$?

In linear regression I have come across a delightful result that if we fit the model $$E[Y] = \beta_1 X_1 + \beta_2 X_2 + c,$$ then, if we standardize and centre the $Y$, $X_1$ and $X_2$ data, $$R^...
20
votes
5answers
77k views

Assumptions of linear models and what to do if the residuals are not normally distributed

I am a little bit confused on what the assumptions of linear regression are. So far I checked whether: all of the explanatory variables correlated linearly with the response variable. (This was the ...
16
votes
1answer
2k views

Where do the assumptions for linear regression come from? [duplicate]

I'v already known that there are several assumpations when using linear regression model. But I cannot understand why some of them exists. They are: independent errors normal distribution of errors ...
22
votes
3answers
13k views

Regression modelling with unequal variance

I would like to fit a linear model (lm) where the residuals variance is clearly dependent on the explanatory variable. The way I know to do this is by using glm with the Gamma family to model the ...
9
votes
1answer
1k views

Normality assumption in linear regression

As an assumption of linear regression, the normality of the distribution of the error is sometimes wrongly "extended" or interpreted as the need for normality of the y or x. Is it possible to ...
14
votes
1answer
2k views

Recovering raw coefficients and variances from orthogonal polynomial regression

It seems that if I have a regression model such as $y_i \sim \beta_0 + \beta_1 x_i+\beta_2 x_i^2 +\beta_3 x_i^3$ I can either fit a raw polynomial and get unreliable results or fit an orthogonal ...
7
votes
1answer
29k views

The main effect will be non-significant if the interaction is significant? [duplicate]

I am using linear mixed models to identify important factors, and it turns out that: A: significant B: not significant ...
3
votes
3answers
946 views

Identity in Simple Linear Model

I'm working on this identity $$\sum_{i-1}^n (y_i - \hat {\beta_0} - \hat {\beta_1}x_i)^2 = \sum_{i=1}^n y_i^2 - \hat {\beta_0}\sum_{i=1}^n y_i - \hat {\beta_1} \sum_{i=1}^n x_iy_i$$ I have these ...
33
votes
4answers
16k views

(Why) do overfitted models tend to have large coefficients?

I imagine that the larger a coefficient on a variable is, the more ability the model has to "swing" in that dimension, providing an increased opportunity to fit noise. Although I think I've got a ...
45
votes
3answers
3k views

Where does the misconception that Y must be normally distributed come from?

Seemingly reputable sources claim that the dependent variable must be normally distributed: Model assumptions: $Y$ is normally distributed, errors are normally distributed, $e_i \sim N(0,\sigma^2)...
16
votes
2answers
5k views

Why is GLM different than an LM with transformed variable

As explained in this course handout (page 1), a linear model can be written in the form: $$ y = \beta_1 x_{1} + \cdots + \beta_p x_{p} + \varepsilon_i,$$ where $y$ is the response variable and $x_{...
15
votes
3answers
9k views

For linear classifiers, do larger coefficients imply more important features?

I'm a software engineer working on machine learning. From my understanding, linear regression (such as OLS) and linear classification (such as logistic regression and SVM) make a prediction based on ...
15
votes
3answers
44k views

When can we speak of collinearity

In linear models we need to check if a relationship exists among the explanatory variables. If they correlate too much then there is collinearity (i.e., the variables partly explain each other). I am ...
15
votes
1answer
8k views

Understanding QR Decomposition

I've got a worked example (in R), that I'm trying to understand further. I'm using Limma to create a linear model and I'm trying to understand what's happening step by step in the fold change ...
12
votes
3answers
7k views

Perform linear regression, but force solution to go through some particular data points

I know how to perform a linear regression on a set of points. That is, I know how to fit a polynomial of my choice, to a given data set, (in the LSE sense). However, what I do not know, is how to ...
15
votes
2answers
27k views

VIF, condition Index and eigenvalues

I am currently assessing multicollinearity in my datasets. What threshold values of VIF and condition index below/above suggest a problem? VIF: I have heard that VIF $\geq 10$ is a problem. After ...
7
votes
2answers
1k views

Likelihood in Linear Regression

I am trying to understand how people derive the Likelihood for simple linear regression. Lets say that we just have one feature x and the outcome y. I do not doubt the expression with the normal ...
25
votes
2answers
26k views

General Linear Model vs. Generalized Linear Model (with an identity link function?)

This is my first post, so please take it easy on me if I am not following some standards! I did a search for my question and nothing came up. My question relates mostly around the practical ...
12
votes
4answers
15k views

Distinction between linear and nonlinear model

I have read some explanations about the properties of linear vs nonlinear models, but still I am sometimes not sure if a model on hand is a linear or a nonlinear one. For example, is the following ...
15
votes
4answers
2k views

Classic linear model - model selection

I have a classic linear model, with 5 possible regressors. They are uncorrelated with one another, and have quite low correlation with the response. I have arrived at a model where 3 of the regressors ...
9
votes
1answer
5k views

Should the difference between control and treatment be modelled explicitly or implicitly?

Given the following experimental setup: Multiple samples are taken from a subject and each sample is treated multiple ways (including a control treatment). What is mainly interesting is the ...
14
votes
2answers
8k views

Matrix notation for logistic regression

In linear regression (squared loss), using matrix we have a very concise notation for the objective $$\text{minimize}~~ \|Ax-b\|^2$$ Where $A$ is the data matrix, $x$ is the coefficients, and $b$ ...
13
votes
2answers
994 views

Linear vs. nonlinear regression

I have a set of values $x$ and $y$ which are theoretically related exponentially: $y = ax^b$ One way to obtain the coefficients is by applying natural logarithms in both sides and fitting a linear ...
10
votes
3answers
1k views

Is the linearity assumption in linear regression merely a definition of $\epsilon$?

I am revising linear regression. The textbook by Greene states: Now, of course there will be other assumptions on the linear regression model, such as $E(\epsilon|X)=0$. This assumption ...
8
votes
2answers
7k views

Interpret Regression Coefficients After various Differencing

There are few explanations I can find that describe how to interpret linear regression coefficients after differencing a time series (to eliminate a unit root). Is it just so simple that there is no ...
13
votes
2answers
4k views

How can I use the value of $R^2$ to test the linearity assumption in multiple regression analysis?

The below graphs are residual scatter plots of a regression test for which "normality", "homoscedasticity" and "independence" assumptions have already been met for sure! For testing the "linearity" ...
11
votes
5answers
5k views

What makes mean square error so good? [duplicate]

Our statistical inference course material states the following: The principle of mean square error can be derived from the principle of maximum likelihood (after we set a linear model where ...
1
vote
1answer
570 views

Difference between linear regression and neural network

I am obviously confused with terms, and different concepts behind it. Each websites gives different intuitions. With all intuitions my brain is full of confusion now. Please help me to address what is ...
11
votes
4answers
14k views

What does “curvilinear” mean?

As far as I can tell, curvilinear is defined vaguely but means the same as nonlinear. Is that correct? Or does curvilinear have a distinct definition?