49
votes
Accepted
Linearity of PCA
When we say that PCA is a linear method, we refer to the dimensionality reducing mapping $f:\mathbf x\mapsto \mathbf z$ from high-dimensional space $\mathbb R^p$ to a lower-dimensional space $\mathbb ...
- 106k
32
votes
Why Normality assumption in linear regression
We do choose other error distributions. You can in many cases do so fairly easily; if you are using maximum likelihood estimation, this will change the loss function. This is certainly done in ...
- 287k
28
votes
What does it mean for a linear regression to be statistically significant but has very low r squared?
It means that you can explain a small portion of the variance in the data. For instance, you can establish that a college degree impacts salaries, but at the same time it's just a small factor. There ...
- 61.9k
27
votes
Accepted
Adding a linear regression predictor decreases R squared
Could it be that you have missing values in Q that are getting auto-dropped? That'd have implications on the sample, making the two regressions not comparable.
- 13.6k
26
votes
In linear regression why does the response variable have to be continuous?
There's nothing stopping you using linear regression on any two columns of numbers you like. There are times when it might even be a quite sensible choice.
However, the properties of what you get out ...
- 287k
26
votes
Accepted
What is the best programmatic way for determining whether two variables are linearly or non-linearly or not even related
It is very difficult to achieve what you want programmatically because there are so many different forms of nonlinear associations. Even looking at correlation or regression coefficients will not ...
- 63.6k
25
votes
Accepted
Is linear regression obsolete?
It is true that the assumptions of linear regression aren't realistic. However, this is true of all statistical models. "All models are wrong, but some are useful."
I guess you're under the ...
23
votes
Accepted
Is a decision stump a linear model?
No, unless you transform the data.
It is a linear model if you transform $x$ using indicator function:
$$
x' = \mathbb I \left(\{x>2\}\right) = \begin{cases}\begin{align} 0 \quad &x\leq 2\\ 1 \...
- 12.8k
20
votes
Accepted
Are linear regression and least squares regression necessarily the same thing?
An explanation rather depends on what your background is.
Suppose you have some so-called independent variables $x_1,x_2,\ldots, x_k$ (they do not have to be independent of each other) where each $x_i$...
- 40.8k
20
votes
Accepted
If X=Y+Z, Is it ever useful to regress X on Y?
If you know $X = Y + Z$ and you have $Y$ and $Z$ measured, why would you need to run a regression of $X$ on $Y$ and $Z$? It provides no additional information and does not allow you to make "...
- 35.7k
19
votes
Accepted
Linear regression what does the F statistic, R squared and residual standard error tell us?
The best way to understand these terms is to do a regression calculation by hand. I wrote two closely related answers (here and here), however they may not fully help you understanding your particular ...
- 6,541
19
votes
Are linear regression and least squares regression necessarily the same thing?
Least squares is the processes of minimizing the sum of squared errors from some model. Given a function $f$ which depends on parameters $\theta$, the least squares estimates of $\theta$ are
$$ \hat{\...
- 38.6k
18
votes
Why do we need regularization for linear least squares given that a line is the simplest model possible?
Great question! The need for regularization always depends on your sample size.
Imagine you do not have a lot of data, just three samples. I plotted three possible linear regression lines. The red one ...
- 829
17
votes
What are the differences between ANOVAs and GLMs?
This is a common point of confusion, as the word ANOVA is used with different meanings in different textbooks / software packages. I'll try to sort this out a bit:
Historically, ANOVA is a method to ...
- 8,409
17
votes
Why does linear regression use a cost function based on the vertical distance between the hypothesis and the input data point?
When you have noise in both the dependent variable (vertical errors) and the independent variable (horizontal errors), the least squares objective function can be modified to incorporate these ...
- 37.8k
17
votes
Beginner Q: Residual Sum Squared (RSS) and R2
No, these numbers do not contradict each other. It sounds like you have an understanding of what $R^2$ means: it represents the proportion of the variance in your data which is explained by your model;...
- 380
15
votes
Why do we call the equations of least square estimation in linear regression the *normal equations*?
I'll give what is perhaps the most common understanding, then some additional details.
Normal is a term in geometry (Wikipedia):
In geometry, a normal is an object such as a line or vector that is ...
- 287k
15
votes
Correct formula for MSE
Both are correct. As said by blooraven (+1), this is the same kind of correction as in the unbiased estimator for sample variance. The second formula is used with linear regression corrects for the ...
- 140k
15
votes
Difference between E(Y) and E(Y|X) in regression
Disclaimer: I only read your question correctly after i wrote this. If $X$ is non random then yeah $E(Y|X) = E(Y)$.
Your mistake comes down to an abuse of notation. Within usual notation $X$ is a ...
- 2,972
14
votes
Accepted
Understand Link Function in Generalized Linear Model
So when you have binary response data, you have a "yes/no" or "1/0" outcome for each observation. However, what you are trying to estimate when doing a binary response regression is not a 1/0 outcome ...
- 992
14
votes
Why linear regression has assumption on residual but generalized linear model has assumptions on response?
The assumptions are not inconsistent. If, for $i = 1, \ldots, n$, you assume
$$
Y_i = \beta_0 + \beta_1 X_{i1} + \ldots + \beta_k X_{ik} + \epsilon_i,
$$
with the errors $\epsilon_i$ being normally ...
- 3,288
14
votes
Accepted
Why linear regression has assumption on residual but generalized linear model has assumptions on response?
Simple linear regression having Gaussian errors is a very nice attribute that does not generalize to generalized linear models.
In generalized linear models, the response follows some given ...
- 21.5k
14
votes
Linear regression when Y is bounded and discrete
When a response or outcome $Y$ is bounded, various questions arise in fitting a model, including the following:
Any model that could predict values for the response outside those bounds is in ...
- 58.7k
13
votes
Correcting log-transformation bias in a linear model
To understand the bias correction used in Miller (1984) you need to understand a little bit about the log-normal distribution. From the properties of the log-normal distribution, if $\ln Y \sim \text{...
- 130k
13
votes
Interpretation of statistically non-significant coefficient
It is not true in general that an insignificant variable has no effect on the response. A variable can be insignificant because the sample size is too low or the random variation too large to find a ...
- 26.8k
12
votes
How to run linear regression in a parallel/distributed way for big data setting?
Short Answer:
Yes, running linear regression in parallel has been done. For example, Xiangrui Meng et al. (2016) for Machine Learning in Apache Spark. The way it works is using stochastic gradient ...
- 37.2k
12
votes
Accepted
MLE for Linear Regression, student-t distributed error
The MLE is obtained by maximising the log-likelihood function, so the first thing you will want to do is have a look at this function. Using the density function for the Student T-distribution with ...
- 130k
12
votes
Different regression coefficients in R and Excel
The difference between coefficients is in the relation x versus y which is reversed in the one case.
Note that
in your R case the coefficient relates to 'suva'
and in your Excel case the ...
- 82.8k
12
votes
What is the problem with $p > n$?
This is a very good question. When the number of candidate predictors $p$ is more than the effective sample size $n$, and one does not place any restrictions on the regression coefficients (e.g., one ...
- 96.4k
12
votes
In the case of linear regression, if the parameters are uncorrelated, does this make the model better? If yes, why?
This depends on what you mean by "make the model better". Do you want to use this model to say something about how the world works, or to make predictions?
if the covariates are uncorrelated, then ...
- 1,414
Only top scored, non community-wiki answers of a minimum length are eligible