Linked Questions
20 questions linked to/from Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression
630
votes
5
answers
517k
views
Relationship between SVD and PCA. How to use SVD to perform PCA?
Principal component analysis (PCA) is usually explained via an eigen-decomposition of the covariance matrix. However, it can also be performed via singular value decomposition (SVD) of the data matrix ...
- 107k
187
votes
10
answers
126k
views
Bottom to top explanation of the Mahalanobis distance?
I'm studying pattern recognition and statistics and almost every book I open on the subject I bump into the concept of Mahalanobis distance. The books give sort of intuitive explanations, but still ...
- 5,907
137
votes
7
answers
30k
views
Is there an intuitive interpretation of $A^TA$ for a data matrix $A$?
For a given data matrix $A$ (with variables in columns and data points in rows), it seems like $A^TA$ plays an important role in statistics. For example, it is an important part of the analytical ...
- 2,415
46
votes
5
answers
145k
views
How to derive the least square estimator for multiple linear regression?
In the simple linear regression case $y=\beta_0+\beta_1x$, you can derive the least square estimator $\hat\beta_1=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}$ such that you don't have to ...
- 859
19
votes
8
answers
3k
views
What is the intuition behind the idea that for linear regression, the number of observations should exceed the number of parameters?
If a population model has k independent variables and 1 intercept, why are k+1 observations required to perform OLS estimates?
What is the intuition behind this?
- 309
24
votes
2
answers
13k
views
Relationship between Gram and covariance matrices
For a $n\times p$ matrix $X$, where $p \gg n$, what is the relationship between $X^{T}X$ (scatter matrix, on which covariance matrix is based) and $XX^{T}$ (outer product sometimes called Gram matrix)?...
- 439
11
votes
2
answers
2k
views
How does R lm() function calculate standard error of slopes with more than one predictor? [duplicate]
I am confused as to why the standard error of slopes calculated by the R function 'lm()' differs from the following formula when there is more than one predictor:
$$
SE(\hat{\beta}_j) = \sqrt{\frac{\...
- 133
7
votes
5
answers
4k
views
regression with multiple independent variables vs multiple regressions with one independent variable
For example, we want to use age and IQ to predict GPA.
Of course we can do a multiple linear regression, i.e. regress GPA on age and IQ.
My question is: can we do two simple regressions instead? ...
- 71
12
votes
2
answers
4k
views
What is the geometric relationship between the covariance matrix and the inverse of the covariance matrix?
The covariance matrix represents the dispersion of data points while the inverse of the covariance matrix represents the tightness of data points. How is the dispersion and tightness related ...
- 252
10
votes
2
answers
2k
views
Is there an elegant/insightful way to understand this linear regression identity for multiple $R^2$?
In linear regression I have come across a delightful result that if we fit the model
$$E[Y] = \beta_1 X_1 + \beta_2 X_2 + c,$$
then, if we standardize and centre the $Y$, $X_1$ and $X_2$ data,
$$R^...
- 5,415
15
votes
1
answer
3k
views
Why is multicollinearity different than correlation?
I know that someone will probably say that this question is repeated and I will get a negative vote, but I'm very convinced that it's not, or at least it wasn't properly answered. See, we have lots of ...
- 327
3
votes
3
answers
2k
views
Demeaning variables in OLS
I am analysing some data in R where I have information on $y$ and $x$.
When I run $y = \alpha + \beta\cdot x$ I get the same coefficient on $x$ (i.e. $\beta$) as ...
- 55
4
votes
3
answers
250
views
Different OLS regression procedures that lead to the same coefficients
I've rewritten this question, because my phrasing and notation was confusing.
We're assuming OLS regression throughout this post.
If we have the data $\mathbf{y} \in \mathbb{R}^N$, $\mathbf{X}\in \...
- 387
6
votes
2
answers
531
views
Why do my (coefficients, standard errors & CIs, p-values & significance) change when I add a term to my regression model?
Lots of people seem to be asking this. They often seem to get shallow answers that merely assert what is true, instead of drawing or explaining the mechanism. They also seem to not find each other -- ...
- 5,288
4
votes
2
answers
2k
views
why does the same variable have a different slope when incorporated into a linear model with multiple x variables
When I call for the summary of my linear model it shows X2 to have a negative slope. But
when i call the same variable in a linear model of its own it has a positive slope. why is it not just the same ...
- 81