Linked Questions
11 questions linked to/from Why do normal equations always have at least one solution?
0
votes
1
answer
79
views
In a linear model, why do we have $-2X^T \vec{y} + 2X^T X \vec{\beta}=0$? [duplicate]
When we derive the estimates of $\vec{\beta}$ such that they minimize the sum of squared error ($SSE$) we begin with $\sum_{i=1}^{n} (y_i - (\beta_0 + \beta_1x_1 + ... + \beta_kx_k))^2$. This is ...
- 126
0
votes
0
answers
27
views
A simple optimization problem [duplicate]
I am trying to derive ELM going through the basics , please help me out here :
$$f = x^Tx$$
$$g = Ax-b $$
The constraint is $Ax-b = 0$
I calculated $J' = f'+\lambda^T g'$
which is $2x+(\lambda^T ...
- 227
137
votes
7
answers
30k
views
Is there an intuitive interpretation of $A^TA$ for a data matrix $A$?
For a given data matrix $A$ (with variables in columns and data points in rows), it seems like $A^TA$ plays an important role in statistics. For example, it is an important part of the analytical ...
- 2,415
8
votes
1
answer
4k
views
How do we know $X'X$ is nonsingular in OLS?
I am currently working through understanding the mechanics of OLS estimates and the hat matrix. One thing I have been searching for without luck is how we know that the term $X'X$ is invertible where $...
- 335
9
votes
1
answer
5k
views
Intuition using linear algebra that the rank of the projection matrix equals the rank of the design matrix
Using linear algebra to explain, can someone show the intuition? I can show that the ranks are the same by using properties of rank but can't get my head around the whole projection thing more than ...
- 649
5
votes
1
answer
2k
views
Why is R-squared equal to the sum of standardized coefficients times the correlation?
Reading about standardized coefficients I came across the following formula: $$R^2=\sum\beta_ir_{yi}$$
Where $\beta$ is the standardized coefficient for the independent variable $i$ and $r_{yi}$ is ...
- 51
0
votes
2
answers
293
views
Orthogonality of columns of the augmented design matrix for ridge regression
In the question: How to derive the ridge regression solution? there is a solution by whuber, which describes how the columns of the augmented matrix approach pairwise orthogonality as the ...
- 109
4
votes
2
answers
256
views
Theoretical reason for multiple linear regression predictions being the same when adding and subtracting predictors
Say I have two variables $x_1$ and $x_2$, now I build a linear regression model as below
$$\hat{Y} = n_1 x_1 + n_2 x_2.$$
Then I build another model as below
$$\hat{Z} = m_1 (x_1 + x_2) + m_2 (x_1 - ...
- 713
3
votes
1
answer
98
views
Geometric understanding of linear regression
I am reading up on linear regression from mit 16.850
Here is how the lecture goes:
Given: $Y_{n,1}$ (targets), $X_{n, p}$ (data), $t_{p, 1}$ (the parameters I'm optimizing over), True model: $Y = \...
- 2,693
2
votes
1
answer
161
views
How do we get that $\hat\beta_1\,\sigma_x = \widehat{\beta_1\sigma_x} = \frac{1}{n}\sum_{i=1}^n \xi_i y_i = \sum_{i=1}^n \frac{\xi_i}{n} y_i$?
I am trying to understand this answer by @whuber to this question of how to get the standard errors of the regression estimators. @whuber says the following:
Variance of the slope estimate
The ...
- 2,204
1
vote
0
answers
151
views
OLS - The relationship between "minimizing SSR" and "the ration between cov(X,Y) and Var(X)" [closed]
Question
What would be the intuitive explanation for the slope of Ordinary Least Squares(OLS), which is $\frac{cov(X,Y)}{var(X)}$ contributes minimizing the sum of squared residuals?
In the same ...
- 377