Skip to main content
35 votes
Accepted

Relationship between Cholesky decomposition and matrix inversion?

Gaussian process models often involve computing some quadratic form, such as $$ y = x^\top\Sigma^{-1}x $$ where $\Sigma$ is positive definite, $x$ is a vector of appropriate dimension, and we wish to ...
Sycorax's user avatar
  • 92.2k
27 votes

What is an example of perfect multicollinearity?

Here are a couple of fairly common scenarios producing perfect multicollinearity, i.e. situations in which the columns of the design matrix are linearly dependent. Recall from linear algebra that this ...
Silverfish's user avatar
  • 23.7k
25 votes

Why does inversion of a covariance matrix yield partial correlations between random variables?

Here is a proof with just matrix calculations. I appreciate the answer by whuber. It is very insightful on the math behind the scene. However, it is still not so trivial how to use his answer to ...
Po C.'s user avatar
  • 350
17 votes
Accepted

Why can't we cancel these two matrices in the OLS estimator?

Why can't we simply cancel out the $X^T$ I remember asking myself almost exactly the same question 300 years ago upon seeing the regression equation $y=X\beta$ for the first time in my life. The ...
Aksakal's user avatar
  • 61.6k
13 votes
Accepted

What is an example of perfect multicollinearity?

Here is an example with 3 variables, $y$, $x_1$ and $x_2$, related by the equation $$ y = x_1 + x_2 + \varepsilon $$ where $\varepsilon \sim N(0,1)$ The particular data are ...
Robert Long's user avatar
  • 62.5k
12 votes
Accepted

Why doesn't $(e^{A})^{-1} = e^{-A}$ hold for a symmetric matrix in Python?

This is a case of reasoning from a false premise. The matrix exponential is not defined as the exponentiation of each element of a matrix. Instead, the definition of a matrix exponential is $$ \exp(X)...
Sycorax's user avatar
  • 92.2k
11 votes
Accepted

Obtaining the possible least squares solutions when $X^TX$ is not invertible

You want all the possible solutions? Write the linear model in matrix form as $$ Y=X\beta + \epsilon, $$ and let the Moore-Penrose inverse of matrix $A$ be denoted by $A^+$. The normal equations for ...
kjetil b halvorsen's user avatar
11 votes

Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression

Geometric viewpoint A geometric viewpoint can be like the n-dimensional vectors $y$ and $X\beta$ being points in n-dimensional-space $V$. Where $X\beta$ is also in the subspace $W$ spanned by the ...
Sextus Empiricus's user avatar
10 votes
Accepted

How do we know $X'X$ is nonsingular in OLS?

It's a property of the $\text{rank}$ operator when its used on real matrices $\mathbf{A}$: $$ \text{rank}(\mathbf{A}) = \text{rank}(\mathbf{A}') = \text{rank}(\mathbf{A}'\mathbf{A}) = \text{rank}(\...
Taylor's user avatar
  • 20.9k
9 votes

Matrices: system that is "computationally singular" versus "exactly singular"

The condition number that is typically used in linear algebra to describe a matrix $A$ indicates, very roughly, how many significant digits of accuracy you can expect to lose when solving a linear ...
jbowman's user avatar
  • 40k
9 votes
Accepted

Updating the inverse covariance matrix after deleting the i-th column and row of the covariance matrix

When $i=n,$ write $\mathbb{A}$ in block matrix form $$\mathbb A = \pmatrix{A & B \\ C & D}$$ where $A$ is the $n-1 \times n-1$ matrix obtained by omitting the last row and column of $\...
whuber's user avatar
  • 327k
8 votes

Inverse covariance matrix, off-diagonal entries

The underlying intuition is quite general: because multiplying a matrix by its inverse has to produce a matrix with a lot of zeros, if the original matrix contains only positive values then obviously ...
whuber's user avatar
  • 327k
8 votes
Accepted

Usefulness of convexity of linear regression when there is no closed form solution

When $(X^TX)$ is not invertible there is not one solution but several: an affine subspace. But they are still closed form solutions in a way. They are solutions of the linear system: $(X^TX)\beta=X^Ty$...
Benoit Sanchez's user avatar
7 votes

What is the physical significance of inverse of a matrix?

Matrix Inverse in Terms of Geometry: If a matrix works on a set of vectors by rotating and scaling the vectors, then the matrix's inverse will undo the rotations and scalings and return the original ...
Beyer's user avatar
  • 1,232
7 votes

Why does inversion of a covariance matrix yield partial correlations between random variables?

Note that the sign of the answer actually depends on how you define partial correlation. There is a difference between regressing $X_i$ and $X_j$ on the other $n - 1$ variables separately vs. ...
Johnny Ho's user avatar
7 votes

Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression

Assuming you're familiar with the simple linear regression: $$y_i=\alpha+\beta x_i+\varepsilon_i$$ and its solution: $$\beta=\frac{\mathrm{cov}[x_i,y_i]}{\mathrm{var}[x_i]}$$ It's easy to see how $X'...
Aksakal's user avatar
  • 61.6k
7 votes

Explanation of generalization of Newton's Method for multiple dimensions

Overview Suppose we want to minimize an objective function $f$ that maps a parameter vector $x \in \mathbb{R}^d$ to a scalar value. The idea behind Newton's method is to locally approximate $f$ with ...
user20160's user avatar
  • 32.8k
7 votes

Which is more numerically stable for OLS: pinv vs QR

Using the Moore-Penrose pseudo-inverse $X^{\dagger}$ of an matrix $X$ is more stable in the sense that can directly account for rank-deficient design matrices $X$. $X^{\dagger}$ allows us to ...
usεr11852's user avatar
  • 44.7k
7 votes

Parameter estimate in linear regression model with singular matrix $X^TX$

The question asks The matrix above tends to be singular and I can not get a unique solution. And I am trying to find the possible solutions of $\hat{\beta}$. The solution set to the specific problem ...
Sycorax's user avatar
  • 92.2k
6 votes
Accepted

Residual Sum of squares in Weighted regression

There are several good ways to do this using R. One classical method is to compute the Choleski factor of the covariance matrix: ...
Gordon Smyth's user avatar
  • 13.1k
6 votes
Accepted

Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression

I found these posts particularly helpful: How to derive the least square estimator for multiple linear regression? Relationship between SVD and PCA. How to use SVD to perform PCA? http://www.math....
James McKeown's user avatar
6 votes
Accepted

What do you do in regression if $(X^\text{T} X)^{-1}$ does not exist?

The matrix $X^\text{T} X$ is the Gramian matrix of the design matrix (assuming here that the design matrix has elements that are all real numbers). If the Gram-determinant is zero (i.e., if $\text{...
Ben's user avatar
  • 127k
6 votes
Accepted

Why do large LMs use the transpose of the word embeddings matrix in the classification head?

Turning my comment into an answer. This practice is called parameter tying. They’ve set $U$ and $V$ to be equal to each other as a constraint, which regularizes the model. The information about token ...
Arya McCarthy's user avatar
5 votes

What is an example of perfect multicollinearity?

Some trivial examples to help intuition: $\mathbf{x_1}$ is height in centimeters. $\mathbf{x_2}$ is height in meters. Then: $\mathbf{x_1} = 100 \mathbf{x_2}$, and your design matrix $X$ will not ...
Matthew Gunn's user avatar
  • 22.5k
5 votes
Accepted

Least Squares removing first $k$ observations Woodbury formula?

You've basically laid out the key facts, I think you just need a hint on how to fit them all together. Here's a quick-and-dirty overview. I think it's easier to see how to accomplish your goal if you ...
Sycorax's user avatar
  • 92.2k
5 votes
Accepted

How to prove $(P^{-1} + B^T R^{-1} B)^{-1} B^T R^{-1} = PB^T(BPB^T + R)^{-1}$

We have: \begin{align*} (P^{-1}+B^TR^{-1}B)^{-1}B^TR^{-1}&=PB^T(BPB^T+R)^{-1}\\ (P^{-1}+B^TR^{-1}B)^{-1}B^TR^{-1}(BPB^T+R)&=PB^T(BPB^T+R)^{-1}(BPB^T+R)\\ (P^{-1}+B^TR^{-1}B)^{-1}(B^TR^{-1}BPB^...
Adrian Keister's user avatar
5 votes
Accepted

Why does a valid Kernel only have to be positive semi-definite instead of positive definite?

You have not told us what you consider a valid kernel, so I can't comment on that. But I can give you a reason why one likes to consider positive semi-definite kernels and not just positive definite ...
g g's user avatar
  • 2,708
4 votes
Accepted

Inverse of block covariance matrix

The inverse of a block (or partitioned) matrix is given by $$ \left[ \begin{array}{cc} M_{11} & M_{12} \\ M_{21} & M_{22} \end{array} \right] ^{-1} = \left[ \begin{array}{cc} K_1^{-1} & -...
Elizabeth Santorella's user avatar
4 votes
Accepted

Non-Singularity due to inclusion of non-zero lambda in ridge regression

Full rank matrix is invertible. Adding $\lambda I$ to $X^TX$ make it a full rank matrix. Here is an example. Suppose $X$ has two identical columns: ...
Haitao Du's user avatar
  • 37.1k
4 votes
Accepted

The probability limit of an inverse matrix

I'm going to change your notation a little bit to avoid so many subscripts and to make things clearer. It seems that you're considering a sequence of random matrices $$ M_n = \begin{bmatrix} a_n & ...
jld's user avatar
  • 20.4k

Only top scored, non community-wiki answers of a minimum length are eligible