# Tag Info

Accepted

### Relationship between Cholesky decomposition and matrix inversion?

Gaussian process models often involve computing some quadratic form, such as $$y = x^\top\Sigma^{-1}x$$ where $\Sigma$ is positive definite, $x$ is a vector of appropriate dimension, and we wish to ...
• 92.2k

### What is an example of perfect multicollinearity?

Here are a couple of fairly common scenarios producing perfect multicollinearity, i.e. situations in which the columns of the design matrix are linearly dependent. Recall from linear algebra that this ...
• 23.7k

### Why does inversion of a covariance matrix yield partial correlations between random variables?

Here is a proof with just matrix calculations. I appreciate the answer by whuber. It is very insightful on the math behind the scene. However, it is still not so trivial how to use his answer to ...
• 350
Accepted

### Why can't we cancel these two matrices in the OLS estimator?

Why can't we simply cancel out the $X^T$ I remember asking myself almost exactly the same question 300 years ago upon seeing the regression equation $y=X\beta$ for the first time in my life. The ...
• 61.6k
Accepted

### What is an example of perfect multicollinearity?

Here is an example with 3 variables, $y$, $x_1$ and $x_2$, related by the equation $$y = x_1 + x_2 + \varepsilon$$ where $\varepsilon \sim N(0,1)$ The particular data are ...
• 62.5k
Accepted

• 20.9k

### Matrices: system that is "computationally singular" versus "exactly singular"

The condition number that is typically used in linear algebra to describe a matrix $A$ indicates, very roughly, how many significant digits of accuracy you can expect to lose when solving a linear ...
• 40k
Accepted

• 61.6k

### Explanation of generalization of Newton's Method for multiple dimensions

Overview Suppose we want to minimize an objective function $f$ that maps a parameter vector $x \in \mathbb{R}^d$ to a scalar value. The idea behind Newton's method is to locally approximate $f$ with ...
• 32.8k

### Which is more numerically stable for OLS: pinv vs QR

Using the Moore-Penrose pseudo-inverse $X^{\dagger}$ of an matrix $X$ is more stable in the sense that can directly account for rank-deficient design matrices $X$. $X^{\dagger}$ allows us to ...
• 44.7k

### Parameter estimate in linear regression model with singular matrix $X^TX$

The question asks The matrix above tends to be singular and I can not get a unique solution. And I am trying to find the possible solutions of $\hat{\beta}$. The solution set to the specific problem ...
• 92.2k
Accepted

### Residual Sum of squares in Weighted regression

There are several good ways to do this using R. One classical method is to compute the Choleski factor of the covariance matrix: ...
• 13.1k
Accepted

### Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression

I found these posts particularly helpful: How to derive the least square estimator for multiple linear regression? Relationship between SVD and PCA. How to use SVD to perform PCA? http://www.math....
Accepted

The matrix $X^\text{T} X$ is the Gramian matrix of the design matrix (assuming here that the design matrix has elements that are all real numbers). If the Gram-determinant is zero (i.e., if $\text{... • 127k 6 votes Accepted ### Why do large LMs use the transpose of the word embeddings matrix in the classification head? Turning my comment into an answer. This practice is called parameter tying. They’ve set$U$and$V$to be equal to each other as a constraint, which regularizes the model. The information about token ... • 8,826 5 votes ### What is an example of perfect multicollinearity? Some trivial examples to help intuition:$\mathbf{x_1}$is height in centimeters.$\mathbf{x_2}$is height in meters. Then:$\mathbf{x_1} = 100 \mathbf{x_2}$, and your design matrix$X$will not ... • 22.5k 5 votes Accepted ### Least Squares removing first$k$observations Woodbury formula? You've basically laid out the key facts, I think you just need a hint on how to fit them all together. Here's a quick-and-dirty overview. I think it's easier to see how to accomplish your goal if you ... • 92.2k 5 votes Accepted ### How to prove$(P^{-1} + B^T R^{-1} B)^{-1} B^T R^{-1} = PB^T(BPB^T + R)^{-1}\$

We have: \begin{align*} (P^{-1}+B^TR^{-1}B)^{-1}B^TR^{-1}&=PB^T(BPB^T+R)^{-1}\\ (P^{-1}+B^TR^{-1}B)^{-1}B^TR^{-1}(BPB^T+R)&=PB^T(BPB^T+R)^{-1}(BPB^T+R)\\ (P^{-1}+B^TR^{-1}B)^{-1}(B^TR^{-1}BPB^...
• 5,842
Accepted

### Why does a valid Kernel only have to be positive semi-definite instead of positive definite?

You have not told us what you consider a valid kernel, so I can't comment on that. But I can give you a reason why one likes to consider positive semi-definite kernels and not just positive definite ...
• 2,708