Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
42 votes
3 answers
20k views

Distribution of scalar products of two random unit vectors in $D$ dimensions

If $\mathbf{x}$ and $\mathbf{y}$ are two independent random unit vectors in $\mathbb{R}^D$ (uniformly distributed on a unit sphere), what is the distribution of their scalar product (dot product) $\...
amoeba's user avatar
  • 104k
91 votes
3 answers
25k views

What is the intuition behind SVD?

I have read about singular value decomposition (SVD). In almost all textbooks it is mentioned that it factorizes the matrix into three matrices with given specification. But what is the intuition ...
SHASHANK GUPTA's user avatar
194 votes
10 answers
47k views

Why the sudden fascination with tensors?

I've noticed lately that a lot of people are developing tensor equivalents of many methods (tensor factorization, tensor kernels, tensors for topic modeling, etc) I'm wondering, why is the world ...
84 votes
6 answers
44k views

What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?

I've read a lot about PCA, including various tutorials and questions (such as this one, this one, this one, and this one). The geometric problem that PCA is trying to optimize is clear to me: PCA ...
stackoverflowuser2010's user avatar
36 votes
4 answers
35k views

Why is a sample covariance matrix singular when sample size is less than number of variables?

Let's say I have a $p$-dimensional multivariate Gaussian distribution. And I take $n$ observations (each of them a $p$-vector) from this distribution and calculate the sample covariance matrix $S$. In ...
user34790's user avatar
  • 6,677
64 votes
5 answers
45k views

Is every covariance matrix positive definite?

I guess the answer should be yes, but I still feel something is not right. There should be some general results in the literature, could anyone help me?
Jingjings's user avatar
  • 1,283
8 votes
1 answer
9k views

completing the square for Gaussian multivariate estimation

I have been trying to derive the posterior distribution in the case of weighted Bayesian regression in the case of multivariate normal distribution for a few days and have been stuck. I am not sure if ...
Luca's user avatar
  • 4,650
4 votes
1 answer
3k views

Understanding linear algebra in Ordinary Least Squares derivation

In ordinary least squared there is this equation (Kevin Murphy book page 221, latest edition) $$NLL(w)=\frac{1}{2}({y-Xw})^T(y-Xw)=\frac{1}{2}w^T(X^TX)w-w^T(X^T)y$$ I am not sure how the RHS equals ...
mathopt's user avatar
  • 235
61 votes
12 answers
22k views

Reference book for linear algebra applied to statistics?

I have been working in R for a bit and have been faced with things like PCA, SVD, QR decompositions and many such linear algebra results (when inspecting estimating weighted regressions and such) so I ...
53 votes
4 answers
30k views

Why does inversion of a covariance matrix yield partial correlations between random variables?

I heard that partial correlations between random variables can be found by inverting the covariance matrix and taking appropriate cells from such resulting precision matrix (this fact is mentioned in ...
michal's user avatar
  • 1,278
21 votes
3 answers
2k views

Weird correlations in the SVD results of random data; do they have a mathematical explanation or is it a LAPACK bug?

I observe a very weird behaviour in the SVD outcome of random data, which I can reproduce in both Matlab and R. It looks like some numerical issue in the LAPACK library; is it? I draw $n=1000$ ...
amoeba's user avatar
  • 104k
26 votes
1 answer
24k views

How to whiten the data using principal component analysis?

I want to transform my data $\mathbf X$ such that the variances will be one and the covariances will be zero (i.e I want to whiten the data). Furthermore the means should be zero. I know I will get ...
Angelorf's user avatar
  • 1,631
25 votes
3 answers
17k views

Why is the rank of covariance matrix at most $n-1$?

As stated in this question, the maximum rank of covariance matrix is $n-1$ where $n$ is sample size and so if the dimension of covariance matrix is equal to the sample size, it would be singular. I ...
user3070752's user avatar
10 votes
4 answers
1k views

Intuition for nonmonotonicity of coefficient paths in ridge regression

Intuitively, why may some of the slope coefficients in ridge regression increase in magnitude when the penalty parameter $\lambda$ is increased? Or in other words, why are the coefficient paths ...
Richard Hardy's user avatar
9 votes
3 answers
5k views

Is every correlation matrix positive semi-definite?

I am generating correlation matrix by some new algorithm. Generated matrix is non positive semi-definite matrix. I'm getting a few negative eigenvalues. The rest of eigenvalues are quite equal to the ...
Vanita's user avatar
  • 91
7 votes
1 answer
3k views

When to use dot-product as a similarity metric

I'm trying to understand which similarity measure should be used in which situations. Roughly speaking, when should someone use the dot product to assess similarity between vectors? Roughly speaking, ...
JacKeown's user avatar
  • 718
1 vote
1 answer
1k views

Conditional multivariate Gaussian distribution

I am trying to understand how 'completing the square' method is used to determine mean and co-variance of multivariate Gaussian distribution but I am not able to make much sense of the following step $...
Rakesh K's user avatar
35 votes
7 answers
26k views

Why are symmetric positive definite (SPD) matrices so important?

I know the definition of symmetric positive definite (SPD) matrix, but want to understand more. Why are they so important, intuitively? Here is what I know. What else? For a given data, Co-...
Haitao Du's user avatar
  • 36.6k
29 votes
1 answer
31k views

Multivariate normal posterior

This is a very simple question but I can't find the derivation anywhere on the internet or in a book. I would like to see the derivation of how one Bayesian updates a multivariate normal distribution....
Alex's user avatar
  • 853
28 votes
2 answers
9k views

Why is the Fisher Information matrix positive semidefinite?

Let $\theta \in R^{n}$. The Fisher Information Matrix is defined as: $$I(\theta)_{i,j} = -E\left[\frac{\partial^{2} \log(f(X|\theta))}{\partial \theta_{i} \partial \theta_{j}}\bigg|\theta\right]$$ ...
madprob's user avatar
  • 383
20 votes
1 answer
3k views

Geometric understanding of PCA in the subject (dual) space

I am trying to get an intuitive understanding of how principal component analysis (PCA) works in subject (dual) space. Consider 2D dataset with two variables, $x_1$ and $x_2$, and $n$ data points (...
amoeba's user avatar
  • 104k
11 votes
2 answers
1k views

Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

I am trying to do SVD by hand: ...
failedstatistician's user avatar
10 votes
2 answers
3k views

Appropriate measure to find smallest covariance matrix

In the textbook I am reading they use positive definiteness (semi-positive definiteness) to compare two covariance matrices. The idea being that if $A-B$ is pd then $B$ is smaller than $A$. But I'm ...
Baz's user avatar
  • 1,763
8 votes
2 answers
2k views

Prove that $\mathrm{Cov}(x^TAx,x^TBx) = 2 \mathrm{Tr}(A \Sigma B \Sigma) + 4 \mu^TA \Sigma B \mu$

Suppose $\vec x \sim N(\vec \mu, \Sigma)$ is multivariate normal. I want to see that $$\mathrm{Cov}(\vec x^TA\vec x,\vec x^TB\vec x) = 2 \mathrm{Tr}(A \Sigma B \Sigma) + 4 \vec \mu^TA \Sigma B \vec \...
Mikkel Rev's user avatar
25 votes
1 answer
13k views

Updating SVD decomposition after adding one new row to the matrix

Suppose that I have a dense matrix $ \textbf{A}$ of $m \times n$ size, with SVD decomposition $$\mathbf{A}=\mathbf{USV}^\top.$$ In R I can calculate the SVD as ...
user1436187's user avatar
17 votes
1 answer
14k views

How does NumPy solve least squares for underdetermined systems?

Let's say that we have X of shape (2, 5) and y of shape (2,) This works: np.linalg.lstsq(X, y) We would expect this to work only if X was of shape (N,5) where N>=...
George Pligoropoulos's user avatar
10 votes
1 answer
9k views

Why does the rank of the design matrix X equal the rank of X'X?

Why does the rank of the design matrix $\boldsymbol X$ equal the rank of $\boldsymbol{X'X}$? Is this true in all circumstances? If X is not linearly independent, what would the rank of X'X be?
kurt's user avatar
  • 123
8 votes
2 answers
3k views

Is PCA invariant to orthogonal transformations?

Let $A$ a be an $n$ x $p$ matrix and let $B$ be the transformed data set of $A$ under $Q$: $$ B = A Q $$ where Q is a $p$ x $p$ orthogonal matrix: $$ Q Q^T = I $$ $n$ is the number of samples (...
turdus-merula's user avatar
7 votes
1 answer
259 views

Interpretation of $\mathbf{y}^T(\mathbf{I}-\mathbf{H})\mathbf{y}$ in OLS

In classic OLS regression it is well-known that $(\mathbf{I}-\mathbf{H})\mathbf{y}=\mathbf{r}$, where $\mathbf{I}$ is the identity matrix, $\mathbf{H}$ is the hat matrix, $\mathbf{y}$ is the vector of ...
boscovich's user avatar
  • 1,656
6 votes
1 answer
859 views

Deriving the maximum likelihood for the parameters in linear regression

Notation: $\textbf{w}$ is an M-dimensional vector of parameters (including the bias parameter), $\textbf{x}_n$ is an M-dimensional vector of the features of each training example, $\textbf{t}$ is an N-...
BitRiver's user avatar
  • 467
2 votes
1 answer
373 views

Why is it necessary to "ignore" a level when applying sum contrasts?

I am confused about how sum contrasts are set up. As I understand, if I have some $K$-leveled factor, I can use sum contrasts to compare each level to the grand mean ($M_G$), effectively testing ...
Alex Ten's user avatar
1 vote
1 answer
3k views

Why does logistic regression's likelihood function have no closed form?

The derivative of Log likelihood function of logistic regression with respect to theta is Why can't we equate it to zero and solve for theta so that we can obtain a 'closed form solution' for theta? ...
Preetham_tsp's user avatar
51 votes
7 answers
32k views

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
DongukJu's user avatar
  • 653
16 votes
4 answers
4k views

Sufficient and necessary conditions for zero eigenvalue of a correlation matrix

Given $n$ random variable $X_i$, with probability distribution $P(X_1,\ldots,X_n)$, the correlation matrix $C_{ij}=E[X_i X_j]-E[X_i]E[X_j]$ is positive semi-definite, i.e. its eigenvalues are positive ...
Adam's user avatar
  • 203
14 votes
1 answer
7k views

Why are principal component scores uncorrelated?

Supose $\mathbf A$ is a matrix of mean-centred data. The matrix $\mathbf S=\text{cov}(\mathbf A)$ is $m\times m$, has $m$ distinct eigenvalues, and eigenvectors $\mathbf s_1$, $\mathbf s_2$ ... $\...
Ernest A's user avatar
  • 2,312
8 votes
1 answer
373 views

Null distribution of subspaces similarity, or what is the distribution of $\mathrm{tr}(AA'BB')$?

What is the distribution of $\mathrm{tr}(AA'BB')$ where $A$ and $B$ are two random matrices of $d \times k$ size with orthonormal columns? Maybe the expected value is easier to compute? A fallback ...
M. Toya's user avatar
  • 477
8 votes
1 answer
310 views

Why is $B^TB+\lambda\Omega$ positive definite?

In spline regression, it is not uncommon for the basis expansion to create a rank-deficient design matrix $B_{n\times p}$, but it is well-known that penalization of the estimation procedure solves the ...
Sycorax's user avatar
  • 90.4k
6 votes
2 answers
28k views

Linear Transformation of Gaussian Random Variable

I've been trying to prove that if $\mathbf{x}$ is a random variable with multivariable normal distribution $Pr(\mathbf{x}) = Norm_\mathbf{x}[\mathbf{\mu}, \mathbf{\Sigma}]$ and $\mathbf{y}$ is a ...
Removed's user avatar
  • 135
5 votes
2 answers
929 views

What is the Joint Density Function of a Three-Level Mixed-Effects Model?

This is a follow-up question to a question I posted earlier. Obviously, maximum-likelihood estimation of mixed-effects models requires the joint density function. Let us assume a two-level mixed ...
DomB's user avatar
  • 531
2 votes
1 answer
950 views

Maximize log-likelihood of logistic regression

I'm trying to understand the derivation of the equations for the logistic regression. I'm following the cs229 notes: http://cs229.stanford.edu/notes/cs229-notes1.pdf At some point in the derivation, ...
jagg's user avatar
  • 23
1 vote
2 answers
1k views

How can I map data to lower dimension?

I am trying to learn data in higher space into lower space. To have a clue, I'd like to know how to transform the data in the image below into a lower dimension preserving the structure. Hope to hear ...
user122358's user avatar
  • 1,673
0 votes
0 answers
123 views

Show by derivation why both Order and Rank conditions are needed for identification with instrumental variables

Why are both Order and Rank conditions needed for identification with instrumental variables?
Dr. T's user avatar
  • 23
16 votes
2 answers
4k views

Mixed Models: How to derive Henderson's mixed-model equations?

In the context of best linear unbiased predictors (BLUP), Henderson specified the mixed-model equations (see Henderson (1950): Estimation of Genetic Parameters. Annals of Mathematical Statistics, 21, ...
DomB's user avatar
  • 531
12 votes
2 answers
2k views

Incremental Gaussian Process Regression

I want to implement an incremental gaussian process regression using a sliding window over the data points which arrives one by one through a stream. Let $d$ denote the dimensionality of the input ...
bfaskiplar's user avatar
10 votes
1 answer
9k views

Gradient and vector derivatives: row or column vector?

Quite a lot of references (including wikipedia, and http://www.atmos.washington.edu/~dennis/MatrixCalculus.pdf and http://michael.orlitzky.com/articles/the_derivative_of_a_quadratic_form.php) define ...
Simplefish's user avatar
10 votes
4 answers
2k views

Is "random projection" strictly speaking not a projection?

Current implementations of the Random Projection algorithm reduce the dimensionality of data samples by mapping them from $\mathbb R^d$ to $\mathbb R^k$ using a $d\times k$ projection matrix $R$ whose ...
Daniel López's user avatar
9 votes
1 answer
4k views

What are the steps to convert weighted sum of squares to matrix form?

I'm new to converting formulas to matrix form. But this is required for efficient machine learning code. So I want to understand the "right" way, not the cowboy stuff I do. Alright here we go, I'm ...
vega's user avatar
  • 347
9 votes
1 answer
2k views

Relationship between eigenvectors of $\frac{1}{N}XX^\top$ and $\frac{1}{N}X^\top X$ in the context of PCA

In Christopher Bishop's book Pattern Recognition and Machine Learning, the section on PCA contains the following: Given a centred data matrix $\mathbf{X}$ with covariance matrix $N^{-1}\mathbf{X}^T\...
Danny's user avatar
  • 93
8 votes
1 answer
5k views

What exactly should be called "projection matrix" in the context of PCA?

At the end of the PCA algorithm one gets a $D\times d$ matrix $U$ such that $z=U^Tx$ (here $x$ is $D$-dimensional and $z$ is $d$ dimensional with $d\leq D$). In multiple sources on the Web I found ...
user_anon's user avatar
  • 1,047
8 votes
2 answers
11k views

Prove that $(A^{-1} + B^{-1})^{-1}=A(A+B)^{-1}B$

I have this equality $$(A^{-1} + B^{-1})^{-1}=A(A+B)^{-1}B$$ where $A$ and $B$ are square symmetric matrices. I have done many test of R and Matlab that show that this holds, however I do not know ...
Wis's user avatar
  • 2,164