Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
185
votes
10answers
41k views

Why the sudden fascination with tensors?

I've noticed lately that a lot of people are developing tensor equivalents of many methods (tensor factorization, tensor kernels, tensors for topic modeling, etc) I'm wondering, why is the world ...
77
votes
3answers
18k views

What is the intuition behind SVD?

I have read about singular value decomposition (SVD). In almost all textbooks it is mentioned that it factorizes the matrix into three matrices with given specification. But what is the intuition ...
76
votes
6answers
35k views

What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?

I've read a lot about PCA, including various tutorials and questions (such as this one, this one, this one, and this one). The geometric problem that PCA is trying to optimize is clear to me: PCA ...
60
votes
5answers
37k views

Is every covariance matrix positive definite?

I guess the answer should be yes, but I still feel something is not right. There should be some general results in the literature, could anyone help me?
56
votes
9answers
19k views

Reference book for linear algebra applied to statistics?

I have been working in R for a bit and have been faced with things like PCA, SVD, QR decompositions and many such linear algebra results (when inspecting estimating weighted regressions and such) so I ...
44
votes
4answers
22k views

Why does inversion of a covariance matrix yield partial correlations between random variables?

I heard that partial correlations between random variables can be found by inverting the covariance matrix and taking appropriate cells from such resulting precision matrix (this fact is mentioned in ...
44
votes
7answers
24k views

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
38
votes
3answers
15k views

Distribution of scalar products of two random unit vectors in $D$ dimensions

If $\mathbf{x}$ and $\mathbf{y}$ are two independent random unit vectors in $\mathbb{R}^D$ (uniformly distributed on a unit sphere), what is the distribution of their scalar product (dot product) $\...
33
votes
3answers
29k views

Why is a sample covariance matrix singular when sample size is less than number of variables?

Let's say I have a $p$-dimensional multivariate Gaussian distribution. And I take $n$ observations (each of them a $p$-vector) from this distribution and calculate the sample covariance matrix $S$. In ...
27
votes
7answers
19k views

Why are symmetric positive definite (SPD) matrices so important?

I know the definition of symmetric positive definite (SPD) matrix, but want to understand more. Why are they so important, intuitively? Here is what I know. What else? For a given data, Co-...
26
votes
2answers
7k views

Why is the Fisher Information matrix positive semidefinite?

Let $\theta \in R^{n}$. The Fisher Information Matrix is defined as: $$I(\theta)_{i,j} = -E\left[\frac{\partial^{2} \log(f(X|\theta))}{\partial \theta_{i} \partial \theta_{j}}\bigg|\theta\right]$$ ...
26
votes
1answer
23k views

Multivariate normal posterior

This is a very simple question but I can't find the derivation anywhere on the internet or in a book. I would like to see the derivation of how one Bayesian updates a multivariate normal distribution....
22
votes
1answer
19k views

How to whiten the data using principal component analysis?

I want to transform my data $\mathbf X$ such that the variances will be one and the covariances will be zero (i.e I want to whiten the data). Furthermore the means should be zero. I know I will get ...
22
votes
2answers
12k views

Why is the rank of covariance matrix at most $n-1$?

As stated in this question, the maximum rank of covariance matrix is $n-1$ where $n$ is sample size and so if the dimension of covariance matrix is equal to the sample size, it would be singular. I ...
21
votes
3answers
2k views

Weird correlations in the SVD results of random data; do they have a mathematical explanation or is it a LAPACK bug?

I observe a very weird behaviour in the SVD outcome of random data, which I can reproduce in both Matlab and R. It looks like some numerical issue in the LAPACK library; is it? I draw $n=1000$ ...
21
votes
1answer
10k views

Updating SVD decomposition after adding one new row to the matrix

Suppose that I have a dense matrix $ \textbf{A}$ of $m \times n$ size, with SVD decomposition $$\mathbf{A}=\mathbf{USV}^\top.$$ In R I can calculate the SVD as ...
19
votes
1answer
2k views

Geometric understanding of PCA in the subject (dual) space

I am trying to get an intuitive understanding of how principal component analysis (PCA) works in subject (dual) space. Consider 2D dataset with two variables, $x_1$ and $x_2$, and $n$ data points (...
18
votes
1answer
10k views

How does NumPy solve least squares for underdetermined systems?

Let's say that we have X of shape (2, 5) and y of shape (2,) This works: np.linalg.lstsq(X, y) We would expect this to work only if X was of shape (N,5) where N>=...
17
votes
3answers
6k views

Why the default matrix norm is spectral norm and not Frobenius norm?

For vector norm, the L2 norm or "Euclidean distance" is the widely used and intuitive definition. But why "most used" or "default" norm definition for a matrix is spectral norm, but not Frobenius norm ...
15
votes
4answers
3k views

Sufficient and necessary conditions for zero eigenvalue of a correlation matrix

Given $n$ random variable $X_i$, with probability distribution $P(X_1,\ldots,X_n)$, the correlation matrix $C_{ij}=E[X_i X_j]-E[X_i]E[X_j]$ is positive semi-definite, i.e. its eigenvalues are positive ...
14
votes
1answer
22k views

What do the arrows in a PCA biplot mean?

Consider the following PCA biplot: ...
13
votes
1answer
3k views

What is the meaning of double bars and 2 at the bottom in ordinary least squares?

I saw this notation for ordinary least squares here. $$ \min_w \left\| Xw - y \right\|^2_2$$ I have never seen the double bars and the 2 at the bottom. What do these symbols mean? Do they have ...
12
votes
2answers
3k views

Is expectation the same as mean?

I am doing ML at my university, and the professor mentioned the term Expectation (E), while he was trying to explain some things on Gaussian processes to us. But from the way he explained it, I ...
12
votes
2answers
2k views

Mixed Models: How to derive Henderson's mixed-model equations?

In the context of best linear unbiased predictors (BLUP), Henderson specified the mixed-model equations (see Henderson (1950): Estimation of Genetic Parameters. Annals of Mathematical Statistics, 21, ...
12
votes
1answer
4k views

Why are principal component scores uncorrelated?

Supose $\mathbf A$ is a matrix of mean-centred data. The matrix $\mathbf S=\text{cov}(\mathbf A)$ is $m\times m$, has $m$ distinct eigenvalues, and eigenvectors $\mathbf s_1$, $\mathbf s_2$ ... $\...
11
votes
4answers
9k views

Intuitive meaning of vector multiplication with covariance matrix

I often see multiplications with covariance matrices in literature. However I never really understood what is achieved by multiplication with the covariance matrix. Given $\Sigma * r = s$ with $\...
11
votes
2answers
2k views

Incremental Gaussian Process Regression

I want to implement an incremental gaussian process regression using a sliding window over the data points which arrives one by one through a stream. Let $d$ denote the dimensionality of the input ...
10
votes
3answers
376 views

What is this bias-variance tradeoff for regression coefficients and how to derive it?

In this paper, (Bayesian Inference for Variance Components Using Only Error Contrasts, Harville, 1974), the author claims $$(y-X\beta)'H^{-1}(y-X\beta)=(y-X\hat\beta)'H^{-1}(y-X\hat\beta)+(\beta-\hat\...
10
votes
1answer
3k views

invariance of correlation to linear transformation: $\text{corr}(aX+b, cY+d) = \text{corr}(X,Y)$

This is actually one of the problems in Gujarati's Basic Econometrics 4th edition (Q3.11) and says that the correlation coefficient is invariant with respect to the change of origin and scale, that is ...
10
votes
2answers
2k views

Appropriate measure to find smallest covariance matrix

In the textbook I am reading they use positive definiteness (semi-positive definiteness) to compare two covariance matrices. The idea being that if $A-B$ is pd then $B$ is smaller than $A$. But I'm ...
10
votes
1answer
3k views

How to get "eigenvalues" (percentages of explained variance) of vectors that are not PCA eigenvectors?

I would like to understand how I can get the percentage of variance of a data set, not in the coordinate space provided by PCA, but against a slightly different set of (rotated) vectors. ...
9
votes
2answers
24k views

Finding matrix eigenvectors using QR decomposition

First, a general linear algebra question: Can a matrix have more than one set of (unit size) eigenvectors? From a different angle: Is it possible that different decomposition methods/algorithms (QR, ...
9
votes
2answers
883 views

Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

I am trying to do SVD by hand: ...
9
votes
1answer
7k views

Why does the rank of the design matrix X equal the rank of X'X?

Why does the rank of the design matrix $\boldsymbol X$ equal the rank of $\boldsymbol{X'X}$? Is this true in all circumstances? If X is not linearly independent, what would the rank of X'X be?
9
votes
2answers
4k views

How does cosine similarity change after a linear transformation?

Is there a mathematical relationship between: the cosine similarity $\operatorname{sim}(A, B)$ of two vectors $A$ and $B$, and the cosine similarity $\operatorname{sim}(MA, MB)$ of $A$ and $B$, non-...
9
votes
3answers
21k views

Simple linear regression fit manually via matrix equations does not match lm() output

I am trying to fit a linear model using matrices to my data set even though I can use OLS and do it without matrices as a simple tutorial for myself to better understand both ...
9
votes
4answers
1k views

Is "random projection" strictly speaking not a projection?

Current implementations of the Random Projection algorithm reduce the dimensionality of data samples by mapping them from $\mathbb R^d$ to $\mathbb R^k$ using a $d\times k$ projection matrix $R$ whose ...
9
votes
2answers
3k views

What's the intuition for ABA' in linear algebra?

I've seen the pattern ABA', where A and B are matrices, and ' stands for the transpose, many times so I want to know if there is an intuition for this pattern. I did some development to see what the ...
9
votes
1answer
7k views

Gradient and vector derivatives: row or column vector?

Quite a lot of references (including wikipedia, and http://www.atmos.washington.edu/~dennis/MatrixCalculus.pdf and http://michael.orlitzky.com/articles/the_derivative_of_a_quadratic_form.php) define ...
8
votes
2answers
9k views

Prove that $(A^{-1} + B^{-1})^{-1}=A(A+B)^{-1}B$

I have this equality $$(A^{-1} + B^{-1})^{-1}=A(A+B)^{-1}B$$ where $A$ and $B$ are square symmetric matrices. I have done many test of R and Matlab that show that this holds, however I do not know ...
8
votes
2answers
7k views

How to show this matrix is positive semidefinite?

Let $$K=\begin{pmatrix} K_{11} & K_{12}\\ K_{21} & K_{22} \end{pmatrix}$$ be a symmetric positive semidefinite real matrix (PSD) with $K_{12}=K_{21}^T$. Then, for $|r| \le 1$, $$K^*=\...
8
votes
1answer
1k views

Relationship between eigenvectors of $\frac{1}{N}XX^\top$ and $\frac{1}{N}X^\top X$ in the context of PCA

In Christopher Bishop's book Pattern Recognition and Machine Learning, the section on PCA contains the following: Given a centred data matrix $\mathbf{X}$ with covariance matrix $N^{-1}\mathbf{X}^T\...
8
votes
3answers
2k views

Proof that if covariance is zero then there is no linear relationship

I get that a zero covariance doesn´t imply independence, but everybody says that if there is dependence and the covariance is zero then it is a non linear dependence. People base their interpretation ...
8
votes
2answers
1k views

What is the problem with $p > n$?

I know that this is the solving system of linear equation problem. But my question is why it is a problem the number of observation is lower than the number of predictors how can that thing happen? ...
8
votes
1answer
269 views

Null distribution of subspaces similarity, or what is the distribution of $\mathrm{tr}(AA'BB')$?

What is the distribution of $\mathrm{tr}(AA'BB')$ where $A$ and $B$ are two random matrices of $d \times k$ size with orthonormal columns? Maybe the expected value is easier to compute? A fallback ...
8
votes
1answer
6k views

completing the square for Gaussian multivariate estimation

I have been trying to derive the posterior distribution in the case of weighted Bayesian regression in the case of multivariate normal distribution for a few days and have been stuck. I am not sure if ...
8
votes
1answer
3k views

QR factorization and linear regression

I have been reading "Generalized Additive Models an Introduction with R" by Simon Wood and have come across a section I'm having trouble with. On page 13 it is stated that the model or design matrix ...
8
votes
1answer
162 views

Mixed Model Equations

In this paper on page 1924 it is stated that \begin{equation} \text{Var}(u \mid y) = \sigma^2[G - GZ^\top H^{-1}ZG] \end{equation} can be written as \begin{equation} \text{Var}(u \mid y) = \sigma^...
8
votes
1answer
299 views

If an inverse covariance matrix is sparse, what can I say about the covariance matrix?

How does the sparsity condition on an inverse covariance matrix affect the actual covariance matrix?
8
votes
1answer
3k views

What are the steps to convert weighted sum of squares to matrix form?

I'm new to converting formulas to matrix form. But this is required for efficient machine learning code. So I want to understand the "right" way, not the cowboy stuff I do. Alright here we go, I'm ...

1
2 3 4 5
12