Skip to main content

Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

Filter by
Sorted by
Tagged with
194 votes
10 answers
48k views

Why the sudden fascination with tensors?

I've noticed lately that a lot of people are developing tensor equivalents of many methods (tensor factorization, tensor kernels, tensors for topic modeling, etc) I'm wondering, why is the world ...
94 votes
3 answers
26k views

What is the intuition behind SVD?

I have read about singular value decomposition (SVD). In almost all textbooks it is mentioned that it factorizes the matrix into three matrices with given specification. But what is the intuition ...
SHASHANK GUPTA's user avatar
84 votes
6 answers
45k views

What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?

I've read a lot about PCA, including various tutorials and questions (such as this one, this one, this one, and this one). The geometric problem that PCA is trying to optimize is clear to me: PCA ...
stackoverflowuser2010's user avatar
65 votes
5 answers
45k views

Is every covariance matrix positive definite?

I guess the answer should be yes, but I still feel something is not right. There should be some general results in the literature, could anyone help me?
Jingjings's user avatar
  • 1,293
62 votes
12 answers
22k views

Reference book for linear algebra applied to statistics?

I have been working in R for a bit and have been faced with things like PCA, SVD, QR decompositions and many such linear algebra results (when inspecting estimating weighted regressions and such) so I ...
53 votes
4 answers
31k views

Why does inversion of a covariance matrix yield partial correlations between random variables?

I heard that partial correlations between random variables can be found by inverting the covariance matrix and taking appropriate cells from such resulting precision matrix (this fact is mentioned in ...
michal's user avatar
  • 1,288
52 votes
7 answers
33k views

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
DongukJu's user avatar
  • 663
42 votes
3 answers
21k views

Distribution of scalar products of two random unit vectors in $D$ dimensions

If $\mathbf{x}$ and $\mathbf{y}$ are two independent random unit vectors in $\mathbb{R}^D$ (uniformly distributed on a unit sphere), what is the distribution of their scalar product (dot product) $\...
amoeba's user avatar
  • 106k
37 votes
4 answers
36k views

Why is a sample covariance matrix singular when sample size is less than number of variables?

Let's say I have a $p$-dimensional multivariate Gaussian distribution. And I take $n$ observations (each of them a $p$-vector) from this distribution and calculate the sample covariance matrix $S$. In ...
user34790's user avatar
  • 6,837
36 votes
7 answers
27k views

Why are symmetric positive definite (SPD) matrices so important?

I know the definition of symmetric positive definite (SPD) matrix, but want to understand more. Why are they so important, intuitively? Here is what I know. What else? For a given data, Co-...
Haitao Du's user avatar
  • 37.2k
29 votes
1 answer
32k views

Multivariate normal posterior

This is a very simple question but I can't find the derivation anywhere on the internet or in a book. I would like to see the derivation of how one Bayesian updates a multivariate normal distribution....
Alex's user avatar
  • 853
28 votes
2 answers
10k views

Why is the Fisher Information matrix positive semidefinite?

Let $\theta \in R^{n}$. The Fisher Information Matrix is defined as: $$I(\theta)_{i,j} = -E\left[\frac{\partial^{2} \log(f(X|\theta))}{\partial \theta_{i} \partial \theta_{j}}\bigg|\theta\right]$$ ...
madprob's user avatar
  • 383
26 votes
1 answer
24k views

How to whiten the data using principal component analysis?

I want to transform my data $\mathbf X$ such that the variances will be one and the covariances will be zero (i.e I want to whiten the data). Furthermore the means should be zero. I know I will get ...
Angelorf's user avatar
  • 1,631
26 votes
1 answer
13k views

Updating SVD decomposition after adding one new row to the matrix

Suppose that I have a dense matrix $ \textbf{A}$ of $m \times n$ size, with SVD decomposition $$\mathbf{A}=\mathbf{USV}^\top.$$ In R I can calculate the SVD as ...
user1436187's user avatar
25 votes
3 answers
18k views

Why is the rank of covariance matrix at most $n-1$?

As stated in this question, the maximum rank of covariance matrix is $n-1$ where $n$ is sample size and so if the dimension of covariance matrix is equal to the sample size, it would be singular. I ...
user3070752's user avatar
21 votes
2 answers
6k views

Why is XOR not linearly separable?

Let the function $XOR:\{0,1\} \times \{0,1\} \to \{0,1\}$ be the function defined by $$\begin{align} XOR(0,0) &= 0, \\[6pt] XOR(0,1) &= 1, \\[6pt] XOR(1,0) &= 1, \\[6pt] XOR(1,1) &= 0. ...
lap lapan's user avatar
  • 313
21 votes
3 answers
2k views

Weird correlations in the SVD results of random data; do they have a mathematical explanation or is it a LAPACK bug?

I observe a very weird behaviour in the SVD outcome of random data, which I can reproduce in both Matlab and R. It looks like some numerical issue in the LAPACK library; is it? I draw $n=1000$ ...
amoeba's user avatar
  • 106k
20 votes
1 answer
3k views

Geometric understanding of PCA in the subject (dual) space

I am trying to get an intuitive understanding of how principal component analysis (PCA) works in subject (dual) space. Consider 2D dataset with two variables, $x_1$ and $x_2$, and $n$ data points (...
amoeba's user avatar
  • 106k
19 votes
3 answers
8k views

Why the default matrix norm is spectral norm and not Frobenius norm?

For vector norm, the L2 norm or "Euclidean distance" is the widely used and intuitive definition. But why "most used" or "default" norm definition for a matrix is spectral norm, but not Frobenius norm ...
Haitao Du's user avatar
  • 37.2k
18 votes
1 answer
14k views

How does NumPy solve least squares for underdetermined systems?

Let's say that we have X of shape (2, 5) and y of shape (2,) This works: np.linalg.lstsq(X, y) We would expect this to work only if X was of shape (N,5) where N>=...
George Pligoropoulos's user avatar
18 votes
4 answers
15k views

Intuitive meaning of vector multiplication with covariance matrix

I often see multiplications with covariance matrices in literature. However I never really understood what is achieved by multiplication with the covariance matrix. Given $\Sigma * r = s$ with $\...
tierriminator's user avatar
17 votes
2 answers
16k views

Dot product vs Element-wise multiplication

What is the different between the dot product "$\cdot$" and the element-wise multiplication notation $\odot$ in Statistics? I referred to Hamilton's Time-Series Analysis, and these seem to ...
Carl's user avatar
  • 1,226
17 votes
2 answers
4k views

Mixed Models: How to derive Henderson's mixed-model equations?

In the context of best linear unbiased predictors (BLUP), Henderson specified the mixed-model equations (see Henderson (1950): Estimation of Genetic Parameters. Annals of Mathematical Statistics, 21, ...
DomB's user avatar
  • 541
16 votes
4 answers
4k views

Sufficient and necessary conditions for zero eigenvalue of a correlation matrix

Given $n$ random variable $X_i$, with probability distribution $P(X_1,\ldots,X_n)$, the correlation matrix $C_{ij}=E[X_i X_j]-E[X_i]E[X_j]$ is positive semi-definite, i.e. its eigenvalues are positive ...
Adam's user avatar
  • 203
14 votes
1 answer
7k views

Why are principal component scores uncorrelated?

Supose $\mathbf A$ is a matrix of mean-centred data. The matrix $\mathbf S=\text{cov}(\mathbf A)$ is $m\times m$, has $m$ distinct eigenvalues, and eigenvectors $\mathbf s_1$, $\mathbf s_2$ ... $\...
Ernest A's user avatar
  • 2,372
14 votes
1 answer
26k views

What do the arrows in a PCA biplot mean?

Consider the following PCA biplot: ...
Luna's user avatar
  • 2,355
13 votes
1 answer
4k views

What is the meaning of double bars and 2 at the bottom in ordinary least squares?

I saw this notation for ordinary least squares here. $$ \min_w \left\| Xw - y \right\|^2_2$$ I have never seen the double bars and the 2 at the bottom. What do these symbols mean? Do they have ...
Aseem Bansal's user avatar
13 votes
2 answers
15k views

Why take transpose of regressor variable in linear regression

I am stuck trying to understand the basic calculation of ordinary least squares. From Wikipedia: $$y = \beta X^T + \varepsilon$$ where $X$ is the independent variable, $Y$ is the dependent variable ...
Victor's user avatar
  • 6,605
13 votes
2 answers
4k views

Is expectation the same as mean?

I am doing ML at my university, and the professor mentioned the term Expectation (E), while he was trying to explain some things on Gaussian processes to us. But from the way he explained it, I ...
Jim Blum's user avatar
  • 634
12 votes
2 answers
32k views

Finding matrix eigenvectors using QR decomposition

First, a general linear algebra question: Can a matrix have more than one set of (unit size) eigenvectors? From a different angle: Is it possible that different decomposition methods/algorithms (QR, ...
Bliss's user avatar
  • 453
12 votes
2 answers
6k views

How does cosine similarity change after a linear transformation?

Is there a mathematical relationship between: the cosine similarity $\operatorname{sim}(A, B)$ of two vectors $A$ and $B$, and the cosine similarity $\operatorname{sim}(MA, MB)$ of $A$ and $B$, non-...
turdus-merula's user avatar
12 votes
3 answers
26k views

Simple linear regression fit manually via matrix equations does not match lm() output

I am trying to fit a linear model using matrices to my data set even though I can use OLS and do it without matrices as a simple tutorial for myself to better understand both ...
nicefella's user avatar
  • 1,323
12 votes
2 answers
3k views

Incremental Gaussian Process Regression

I want to implement an incremental gaussian process regression using a sliding window over the data points which arrives one by one through a stream. Let $d$ denote the dimensionality of the input ...
bfaskiplar's user avatar
11 votes
2 answers
1k views

Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

I am trying to do SVD by hand: ...
failedstatistician's user avatar
11 votes
2 answers
6k views

invariance of correlation to linear transformation: $\text{corr}(aX+b, cY+d) = \text{corr}(X,Y)$

This is actually one of the problems in Gujarati's Basic Econometrics 4th edition (Q3.11) and says that the correlation coefficient is invariant with respect to the change of origin and scale, that is ...
Daniel's user avatar
  • 213
11 votes
1 answer
4k views

How to get "eigenvalues" (percentages of explained variance) of vectors that are not PCA eigenvectors?

I would like to understand how I can get the percentage of variance of a data set, not in the coordinate space provided by PCA, but against a slightly different set of (rotated) vectors. ...
Thomas Browne's user avatar
10 votes
1 answer
9k views

Why does the rank of the design matrix X equal the rank of X'X?

Why does the rank of the design matrix $\boldsymbol X$ equal the rank of $\boldsymbol{X'X}$? Is this true in all circumstances? If X is not linearly independent, what would the rank of X'X be?
kurt's user avatar
  • 123
10 votes
3 answers
466 views

What is this bias-variance tradeoff for regression coefficients and how to derive it?

In this paper, (Bayesian Inference for Variance Components Using Only Error Contrasts, Harville, 1974), the author claims $$(y-X\beta)'H^{-1}(y-X\beta)=(y-X\hat\beta)'H^{-1}(y-X\hat\beta)+(\beta-\hat\...
Sibbs Gambling's user avatar
10 votes
3 answers
4k views

Why does Hutchinson's trace estimator reduce computation complexity?

Given a matrix $A$, we want to compute its trace, in which we can use a trick name Hutchinson's trace estimator \begin{align} tr(A) = tr(A\mathbb{E}[\epsilon \epsilon^T])=\mathbb{E}[tr(A \epsilon \...
jzin's user avatar
  • 337
10 votes
2 answers
3k views

Appropriate measure to find smallest covariance matrix

In the textbook I am reading they use positive definiteness (semi-positive definiteness) to compare two covariance matrices. The idea being that if $A-B$ is pd then $B$ is smaller than $A$. But I'm ...
Baz's user avatar
  • 1,763
10 votes
4 answers
1k views

Intuition for nonmonotonicity of coefficient paths in ridge regression

Intuitively, why may some of the slope coefficients in ridge regression increase in magnitude when the penalty parameter $\lambda$ is increased? Or in other words, why are the coefficient paths ...
Richard Hardy's user avatar
10 votes
4 answers
2k views

Is "random projection" strictly speaking not a projection?

Current implementations of the Random Projection algorithm reduce the dimensionality of data samples by mapping them from $\mathbb R^d$ to $\mathbb R^k$ using a $d\times k$ projection matrix $R$ whose ...
Daniel López's user avatar
10 votes
2 answers
282 views

Does the conditional expectation operator have an interpretable decomposition like the projection matrix does in linear algebra?

I'm trying to draw a parallel between the concept of projections in a finite linear space to an infinite linear space. Here is the set-up, first in the finite dimensional case, and then second in the ...
absolutelyzeroEQ's user avatar
10 votes
1 answer
9k views

Gradient and vector derivatives: row or column vector?

Quite a lot of references (including wikipedia, and http://www.atmos.washington.edu/~dennis/MatrixCalculus.pdf and http://michael.orlitzky.com/articles/the_derivative_of_a_quadratic_form.php) define ...
Simplefish's user avatar
9 votes
1 answer
2k views

Relationship between eigenvectors of $\frac{1}{N}XX^\top$ and $\frac{1}{N}X^\top X$ in the context of PCA

In Christopher Bishop's book Pattern Recognition and Machine Learning, the section on PCA contains the following: Given a centred data matrix $\mathbf{X}$ with covariance matrix $N^{-1}\mathbf{X}^T\...
Danny's user avatar
  • 93
9 votes
3 answers
5k views

Is every correlation matrix positive semi-definite?

I am generating correlation matrix by some new algorithm. Generated matrix is non positive semi-definite matrix. I'm getting a few negative eigenvalues. The rest of eigenvalues are quite equal to the ...
Vanita's user avatar
  • 91
9 votes
1 answer
31k views

Are 1-dimensional numpy arrays equivalent to vectors? [closed]

I'm new to both linear algebra and numpy, so please bear with me. I'm taking a course on linear regression, where I learned that we can express our hypothesis as $\theta^TX$ where $\theta$ is our ...
user avatar
9 votes
1 answer
3k views

Covariance matrix as linear transformation

Why applying the covariance matrix to the original vector will turn the vector to the direction of largest variance? How do I intuitively understand this? And what does the inverse of the covariance ...
user34829's user avatar
  • 330
9 votes
2 answers
2k views

What is the problem with $p > n$?

I know that this is the solving system of linear equation problem. But my question is why it is a problem the number of observation is lower than the number of predictors how can that thing happen? ...
EconBoy's user avatar
  • 147
9 votes
2 answers
5k views

What's the intuition for ABA' in linear algebra?

I've seen the pattern ABA', where A and B are matrices, and ' stands for the transpose, many times so I want to know if there is an intuition for this pattern. I did some development to see what the ...
Lay González's user avatar

1
2 3 4 5
14