Questions tagged [linear-algebra]
A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.
102
questions
42
votes
3
answers
21k
views
Distribution of scalar products of two random unit vectors in $D$ dimensions
If $\mathbf{x}$ and $\mathbf{y}$ are two independent random unit vectors in $\mathbb{R}^D$ (uniformly distributed on a unit sphere), what is the distribution of their scalar product (dot product) $\...
- 106k
94
votes
3
answers
27k
views
What is the intuition behind SVD?
I have read about singular value decomposition (SVD). In almost all textbooks it is mentioned that it factorizes the matrix into three matrices with given specification.
But what is the intuition ...
- 1,349
194
votes
10
answers
48k
views
Why the sudden fascination with tensors?
I've noticed lately that a lot of people are developing tensor equivalents of many methods (tensor factorization, tensor kernels, tensors for topic modeling, etc) I'm wondering, why is the world ...
37
votes
4
answers
36k
views
Why is a sample covariance matrix singular when sample size is less than number of variables?
Let's say I have a $p$-dimensional multivariate Gaussian distribution. And I take $n$ observations (each of them a $p$-vector) from this distribution and calculate the sample covariance matrix $S$. In ...
- 6,817
85
votes
6
answers
46k
views
What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?
I've read a lot about PCA, including various tutorials and questions (such as this one, this one, this one, and this one).
The geometric problem that PCA is trying to optimize is clear to me: PCA ...
- 3,610
66
votes
5
answers
46k
views
Is every covariance matrix positive definite?
I guess the answer should be yes, but I still feel something is not right. There should be some general results in the literature, could anyone help me?
- 1,303
8
votes
1
answer
9k
views
completing the square for Gaussian multivariate estimation
I have been trying to derive the posterior distribution in the case of weighted Bayesian regression in the case of multivariate normal distribution for a few days and have been stuck. I am not sure if ...
- 4,700
5
votes
1
answer
3k
views
Understanding linear algebra in Ordinary Least Squares derivation
In ordinary least squared there is this equation (Kevin Murphy book page 221, latest edition)
$$NLL(w)=\frac{1}{2}({y-Xw})^T(y-Xw)=\frac{1}{2}w^T(X^TX)w-w^T(X^T)y$$
I am not sure how the RHS equals ...
- 245
63
votes
12
answers
22k
views
Reference book for linear algebra applied to statistics?
I have been working in R for a bit and have been faced with things like PCA, SVD, QR decompositions and many such linear algebra results (when inspecting estimating weighted regressions and such) so I ...
53
votes
4
answers
31k
views
Why does inversion of a covariance matrix yield partial correlations between random variables?
I heard that partial correlations between random variables can be found by inverting the covariance matrix and taking appropriate cells from such resulting precision matrix (this fact is mentioned in ...
- 1,288
21
votes
3
answers
3k
views
Weird correlations in the SVD results of random data; do they have a mathematical explanation or is it a LAPACK bug?
I observe a very weird behaviour in the SVD outcome of random data, which I can reproduce in both Matlab and R. It looks like some numerical issue in the LAPACK library; is it?
I draw $n=1000$ ...
- 106k
26
votes
1
answer
24k
views
How to whiten the data using principal component analysis?
I want to transform my data $\mathbf X$ such that the variances will be one and the covariances will be zero (i.e I want to whiten the data). Furthermore the means should be zero.
I know I will get ...
- 1,631
10
votes
4
answers
1k
views
Intuition for nonmonotonicity of coefficient paths in ridge regression
Intuitively, why may some of the slope coefficients in ridge regression increase in magnitude when the penalty parameter $\lambda$ is increased? Or in other words, why are the coefficient paths ...
- 68.8k
25
votes
3
answers
18k
views
Why is the rank of covariance matrix at most $n-1$?
As stated in this question, the maximum rank of covariance matrix is $n-1$ where $n$ is sample size and so if the dimension of covariance matrix is equal to the sample size, it would be singular. I ...
- 406
9
votes
3
answers
5k
views
Is every correlation matrix positive semi-definite?
I am generating correlation matrix by some new algorithm. Generated matrix is non positive semi-definite matrix.
I'm getting a few negative eigenvalues. The rest of eigenvalues are quite equal to the ...
- 91
7
votes
1
answer
3k
views
When to use dot-product as a similarity metric
I'm trying to understand which similarity measure should be used in which situations.
Roughly speaking, when should someone use the dot product to assess similarity between vectors?
Roughly speaking, ...
- 728
1
vote
1
answer
1k
views
Conditional multivariate Gaussian distribution
I am trying to understand how 'completing the square' method is used to determine mean and co-variance of multivariate Gaussian distribution but I am not able to make much sense of the following step
$...
- 21
36
votes
7
answers
27k
views
Why are symmetric positive definite (SPD) matrices so important?
I know the definition of symmetric positive definite (SPD) matrix, but want to understand more.
Why are they so important, intuitively?
Here is what I know. What else?
For a given data, Co-...
- 37.2k
29
votes
1
answer
32k
views
Multivariate normal posterior
This is a very simple question but I can't find the derivation anywhere on the internet or in a book. I would like to see the derivation of how one Bayesian updates a multivariate normal distribution....
- 853
28
votes
2
answers
10k
views
Why is the Fisher Information matrix positive semidefinite?
Let $\theta \in R^{n}$. The Fisher Information Matrix is defined as:
$$I(\theta)_{i,j} = -E\left[\frac{\partial^{2} \log(f(X|\theta))}{\partial \theta_{i} \partial \theta_{j}}\bigg|\theta\right]$$
...
- 383
20
votes
1
answer
3k
views
Geometric understanding of PCA in the subject (dual) space
I am trying to get an intuitive understanding of how principal component analysis (PCA) works in subject (dual) space.
Consider 2D dataset with two variables, $x_1$ and $x_2$, and $n$ data points (...
- 106k
11
votes
2
answers
1k
views
Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?
I am trying to do SVD by hand:
...
10
votes
2
answers
3k
views
Appropriate measure to find smallest covariance matrix
In the textbook I am reading they use positive definiteness (semi-positive definiteness) to compare two covariance matrices. The idea being that if $A-B$ is pd then $B$ is smaller than $A$. But I'm ...
- 1,763
8
votes
2
answers
2k
views
Prove that $\mathrm{Cov}(x^TAx,x^TBx) = 2 \mathrm{Tr}(A \Sigma B \Sigma) + 4 \mu^TA \Sigma B \mu$
Suppose $\vec x \sim N(\vec \mu, \Sigma)$ is multivariate normal. I want to see that
$$\mathrm{Cov}(\vec x^TA\vec x,\vec x^TB\vec x) = 2 \mathrm{Tr}(A \Sigma B \Sigma) + 4 \vec \mu^TA \Sigma B \vec \...
- 821
26
votes
1
answer
13k
views
Updating SVD decomposition after adding one new row to the matrix
Suppose that I have a dense matrix $ \textbf{A}$ of $m \times n$ size, with SVD decomposition $$\mathbf{A}=\mathbf{USV}^\top.$$ In R I can calculate the SVD as ...
- 573
18
votes
1
answer
14k
views
How does NumPy solve least squares for underdetermined systems?
Let's say that we have X of shape (2, 5)
and y of shape (2,)
This works: np.linalg.lstsq(X, y)
We would expect this to work only if X was of shape (N,5) where N>=...
10
votes
1
answer
9k
views
Why does the rank of the design matrix X equal the rank of X'X?
Why does the rank of the design matrix $\boldsymbol X$ equal the rank of $\boldsymbol{X'X}$? Is this true in all circumstances?
If X is not linearly independent, what would the rank of X'X be?
- 123
8
votes
2
answers
4k
views
Is PCA invariant to orthogonal transformations?
Let $A$ a be an $n$ x $p$ matrix and let $B$ be the transformed data set of $A$ under $Q$:
$$ B = A Q $$
where Q is a $p$ x $p$ orthogonal matrix:
$$ Q Q^T = I $$
$n$ is the number of samples (...
- 1,621
7
votes
1
answer
260
views
Interpretation of $\mathbf{y}^T(\mathbf{I}-\mathbf{H})\mathbf{y}$ in OLS
In classic OLS regression it is well-known that $(\mathbf{I}-\mathbf{H})\mathbf{y}=\mathbf{r}$, where $\mathbf{I}$ is the identity matrix, $\mathbf{H}$ is the hat matrix, $\mathbf{y}$ is the vector of ...
- 1,676
6
votes
1
answer
889
views
Deriving the maximum likelihood for the parameters in linear regression
Notation:
$\textbf{w}$ is an M-dimensional vector of parameters (including the bias parameter), $\textbf{x}_n$ is an M-dimensional vector of the features of each training example, $\textbf{t}$ is an N-...
- 477
2
votes
1
answer
433
views
Why is it necessary to "ignore" a level when applying sum contrasts?
I am confused about how sum contrasts are set up. As I understand, if I have some $K$-leveled factor, I can use sum contrasts to compare each level to the grand mean ($M_G$), effectively testing ...
- 43
53
votes
7
answers
33k
views
Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?
I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
- 673
16
votes
4
answers
4k
views
Sufficient and necessary conditions for zero eigenvalue of a correlation matrix
Given $n$ random variable $X_i$, with probability distribution $P(X_1,\ldots,X_n)$, the correlation matrix $C_{ij}=E[X_i X_j]-E[X_i]E[X_j]$ is positive semi-definite, i.e. its eigenvalues are positive ...
- 203
14
votes
1
answer
7k
views
Why are principal component scores uncorrelated?
Supose $\mathbf A$ is a matrix of mean-centred data. The matrix $\mathbf S=\text{cov}(\mathbf A)$ is $m\times m$, has $m$ distinct eigenvalues, and eigenvectors $\mathbf s_1$, $\mathbf s_2$ ... $\...
- 2,372
8
votes
1
answer
388
views
Null distribution of subspaces similarity, or what is the distribution of $\mathrm{tr}(AA'BB')$?
What is the distribution of $\mathrm{tr}(AA'BB')$ where $A$ and $B$ are two random matrices of $d \times k$ size with orthonormal columns?
Maybe the expected value is easier to compute? A fallback ...
- 477
8
votes
1
answer
318
views
Why is $B^TB+\lambda\Omega$ positive definite?
In spline regression, it is not uncommon for the basis expansion to create a rank-deficient design matrix $B_{n\times p}$, but it is well-known that penalization of the estimation procedure solves the ...
- 92.6k
7
votes
2
answers
30k
views
Linear Transformation of Gaussian Random Variable
I've been trying to prove that if $\mathbf{x}$ is a random variable with multivariable normal distribution $Pr(\mathbf{x}) = Norm_\mathbf{x}[\mathbf{\mu}, \mathbf{\Sigma}]$ and $\mathbf{y}$ is a ...
- 155
5
votes
2
answers
996
views
What is the Joint Density Function of a Three-Level Mixed-Effects Model?
This is a follow-up question to a question I posted earlier. Obviously, maximum-likelihood estimation of mixed-effects models requires the joint density function.
Let us assume a two-level mixed ...
- 541
2
votes
1
answer
953
views
Maximize log-likelihood of logistic regression
I'm trying to understand the derivation of the equations for the logistic regression. I'm following the cs229 notes:
http://cs229.stanford.edu/notes/cs229-notes1.pdf
At some point in the derivation, ...
- 23
1
vote
1
answer
4k
views
Why does logistic regression's likelihood function have no closed form?
The derivative of Log likelihood function of logistic regression with respect to theta is
Why can't we equate it to zero and solve for theta so that we can obtain a 'closed form solution' for theta?
...
- 333
1
vote
2
answers
1k
views
How can I map data to lower dimension?
I am trying to learn data in higher space into lower space. To have a clue, I'd like to know how to transform the data in the image below into a lower dimension preserving the structure. Hope to hear ...
- 1,723
17
votes
2
answers
4k
views
Mixed Models: How to derive Henderson's mixed-model equations?
In the context of best linear unbiased predictors (BLUP), Henderson specified the mixed-model equations (see Henderson (1950): Estimation of Genetic Parameters. Annals of Mathematical Statistics, 21, ...
- 541
17
votes
2
answers
17k
views
Dot product vs Element-wise multiplication
What is the different between the dot product "$\cdot$" and the element-wise multiplication notation $\odot$ in Statistics? I referred to Hamilton's Time-Series Analysis, and these seem to ...
- 1,216
13
votes
2
answers
15k
views
Why take transpose of regressor variable in linear regression
I am stuck trying to understand the basic calculation of ordinary least squares. From Wikipedia:
$$y = \beta X^T + \varepsilon$$
where $X$ is the independent variable, $Y$ is the dependent variable ...
- 6,635
12
votes
2
answers
3k
views
Incremental Gaussian Process Regression
I want to implement an incremental gaussian process regression using a sliding window over the data points which arrives one by one through a stream.
Let $d$ denote the dimensionality of the input ...
- 632
10
votes
4
answers
2k
views
Is "random projection" strictly speaking not a projection?
Current implementations of the Random Projection algorithm reduce the dimensionality of data samples by mapping them from $\mathbb R^d$ to $\mathbb R^k$ using a $d\times k$ projection matrix $R$ whose ...
- 5,686
10
votes
1
answer
9k
views
Gradient and vector derivatives: row or column vector?
Quite a lot of references (including wikipedia, and http://www.atmos.washington.edu/~dennis/MatrixCalculus.pdf and http://michael.orlitzky.com/articles/the_derivative_of_a_quadratic_form.php) define ...
- 203
9
votes
1
answer
4k
views
What are the steps to convert weighted sum of squares to matrix form?
I'm new to converting formulas to matrix form. But this is required for efficient machine learning code. So I want to understand the "right" way, not the cowboy stuff I do.
Alright here we go, I'm ...
- 347
9
votes
2
answers
362
views
Sums of exponentials joint probability
If we have that: $\tau_i \overset{\text{independent}}{\sim}
\exp(\lambda_i)$, for $i=1,2,3,...,n$, where $\lambda_i\neq \lambda_j, \forall i\neq j$ then I would like to find a general form for the ...
- 930
9
votes
1
answer
2k
views
Relationship between eigenvectors of $\frac{1}{N}XX^\top$ and $\frac{1}{N}X^\top X$ in the context of PCA
In Christopher Bishop's book Pattern Recognition and Machine Learning, the section on PCA contains the following:
Given a centred data matrix $\mathbf{X}$ with covariance matrix $N^{-1}\mathbf{X}^T\...
- 93