# Questions tagged [linear-algebra]

A field of mathematics concerned with the study of finite dimensional vector spaces, including matrices and their manipulation, which are important in statistics.

102 questions
Filter by
Sorted by
Tagged with
21k views

• 21
27k views

### Why are symmetric positive definite (SPD) matrices so important?

I know the definition of symmetric positive definite (SPD) matrix, but want to understand more. Why are they so important, intuitively? Here is what I know. What else? For a given data, Co-...
• 37.2k
32k views

### Multivariate normal posterior

This is a very simple question but I can't find the derivation anywhere on the internet or in a book. I would like to see the derivation of how one Bayesian updates a multivariate normal distribution....
• 853
10k views

### Why is the Fisher Information matrix positive semidefinite?

Let $\theta \in R^{n}$. The Fisher Information Matrix is defined as: $$I(\theta)_{i,j} = -E\left[\frac{\partial^{2} \log(f(X|\theta))}{\partial \theta_{i} \partial \theta_{j}}\bigg|\theta\right]$$ ...
• 383
3k views

### Geometric understanding of PCA in the subject (dual) space

I am trying to get an intuitive understanding of how principal component analysis (PCA) works in subject (dual) space. Consider 2D dataset with two variables, $x_1$ and $x_2$, and $n$ data points (...
• 106k
1k views

### Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

I am trying to do SVD by hand: ...
3k views

### Appropriate measure to find smallest covariance matrix

In the textbook I am reading they use positive definiteness (semi-positive definiteness) to compare two covariance matrices. The idea being that if $A-B$ is pd then $B$ is smaller than $A$. But I'm ...
• 1,763
2k views

• 6,635
3k views

### Incremental Gaussian Process Regression

I want to implement an incremental gaussian process regression using a sliding window over the data points which arrives one by one through a stream. Let $d$ denote the dimensionality of the input ...
• 632
2k views

### Is "random projection" strictly speaking not a projection?

Current implementations of the Random Projection algorithm reduce the dimensionality of data samples by mapping them from $\mathbb R^d$ to $\mathbb R^k$ using a $d\times k$ projection matrix $R$ whose ...
• 5,686
9k views

### Gradient and vector derivatives: row or column vector?

Quite a lot of references (including wikipedia, and http://www.atmos.washington.edu/~dennis/MatrixCalculus.pdf and http://michael.orlitzky.com/articles/the_derivative_of_a_quadratic_form.php) define ...
• 203
4k views

### What are the steps to convert weighted sum of squares to matrix form?

I'm new to converting formulas to matrix form. But this is required for efficient machine learning code. So I want to understand the "right" way, not the cowboy stuff I do. Alright here we go, I'm ...
• 347
362 views

### Sums of exponentials joint probability

If we have that: $\tau_i \overset{\text{independent}}{\sim} \exp(\lambda_i)$, for $i=1,2,3,...,n$, where $\lambda_i\neq \lambda_j, \forall i\neq j$ then I would like to find a general form for the ...
• 930
### Relationship between eigenvectors of $\frac{1}{N}XX^\top$ and $\frac{1}{N}X^\top X$ in the context of PCA
In Christopher Bishop's book Pattern Recognition and Machine Learning, the section on PCA contains the following: Given a centred data matrix $\mathbf{X}$ with covariance matrix \$N^{-1}\mathbf{X}^T\...