Skip to main content
43 votes

Relationship between SVD and PCA. How to use SVD to perform PCA?

I wrote a Python & Numpy snippet that accompanies @amoeba's answer and I leave it here in case it is useful for someone. The comments are mostly taken from @amoeba's answer. ...
43 votes
Accepted

If I generate a random symmetric matrix, what's the chance it is positive definite?

If your matrices are drawn from standard-normal iid entries, the probability of being positive-definite is approximately $p_N\approx 3^{-N^2/4}$, so for example if $N=5$, the chance is 1/1000, and ...
Alex R.'s user avatar
  • 14k
33 votes

Relationship between SVD and PCA. How to use SVD to perform PCA?

Let me start with PCA. Suppose that you have $n$ data points comprised of $d$ numbers (or dimensions) each. If you center this data (subtract the mean data point $\mu$ from each data vector $x_i$) you ...
Andre P's user avatar
  • 511
27 votes

What is an example of perfect multicollinearity?

Here are a couple of fairly common scenarios producing perfect multicollinearity, i.e. situations in which the columns of the design matrix are linearly dependent. Recall from linear algebra that this ...
Silverfish's user avatar
  • 23.7k
27 votes
Accepted

Is the average of positive-definite matrices also positive-definite?

Yes, it is. jth asnwer is correct (+1) but I think you can get a much simple explanation with just basic Linear Algebra. Assume $A$ and $B$ are positive definite matrices for size $n$. By definition ...
usεr11852's user avatar
  • 44.8k
24 votes
Accepted

Completing a $3 \times 3$ correlation matrix — $2$ coefficients of the $3$ given

We already know $\gamma$ is bounded between $[-1,1]$ The correlation matrix should be positive semidefinite and hence its principal minors should be nonnegative Thus, \begin{align*} 1(1-\gamma^2)-0.6(...
Saket Choudhary's user avatar
21 votes

What is the intuition behind SVD?

Let $A$ be a real $m \times n$ matrix. I'll assume that $m \geq n$ for simplicity. It's natural to ask in which direction $v$ does $A$ have the most impact (or the most explosiveness, or the most ...
littleO's user avatar
  • 610
19 votes
Accepted

What justifies this calculation of the derivative of a matrix function?

There is a subtle but heavy abuse of the notation that renders many of the steps confusing. Let's address this issue by going back to the definitions of matrix multiplication, transposition, traces, ...
whuber's user avatar
  • 327k
18 votes

Why the sudden fascination with tensors?

As someone who studies and builds neural networks and has repeatedly asked this question, I've come to the conclusion that we borrow useful aspects of tensor notation simply because they make ...
18 votes
Accepted

Are 1-dimensional numpy arrays equivalent to vectors?

A NumPy array is a N-dimensional container of items of the same type and size. As a computer programming data structure, it is limited by resources and dtype --- there are values which are not ...
unutbu's user avatar
  • 579
18 votes

A term for "number of columns" of a matrix

There is a concept of wide and narrow data, so maybe you could use the term „width“ for the number of columns after you define it in order to avoid the ambiguity.
17 votes
Accepted

Why can't we cancel these two matrices in the OLS estimator?

Why can't we simply cancel out the $X^T$ I remember asking myself almost exactly the same question 300 years ago upon seeing the regression equation $y=X\beta$ for the first time in my life. The ...
Aksakal's user avatar
  • 61.7k
17 votes

In a convolutional neural network (CNN), when convolving the image, is the operation used the dot product or the sum of element-wise multiplication?

Any given layer in a CNN has typically 3 dimensions (we'll call them height, width, depth). The convolution will produce a new layer with a new (or same) height, width and depth. The operation however ...
Djib2011's user avatar
  • 6,023
16 votes
Accepted

Why the default matrix norm is spectral norm and not Frobenius norm?

In general, I am unsure that the spectral norm is the most widely used. For example the Frobenius norm is used for to approximate solution on non-negative matrix factorisation or correlation/...
usεr11852's user avatar
  • 44.8k
16 votes

A term for "number of columns" of a matrix

Let's review your objectives: You want a short, meaningful term. You want it to be memorable and readable, rather than some clunky abstract mathematical or computerese construction like "let $\mathbb{...
15 votes

Is the average of positive-definite matrices also positive-definite?

Of course. The set of positive definite matrices forms a cone, meaning it is closed under positive linear combinations and scaling.
nth's user avatar
  • 806
14 votes

How to show this matrix is positive semidefinite?

This is a nice opportunity to apply the definitions: no advanced theorems are needed. To simplify the notation, for any number $\rho$ let $$\mathbb{A}(\rho)=\pmatrix{A&\rho B\\\rho B^\prime&D}...
whuber's user avatar
  • 327k
13 votes

Multivariate normal posterior

With the distributions on our random vectors: $\mathbf x_i | \mathbf \mu \sim N(\mu , \mathbf \Sigma)$ $\mathbf \mu \sim N(\mathbf \mu_0, \mathbf \Sigma_0)$ By Bayes's rule the posterior ...
conjectures's user avatar
  • 4,286
13 votes

What is the intuition behind SVD?

Take an hour of your day and watch this lecture. This guy is super straight-forward; It's important not to skip any of it because it all comes together in the end. Even if it might seem a little slow ...
Tim Johnsen's user avatar
13 votes
Accepted

What is an example of perfect multicollinearity?

Here is an example with 3 variables, $y$, $x_1$ and $x_2$, related by the equation $$ y = x_1 + x_2 + \varepsilon $$ where $\varepsilon \sim N(0,1)$ The particular data are ...
Robert Long's user avatar
13 votes

A term for "number of columns" of a matrix

Personally I would denote the matrix as $$X \in \mathbb{R}^{n \times p}$$ and use $p$ as a reference (assuming your matrix is composed of real values!). Also note that the notation p >> n is quite ...
13 votes
Accepted

Why the second term is transposed, but not the first one?

If you use the convention that $(\boldsymbol{x} - \boldsymbol{\mu})$ is a column vector, i.e. $(\boldsymbol{x} - \boldsymbol{\mu}) = \begin{bmatrix} x_{1} - \mu_1\\ x_{2} - \...
BalaGizeh's user avatar
  • 348
12 votes

Transposition of first matrix in crossprod in R

As you indicated, %*% already does multiplication; there's not really a need for second function to do the same job. The function ...
Sycorax's user avatar
  • 92.3k
12 votes
Accepted

In a convolutional neural network (CNN), when convolving the image, is the operation used the dot product or the sum of element-wise multiplication?

I believe the key is that when the filter is convolving some part of the image (the "receptive field") each number in the filter (i.e. each weight) is first flattened into vector format. Likewise, ...
Ryan Chase's user avatar
12 votes
Accepted

Probability that $2\times2$ matrix of random variables is invertible

In the case of a $2 \times 2$ matrix there is a simple formula for the determinant: $$\det \mathbf{X} = \det \begin{bmatrix} X_1 & X_2 \\ X_3 & X_4 \end{bmatrix} = X_1 \cdot X_4 - X_2 \cdot ...
Ben's user avatar
  • 127k
12 votes

What is the problem with $p > n$?

This is a very good question. When the number of candidate predictors $p$ is more than the effective sample size $n$, and one does not place any restrictions on the regression coefficients (e.g., one ...
Frank Harrell's user avatar
12 votes
Accepted

Why doesn't $(e^{A})^{-1} = e^{-A}$ hold for a symmetric matrix in Python?

This is a case of reasoning from a false premise. The matrix exponential is not defined as the exponentiation of each element of a matrix. Instead, the definition of a matrix exponential is $$ \exp(X)...
Sycorax's user avatar
  • 92.3k
12 votes

Why does Hutchinson's trace estimator reduce computation complexity?

You are right that for calculating the trace of a matrix this does not reduce cost vs a simple calculation...but this trick is very useful when we need to compute the trace of a function of a matrix, $...
HappyDog's user avatar
  • 421
12 votes

Frobenius norm of a product of Gaussian matrices

Let $X$ be $d\times d$ a random matrix with iid $\mathcal N(0, 1/d)$ elements. Let us first show that $$\| XX^\top \|^2 \approx 2d.$$ Induction base We want to compute expectations of squared elements ...
amoeba's user avatar
  • 106k
11 votes

Appropriate measure to find smallest covariance matrix

The ordering of matrices you refer to is known as the Loewner order and is a partial order much used in the study of positive definite matrices. A book-length treatment of the geometry on the manifold ...
kjetil b halvorsen's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible