Skip to main content
81 votes
Accepted

What is an intuitive explanation for how PCA turns from a geometric problem (with distances) to a linear algebra problem (with eigenvectors)?

Problem statement The geometric problem that PCA is trying to optimize is clear to me: PCA tries to find the first principal component by minimizing the reconstruction (projection) error, which ...
amoeba's user avatar
  • 105k
51 votes
Accepted

Why is XOR not linearly separable?

Draw a picture. The question asks you to show it is not possible to find a half-plane and its complement that separate the blue points where XOR is zero from the red points where XOR is one (in the ...
whuber's user avatar
  • 326k
33 votes
Accepted

What is the meaning of double bars and 2 at the bottom in ordinary least squares?

You're talking about the $\ell_2$-norm (Euclidean norm) of the vector ($Xw - y$). If this foreign to you, briefly, the $\ell_p$-norm of a vector $u \in \mathbb{R}^{n}$, is: $$\|u\|_p = \big(\sum_{i=1}...
ilanman's user avatar
  • 4,787
32 votes
Accepted

Dot product vs Element-wise multiplication

The difference operationally is the aggregation by summation. With the dot product, you multiply the corresponding components and add those products together. With the Hadamard product (element-wise ...
Galen's user avatar
  • 9,004
28 votes

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

amoeba already gave a good answer in the comments, but if you want a formal argument, here it goes. The singular value decomposition of a matrix $A$ is $A=U\Sigma V^T$, where the columns of $V$ are ...
cangrejo's user avatar
  • 2,251
26 votes
Accepted

How does NumPy solve least squares for underdetermined systems?

My understanding is that numpy.linalg.lstsq relies on the LAPACK routine dgelsd. The problem is to solve: $$ \text{minimize} (\text{over} \; \mathbf{x}) \quad \| A\mathbf{x} - \mathbf{b} \|_2$$ Of ...
Matthew Gunn's user avatar
  • 22.5k
25 votes

Why does inversion of a covariance matrix yield partial correlations between random variables?

Here is a proof with just matrix calculations. I appreciate the answer by whuber. It is very insightful on the math behind the scene. However, it is still not so trivial how to use his answer to ...
Po C.'s user avatar
  • 350
24 votes
Accepted

Why are symmetric positive definite (SPD) matrices so important?

A (real) symmetric matrix has a complete set of orthogonal eigenvectors for which the corresponding eigenvalues are are all real numbers. For non-symmetric matrices this can fail. For example, a ...
Matthew Drury's user avatar
19 votes

What is the intuition behind SVD?

Let $A$ be a real $m \times n$ matrix. I'll assume that $m \geq n$ for simplicity. It's natural to ask in which direction $v$ does $A$ have the most impact (or the most explosiveness, or the most ...
littleO's user avatar
  • 590
18 votes

Why the sudden fascination with tensors?

As someone who studies and builds neural networks and has repeatedly asked this question, I've come to the conclusion that we borrow useful aspects of tensor notation simply because they make ...
18 votes
Accepted

Are 1-dimensional numpy arrays equivalent to vectors?

A NumPy array is a N-dimensional container of items of the same type and size. As a computer programming data structure, it is limited by resources and dtype --- there are values which are not ...
unutbu's user avatar
  • 579
17 votes
Accepted

Intuitive explanation of Minimum Covariance Determinant (MCD)

One way to detect anomalies is to assume that regular (non-anomalous) data are generated by a particular probability distribution, and to declare points with low probability density as anomalies. For ...
user20160's user avatar
  • 32.7k
16 votes
Accepted

Why the default matrix norm is spectral norm and not Frobenius norm?

In general, I am unsure that the spectral norm is the most widely used. For example the Frobenius norm is used for to approximate solution on non-negative matrix factorisation or correlation/...
usεr11852's user avatar
  • 44.5k
16 votes
Accepted

Covariance matrix as linear transformation

This is not true for all (non-zero) vectors, but let's explore. The covariance matrix $A$ has an orthonormal basis $v_1, \ldots, v_n$ of eigenvectors with eigenvalues $\lambda_1 \geq \lambda_2 \geq \...
WimC's user avatar
  • 306
16 votes
Accepted

Is the sum of the diagonal elements of a covariance matrix always equal or larger than the sum of its off-diagonal elements?

Consider the general equi-correlation covariance matrix: \begin{align} \Sigma = \begin{bmatrix} 1 & \rho & \cdots & \rho \\ \rho & 1 & \cdots & \rho \\ \vdots & \vdots &...
Zhanxiong's user avatar
  • 20k
16 votes

Is the expectation of a random vector multiplied by its transpose equal to the product of the expectation of the vector and that of the transpose

To confirm a claim is not true, you don't "prove it". Instead, just provide a counterexample would be sufficient. You are actually on the right track. Any random vector $z$ with positive ...
Zhanxiong's user avatar
  • 20k
15 votes
Accepted

Why cannot I obtain a valid SVD of X via eigenvalue decomposition of XX' and X'X?

Analysis of the Problem The SVD of a matrix is never unique. Let matrix $A$ have dimensions $n\times k$ and let its SVD be $$A = U D V^\prime$$ for an $n\times p$ matrix $U$ with orthonormal ...
whuber's user avatar
  • 326k
15 votes

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

@amoeba had excellent answers to PCA questions, including this one on relation of SVD to PCA. Answering to your exact question I'll make three points: mathematically there is no difference whether ...
Aksakal's user avatar
  • 61.5k
14 votes

How to show this matrix is positive semidefinite?

This is a nice opportunity to apply the definitions: no advanced theorems are needed. To simplify the notation, for any number $\rho$ let $$\mathbb{A}(\rho)=\pmatrix{A&\rho B\\\rho B^\prime&D}...
whuber's user avatar
  • 326k
14 votes
Accepted

Should I gloss over the linear algebra chapter in the book "Deep Learning" by Ian Goodfellow?

This is a question that often pops up when reading mathematical literature. The initial chapters, of this book or any other math book, lay out tools that you will be using in later chapters, so ...
Stephan Kolassa's user avatar
13 votes

What is the intuition behind SVD?

Take an hour of your day and watch this lecture. This guy is super straight-forward; It's important not to skip any of it because it all comes together in the end. Even if it might seem a little slow ...
Tim Johnsen's user avatar
13 votes

Multivariate normal posterior

With the distributions on our random vectors: $\mathbf x_i | \mathbf \mu \sim N(\mu , \mathbf \Sigma)$ $\mathbf \mu \sim N(\mathbf \mu_0, \mathbf \Sigma_0)$ By Bayes's rule the posterior ...
conjectures's user avatar
  • 4,236
12 votes

Why are symmetric positive definite (SPD) matrices so important?

You'll find some intuition in the many elementary ways of showing the eigenvalues of a real symmetric matrix are all real: https://mathoverflow.net/questions/118626/real-symmetric-matrix-has-real-...
Alex R.'s user avatar
  • 14k
12 votes

What is the problem with $p > n$?

This is a very good question. When the number of candidate predictors $p$ is more than the effective sample size $n$, and one does not place any restrictions on the regression coefficients (e.g., one ...
Frank Harrell's user avatar
12 votes

Why does Hutchinson's trace estimator reduce computation complexity?

You are right that for calculating the trace of a matrix this does not reduce cost vs a simple calculation...but this trick is very useful when we need to compute the trace of a function of a matrix, $...
HappyDog's user avatar
  • 421
11 votes

Appropriate measure to find smallest covariance matrix

The ordering of matrices you refer to is known as the Loewner order and is a partial order much used in the study of positive definite matrices. A book-length treatment of the geometry on the manifold ...
kjetil b halvorsen's user avatar
11 votes
Accepted

How does cosine similarity change after a linear transformation?

Because $M$ is quite general, and the change in cosine similarity depends on the particular $A$ and $B$ and their relationship to $M$, no definite formula is possible. However, there are practically ...
whuber's user avatar
  • 326k
11 votes

Why are symmetric positive definite (SPD) matrices so important?

With respect to optimization (because you tagged your question with the optimization tag), SPD matrices are extremely important for one simple reason - an SPD Hessian guarantees that the search ...
Bill Woessner's user avatar
11 votes
Accepted

Obtaining the possible least squares solutions when $X^TX$ is not invertible

You want all the possible solutions? Write the linear model in matrix form as $$ Y=X\beta + \epsilon, $$ and let the Moore-Penrose inverse of matrix $A$ be denoted by $A^+$. The normal equations for ...
kjetil b halvorsen's user avatar
11 votes
Accepted

Mixed Models: How to derive Henderson's mixed-model equations?

One approach is to form the log-likelihood and differentiate this with respect to the random effects $\mathbf{u}$ and set this equal to zero, then repeat, but differentiate with respect to the fixed ...
Robert Long's user avatar
  • 61.9k

Only top scored, non community-wiki answers of a minimum length are eligible