Skip to main content
35 votes
Accepted

Relationship between Cholesky decomposition and matrix inversion?

Gaussian process models often involve computing some quadratic form, such as $$ y = x^\top\Sigma^{-1}x $$ where $\Sigma$ is positive definite, $x$ is a vector of appropriate dimension, and we wish to ...
Sycorax's user avatar
  • 92.3k
18 votes
Accepted

Deriving Multiplicative Update Rules for NMF

$$\min_{W \in \mathbb{R}^{n \times k},H \in \mathbb{R}^{k \times m}} \left \| V- WH \right \|^{2}_F \text{ s.t. }W,H \geq 0 $$ $$\;\;\;\;\;\;Tr((V-WH)^T(V-WH)) \;\;\;\;\;\; \scriptsize \left [ \...
Satwik Bhattamishra's user avatar
15 votes
Accepted

Why is non-negativity important for collaborative filtering/recommender systems?

I am not a specialist in recommender systems, but as far I understand, the premise of this question is wrong. Non-negativity is not that important for collaborative filtering. The Netflix prize was ...
amoeba's user avatar
  • 106k
12 votes
Accepted

Matrix factorization in recommender systems: adding a new user

Since your training matrix decomposition with gradient descent, I assume you have some loss function $L(X - PQ)$ where $L$ is squared Frobenius norm or something similar. When you add a new user (let'...
Jakub Bartczuk's user avatar
12 votes
Accepted

Why does the resulting matrix from Cholesky decomposition of a covariance matrix when multiplied by its transpose not give back the covariance matrix?

As explained in my comment, the inconvenient truth is that the Cholesky decomposition while usually defined as $K=LL^T$ where $L$ is lower triangular, is equally valid as $K=U^TU$ where $U$ is upper ...
usεr11852's user avatar
  • 44.8k
10 votes
Accepted

Why do we need the regularization term for NMF but not for SVD?

NMF does not always include regularization -- for example, see the first two cost functions here. But, regularized NMF can be useful: If you're willing to add extra constraints beyond nonnegativity, ...
eric_kernfeld's user avatar
9 votes

Matrices: system that is "computationally singular" versus "exactly singular"

The condition number that is typically used in linear algebra to describe a matrix $A$ indicates, very roughly, how many significant digits of accuracy you can expect to lose when solving a linear ...
jbowman's user avatar
  • 40k
9 votes

SVD : Why right singular matrix is written as transpose

It's written as a transpose for linear algebraic reasons. Consider the trivial rank-one case $A = uv^T$, where $u$ and $v$ are, say, unit vectors. This expression tells you that, as a linear ...
8 votes
Accepted

What is low rank intuitively for an adjacency matrix?

I'm not sure if your question is easy to answer, but I will try to provide an intuition. I am not an expert in non-negative matrix factorisation so I can't explain the connections there. Let's ...
MachineEpsilon's user avatar
8 votes
Accepted

CP decomposition for tensor factorization

Here's an old but classic overview on this problem and related ones. PDF: https://www.kolda.net/publication/TensorReview.pdf DOI: https://doi.org/10.1137/07070111X To answer your exact question: $...
eric_kernfeld's user avatar
7 votes
Accepted

Confirming an understanding of SVD

I have tried to dispel some of the popular myths regarding the SVD. ``A common application of SVD is to make low-rank approximations to a matrix, $A$'' This is one of the applications of the SVD ...
Vini's user avatar
  • 266
7 votes

Machine learning algorithms as matrix factorization

Generalized Low Rank Models paper deals with exactly this. From the abstract: This framework encompasses many well known techniques in data analysis, such as nonnegative matrix factorization, ...
Jakub Bartczuk's user avatar
7 votes
Accepted

What is the relation between SVD and ALS?

... I confused about what ALS actually is? I thought it was something akin to SVD, or any other matrix factorization algorithm. Would It matter that i'm using explicit data instead of implicit data? ...
Rob's user avatar
  • 2,100
7 votes

Which is more numerically stable for OLS: pinv vs QR

Using the Moore-Penrose pseudo-inverse $X^{\dagger}$ of an matrix $X$ is more stable in the sense that can directly account for rank-deficient design matrices $X$. $X^{\dagger}$ allows us to ...
usεr11852's user avatar
  • 44.8k
7 votes
Accepted

What is finite precision arithmetic and how does it affect SVD when computed by computers?

Floating point arithmetic is an approximation to arithmetic with real numbers. It's an approximation in the sense that all digits of a number aren't stored, but instead are truncated to a certain ...
Sycorax's user avatar
  • 92.3k
7 votes

What is finite precision arithmetic and how does it affect SVD when computed by computers?

TLDR; In computers numbers are stored in finite slots of memory. For instance, an integer number in mathematics is whole number such as ...,-2,-1,0,1,2,3,... that can go in both directions from ...
Aksakal's user avatar
  • 61.7k
6 votes

How to choose an optimal number of latent factors in non-negative matrix factorization?

To my knowledge, there are two good criteria: 1) the cophenetic correlation coefficient and 2) comparing the residual sum of squares against randomized data for a set of ranks (maybe there is a name ...
Jean-Paul Abbuehl's user avatar
6 votes
Accepted

$LDL^T$ decomposition from Cholesky decomposition

Let $S=D^{\frac12}$, so that $S$ is a diagonal matrix with diagonal $s$. Note that when you compute $A=LS$, multiplying $L$ by $S$ multiplies each column of $L$ by the corresponding element of $s$, ...
Glen_b's user avatar
  • 285k
6 votes

Covariance matrix decomposition and coregionalization

A covariance matrix has ${n \choose 2} + n = \frac{n(n+1)}{2}$ free elements. The constraints for the spectral decomposition are: The eigenvalues are positive The eigenvectors are orthogonal The ...
Taylor's user avatar
  • 20.9k
6 votes

is the difference of two positive definite matrices also positive definite?

Let $G$ and $H$ be positive definite, and let $v$ be any vector. Because the matrices are positive self definite, $\exists$ $a$ and $b$ such that $v^T G v = a >0$ and $v^T H v = b>0$. Without ...
Gregg H's user avatar
  • 5,524
6 votes
Accepted

Difference between Cholesky decomposition and log-cholesky Decomposition

I think it's less confusing to call it the log-Cholesky paramet(e)rization rather than the log-Cholesky decomposition (i.e., the "decomposition" part doesn't change ...) From Pinheiro's thesis (1994, ...
Ben Bolker's user avatar
  • 44.2k
5 votes

Understanding the spectral decomposition of a Markov matrix?

This is all from here: http://cims.nyu.edu/~holmes/teaching/asa15/Lecture2.pdf TLDR: you're still using the spectral decomposition theorem; you just have to find the right symmetric matrix. Detailed ...
Taylor's user avatar
  • 20.9k
5 votes
Accepted

Find best fit diagonal matrix for error minimization

Let the columns of $X$ be $X_1,X_2,\ldots, X_n$, the corresponding entries of $A$ be $a_1, a_2, \ldots, a_n$, the columns of $Y$ be $Y_1, Y_2, \ldots, Y_n$, and the error columns be $e_1, e_2, \ldots, ...
whuber's user avatar
  • 327k
5 votes
Accepted

Iteratively updating the decomposition of a covariance matrix

I think you can achieve part of what you want by using an incremental SVD and/or an online PCA algorithm. Given a known decomposition we update it to take into account a new data-point. In terms of ...
usεr11852's user avatar
  • 44.8k
5 votes

Relationship between Alternating Least Squares and SVD

Actually, ALS is generally less computationally efficient than directly computing the SVD solution, with some special cases. An interesting results of the SVD decomposition is that one gets the ...
Cliff AB's user avatar
  • 21.4k
5 votes
Accepted

Prove that sample covariance matrix is positive definite

First, let's simplify the equation for your sample covariance matrix. Using the fact that the centering matrix is symmetric and idempotent you get the $p \times p$ form: $$\begin{align} \mathbf{S} &...
Ben's user avatar
  • 127k
4 votes
Accepted

Can singular value decomposition be applied to a matrix of $n\times 1$ size?

SVD (singular value decomposition) can be applied to an n by m matrix, as long as n and m are both positive integers. n = 1 and/or m = 1 are perfectly fine. Let M be an n by m matrix. Then the SVD ...
Mark L. Stone's user avatar
4 votes

Matrix Factorization Recommendation Systems with Only "Like" Ratings

This problem is usually called implicit feedback. The typical solution is similar to word2vec noise-contrastive estimation: predict likes, with log-loss, use your set of actual likes (p=1) and ...
Piotr Migdal's user avatar
  • 5,806
4 votes

Relationship between Poisson generation and generalized Kullback-Leibler divergence

We want to proof that \begin{align} argmin_{W,H} \qquad D_{KL}(\boldsymbol{V} | \boldsymbol {WH}) \quad = \quad argmax_{W,H} \qquad p(\boldsymbol{V} | \boldsymbol{W}\boldsymbol{H}) \end{align} under a ...
alberto's user avatar
  • 3,056

Only top scored, non community-wiki answers of a minimum length are eligible