Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Singular value decomposition (SVD) of a matrix $\mathbf{A}$ is given by $\mathbf{A} = \mathbf{USV}^\top$ where $\mathbf{U}$ and $\mathbf{V}$ are orthogonal matrices and $\mathbf{S}$ is a diagonal matrix.

0
votes
0answers
13 views

Eigenvalue decomposition/SVD and the filtering perspective

I have been studying the SVD algorithm recently and I can understand how it might be used for compression but I am trying to figure out if there is a perspective of SVD where it can be seen as a low ...
2
votes
0answers
27 views

Why truncated SVD can denoise images

There are a lot of empirical results about that truncated SVD (TSVD) can help denoise the noises of images, but I wonder what is the theoretical support behind that? We know that TSVD is the best low-...
0
votes
0answers
26 views

Does Column ordering matter in QR decomposition?

I am trying to understand if the ordering of columns matters in QR decompsoition. In general it seems that column ordering won't matter. I guess for SVD or any matrix factorization the way columns ...
0
votes
0answers
11 views

Signs in SPSS's PCA with rotations with the FACTOR algorithm

I am trying to reproduce the results of the PCA with rotations from SPSS in python. But there is some information I didn't find in their documentation. I am trying to do the PCA like in the FACTOR ...
4
votes
2answers
85 views

Eckart-Young-Mirsky theorem: rank $≤k$ or rank $=k$

The Eckart-Young-Mirsky theorem is sometimes stated with rank $\le k$ and sometimes with rank $= k$. Why? More specifically, given a matrix $X \in \mathbb{R}^{n \times d}$, and a natural number $k \...
1
vote
0answers
19 views

Extracting latent vectors from autoencoder similar to SVD

I have read that there is an equivalency between a linear autoencoder and performing SVD. SVD can be used in collaborative filtering, for example, factorization of a user-movies matrix $\mathbf{M}$ ...
0
votes
1answer
104 views

PCA in psych package with more columns than rows

Why is it impossible to do a PCA in R using principal from psych package without warnings with a matrix, which has more columns ...
1
vote
0answers
25 views

PCA for three-dimensional linear fit on time-resolved trajectory

I study the behavior of organisms that are able of self-locomotion and that show directed motion toward one another. This directed motion occurs through the detection of chemical trails released by ...
3
votes
1answer
48 views

Derivation of Procrustes transformation

I'm curious about the derivation for the Procrustes transformation. I'm following ESLII: see figure 14.25 and problem 14.8 (Procrustes distance with scaling). Given matrices $\mathbf{X}_1$ and $\...
1
vote
0answers
30 views

matrix factorization with non-negative constraint only on one of the factors

I have a 2D spectral data time series with a wavelength dimension and a time dimension, and I'd like to decompose it to the time evolution ($SV^T$ for SVD and $H$ for NNMF) of several spectral ...
2
votes
0answers
24 views

Is there any sort of higher-order SVD (quadratic and above) for dimensionality reduction?

X-Posted on math.stackexchange, apologies, though I thought this was equally relevant to both communities. I'm wondering if there exists any higher-order SVD for dimensionality reduction. Note that ...
0
votes
0answers
85 views

How SVD is used for dimensionality reduction? [duplicate]

Given a $M \times N$ data matrix $ D = (x_1, x_2, \cdots, x_M)^{T}$. Applying singular value decomposition to $D$ yields $$D = USV^{T} = (u_1, u_2, \cdots, u_M) \begin{pmatrix} s_1 &&0\\...
0
votes
0answers
55 views

Why is the size of fitted truncated svd model is so big?

I have a dataset with tfidf matrix of shape (200000, 565000). I am fitting truncated svd of 500 dimensions from sklearn onto it and pickling the resulting svd object for later use. The pickle file is ...
4
votes
2answers
105 views

The Curse of high Dimension And Distance

For extracting features from video frames (2 sample/sec) I use keras framework in python and load VGG16 that input size is (150,150,3) and output size is (4,4,512). After the feature extraction step I ...
5
votes
1answer
61 views

Evaluate approximation of PCA from randomized algorithms

I have been comparing different PCA implementations (some via explicit calculation of the covariance matrix, some with randomized/truncated SVD) in terms of speed, and now wanted to compare how good ...
2
votes
1answer
87 views

Latent variable and Factor analysis ICA

While I was going through the factor analysis for Independent component analysis, I got stuck in one statement. How does it come to co-variance of S* is I? Is A* =ART ? Following is what I was going ...
0
votes
0answers
39 views

How to choose reduced features using SVD from a dataset?

How svd reduces features from a matrix. suppose, Matrix A(m,n). if we apply svd to A then we will find matrix U(m,m), S(m,n), v(n,n) matrix. S is the strength/ diagonal matrix as I know. We can ignore ...
1
vote
1answer
84 views

Orthogonality as found by the Gram-Schmidt process vs. uncorrelated basis vectors

I have a data matrix $Y$ of size $n \times p$, a basis vector in $\mathbb{R}^p$ $v_1$, and a potential basis vector in $\mathbb{R}^p$ $v_2'$. Then, if I use the Gram-Schmidt process on $[v_1, v_2']$ ...
0
votes
0answers
11 views

How to return the variance size of a recommendation system using SVD

One problem that many people have when making a recommendation system is the reasonableness of the suggestion / prediction, so I wanted to know how we can calculate the variance size of a generic ...
2
votes
1answer
42 views

Has anyone seen Gibbs phenomenon in SVD?

I read the notes on online about Regularized matrix computation. It said The truncated SVD solution has “ringing,” e.g., Gibbs’s phenomenon in truncated Fourier series I haven't seen any work ...
4
votes
2answers
1k views

How does eigenvalues measure variance along the principal components in PCA? [duplicate]

I understand that eigenvalues measure variance along the principal components. Questions How are eigenvalues and variance same for PCA? What is the intuition behind this being the same? What is ...
0
votes
0answers
17 views

Confused about how to interpret principle components [duplicate]

I think I understand how PCA works. In summary... I have a set of mean-deviated observations. The covariance matrix $S$ for my observations is not diagonal, so for some reason it's hard to interpret. ...
0
votes
1answer
146 views

Relationship between Alternating Least Squares and SVD

I have been assuming that ALS is simply an alternative algorithm for doing matrix decomposition that is more efficient, but in the end produces the same $U$ & $V$ matrices that SVD does. Is this ...
0
votes
0answers
53 views

Methods for selecting the required n_components for TruncatedSVD?

Methods for selecting the required n_components for TruncatedSVD? I found this (https://chrisalbon.com/machine_learning/feature_engineering/...
2
votes
1answer
202 views

PCA - Reconstruction from a “clean” set of eigenvectors?

This is a question related to the explanation here on how to reconstruct data from PCs found here: How to reverse PCA and reconstruct original variables from several principal components? I have two ...
0
votes
0answers
49 views

Singular value decomposition

Can singular value decomposition used to impute missing values in highly nonlinear process under multiple input and multiple output behavior?
1
vote
2answers
419 views

Explaining dimensionality reduction using SVD (without reference to PCA)

I have seen Dimensionality reduction mentioned as one of the practical usages of SVD. However, the explanation for me has always been Let me find the directions in which the variance of the data is ...
0
votes
1answer
111 views

TruncatedSVD always reduces dataset to 1D

I know that I have a large sparse matrix which I'm using TruncatedSVD to condense into a smaller number of dimensions. Here is my code: ...
1
vote
0answers
41 views

Understanding Diagonal Matrix of SVD [closed]

I have a matrix A with dimensions 4x3. I performed SVD on the matrix using numpy (np.linalg.svd) on matrix A. The output dimensions of U, V, and S are (4x4),(3x3),(3,). So V comprise orthonormal ...
0
votes
0answers
57 views

Latent Semantic Indexing

I watched Videos on LSA and have a litte trouble to understand, which parts of the SVD are used. This is the Video I have trouble with. Starting with 20:15min the guy uses only the train.irlba$v. ...
0
votes
0answers
60 views

How do I assign meaning or give names to the latent variables in exploratory factor analysis?

I read the book Latent Variable Models, and in the chapter dealing with exploratory factor analysis the author shows a way to learn latent variables (factors) from ...
0
votes
0answers
134 views

What if my kNN model performs well on PCA data even if I only keep 30% of the variance?

I have a data set with 10 continuous and 1000 categorical (binary) features - as a result the data set is very sparse. Each observation belongs to one of two classes. I perform Principal Component ...
0
votes
1answer
204 views

What is the correct way to scale data, apply PCA and fit a Multivariate Normal Distribution for anomaly detection?

I want to train an anomaly detection model in python. I have a training data set with some 30,000 observations, 700 of which are anomalies, and I can distinguish between normal and anomalous cases (I ...
1
vote
0answers
311 views

Imputing missing values and SVD

Similar questions have been asked a lot of times but I have not found an answer that gives an intuitive explanation as to why this works. For reference I have read the answers here and here. As I ...
1
vote
1answer
65 views

For an eigenvector v, is it always true that v^tv =1?

I am reading this link PCA which is a very insightful tutorial however, in this tutorial, the author mentioned a constraint on PCA: $C^{T}C=1$ when we look at eigenvalue/eigenvector definition $...
5
votes
2answers
200 views

Find a matrix with orthonormal columns with minimal Frobenius distance to a given matrix

Let $\mathbf A$ be an arbitrary $n \times m$ matrix with $n \ge m$. I want to find $\mathbf X$ of the same size with orthonormal columns that minimizes the Frobenius norm of the difference between $\...
1
vote
0answers
175 views

SVD item similarity calculation

I am performing SVD on a rating matrix of Users and Items and I get 3 matrices out of which Vt provides latent feature for items. How do I compute similarities between a pair of items using these ...
1
vote
0answers
23 views

Theoretical form of the rank of the low rank approximation matrix

The low-rank matrix approximation problem involves finding of a rank k version of a m × n matrix A, labeled $A_k$, such that $A_k$ is as ”close” as possible to the best SVD approximation version of A ...
5
votes
1answer
422 views

For symmetric matrices, is the Cholesky decomposition better than the SVD? [closed]

I am inverting a sparse, symmetric, ill-conditioned matrix. I have used both SVD and the LDL decomposition. I find that my results are better with the latter. Why? I understand that LDL ...
3
votes
1answer
384 views

Whitening/Decorrelation - why does it work?

Given some whitening transform, we change some vectors $\textbf{x}$, where features are correlated, into some vector $\textbf{y}$, where components are uncorrelated. Then we run some learning ...
0
votes
1answer
33 views

show that minimising the distance to approximated vector is equivalent to selecting features with the most variance

First off, a little disclaimer: I am basing the question on my own interpretation of the problem. It's well possible that I am mistaken. Setup: We are given: feature vectors $x_1,\ldots, x_k$ with ...
1
vote
0answers
52 views

Interpreting Sign of Loadings in Truncated SVD

I'm performing Truncated SVD on a word co-occurrence matrix for topic extraction. I'm using the scikit-learn implementation. I have read conflicting ideas on how to interpret the sign of the ...
1
vote
1answer
508 views

SVD decomposition and original matrix are not equal [closed]

When l compute the SVD of my matrix x as defined in kernel_hist_to_SVD(). The resulted decomposition is not equal (approximatively) to the original matrix k. Here ...
1
vote
1answer
1k views

Which features to include for Truncated SVD?

I have a dataset of ~31000 8k-filings (ad-hoc announcements from companies listed on the stock exchange). Every document consists of a string (the actual filing, stemmed and stopwords removed) and a ...
0
votes
0answers
100 views

Why PMI + SVD works for similarities arithmetics?

Recently Julia Silge blogged here and here, quoting blog entry by Chris Moody, who suggested that the similarities arithmetic in word2vec can be approximated by using PMI indexes followed by SVD ...
6
votes
1answer
198 views

Why is my LDA performance a non-monotonic function of the amount of training data?

Short story: I have a classification pipeline consisting of some feature extractors and an LDA classifier. When evaluating the pipeline in a cross-validation I get a decent test accuracy of 94% (for ...
0
votes
0answers
39 views

Why reverse SVD has an 'S' term?

When working on PCA, in order to map points in the PC space back to the original space, we can use formula $\hat{\mathbf X} = \mathbf{ZV}^\top$. But for reverse SVD, the formula change to $\hat{\...
24
votes
4answers
7k views

Why does Andrew Ng prefer to use SVD and not EIG of covariance matrix to do PCA?

I am studying PCA from Andrew Ng's Coursera course and other materials. In the Stanford NLP course cs224n's first assignment, and in the lecture video from Andrew Ng, they do singular value ...
1
vote
1answer
2k views

What is pca.components_ in sk-learn?

I've been reading some documentation about PCA and trying to use scikit-learn to implement it. But I struggle to understand what are the attributes returned by sklearn.decompositon.PCA From what I ...
5
votes
1answer
129 views

Confusion regarding the dimensions of matrices in the formula for SVD

I wanted to understand Singular Value Decomposition (SVD) hence consulted a few resources. In general I came across 2 different forms of SVD and hence got confused as to which one is correct or ...