Questions tagged [svd]

Singular value decomposition (SVD) of a matrix $\mathbf{A}$ is given by $\mathbf{A} = \mathbf{USV}^\top$ where $\mathbf{U}$ and $\mathbf{V}$ are orthogonal matrices and $\mathbf{S}$ is a diagonal matrix.

125 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
0answers
173 views

What is the fastest way to compute PC1 scores, without performing the whole PCA?

I want to compute only the first principal component's scores $t_1$ of a large number $n$ of data points x with a high dimensionality $p$. Assume the data has been centered about zero. Data points ...
4
votes
0answers
78 views

Physical interpretation of $U$ and $V$ matrices in SVD

I have a question about the physical interpretation of $U$ and $V$ matrices in SVD. I collect measurements at multiple devices across time are collected into an $m$ × $T$ matrix $M$, where m is the ...
4
votes
0answers
264 views

Why PMI + SVD works for similarities arithmetics?

Recently Julia Silge blogged here and here, quoting blog entry by Chris Moody, who suggested that the similarities arithmetic in word2vec can be approximated by using PMI indexes followed by SVD ...
4
votes
0answers
1k views

Why would SVD be 'unstable' if you don't standardize your data first?

I'm reading an article about Direct Linear Transformation which processes data using SVD, and the data set is standardized so that it has zero mean and unit standard deviation (n.b., some people call ...
4
votes
0answers
1k views

Prediction using SVD and Fisher's linear discriminant

Where can I get an explanation of the procedure used when making a prediction using SVD? Let me elaborate a bit more. Suppose you have data in a matrix $A$ containing two classes. In particular, you ...
4
votes
0answers
3k views

What is the correct way to calculate the explained variance of each EOF as calculated from a gappy data set?

I am trying to determine the correct amount of variance explained by each mode of an Empirical Orthogonal Function (EOF) analysis (similar to "PCA") as applied to a gappy data set. (i.e., containing ...
3
votes
0answers
218 views

Why is it much quicker to compute ridge regression than regular linear regression?

By my understanding, for a matrix with n samples and p features: Ridge regression using cholesky takes O(p^3) time Ordinary linear regression takes O(p^3) time Singular value decomposition if u, v ...
3
votes
0answers
120 views

Why truncated SVD can denoise images

There are a lot of empirical results about that truncated SVD (TSVD) can help denoise the noises of images, but I wonder what is the theoretical support behind that? We know that TSVD is the best low-...
3
votes
1answer
64 views

Has anyone seen Gibbs phenomenon in SVD?

I read the notes on online about Regularized matrix computation. It said The truncated SVD solution has “ringing,” e.g., Gibbs’s phenomenon in truncated Fourier series I haven't seen any work ...
3
votes
0answers
739 views

Imputing missing values and SVD

Similar questions have been asked a lot of times but I have not found an answer that gives an intuitive explanation as to why this works. For reference I have read the answers here and here. As I ...
3
votes
0answers
433 views

Identifying variables contributing to near multicollinearty in linear regression using VIF's and multiple R squared's

When trying to detect collinear columns in $X$ a high proportion of cases give a $R_k^2$ close to 1 for independent columns (see figure). When near multicollinearity arise in a $n\times m$ data ...
2
votes
0answers
22 views

Singular value decomposition used for dimensionality reduction in brain signal topographic data

I am trying to replicate the localizer method described in this paper (page 4). I am stuck on a step which I don't completely understand, and I would like your input and interpretation to progress. ...
2
votes
1answer
71 views

Orthonormalization to use closed form Lasso solution

Given the Lasso problem $$ min_\beta (Y-X\beta)^\top(Y-X\beta) \quad s.t. \|\beta\|_1\leq\lambda, $$ and assuming that X is orthonormal such that $X^\top X=I$, we know that the closed form solution ...
2
votes
0answers
16 views

How to approximate a Hermitian matrix with a transposed cross product of a single matrix?

I have a complex square matrix, and wish to learn latent factors (equally weighted latent factors, so not SVD) from this matrix. Given a Hermitian matrix A of ...
2
votes
0answers
108 views

Row similarity in matrix vs in different factorizations

Suppose an arbitrary $m \times n$ matrix $M$ and the factorizations: Arbitrary: $M = U_a V_a^T$, where $U_a$ is $m \times k$, $V_a$ is $n \times k$ ($k < m,n$), and $rank(U_a)=rank(V_a)=k$. SVD: $...
2
votes
0answers
36 views

Signal Decomposition

I have two time dependent signal sources X & Y. Both can be modeled as having a linear combination of time dependent individual components and common components; so X(t)=a(t)+C(t)+noise, Y(t)=b(t)+...
2
votes
0answers
34 views

Is low rank finite-iteration manifold identification possible?

In sparse optimization, I am trying to solve the problem $$ \min_{x\in \mathbb R^{n}} \quad f(x) + \|x\|_1 $$ and at optimality, $x^*$ may be sparse. If I define the sparse manifold as $\mathcal M = ...
2
votes
1answer
96 views

matrix factorization with non-negative constraint only on one of the factors

I have a 2D spectral data time series with a wavelength dimension and a time dimension, and I'd like to decompose it to the time evolution ($SV^T$ for SVD and $H$ for NNMF) of several spectral ...
2
votes
0answers
65 views

Is there any sort of higher-order SVD (quadratic and above) for dimensionality reduction?

X-Posted on math.stackexchange, apologies, though I thought this was equally relevant to both communities. I'm wondering if there exists any higher-order SVD for dimensionality reduction. Note that ...
2
votes
0answers
549 views

SVD item similarity calculation

I am performing SVD on a rating matrix of Users and Items and I get 3 matrices out of which Vt provides latent feature for items. How do I compute similarities between a pair of items using these ...
2
votes
0answers
225 views

Data compression using either Singular Values or Eigenvalues

In many applications, an SVD of a matrix is used to determine which features are important and which ones less important. For example, in image compression, the smallest singular values are often ...
2
votes
0answers
430 views

SVD based recommender system C#

I'm trying to reduce the number of dimensions in my dataset for a movie recommender system using SVD. I'm using the 'MovieLens 1M Dataset' from GroupLens.org. I've used the MathNet library for ...
2
votes
0answers
107 views

SVD of a matrix normal: practical applications?

What are some practical applications of the distributions of the components of an SVD of a matrix of normals? In particular, assume $Y \sim N_{n \times p}({\bf 0}, \Sigma \otimes I)$, i.e. the rows ...
2
votes
0answers
1k views

What is the difference of learning Latent features using SVD and using embedding vectors in deep network

Traditionally, singular value decomposition (SVD) can be used to learn latent feature of user and items according to user-item rating matrix. Recently, researchers use embedding layers as the input ...
2
votes
0answers
154 views

Why 1 norm of reconstruction error is not used/minimized for low rank approximation using PCA?

In pca, I see reconstruction error is calculated in terms of either frobenius norm or spectral norm. And I also saw they have a closed bound in terms of singular values. My question is why 1 norm of ...
2
votes
0answers
285 views

Is there a supervised/semi supervised version of pca for dimensionality reduction?

PCA can give me the proper result if "Large variances have important dynamics" holds true for the data. In other words if I want to know along which components the variance of my data is maximized, ...
2
votes
0answers
2k views

Different order and signs of eigenvectors when doing PCA via eig() or svd() functions in Matlab

Assume we have a matrix X = randn(5,3). I am doing two things: 1) [S D1 V1] = svd(X); 2) ...
2
votes
0answers
224 views

LSA projections of documents and terms

I am trying to understand how Latent Semantic Analysis works, reading demonstrations based on singular value decomposition. Let's denote $X$ a $D \times W$ document-term matrix. The $D$ rows of $X$ ...
2
votes
0answers
596 views

How to go from sparse matrix to linear regression model (using SVD)?

I am trying to replicate the Kosinski, Stillwell, & Graepel (2013) study about predicting private traits and attributes from Facebook like data for study purposes. First I have admit, however, ...
2
votes
0answers
155 views

Truncated singular value decomposition

Is it possible to get a "truncated SVD"-regularized solution for L1 norm min errors problem? $$min\|Ax-b\|_{1}$$ In L2 universe results are derived easily analytically. I want to formulate a problem ...
2
votes
0answers
628 views

How would you preprocess data for SVD?

I am computing SVD on a matrix which is the empirical version of $E[XY^{\top}]$ for some $X \in \mathbb{R}^{m \times 1}$ and $Y \in \mathbb{R}^{n \times 1}$. I am wondering if there are standard ways ...
2
votes
0answers
249 views

Rank-one nonnegative matrix factorization

For non-negative matrix factorization with Frobenius norm: $$\min\limits_{U\in\mathbb{R}_+^{m\times r}, V\in\mathbb{R}_+^{r\times n}}||A-UV||_F^2, A\in\mathbb{R}_+^{m\times n}$$ $r=1$ is a very ...
1
vote
0answers
107 views

Adding explicit user info to matrix factorization

In the paper Matrix Factorization Techniques for Recommender Systems, it is claimed that we can incorporate extra user information into our recommender model by doing something like this: $$ \hat{r}_{...
1
vote
0answers
33 views

Determining ML Approach for calculating SVD using neural networks

I am currently working on a project where I need to perform SVD (Singular Value Decomposition) computation on a noisy data using neural networks. It doesn't have to be exact SVD, certain degree of ...
1
vote
0answers
29 views

Does it make sense to use SVD to do a sort of “lossy compression”?

So - I know if you perform SVD to a matrix $X$, you can then use Echkart Young theorem to get the best rank $r$ approximation $\overline{X}$ to $X$ possible. Since the resultant ${\overline{X}}$ will ...
1
vote
0answers
36 views

Computational advantage for soft-impute method over other methods

I am reading in the soft-imputing paper for low-rank-based matrix completion. They suggested another solution for $$\hat{Z} = \text{argmin}_Z\lVert X - Z \rVert_F^2 + \lambda \lVert Z \rVert_*$$ ...
1
vote
0answers
23 views

Fastest way to find Leading singular value and vector (power iteration, rsvd etc)

I want to know the fastest way to find out the leading singular value and vector of a large rectangular matrix. I have seen 2 suggestions and have questions on both of them : Power Method : For this ...
1
vote
0answers
26 views

Kernel matrix decomposition

I had a look at the sklearn.kernel_approxiamtion.Nystroem implementation, which is also described in this post: Nystroem Method for Kernel Approximation Here, a ...
1
vote
0answers
24 views

What is so special about the least norm solution in case of an undetermined system of equations

In particular this is the go to approach in case of solving a least squares problem that lacks a unique solution, how does being the closest point to the origin among all the solutions make it any ...
1
vote
0answers
36 views

SVD for a complex data matrix — what is the meaning of the columns of $V$?

I've read this wonderful explanation of SVD, where the writer mentions that the columns of $V$ are the principal directions (Summary, #1). Is this also true when the data matrix $X$ is complex? If I'm ...
1
vote
0answers
44 views

Relationship between the SVD and correlation matrices

I'm reading Data Driven Science and Engineering by Kutz and Brunton to understand more about the SVD. Consider $X = U\Sigma V$, $XX^*$, and $X^*X$ where $X \in \mathbb{R}^{m\times n} $ In particular, ...
1
vote
0answers
21 views

Use Matrix Factorization to predict probability of a recommendation system?

I have a dataset where I have a sparse utility matrix (user-product) with binary input: 1 if the user $i$ bought the product $j$, and 0 if it hasn't. However it has a different meaning on the test set,...
1
vote
1answer
39 views

PCA with SVD exercice 23.5 understanding machine learning

In understanding machine learning Shai Sharev-Scwartz and Shai Ben-David exercice 23.5. I would like to use SVD to minimize : $$ \text{argmin}_{W \in \mathbb{R}^{n,d}, U \in \mathbb{R}^{d,n}}{\text{ }...
1
vote
0answers
20 views

Principal Component Analysis on Numerical Predictors alone for Dimension Reduction

I'm trying to reduce the number of dimensions for this 'Network Anamoly Detection' dataset: https://www.kaggle.com/anushonkar/network-anamoly-detection The dataset has a total of 40 features out of ...
1
vote
0answers
35 views

Can I combine independent components from different models using PCA?

I have a set of independent components for each subject in my dataset (i.e. an ica model was generated for each subject). The samples used to generate each set of ICs are aligned across subjects, and ...
1
vote
1answer
497 views

What do the matrix (S, U, V) returned by singular value decomposition represent (in terms of variation)?

I believe SVD on a matrix A returns three matrices: U, S, and V. Let's imagine A is a data matrix with training examples/records/whatever you call them as its rows and attributes as its columns. I ...
1
vote
0answers
59 views

SVD versus RSVD

In the so-called incremental SVD used for collaborative filtering: http://www.machinelearning.org/proceedings/icml2007/papers/407.pdf http://www2.research.att.com/~volinsky/papers/ieeecomputer.pdf ...
1
vote
0answers
259 views

Why does kmeans after SVD result in ideal clusters

I am clustering tweets which are related to eye fashion and they are extracted using keywords like mascara, eyeliner, eyeshadow, etc from twitter. I constructed a Tf-idf matrix (tweets x words) ...
1
vote
0answers
490 views

Calculating PCA coefficients using SVD, PCA (sklearn) and Covariance Matrix

I am trying to understand PCA implemented in different methods on python. I am failing to get equal PCA coefficients in each of the methods. By PCA coefficients I mean data projected in the principle ...
1
vote
0answers
38 views

How SVD factorisation -based recomendation algos deal with new user interaction

Classic SVD and SVD++ alogritms generate predictions based on a current known ratings only for known users and known items. But I need to make prediction for some new user on the old items. In the ...