Questions tagged [svd]

Singular value decomposition (SVD) of a matrix $\mathbf{A}$ is given by $\mathbf{A} = \mathbf{USV}^\top$ where $\mathbf{U}$ and $\mathbf{V}$ are orthogonal matrices and $\mathbf{S}$ is a diagonal matrix.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Singular Value Decomposition (SVD) and Nonnegative Matrix Factorization (NMF) as dimensionality reduction techniques

I am reading about SVD and NMF and how they can be used for dimensionality reduction of a data matrix X . What I did not understand why the data matrix is still have the same dimensions after SVD or ...
USER's user avatar
  • 1
1 vote
0 answers
41 views

Why does ridge regression apply a non-monotone transformation to the singular values of the design matrix?

Per Wikipedia, Ridge Regression is equivalent to transforming the singular values $\sigma_i$ of the design matrix to $\frac{\sigma_i^2 + \alpha^2}{\sigma_i}$, where $\alpha$ is (in Wikipedia's ...
kuzzooroo's user avatar
  • 121
0 votes
0 answers
62 views

Differences Between Independent Components from ICA Directly on Samples vs. Mixing Matrix from ICA on Features After PCA Dimensionality Reduction

My understanding is that for Independent Component Analysis (ICA), it is recommended to have more samples than features to avoid underdetermination which might cause convergence or stability issues. ...
user19402204's user avatar
2 votes
0 answers
88 views

Independent Component Analysis (ICA): Why rotate whitened data by principal components instead of right singular vectors?

I have a data matrix $ X $ that is $n \times m$, where $n$ is the number of features and $m$ is the number of samples and $ n < m$. Let the Singular Value Decomposition (SVD) of $X$ be $$ X = U \...
user19402204's user avatar
1 vote
0 answers
17 views

How to compare different clusters of different size, rotation, scale and translation?

Assume that you have a matrix $X$ that contains the data inside the left image. The data inside $X$ is not classified. The matrix $X$ also contains outliers/noise. On the right, we can se the template ...
euraad's user avatar
  • 415
0 votes
1 answer
119 views

Difference between conducting PCA on $XX^\top$ vs $X^\top X$?

PCA: For a given set of centered data $\mathscr D =\{x_i\}_{i=1}^N \subset \mathbb R^d$, i.e. the data has $N$ examples with dimension $d$. Then the principal directions of PCA can be obtained from ...
Fong Lam's user avatar
4 votes
2 answers
128 views

PCA to reconstruct Binary Data

I'm working with binary 3D matrices. I calculate their PCA (or REOF or SVD) and as a test I would like to reconstruct these matrices from the PCA results. However I realized that because I only keep ...
vdc's user avatar
  • 143
1 vote
1 answer
73 views

Which features corresponds to which eigenvalues when use SVD in PCA?

Today, after learning about performing $PCA$ using $SVD$, I know $PCA$ will choose $K$ components that have the highest eigenvalues. I have a question which feature will correspond to which ...
manh3's user avatar
  • 113
1 vote
0 answers
73 views

Is it correct to do SVD from the latent space of an autoencoder?

Is it correct to do SVD from the latent space of an autoencoder? I am asking because I think that by performing SVD from a latent space, and plotting the singular values, it is possible to know the ...
Carlos's user avatar
  • 11
0 votes
0 answers
31 views

Is there an alternate estimator for a sample covariance matrix when n < p such that the estimator is not singular

Let's say I have $n$ samples which are vectors of length $p$. I know that the $p \times p$ sample covariance matrix is singular if $n \leq p$. Is there another estimator for the covariance that ...
David Wang's user avatar
0 votes
0 answers
17 views

Is it possible to apply matrix decomposition to a vector, injecting additional information to UV decomposition?

As I am reading about recommender systems in Machine Learning, UV decomposition caught my eye (click for an explanation or see below). So I have two questions: Question 1: what are the drawbacks of ...
Jironymo's user avatar
3 votes
0 answers
61 views

Relationship between SVD of Matrix and SVD of same Matrix with deleted entries (Matrix can be Adjacency Matrix of a Graph)

Could somebody direct to me to some literature dealing with this issue. So we have $X = U\Sigma V^{T}$ and we have $M \odot X = U^{'}\Sigma^{'}V^{'^{T}}$ with \begin{equation} M_{i,j} = \begin{cases} ...
Yunus Cobanoglu's user avatar
1 vote
0 answers
22 views

Prediction of Multiple Linear Regression With Constant

Let $X$ be a matrix with $n$ rows and $d$ columns. We know that there exists matrices $U, S, V$, with $U$ of dimensions $(n, d)$, $S$ of dimensions $(d, d)$ and $V$ of dimensions $(d, d)$, which form ...
user35083's user avatar
0 votes
0 answers
44 views

Rotation Matrix from Covariance of 3D point-cloud

I am trying to retreive rotation matrix from a rotated 3D point cloud covariance matrix, using SVD decomposition (as done in SimNet and MVTrans). Here how I computed the covariance matrix from 3D ...
Samuele's user avatar
3 votes
1 answer
155 views

Why is the Scaling Matrix in LDA unnormalized?

I was carrying out LDA (linear Discriminant Analysis) and noticed that the Scaling matrix produced by R is not normalized. Here is an example: ...
Onyambu's user avatar
  • 451
0 votes
0 answers
13 views

Do we always want to do projection when we using PCA with SVD? [duplicate]

Assume that we have a matrix $X$ and we want to do Singular Value Decompotision(SVD) with $X$. First we need to find the average of $X$ row wise, called $\mu$ ...
euraad's user avatar
  • 415
0 votes
0 answers
44 views

How to do column weighted truncated SVD?

I have an unusual case where I need to combine two vector spaces but weight one more than the other. Rather than discussing my specific use case, it's likely easier to imagine we trained two word2vec ...
Josh's user avatar
  • 308
1 vote
1 answer
61 views

Latent Semantic Indexing vs. PCA

I am trying to understand how Latent Semantic Analysis works, reading demonstrations based on singular value decomposition. Let's denote $X$ a $N \times D$ document-term matrix. The $D$ rows of $X$ ...
Victor M's user avatar
  • 289
1 vote
1 answer
655 views

Understanding Leverage Score Sampling to get representative sample

I was reading about Leverage Score Sampling. If I am not wrong then what I know that Leverage Score Sampling help us to select representative sample. But I didn't understand how the whole process is ...
falamiw's user avatar
  • 131
1 vote
0 answers
61 views

is it fair to use a subset of eigenvalues to evaluate the multidimensional variance

I want to find a single metric to assess how spread (or how much variance) a multidimensional dataset (a large number of features) is. I learned that the determinant (or pseudo-determinant) of the ...
qliang's user avatar
  • 21
0 votes
0 answers
21 views

Should I do documents transformation at once or a pair at a time for auto grading with cosine similarity?

I'm developing auto grading essay that compares the similarity between the answer key and student answer with cosine similarity. This one is written in php. Let's say in a course there are 30 - 100 ...
newtocoding's user avatar
1 vote
1 answer
58 views

In principal component regression, how to show that $X \beta_\mathrm{PCR}=U\: \mathrm{diag}\left\{1,\ldots, 1, 0, \ldots 0\right\}U^Ty$ if $X=USV^T$

Regarding the third equation in this answer, I'm struggling to figure out how $ X \beta_\mathrm{PCR} = U\: \mathrm{diag}\left\{1,\ldots, 1, 0, \ldots 0\right\} U^\top y$. Given $X = USV^T$ where $X$ ...
Yandle's user avatar
  • 1,139
3 votes
0 answers
25 views

Time-Continuity of PCA/SVD [duplicate]

I have a daily set of observations (Jacard similarity matrix) that I want to embed in a much lower dimensional space. Thus I run PCA or SVD for each day and plot the projection on the top 2 dimensions ...
sds's user avatar
  • 2,236
6 votes
1 answer
250 views

Procrustes Problem for rank-deficient input

It is well known that the solution to the orthogonal Procrustes problem $$ \textrm{arg min}_{\Omega \in \text{SO}(n)} ||Y - \Omega X||^2_2, $$ can be expressed in terms of an SVD of the covariance ...
const-ae's user avatar
  • 267
0 votes
0 answers
64 views

Why do we multiply the centered data with eigen vector instead of taking inverse of the eigen vector while performing PCA

The question may sound stupid but I really don't understand the logic behind this. Whenever we do a PCA, we take a covariance matrix on the centered data and do eigen decomposition. In order to ...
Jacob Simon Areickal's user avatar
1 vote
0 answers
50 views

SVD to find the common factors

Let's say I have a time series $R\in \mathbb{R}^{T\times M}$ where $T$ shows the dates and $M$ is the number of variables. I do an SVD on it and obtain $R=U\Sigma V^\top$, where $U\in \mathbb{R}^{T\...
statwoman's user avatar
  • 653
0 votes
1 answer
97 views

Prove that P = A†A is orthogonal projection where A† is pseudo-inverse of A

I am new to this but I was wondering how to prove this. I can reduce A†A to (VΣ†UT)(UΣVT). Would I have to reduce the UT and U to the identity matrix and just continue to simplify? I would get ...
albert's user avatar
  • 1
2 votes
0 answers
29 views

PCA on X to capture the most variance of Y

PCA maximises the variation in $X$. Now, suppose $X$ is causal to $Y$. Is there then any analogous way to decompose $X$ into the principal components that cause the most variation in $Y$. For an ...
Funaizhang's user avatar
1 vote
0 answers
22 views

Eckart–Young–Mirsky theorem for $n \gg m$

It has been proven that the best reconstruction error in the $k$ rank matrix estimation problem in terms of Frobenius or $L2$ norm is given by the $k$-truncated SVD as shown here. I've read in ...
jonithani123's user avatar
1 vote
1 answer
96 views

How to derive the three matrices of SVD from eigenvalue decomposition in Kernel PCA?

Kernel PCA is usually done via eigenvalue decomposition of the Kernel Matrix $\mathbf{K}$ and standard PCA via SVD of the input $\mathbf{X}$. In standard PCA as far as I know we can derive $\mathbf{S}...
arod's user avatar
  • 23
2 votes
0 answers
221 views

How do ‘arpack’, ‘propack’ and ‘lobpcg’ compare to each other?

I want to implement singular value decomposition on a sparse matrix using the Scipy implementation. The documentation shows three solvers: ‘arpack’, ‘propack’ and ‘lobpcg’. How do these solvers ...
Emil's user avatar
  • 361
1 vote
0 answers
23 views

quasi-PCA reconstruction of the matrix by orthogonal basis

Let's say I have a "data" matrix $X$ of $N$ rows and $p$ cols with $N \gg p$. Now PCA with $L$ components can be formulated as $$ X_L = \underset{Y:rank(Y) = L}{\text{argmin}} ||X- Y||^2_F, ...
Vladimir Kirilin's user avatar
0 votes
0 answers
53 views

Is it logical to perform PCA and reduction on observations instead of features?

I am currently working with a set of code called GPMSA that was published by LANL. The code serves to create a Gaussian process model of some simulator and perform regression with experimental data. I ...
Stephen Wright's user avatar
1 vote
1 answer
50 views

Rotation-sensitivity of SVD

Suppose I perform a truncated SVD on a symmetric, PSD matrix $A \in R^{N \times d}$ (lowering the dimensionality from $d$ to $k$). Further suppose that there is a rotation matrix $Q$ such that some of ...
user1767774's user avatar
1 vote
0 answers
238 views

Optimal truncation in SVD

I am working with SVD on a matrix $$Y_{m,n} = T_{m,m} \Sigma D^T_{n,n} $$ where $T$ and $D$ describe the row and the column entities of Y, respectively. The truncated SVD takes the first $r$ ...
KArrow'sBest's user avatar
1 vote
0 answers
11 views

PCA: understanding how to use loading vectors of X to recapture the geometry of X

If we suppose that X is an n x d matrix, and then perform PCA to obtain loading vectors ...
Andrew Bell's user avatar
1 vote
0 answers
76 views

Does PCA centering guarantee projection optimality onto an affine set subject to dimensionality constraint?

Say I have $m$ points in $R^n$, not necessarily with sum 0, and I want to project them onto an affine set $S$ with $\dim(S)=d$ for a given $d$ so as to minimize the sum of the squared distances ...
Costin Oncescu's user avatar
0 votes
0 answers
78 views

Dot product and linear combination of Basis

I am currently going through the SVD intuition provided here. In the section "From intuition to definition", It says that, First, note that any vector $\textbf{x}$ can be described using ...
Ganesh Tata's user avatar
1 vote
1 answer
454 views

Use K-means clustering on SVD/PCA of data

In an assignment I was suppose to perform K-means clustering on the MNIST dataset (just the 0's and the 1's) and then use SVD/PCA to visualize the data in two dimensions. I missunderstood this and ...
user202542's user avatar
0 votes
0 answers
33 views

Why does reversing PCA from SVD follow this formula?

In this answer the author writes: PCA is very closely related to singular value decomposition (SVD), see Relationship between SVD and PCA. How to use SVD to perform PCA? for more details. If a $n\...
z611's user avatar
  • 255
0 votes
1 answer
40 views

Practical significance of number of singular values in SVD

I am working on a binary classification problem. SVD is used for dimensionality reduction and the vector with reduced dimension is used as the feature vector. DNN is used as the classifier. There are ...
user7080's user avatar
0 votes
0 answers
256 views

The discrepancy of results of PCA via Eigendecomposition vs via SVD in Python with scipy.linalg [duplicate]

I recently learned about different methods of PCA. I decided to manually implement PCA in Python with Eigendecomposition of cov(X) and the Singular Value ...
Alex.Kh's user avatar
  • 165
1 vote
0 answers
25 views

What does the SVD of the numerical derivative matrix of paired instances tell us about our data? [closed]

Let's say we have real-valued random variables $X$ and $Y$, and that under simple random sampling we obtain paired values $\{(x_1,y_1), \cdots, (x_i,y_i), \cdots, (x_m,y_m) \}$. From this sample we ...
Galen's user avatar
  • 8,195
1 vote
0 answers
153 views

How do I get the stationary distribution of a Markov chain matrix from SVD?

I have a matrix that represents a Markov chain. ...
Jane Wayne's user avatar
  • 1,389
1 vote
0 answers
123 views

Should you train/test split when using softimpute/matrix completion?

I am trying to impute missing values on a large dataset. After reading the paper(s) introducing matrix completion via soft-SVD thresholding, as well as the softImpute R package vignetter by Hastie (...
dwp's user avatar
  • 11
1 vote
0 answers
84 views

What is the difference between two forms of matrix coherence?

I am stuck on the definition of coherence of a matrix. Let $x_1, \dots, x_p \in \mathbb{R}^p$ be the columns of the matrix $X$, which are assumed to be normalized such that $x'_i x_i = 1$. Wikipedia ...
HJ Liang's user avatar
  • 132
1 vote
1 answer
107 views

Why is the first canonical direction equal to the left singular vector i.e. why is $w_1 = a = u_1$ in CCA (Canonical Correlation Analysis)?

I want to understand why the canonical direction $a$ is equal to the left singular value of $M = \Sigma^{-1/2}_X \Sigma_{X, Y} \Sigma^{-1/2}_Y$ and not $a = \sigma_1 u_1$. My calculation tell me that ...
Charlie Parker's user avatar
0 votes
0 answers
298 views

Singular value decomposition for a matrix with missing entries

Suppose $A$ is a matrix consisting of real numbers and nan's. What are some of the robust formulations and the associated algorithms for estimating its singular ...
Hans's user avatar
  • 995
1 vote
0 answers
26 views

Relation between low-rank approximation, nuclear norm of a matrix and Singular Value Decomposition

I'm reading the following paper https://arxiv.org/pdf/2005.10203.pdf which proposes improvements on robustness of large graphs to defend against adversarial attacks that are nothing but slight ...
James Arten's user avatar
0 votes
0 answers
71 views

how to understand the implications of the SVD matrix of the covariance $C_{XY} = X Y^T$

Given an $m \times n$ data matrix $X$, the SVD of its covariance matrix $$C = XX^T = ULU^T$$ provides the orthogonal unit vectors that maximize the variance in these directions. In the case of an $m \...
wsdzbm's user avatar
  • 101

1
2 3 4 5
9