Skip to main content

Questions tagged [svd]

Singular value decomposition (SVD) of a matrix $\mathbf{A}$ is given by $\mathbf{A} = \mathbf{USV}^\top$ where $\mathbf{U}$ and $\mathbf{V}$ are orthogonal matrices and $\mathbf{S}$ is a diagonal matrix.

Filter by
Sorted by
Tagged with
1 vote
0 answers
29 views

Variable weighted PCA

I have seen a lot of "weighted PCA" but they are really all on "observations". For example Weighted principal components analysis if you have K variables, N observations, the ...
Brandon Bolden's user avatar
1 vote
0 answers
33 views

Is zero-phase component analysis (ZCA) equivalent to substituting singular values with $\sqrt{n}$?

Consider the (centered) dataset $X \in R^{n \times p}$, with $n$ samples and $p$ features. According to this solution, the zero-phase component analysis (ZCA) is simply a re-rotation of PCA: $$W_{ZCA} ...
arash's user avatar
  • 309
0 votes
0 answers
32 views

PCA factor model - insignificant factor loadings

I have a time-series of N assets for which I am trying to estimate a factor model. Let $Z_{t}$ be one of these assets' prices at time $t$. We can write it as: $$ Z_{t} = \beta F_{t} + \theta_{t} $$ ...
deblue's user avatar
  • 243
3 votes
1 answer
132 views

Are there any situations where orthogonality is not optimal?

Data reduction is often used to avoid overfitting and to enhance explainability. Popular data reduction techniques, such as SVD or PCA map/project high-dimensional data to a lower-dimensional ...
Chris M's user avatar
  • 39
0 votes
1 answer
69 views

Frobenius norm of rank-constrained matrix product is bounded

Say I have three matrices $\mathbf{W} \in \mathbb{R}^{p \times m}$ and $\mathbf{A}, \mathbf{B} \in \mathbb{R}^{m \times n}$ with $\operatorname{rank}(\mathbf{A}) \leq r$ and $\mathbf{B}$ is ...
Dime's user avatar
  • 125
0 votes
0 answers
22 views

Can principal components changed by a normalization method be used to construct original data shape with SVD

I'm planning to use an algorithm called Harmony, designed for data normalization, particularly in the context of single cell data analysis. Harmony operates by taking principal components (PCs) as ...
MadmanLee's user avatar
  • 133
1 vote
0 answers
51 views

Why does ridge regression apply a non-monotone transformation to the singular values of the design matrix?

Per Wikipedia, Ridge Regression is equivalent to transforming the singular values $\sigma_i$ of the design matrix to $\frac{\sigma_i^2 + \alpha^2}{\sigma_i}$, where $\alpha$ is (in Wikipedia's ...
kuzzooroo's user avatar
  • 121
0 votes
0 answers
78 views

Differences Between Independent Components from ICA Directly on Samples vs. Mixing Matrix from ICA on Features After PCA Dimensionality Reduction

My understanding is that for Independent Component Analysis (ICA), it is recommended to have more samples than features to avoid underdetermination which might cause convergence or stability issues. ...
user19402204's user avatar
2 votes
0 answers
103 views

Independent Component Analysis (ICA): Why rotate whitened data by principal components instead of right singular vectors?

I have a data matrix $ X $ that is $n \times m$, where $n$ is the number of features and $m$ is the number of samples and $ n < m$. Let the Singular Value Decomposition (SVD) of $X$ be $$ X = U \...
user19402204's user avatar
1 vote
0 answers
20 views

How to compare different clusters of different size, rotation, scale and translation?

Assume that you have a matrix $X$ that contains the data inside the left image. The data inside $X$ is not classified. The matrix $X$ also contains outliers/noise. On the right, we can se the template ...
euraad's user avatar
  • 425
0 votes
1 answer
188 views

Difference between conducting PCA on $XX^\top$ vs $X^\top X$?

PCA: For a given set of centered data $\mathscr D =\{x_i\}_{i=1}^N \subset \mathbb R^d$, i.e. the data has $N$ examples with dimension $d$. Then the principal directions of PCA can be obtained from ...
Fong Lam's user avatar
4 votes
2 answers
156 views

PCA to reconstruct Binary Data

I'm working with binary 3D matrices. I calculate their PCA (or REOF or SVD) and as a test I would like to reconstruct these matrices from the PCA results. However I realized that because I only keep ...
vdc's user avatar
  • 143
1 vote
1 answer
84 views

Which features corresponds to which eigenvalues when use SVD in PCA?

Today, after learning about performing $PCA$ using $SVD$, I know $PCA$ will choose $K$ components that have the highest eigenvalues. I have a question which feature will correspond to which ...
kenj's user avatar
  • 113
1 vote
0 answers
81 views

Is it correct to do SVD from the latent space of an autoencoder?

Is it correct to do SVD from the latent space of an autoencoder? I am asking because I think that by performing SVD from a latent space, and plotting the singular values, it is possible to know the ...
Carlos's user avatar
  • 11
0 votes
0 answers
32 views

Is there an alternate estimator for a sample covariance matrix when n < p such that the estimator is not singular

Let's say I have $n$ samples which are vectors of length $p$. I know that the $p \times p$ sample covariance matrix is singular if $n \leq p$. Is there another estimator for the covariance that ...
David Wang's user avatar
3 votes
0 answers
66 views

Relationship between SVD of Matrix and SVD of same Matrix with deleted entries (Matrix can be Adjacency Matrix of a Graph)

Could somebody direct to me to some literature dealing with this issue. So we have $X = U\Sigma V^{T}$ and we have $M \odot X = U^{'}\Sigma^{'}V^{'^{T}}$ with \begin{equation} M_{i,j} = \begin{cases} ...
Yunus Cobanoglu's user avatar
1 vote
0 answers
22 views

Prediction of Multiple Linear Regression With Constant

Let $X$ be a matrix with $n$ rows and $d$ columns. We know that there exists matrices $U, S, V$, with $U$ of dimensions $(n, d)$, $S$ of dimensions $(d, d)$ and $V$ of dimensions $(d, d)$, which form ...
user35083's user avatar
4 votes
1 answer
178 views

Why is the Scaling Matrix in LDA unnormalized?

I was carrying out LDA (linear Discriminant Analysis) and noticed that the Scaling matrix produced by R is not normalized. Here is an example: ...
Onyambu's user avatar
  • 516
2 votes
1 answer
76 views

Latent Semantic Indexing vs. PCA

I am trying to understand how Latent Semantic Analysis works, reading demonstrations based on singular value decomposition. Let's denote $X$ a $N \times D$ document-term matrix. The $D$ rows of $X$ ...
Victor M's user avatar
  • 319
1 vote
1 answer
1k views

Understanding Leverage Score Sampling to get representative sample

I was reading about Leverage Score Sampling. If I am not wrong then what I know that Leverage Score Sampling help us to select representative sample. But I didn't understand how the whole process is ...
falamiw's user avatar
  • 131
1 vote
0 answers
62 views

is it fair to use a subset of eigenvalues to evaluate the multidimensional variance

I want to find a single metric to assess how spread (or how much variance) a multidimensional dataset (a large number of features) is. I learned that the determinant (or pseudo-determinant) of the ...
qliang's user avatar
  • 21
0 votes
0 answers
21 views

Should I do documents transformation at once or a pair at a time for auto grading with cosine similarity?

I'm developing auto grading essay that compares the similarity between the answer key and student answer with cosine similarity. This one is written in php. Let's say in a course there are 30 - 100 ...
newtocoding's user avatar
1 vote
1 answer
60 views

In principal component regression, how to show that $X \beta_\mathrm{PCR}=U\: \mathrm{diag}\left\{1,\ldots, 1, 0, \ldots 0\right\}U^Ty$ if $X=USV^T$

Regarding the third equation in this answer, I'm struggling to figure out how $ X \beta_\mathrm{PCR} = U\: \mathrm{diag}\left\{1,\ldots, 1, 0, \ldots 0\right\} U^\top y$. Given $X = USV^T$ where $X$ ...
Yandle's user avatar
  • 1,189
3 votes
0 answers
26 views

Time-Continuity of PCA/SVD [duplicate]

I have a daily set of observations (Jacard similarity matrix) that I want to embed in a much lower dimensional space. Thus I run PCA or SVD for each day and plot the projection on the top 2 dimensions ...
sds's user avatar
  • 2,246
6 votes
1 answer
296 views

Procrustes Problem for rank-deficient input

It is well known that the solution to the orthogonal Procrustes problem $$ \textrm{arg min}_{\Omega \in \text{SO}(n)} ||Y - \Omega X||^2_2, $$ can be expressed in terms of an SVD of the covariance ...
const-ae's user avatar
  • 301
0 votes
0 answers
78 views

Why do we multiply the centered data with eigen vector instead of taking inverse of the eigen vector while performing PCA

The question may sound stupid but I really don't understand the logic behind this. Whenever we do a PCA, we take a covariance matrix on the centered data and do eigen decomposition. In order to ...
Jacob Simon Areickal's user avatar
1 vote
0 answers
52 views

SVD to find the common factors

Let's say I have a time series $R\in \mathbb{R}^{T\times M}$ where $T$ shows the dates and $M$ is the number of variables. I do an SVD on it and obtain $R=U\Sigma V^\top$, where $U\in \mathbb{R}^{T\...
statwoman's user avatar
  • 703
0 votes
1 answer
127 views

Prove that P = A†A is orthogonal projection where A† is pseudo-inverse of A

I am new to this but I was wondering how to prove this. I can reduce A†A to (VΣ†UT)(UΣVT). Would I have to reduce the UT and U to the identity matrix and just continue to simplify? I would get ...
albert's user avatar
  • 1
2 votes
0 answers
29 views

PCA on X to capture the most variance of Y

PCA maximises the variation in $X$. Now, suppose $X$ is causal to $Y$. Is there then any analogous way to decompose $X$ into the principal components that cause the most variation in $Y$. For an ...
Funaizhang's user avatar
1 vote
0 answers
23 views

Eckart–Young–Mirsky theorem for $n \gg m$

It has been proven that the best reconstruction error in the $k$ rank matrix estimation problem in terms of Frobenius or $L2$ norm is given by the $k$-truncated SVD as shown here. I've read in ...
jonithani123's user avatar
1 vote
1 answer
99 views

How to derive the three matrices of SVD from eigenvalue decomposition in Kernel PCA?

Kernel PCA is usually done via eigenvalue decomposition of the Kernel Matrix $\mathbf{K}$ and standard PCA via SVD of the input $\mathbf{X}$. In standard PCA as far as I know we can derive $\mathbf{S}...
arod's user avatar
  • 23
2 votes
0 answers
239 views

How do ‘arpack’, ‘propack’ and ‘lobpcg’ compare to each other?

I want to implement singular value decomposition on a sparse matrix using the Scipy implementation. The documentation shows three solvers: ‘arpack’, ‘propack’ and ‘lobpcg’. How do these solvers ...
Emil's user avatar
  • 361
1 vote
0 answers
23 views

quasi-PCA reconstruction of the matrix by orthogonal basis

Let's say I have a "data" matrix $X$ of $N$ rows and $p$ cols with $N \gg p$. Now PCA with $L$ components can be formulated as $$ X_L = \underset{Y:rank(Y) = L}{\text{argmin}} ||X- Y||^2_F, ...
Vladimir Kirilin's user avatar
0 votes
0 answers
63 views

Is it logical to perform PCA and reduction on observations instead of features?

I am currently working with a set of code called GPMSA that was published by LANL. The code serves to create a Gaussian process model of some simulator and perform regression with experimental data. I ...
Stephen Wright's user avatar
1 vote
1 answer
55 views

Rotation-sensitivity of SVD

Suppose I perform a truncated SVD on a symmetric, PSD matrix $A \in R^{N \times d}$ (lowering the dimensionality from $d$ to $k$). Further suppose that there is a rotation matrix $Q$ such that some of ...
user1767774's user avatar
1 vote
0 answers
260 views

Optimal truncation in SVD

I am working with SVD on a matrix $$Y_{m,n} = T_{m,m} \Sigma D^T_{n,n} $$ where $T$ and $D$ describe the row and the column entities of Y, respectively. The truncated SVD takes the first $r$ ...
KArrow'sBest's user avatar
1 vote
0 answers
12 views

PCA: understanding how to use loading vectors of X to recapture the geometry of X

If we suppose that X is an n x d matrix, and then perform PCA to obtain loading vectors ...
Andrew Bell's user avatar
1 vote
0 answers
83 views

Does PCA centering guarantee projection optimality onto an affine set subject to dimensionality constraint?

Say I have $m$ points in $R^n$, not necessarily with sum 0, and I want to project them onto an affine set $S$ with $\dim(S)=d$ for a given $d$ so as to minimize the sum of the squared distances ...
Costin Oncescu's user avatar
0 votes
0 answers
83 views

Dot product and linear combination of Basis

I am currently going through the SVD intuition provided here. In the section "From intuition to definition", It says that, First, note that any vector $\textbf{x}$ can be described using ...
Ganesh Tata's user avatar
1 vote
1 answer
478 views

Use K-means clustering on SVD/PCA of data

In an assignment I was suppose to perform K-means clustering on the MNIST dataset (just the 0's and the 1's) and then use SVD/PCA to visualize the data in two dimensions. I missunderstood this and ...
user202542's user avatar
0 votes
0 answers
33 views

Why does reversing PCA from SVD follow this formula?

In this answer the author writes: PCA is very closely related to singular value decomposition (SVD), see Relationship between SVD and PCA. How to use SVD to perform PCA? for more details. If a $n\...
z611's user avatar
  • 255
0 votes
1 answer
41 views

Practical significance of number of singular values in SVD

I am working on a binary classification problem. SVD is used for dimensionality reduction and the vector with reduced dimension is used as the feature vector. DNN is used as the classifier. There are ...
user7080's user avatar
0 votes
0 answers
302 views

The discrepancy of results of PCA via Eigendecomposition vs via SVD in Python with scipy.linalg [duplicate]

I recently learned about different methods of PCA. I decided to manually implement PCA in Python with Eigendecomposition of cov(X) and the Singular Value ...
Alex.Kh's user avatar
  • 175
1 vote
0 answers
26 views

What does the SVD of the numerical derivative matrix of paired instances tell us about our data? [closed]

Let's say we have real-valued random variables $X$ and $Y$, and that under simple random sampling we obtain paired values $\{(x_1,y_1), \cdots, (x_i,y_i), \cdots, (x_m,y_m) \}$. From this sample we ...
Galen's user avatar
  • 9,412
1 vote
0 answers
171 views

How do I get the stationary distribution of a Markov chain matrix from SVD?

I have a matrix that represents a Markov chain. ...
Jane Wayne's user avatar
  • 1,399
1 vote
0 answers
131 views

Should you train/test split when using softimpute/matrix completion?

I am trying to impute missing values on a large dataset. After reading the paper(s) introducing matrix completion via soft-SVD thresholding, as well as the softImpute R package vignetter by Hastie (...
dwp's user avatar
  • 11
1 vote
0 answers
89 views

What is the difference between two forms of matrix coherence?

I am stuck on the definition of coherence of a matrix. Let $x_1, \dots, x_p \in \mathbb{R}^p$ be the columns of the matrix $X$, which are assumed to be normalized such that $x'_i x_i = 1$. Wikipedia ...
HJ Liang's user avatar
  • 132
1 vote
1 answer
111 views

Why is the first canonical direction equal to the left singular vector i.e. why is $w_1 = a = u_1$ in CCA (Canonical Correlation Analysis)?

I want to understand why the canonical direction $a$ is equal to the left singular value of $M = \Sigma^{-1/2}_X \Sigma_{X, Y} \Sigma^{-1/2}_Y$ and not $a = \sigma_1 u_1$. My calculation tell me that ...
Charlie Parker's user avatar
0 votes
0 answers
319 views

Singular value decomposition for a matrix with missing entries

Suppose $A$ is a matrix consisting of real numbers and nan's. What are some of the robust formulations and the associated algorithms for estimating its singular ...
Hans's user avatar
  • 1,005
1 vote
0 answers
26 views

Relation between low-rank approximation, nuclear norm of a matrix and Singular Value Decomposition

I'm reading the following paper https://arxiv.org/pdf/2005.10203.pdf which proposes improvements on robustness of large graphs to defend against adversarial attacks that are nothing but slight ...
James Arten's user avatar

1
2 3 4 5
9