7
$\begingroup$

In a number of singular value decomposition (SVD) applications, for example Latent Semantic Indexing, only the biggest singular values are used to make searches and calculate distances.

Are there useful applications that drop the biggest singular values and use only the smallest ones?

|cite|improve this question
$\endgroup$
  • $\begingroup$ For least-squares regression, the small singular values are much more important than the large ones. That's because the pseudo-inverse has singular values corresponding to 1 divided by the singular values of the original matrix. $\endgroup$ – Thomas Ahle Jul 18 at 13:26
5
$\begingroup$

It acts like a highpass filter in a slightly different space.

There is lots of linear data, and in many cases you are looking for that linear relationship, so a low-pass (high-blocking) filter lets you retain the important part.

For non-linear data, usually stuff that you have applied the simple methods without success to, the high-pass means that you throw away the unimportant (linear) part.

This makes me wonder about computational photography and edging. Thanks..

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Given that the whole idea behind PCA is that your data can be seen as linear combinations of your eigencomponents why would you do standard PCA to non-linear data? I see what you mean in terms of DSP (so +1) and I think you got that right but even then you do assume some stationarity etc. (Also we would not be talking not for the smallest eigenvalues but OK...) $\endgroup$ – usεr11852 Jul 18 '13 at 23:06
  • $\begingroup$ When preprocessing the PCA material it is important to "center then scale" the data. This takes out the central tendency. You could use this to take out the "linear combinations in multiple dimensions" component if it were accounted for elsewhere. How common is it to account for linear combinations of the major components? If there is high value in the higher "frequency" (or wavenumber or whatever) components - then this would generally select for higher value. $\endgroup$ – EngrStudent Jul 18 '13 at 23:56
  • $\begingroup$ Sorry but you are losing me more. Centring the data $X$ does not "take out the linear combinations in multiple dimensions" totally. Usually one computes the eigencomponents $\phi$ from the covariance matrix $C(t,s) = \sum_i \lambda\phi(t)\phi(s)$ (spectral decomposition). You subtract the mean to calculate the projections scores $A$ because you follow a generative model $X(t) = \mu_X(t) + \sum_i A_i \phi(t)$. The "major components" are orthogonal to each other, otherwise you would have non-identifiability. I agree with what you say about low/high-pass filters idea but not with your exposition. $\endgroup$ – usεr11852 Jul 19 '13 at 1:37
  • 1
    $\begingroup$ @user11852, sorry if I wasn't clear. I meant that in an analog to centering and scaling (which counts only for the expectation) one could also presume there to be parallel methods that account for the "linear combinations" and thus find value in removing them from the set and dealing with the remnants. Kind of a conceptual cousin of boosting. $\endgroup$ – EngrStudent Jul 19 '13 at 4:19
  • $\begingroup$ Small in distance becomes large in the FFT analog - wavenumber. $\endgroup$ – EngrStudent May 16 '16 at 12:10
8
$\begingroup$
  • Slow feature analysis (SFA) uses the smalles Eigenvalues of the covariance matrix of temporal differences to find the slowest features in a time series,
  • Minor component analysis (MCA) uses the smallest components in a probabilistic setting--here, not directions of variations are found but constraints,
  • Extreme component analysis (XCA) is a combination of probabilistic PCA and MCA,
  • In Canonical Correlation Analysis (where you analyse the correlation btw two different data sets), the smaller components of the correlation matrix correspond to so called "private" spaces. These represent the subspaces of each variable which do not correlate linearly with each other.
|cite|improve this answer
$\endgroup$
5
$\begingroup$

Total Least Squares regression (aka Orthogonal Distance regression) uses the singular vector corresponding to the smallest singular value of the augmented predictor/criterion matrix.

When there is only one dependent variable (i.e., when $k = 1$), both equation 12.3-5 in my Golub & Van Loan (first edition), and the final equation and Octave code in the "Algebraic point of view" section of the Standard account, use only the singular vector corresponding to the smallest singular value to get the vector of regression coefficients.

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ Could you expand on just how TLS "uses" the smallest singular vector? It looks like Standard accounts of TLS are the opposite of your characterization: the smallest singular values are zeroed out and effectively ignored in order to obtain the fit. $\endgroup$ – whuber♦ Jul 19 '13 at 14:13
  • 1
    $\begingroup$ You're right. I don't know what I was thinking :( $\endgroup$ – Ray Koopman Jul 19 '13 at 17:09
  • 1
    $\begingroup$ On the other hand, when k = 1, both equation 12.3-5 in my Golub & Van Loan (first edition), and the final equation and Octave code in the "Algebraic point of view" section of the page you linked to, appear to use only the singular vector corresponding to the smallest singular value to get the vector of regression coefficients. $\endgroup$ – Ray Koopman Jul 20 '13 at 2:18
  • 3
    $\begingroup$ @whuber, I believe Ray was right: total least squares (TLS) does "use" the smallest singular vector. This is because the regression hyperplane is spanned by all the singular vectors except the smallest one, and so the smallest one is orthogonal to the hyperplane and hence conveniently "defines" it. I have just posted an answer in another thread that extensively covers this relationship: How to perform orthogonal regression (total least squares) via PCA? $\endgroup$ – amoeba Feb 6 '15 at 22:19
  • 1
    $\begingroup$ @amoeba Thank you for the clarification. It comes down to how one understands "use." Typically we think of "using" principal components as generators of subspaces. The sense here (of defining an orthogonal subspace) is substantially different. So long as it is clear what is meant, there is no problem. I recall answering another thread a few months ago that advances the same argument you have here. $\endgroup$ – whuber♦ Feb 6 '15 at 23:08
3
$\begingroup$

Yes, there are. I'm currently working with a professor on a research project where we try to predict very short-term stock market changes based on real-time tweets from Twitter. Unfortunately, the majority of what people say on Twitter about the companies we are tracking is useless rambling. In other words, the largest singular values are useless.

Our plan is to use the biggest singular values to flag vast amounts of garbage tweets so that we can delete them. The remaining tweets and their text content (the small singular values) are candidates for a variable selection process.

We are trying find the needles in a haystack, and dropping the largest singular values is like setting the hay on fire.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ What an interesting idea! Are you going to publish a paper on this topic? $\endgroup$ – Sergey Aug 1 '13 at 11:17
  • $\begingroup$ We hope to publish the paper within about a year. I can post a link to the paper in the comments section here once it is ready. $\endgroup$ – Ryan Zotti Aug 1 '13 at 22:40
  • $\begingroup$ That would be great. $\endgroup$ – Sergey Aug 2 '13 at 10:56
  • $\begingroup$ @Ryan Zotti: Did that paper finnish? $\endgroup$ – kjetil b halvorsen Aug 21 at 8:05
3
$\begingroup$

It's a bit of a stretch, but consider the portfolio optimization problem: minimize $w^{\top} \Sigma w$ subject to $w^{\top} w \ge 1$. You can think of this as the minimum variance portfolio with an $\ell_2$ constraint. After applying Lagrange Multiplier method, you find that $w$ should be the eigenvector associated with the smallest eigenvalue of $\Sigma$. Since $\Sigma$ is typically the sample covariance $(1/N) \sum_{1\le i\le N} X_i X_i^{\top}$, where the $X_i$ have been centered, you can view this problem as an SVD computation where the singular vector associated with the smallest singular value is of importance. Like I said, it's a bit of a stretch.

|cite|improve this answer
$\endgroup$
2
$\begingroup$

There is an interesting LSA related paper which concludes that discarding (a lot of) first SVD features can improve results on semantic tests like TOEFL - "Extracting Semantic Represenations from Word Cooccurrence Statistics: Stop-lists, Stemming and SVD" (Bullinaria and Levy, 2012)

|cite|improve this answer
$\endgroup$
1
$\begingroup$

I'm not aware of any. The smallest singular values correspond to modes that don't contribute much to the reconstruction of the original matrix, or to use the PCA interpretation, don't describe much of the variance in the data. Typically, the modes with smaller singular values are just noise. This doesn't rule out the possibility that some meaning could be found in them, but I think it would be highly dependent on the data which make up the original matrix and — honestly — pretty unlikely.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ Reconstruction or the variance of the data is only one of many statistics that you might be interested in. $\endgroup$ – bayerj Jul 23 '13 at 6:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.