1
$\begingroup$

Assume I have a set of N vectors with M features each. If I want to create a latent space to project these vectors into, there are a variety of techniques available to me, such as Principle Component Analysis (PCA).

However, if I were using PCA and I decide to only consider x of M features for a latent space, I would then have to carry out PCA again for the N vectors with x features each.

Is there an unsupervised learning technique which allows for training with N vectors of M features, but allows for selection of a subset of M features with completely recomputing the space? Is my only option to carry out PCA for all possible subsets of M which might be considered and cache the results? Is there a different term for what I'm trying to do?

|cite|improve this question
$\endgroup$
  • $\begingroup$ Do you mean, run PCA on say 50% rows and recompute for 100% rows? or Use 50% columns and then recompute for 100%? $\endgroup$ – Nishad Aug 24 '18 at 18:35
  • $\begingroup$ When you run PCA on M features, you get M components. Then you choose say top 10% components which make up say 90% of the variance. So you can always choose more or less (no need to recompute). For rows, you can run on say 50% data and use the learned representation to transform the rest of 50% or recompute with 100% for complete representation. $\endgroup$ – Nishad Aug 24 '18 at 18:38
  • $\begingroup$ @Nishad I mean use 50% columns and recompute for 100% of the data. I understand PCA means you choose the top y components, but if I leave components out of the initial computation, does that not change the values of the y components significantly? $\endgroup$ – Seanny123 Aug 24 '18 at 18:43
  • $\begingroup$ Yes, you are right. You will have to recompute if you are adding new columns as the projections are based on representing entire feature space optimally. Same thing with SVD which is another popular technique. $\endgroup$ – Nishad Aug 24 '18 at 18:51
1
$\begingroup$

You need to take a time out and think about the $M$ features you already have, since your OP basically starts with shoving a lot of stuff in an oven and expecting a pizza to come out when done. For example, when performing micro-surgery on your features -- before you do anything with them -- you need to determine whether:

  1. the features were already selected from a class or function approximation prediction analysis
  2. the features represent everything known about a system, i.e., all possible measurements including signals and noise, and PCA would be used to learn about groups of correlated features because I don't know anything about the data.

Firstly, what's the scale and range of all the features, and what's the correlation between the features? If the data are merely all noise, then the correlation would be degenerate (but not zero, orthogonality for noise data only occurs as $n \rightarrow \infty$). PCA is not intended for all possible subsets or random sets of features to understand the spectral information of a set of data. You could always develop something that would analyze the data ad nauseam via random resampling, but the result would be like hyperemesis $\rightarrow$ projectile vomiting.

The beauty of PCA stems from eigendecomposition, and there have been many theoretical aspects of eigendecomposition that are still unexplored in applied numerical analysis.

|cite|improve this answer
$\endgroup$
0
$\begingroup$

You might be looking for something more like independent component analysis (ICA). The goal there is to find the statistically independent components in a latent space that generated the observed features. In the standard, linear version of ICA there are just as many independent features found as features in each sample vector, but there are generalizations of ICA to the nonlinear case which act in a manner similar to an autoencoder and can produce latent factor spaces of lower dimension than the original feature space. This may be related to factor disentanglement or for a linear alternative to low-rank representations, depending on your overall goals and linearity requirements.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ My problem isn't the need to project to a lower-dimensional space, but to accept input into my "reducer" of varying dimensionality. $\endgroup$ – Seanny123 Aug 24 '18 at 19:05
  • $\begingroup$ So, how are you varying the dimensionality? Is there a particular criterion for selecting the particular x of M features that is being satisfied before you feed the result to your reducer? $\endgroup$ – Don Walpola Aug 24 '18 at 19:09
  • $\begingroup$ The set of x is set by a user, so there's no way to determine it ahead of time. However, for the sake of practicality, you can assume x > 3, since otherwise it wouldn't make much sense to do reduction. $\endgroup$ – Seanny123 Aug 24 '18 at 19:11
  • 1
    $\begingroup$ Well, if the value of x is completely arbitrary, and the particular features selected are a random subset of x from the M features, then I'm not sure of any recourse other than your original idea of just having to recompute the PCA. If you wanted a constraint similar to PCA where you want the lower rank to capture the most variance, SVD is a very slight generalization of PCA that gives you a projection onto the lower rank space. If you want something similar to PCA but that operates on an arbitrary similarity matrix rather than the covariance matrix, then spectral clustering might work. $\endgroup$ – Don Walpola Aug 24 '18 at 19:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.