4
$\begingroup$

I want to use dimension reduction method for a high dimensional data set Is there any possible way to assess the "non-linearity" of the data first to give me the insight of whether I should use linear method (e.g. PCA) or nonlinear method (e.g. NLPCA)? I read several literatures and neither of them could provide a justification of when should I use linear of nonlinear method. They always use examples of "swiss roll" to indicate a data set which I should use nonlinear method. But for practical case, I cannot visualize a high dimensional data set first before I choose the method. To put it more straightforward, I want to have a test of the data set first and the result will tell me whether the nonlinear method is more suitable for this data set or vice versus.

Thank you!

$\endgroup$
6
  • 1
    $\begingroup$ "But for practical case, I cannot visualize a high dimensional data set first before I choose the method." - I don't get this part. What do you mean you can't visualize the data first? For visualization purposes 2 or 3-dimensional reduction is used, so if you can't run this, you won't be able to run anything using more dimensions. $\endgroup$ Sep 6, 2017 at 20:01
  • $\begingroup$ @JakubBartczuk Hi Jakub, in my opinion, using dimension reduction will change the original form of the data. Some data will look linearly after dimension reduction but it is nonlinear in the original space, thats what I am saying here. $\endgroup$
    – vicky
    Sep 6, 2017 at 23:37
  • 1
    $\begingroup$ The problem with this question is that there is no such single thing as 'nonlinear structure'. Different manifold learning algorithms optimize for different things, and it seems like there is no single one to outperform the others. $\endgroup$ Sep 9, 2017 at 10:10
  • $\begingroup$ I've had this question for years and still cannot find a clear answer which is general. I always get over simplified examples which do not reflect most real world cases... $\endgroup$ May 28, 2020 at 9:44
  • 1
    $\begingroup$ I often do both and make a comparison between the two. $\endgroup$
    – Galen
    Jun 23, 2021 at 21:26

1 Answer 1

1
$\begingroup$

An example in sklearn.manifold documentation that compares the methods on MNIST digits dataset. It seems that nonlinear methods give more meaningful reduction - you can see clusters corresponding to digits.

As for when to use nonlinear methods - it depends what you want to use. Using PCA or something similar first is a good idea, as it's more scalable than manifold learning methods. If you're unsatisfied with PCA's results, you can try running nonlinear methods, if they can be actually used (they are not as scalable as basic matrix decomposition methods).

For dimensionality reduction there is also another alternative to PCA - some of its generalizations can incorporate prior knowledge (or, in other words, use regularization). Some methods are even designed to work with mixed data (both categorical and continuous) - see PCA on a Data Frame and Generalized Low Rank Models.

$\endgroup$
1
  • $\begingroup$ This answer has nothing to do with the question $\endgroup$ May 28, 2020 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.