2
$\begingroup$

I am trying to reduce the dimensionality of a data set of about 100'000 rows and 1'000 columns, in order to cluster the individual observations with k-means. I tried PCA with rescaling (i.e., subtracting the mean and dividing by the standard deviation), but I am not sure this approach makes much sense, because

  • The majority of the variables are not normally distributed (i.e., they follow exponential distributions, or other skewed distributions)
  • Many variables are 0/1 flags, and most of them are very sparse (i.e., 99.9% of the data is 0, and 0.1% is 1)
  • There are many outliers, and it is not always clear if the best way to remove an outlier is by removing the corresponding column or row

Is there a better way to reduce the dimensionality than PCA? I also tried linear mapping each variable in the interval [0,1] instead of mean subtraction/sigma division rescaling, and I even tried substituting some variables with the corresponding deciles, but then again I don't know if the combination PCA+kmeans is the best way to perform the clustering in this case.

Improve this question
$\endgroup$
4
  • $\begingroup$ There are multiple implementations of Principal Component Analysis. You should look to find one that uses a same data model as you need. However, I think you need to also look at why you want to use PCA or reduce dimensionality? And after you reduce, then what? $\endgroup$
    – Jon
    Commented Oct 31, 2017 at 18:05
  • $\begingroup$ If your set of continuous variables tend to have distributions, you can try applying nonlinear transformations to them to make the distributions more normal-- e.g., sqrt or log. This makes them more amenable to PCA. You may consider attempting PCA on the subset of continuous features separately. Regarding doing PCA on binary features, see this thread. $\endgroup$
    – the higgs broson
    Commented Nov 1, 2017 at 2:20
  • $\begingroup$ Look into homogeneity analysis, homals: gifi.stat.ucla.edu/janspubs/2007/reports/deleeuw_mair_R_07b.pdf $\endgroup$
    – kjetil b halvorsen ♦
    Commented Nov 1, 2017 at 17:43
  • $\begingroup$ IMHO, the use of PCA on such data is highly questionable. You can do it, but I doubt you will find a statistical model of what this optimizes if even just does. To my understanding, PCA should only be used on interval scale attributes. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Nov 2, 2017 at 19:52

2 Answers 2

Reset to default
1
$\begingroup$

There is a paper PCA on a DataFrame that seems trying to solve this problem. The technique used here is called collectively Generalized Low Rank Models (PCA and Sparse PCA are examples of this family of methods).

If you are familiar with Python/R you can try to use GLRMs from H2O library. They can handle both categorical and continuous data in single row.

Improve this answer
$\endgroup$
-1
$\begingroup$

Dimensionality reduction methods like t-SNE or Diffusion maps can be very effective because they integrate local distances.
If two points differ at a feature by a logical $1$, then this information is fully taken into account; the result won't be affected by the distribution of $1$'s over the entire data set. Also, you are free to normalize your data as you like. (Note however, that how you choose to normalize your data affects the way distances are measured between your data points.)

These non-linear methods require more computation time, but in my experience I think $100,000$ points and $1,000$ features will be able to be done within $1$-$2$ days.

Improve this answer
$\endgroup$
4
  • $\begingroup$ You didn't mention the problem of finding hyperparameters. If one evaluation of t-SNE runs for 1 day, then you'll probably need a couple of weeks to find reasonable hyperparameters. $\endgroup$
    – Jakub Bartczuk
    Commented Nov 1, 2017 at 17:40
  • $\begingroup$ tSNE needs a good similarity before you can use it. On data as he has, tSNE doesn't work without solving his problem first. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Nov 2, 2017 at 19:50
  • $\begingroup$ @Anony-Mousse Any dimension reduction method requires a notion of similarity between points. In the case of PCA, this similarity is determined from the way the matrix is normalized. $\endgroup$
    – Open Season
    Commented Nov 2, 2017 at 23:45
  • $\begingroup$ I wouldn't call PCA to be similarity based. It's maximizing variance. But of course, variance is related to Euclidean distance, and the rotations performed by PCA only make sense in Euclidean geometry. But as you can see above, I do not suggest to use PCA on such data either. $\endgroup$
    – Has QUIT--Anony-Mousse
    Commented Nov 3, 2017 at 7:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.