I have a highly dimensional sparse dataset composed of 2.5 million of examples as follow :


Each example of this dataset is of dimension (360,280,18). Examples are very sparse and real-valued. values are between 0 and 1, we have 0 values, values which are near to 0 such as 0.000000000000000000000001245 and high values such 0.8. At each of the 18 dimensions we have at most one value which is high and the remaining either are very near to 0 or equal 0. Hence for each example we have at most 18 values which near to one and the remaining are very small of 0 values. In average in each example of dimension (360,280) we have at most 500 values which are different from zeros.

What l want to do ? l would like to reduce the dimensionality of my dataset and encode these examples as smaller dense vectors.

What are my ideas ?

1)Kernel PCA. However, it seems that my dataset is so huge to fit into memory in order to ouput the K first components that cover 95% variability of my data. To apply PCA on my dataset, we transform dataset_dimension=[2500000,360,280,18] to dataset_dimension=[2500000,360*280*18] where 360*280*18=1814400. It is a very high dimensional sparse vector. It is caracterised by a spartsity of about 500/1814400.

Is sickit learn package able to apply PCA on a dataset of [2500000,1814400] ?

2) Apply an autoencoder and extract the intermediate representation (low level representation , let's a 2048-vector dense representation). However, since my input are at the same time : - In a very high dimension : 1814400 - with a large sparsity : 500/1814400 - values are either equal 0 or few of them near to 0 and the remaining are near to 0 such as 0.0000000000000001245, 0.0000000000000000000000045. Is autoencoder able to learn a dense representation from this kind of inputs ? Is autoencoder not sensible to high dimensional and large sparse float-valued vectors ?

Thank you for your consideration

  • $\begingroup$ Your small, positive values are awfully close to double-float machine precision. Are you sure that they are not merely the result of accumulated numerical error? $\endgroup$
    – Sycorax
    Aug 10 '18 at 21:57
  • $\begingroup$ no they are not. I can ignore them and turn them to 0. because for me the most important values are those > 0.1 $\endgroup$
    – vincet
    Aug 10 '18 at 22:00
  • $\begingroup$ I think you have a slightly issue that you really want sparse PCA rather than kernel PCA. Just doing kernel PCA will still return large dense eigenvectors and scores. That said, for a first step you might want to use something like RSpectra that uses sparse matrices of class dgCMatrix and dgRMatrix. $\endgroup$
    – usεr11852
    Aug 11 '18 at 0:27

Since your matrix has nonnegative values, it might make sense to use nonnegative matrix factorization. There exist toolkits that are able to perform it for sparse data (for example this one and BTW the results can also be constrained to be sparse). Actually if you dig into NMF you'd see that basic algorithms for that problem use direct matrix manipulation, which means there shouldn't be problem with sparse matrices.

If you run into problems with scalability, you can always try to sample a training set and then evaluate results of dimensionality reduction on the rest (just like you'd do in supervised learning, where you train and then are more interested in the test set error).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.