0
$\begingroup$

This question already has an answer here:

From https://jessesw.com/Rec-System/ says

98.3% of the interaction matrix is sparse. For collaborative filtering to work, the maximum sparsity you could get away with would probably be about 99.5% or so. We are well below this, so we should be able to get decent results.

Why is 99.5% ?

$\endgroup$

marked as duplicate by mdewey, Sycorax, kjetil b halvorsen, mkt, Jeremy Miles Dec 10 '18 at 5:07

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ 99.5% is not an accurate number. It's just to emphasis that the matrix used for collaborative filtering is very sparse. Imagine building a CF recommender system for merchandise and customers on Amazon based on order history, we can say for sure that an average customer has purchased less than 0.5% of all merchandise on Amazon. $\endgroup$ – user12075 Sep 18 '18 at 4:08
1
$\begingroup$

The 99.5% is based on experience in the specific data set.

Collaborative filtering includes both matrix factorization (MF) and KNN methods. MF methods are designed to cope with sparsity and therefore can possibly work even with such a sparse data set. On the other hand KNN methods would not be so tolerant in sparsity and in several cases a representative amount of neighbors would not be extracted.

Overall don't try KNN with 99.5 but use MF. A crisp sparsity rule cannot be said since this is a matter of data quality and validation error.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.