0
$\begingroup$

From https://jessesw.com/Rec-System/ says

98.3% of the interaction matrix is sparse. For collaborative filtering to work, the maximum sparsity you could get away with would probably be about 99.5% or so. We are well below this, so we should be able to get decent results.

Why is 99.5% ?

Improve this question
$\endgroup$
1
  • $\begingroup$ 99.5% is not an accurate number. It's just to emphasis that the matrix used for collaborative filtering is very sparse. Imagine building a CF recommender system for merchandise and customers on Amazon based on order history, we can say for sure that an average customer has purchased less than 0.5% of all merchandise on Amazon. $\endgroup$
    – user12075
    Sep 18, 2018 at 4:08

1 Answer 1

Reset to default
1
$\begingroup$

The 99.5% is based on experience in the specific data set.

Collaborative filtering includes both matrix factorization (MF) and KNN methods. MF methods are designed to cope with sparsity and therefore can possibly work even with such a sparse data set. On the other hand KNN methods would not be so tolerant in sparsity and in several cases a representative amount of neighbors would not be extracted.

Overall don't try KNN with 99.5 but use MF. A crisp sparsity rule cannot be said since this is a matter of data quality and validation error.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.