Timeline for Avoiding cluster recalculation in large scale clustering
Current License: CC BY-SA 3.0
5 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Oct 14, 2015 at 20:59 | answer | added | Has QUIT--Anony-Mousse | timeline score: 0 | |
| Oct 14, 2015 at 13:33 | comment | added | user78229 | Can you define "some" dimensions? How many are you talking about? Also, you can simplify your life by taking samples of the 500 million records and projecting results to the remainder. This would also afford an opportunity to cross-validate the results in terms of classification (or misclassification)... | |
| Oct 14, 2015 at 13:33 | comment | added | Ben | I would need some more details about your problem to say for sure, but many clustering algorithms do their clustering based on the distance between items. If that's how clustering will work here. The easiest way it probably to just save the distance matrix between runs. When data points change, you can recalculate the distance between the $m << n$ points that change with all $n$ points. If you use a clustering algorithm like $k$-means, you can initialize today's clustering with yesterday's centroids to speed it up. | |
| Oct 14, 2015 at 13:27 | history | edited | mzm | CC BY-SA 3.0 |
added more info
|
| Oct 14, 2015 at 13:04 | history | asked | mzm | CC BY-SA 3.0 |