5
$\begingroup$

I hope this is an appropriate forum for this question...if not, any pointers on a place to ask would be great. If my questions is not clear, please just let me know and I'll try to add information/explanation where I can.

Say I have a set of data points that I run through a FLAME clustering algorithm, and I get my set of cluster supporting objects, and the fuzzy memberships among them.

Next, let's say I get an additional set of data, for argument's sake, let's say it doubles the amount of data. If I want to add that to my clustering, without having to reprocess the old data (i.e. just using the cluster supporting objects, and adding these new observations to the cluster), will I be introducing some kind of bias? To get the most "accurate" picture, would it be better to reprocess ALL the data, re-establish CSO's, re-compute distances, etc.?

To add to the complexity, let's say I'm going to be periodically adding similar quantities of data points. Does the answer to the above change?

And I guess a final question - is there a clustering algorithm (or family of algorithms) that is not subject to extreme bias in the beginning, such that added data points can be added relatively pain-free? I'm sort of thinking that FLAME clustering, with its nice fuzziness properties is a reasonable way of going about this, but any more suggestions would be helpful.

Thanks!

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ @awshepard: Try stats.stackexchange.com. They also cover data mining, which I would say this falls under. Also, a "which algorithm should I use" question typically requires a lot more information about the size and shape of your data than you've given here. $\endgroup$
    – Larry Wang
    Commented Aug 27, 2010 at 22:47
  • 2
    $\begingroup$ What you want to do is called "online clustering". An "online" algorithm in general means you can add data to it incrementally without retraining from scratch. FLAME is not online; you can add data incrementally to it, but in general the solution will be different than if you retrained. I don't know what would be a standard online clustering algorithm. People often use custom hacks, e.g. add incrementally for 10 iterations, then retrain. $\endgroup$
    – SheldonCooper
    Commented Mar 17, 2011 at 21:23

1 Answer 1

Reset to default
1
$\begingroup$

I think it is possible to introduce the new set of data periodically by assigning its initial value with the interpolated value of its neighbor while the possible set of neighbor for each new data point could be the points in the triangle that data point resides in.

I think what is more important is the order of assigning the initial values for this set of data point since the data point that is closer to CSO will be updated first. To determine the closeness of data point to CSO, we can use Euclidean distance, weighted link or both.

Note: something to think about

  1. Data point that lie outside the contour of current data set.

  2. The number of update recurring. The data point tends to converge to CSO as the number of update increases. This can cause saturation problem. To solve this problem, subsidizing current data points or levitating new data points is necessary.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.