4
$\begingroup$

We have 128-dimensional vectors representing people's identities where the euclidean metric defines the similarity between them.

Ours solution requires them to be clustered and then annotated (assign real identities to the clusters). This procedure has the purpose of reducing the amount of work necessary to annotate every object - the amount of work needed to annotate objects drops to one fifth since we have the groupings obtained from clustering.

One pass solution with DBSCAN works well, but we would like to do clustering on the fly (gradually). Upload a batch of data, cluster it, annotate it, upload additional batch of data, cluster the additional batch (add them to existing clustering), annotate … If we cluster whole data after new batch then we lost the annotated identities. Is it possible to implement such a solution? Furthermore is it possible to be solved in deterministic manner (no matter on an order of the batches to everytime get the same solution)?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Clustering in high dimension is is hard (Klawonn et al, 2015, "What are Clusters in High Dimensions and are they Difficult to Find?"). You seem to be looking for online clustering; I'll add the tag, and previous questions with both tags may be helpful. However, I wouldn't expect there to be an online version that gives results which are invariant with respect to the order of observations. $\endgroup$
    – Stephan Kolassa
    Commented May 24, 2018 at 11:12
  • 1
    $\begingroup$ Sounds like a tough question, but neither off-topic nor unclear. $\endgroup$
    – Jan Kukacka
    Commented May 24, 2018 at 13:16

1 Answer 1

Reset to default
0
$\begingroup$

Adding data to DBSCAN suffers from as parameterization problem: the new data can cause clusters to merge, but not to split. Add enough data, and eventually everything will be connected. To prevent this, you would likely need to decrease epsilon and increase minPts over time.

But why don't you just label clusters based on the data they contain? If 80% are labeled the same, assume that the remaining 20% should also be labeled? Or ask the user to verify the current labels, etc. There are many possibilities to reuse the labels from a previous run.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.