2
$\begingroup$

I am quite confused regarding the evaluation of data stream clustering algorithms.

Assuming I have some data stream (finite for now) $(X_1,X_2,...,X_n)$ with labels $(Z_1,Z_2,...,Z_n)$, where each $X_i$ is a batch arriving at time $i$.

I perform data stream clustering, and for each batch, I now have prediction $Y_i$.

So now how do I evaluate this? I can compare each $Y_i$ with its respective $Z_i$, which could work, but then I am unable to evaluate the clusters' consistency between batches.

I could concatenate the $Z$ and $Y$ vectors and compare them, but many clustering algorithms do not produce consistent labels between batches (e.g. Birch, CluStream).

Evaluating against new test data makes little sense in data streams with concept drift.

$\endgroup$

1 Answer 1

1
$\begingroup$

Self answering for future reference:

Given some horizon $h$, and stream algorithm $S$ trained on $\{X_1,...X_{i-1}\}$ batches of data, each of size $h$, we will predict the next $h$ data points ($X_i$), creating prediction $Y_i$ (where $|Y_i| = h$), and compare it to $Z_i$. Only then we update $S$ using the next $h$ points ($X_i$).

The most standard comparison metric seems to be ARI, but others (Purity, fmeasure, NMI) are also common.

The above procedure gives us a comparison for each batch, we can leave it at that, or take the mean and std.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.