1
$\begingroup$

I got data that was generated from $n$ multivariate gaussians with different means and covariances, which means that they are separateble, and I'm tasked with classifying them using neural network in an unsupervised fashion, meaning without using the true labels.

I wanted to create a simple DNN and define a loss function which score the predictions of the model for the current batch, based on metrics like the clusters distances from each other, cluster covariances, etc.

Is there a canonical loss function I'm missing here? I couldn't find anything in the literature. Also, if I know the data is generated from different distributions (not gaussians), how it will affect the loss function?

$\endgroup$
2
  • $\begingroup$ There are internal and external indexes for clustering. Internal means that only the data but not any "true classification" are used, external indexes measure how close a found clustering is to a reference ("true") one. Your question looks as if you want internal ones, but please confirm! $\endgroup$ Commented Jul 30 at 23:02
  • $\begingroup$ I can confirm. This task is about unsupervised clustering, which sounds like internal indexes. $\endgroup$
    – galah92
    Commented Jul 30 at 23:11

1 Answer 1

1
$\begingroup$

This is a very tough question. To some extent the concept of "loss" is really connected to supervision. Standard loss would compare predictions to true values/classes, but in unsupervised classification/clustering you don't have these.

So "loss" is probably a misnomer for what you need. I think what you are looking for is found in some literature under the name "internal cluster validity index"; these are data-based measurements of the quality of a clustering.

In the absence of a "true clustering" we should acknowledge that clustering is not so much "prediction" but rather construction. The question to address is what characteristics of a clustering are desirable in the given application? There cannot be any guarantee that this coincides with a "true clustering", because no information of such a "truth" is used.

Note that this will depend on the application, and there is no unique way of addressing this. To illustrate this: In some problems meaningful clusters are connected to density modes. However, in other applications it is of vital importance to have all within-cluster distances small as far as possible (with a density mode based cluster concept within-cluster distances can be as large as any distance you find in your data set). There are also applications in which you want to interpret clusters in terms of the marginal variables, which is helped if clusters are locally independent or at least aligned with separated intervals on one or more variable. Also there is no well defined optimum regarding balancing small within-cluster distances against large between-cluster distances when it comes to deciding about the number of clusters ("granularity of solution").

This question has excellent answers, particularly by @ttnphns, explaining many existing cluster validity indexes. Due to what I said above, there is no single "optimal" one; the Silhouette index/Average Silhouette Width is probably the most popular one these days.)

Note that all of these have issues. The problem is just very difficult.

On top of that, in the context of Gaussian mixture modelling, the BIC (Bayesian Information Criterium) is often used to assess the quality of the fit of a specific Gaussian mixture to the data. Recently I see more uses of the ICL for this task; the ICL demands more separation from clusters. Both are explained in this paper.

You could still compute these criteria from your network fit as long as you can retrieve mean, covariance matrix, and component proportion estimators (you could compute them from your clustering). This will rely heavily on the Gaussian mixture assumption though.

My own take on cluster validation indexes is here, published version here, also Chapter 26 of the Handbook of Cluster Analysis.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.