Why does `sklearn`'s validation curve return test scores in unsupervised learning?

Question

I'm using sklearn.model_selection.validation_curve to determine $k$ for my $k$-means clustering. For inputs, the y parameter is optional, which makes sense for unsupervised learning.

y : array-like, optional, default: None

The target variable to try to predict in the case of supervised learning.

So, I just pass None for y.

train_scores, test_scores = validation_curve(KMeans(), X=X, y=None, param_name='n_clusters', param_range=k_range, 
                                             cv=10, n_jobs=3, verbose=2)

The function still returns train_scores and test_scores. I'm having trouble wrapping my mind around this in an unsupervised learning context - my 10 fold CV for each value of $k$ will produce training vs. test folds, but what are the use of these folds if there is no "correct" answer? I get something like below if I plot both train & test, and arguably they're both indicating different $k$ at the elbow.

1. If there's no "correct" answer on which to validate in unsupervised learning, what is the use of test scores?
2. If there is a use to test scores, which would be more appropriate to use for choosing optimal $k$ via the elbow method - the train or test scores, and why?

Answer 1

sklearn is, unfortunately, very much built with a supervised mindset. This shows in many is places, such as the y variable even being present at all in unsupervised "learners" (that don't learn either) fit(X, y=None) methods. Not only is the default None - it will also be ignored by k means if set to anything else.

Normally, you wouldn't do cross validation on unsupervised methods at all - because of you don't have labels, how could you validate?

One of exception is kmeans. Because k-means can actually "predict" cluster numbers for new objects (i.e., find the nearest center), you can perform some kind of validation here: if the model is good, you'd expect unseen objects to have a similar distance to the means as seen objects. If the model is overfitting the training data, then you'd expect the distances to be much smaller for training than for test.

Now the plot you have has some issues: (1) it's inertia, not average inertia; the training setnis much larger than the test set and thus the inertia will be several times larger (more precisely, 9 times with 10x CV). (2) the values are expected to drop with k, hence the plot will have this drop-off curve that by itself does not mean a lot. It would likely make more sense to look at average inertia / (k-1)... Or the quotient of test vs. train average inertia. But even with these modifications I doubt you'll get some useful insights. First of all, the shape of the curve already suggests rather poor performance (no clear "elbow", but rather the exponential drop you get on random data), and secondly the test average inertia will likely always be quite a bit bigger than the average training inertia - so when does it overfit, when not?

Answer 2

validation_curve() is "similar to grid search with one parameter" and "merely a utility for plotting the results". Not a lot of intelligence is built into the function if you look at its code.

In particular, the code does not distinguish between supervised and unsupervised estimators. It simply executes cross-validation, and plots the results for test- and training-set.

1. For unsupervised, the sets still make sense in the sense that test scores are an estimate of how your model performs on unseen data.

2. The test scores, since they tell you more about how your model generalizes.