Turns out there's also a bug in PySpark that has been fixed that relates to the way the average test metric was being calculated (it always looked weird to me, now I know why!). issues.apache.org/jira/browse/…

This question I asked (and got no responses) about AUC in Spark is related to the comment above (and perhaps sheds some light on my question here): stackoverflow.com/questions/39516668/…

It only returns a "training AUC" for the best model selected by CrossValidator. This is what I have listed in the left-hand column for 10 models trained in this way (i.e. me running a script 10 times). The test AUC is me taking the best model from each cross-validation run and testing it on a hold out set (10%) that was randomly selected before cross-validation.

I think you're misunderstanding my question. These are not the results from one run of cross-validation. The above results are from repeated runs of cross-validation each of which is 10-fold. So the "software" (Spark in this case) has already made a selection, 10 times in this case, and the results above are for that selection (i.e. 10 different models each selected by CV).

I'm not asking what cross-validation is. You may be misunderstanding my question. Each result in the table I listed is from a different model, each one of which is created using cross-validation. My question is more about model selection, not parameter optimization.

stackoverflow.com/questions/3822535/… That looks like it will answer the question that was asked. Whether the OP should do this is another question entirely.

@FabianWerner I believe the solution did not update the word-topic matrix from the original training. It just re-runs a Gibbs sampler starting with the trained word-topic matrix, and creates a new document-topic matrix. At any rate, do you know of another way to do what the OP asked (admittedly several years ago)? I'm looking at the same problem.