- You are right to be cautious in the described situation, and also that
- proper validation is needed.
- However, splitting off a test set in addition to the cross validation would not be sufficient to get such a proper validation in the described situation.
I understand the situation where we perform cross validation. We keep a split of the data aside never to be touched during cross validation, and test the model obtained from cross validation on the test set. My understanding is that this is done to evaluate and test the generalizability of the model, but point to note is that the test set also has the labels.
No: cross validation in itself doesn't give you a model. Cross validation does what the single split does (estimating generalization error), with two differences:
- it uses more than one split into training and test,
- cross validation error is used as approximation for generalization error of the model built on the whole available data, whereas for a single split often the model on the training subset is kept (but not necessarily, sometimes that error is used like cross validation error as approximation for the generalization error for the model built on the whole data set).
A second, independent estimate of generalization error is needed iff the first estimate is used during training (e.g. for hyperparameter tuning). And that 2nd estimate can again be e.g. cross validation (see nested aka double cross validation) or a single split (or out-of-bootstrap or a validation study, ...)
If no such hyperparameter tuning takes place, the additional single split does not give any advantage over the cross validation.
I’m running into a scenario where someone created a model from dataset (n=1068) and performed cross validation without splitting the data into training and testing set.
(as explained above, cross validation does split into training and testing sets, repeatedly)
So, unless there was data-driven model selection/hyperparameter tuning, this is as it is supposed to be (subject to the further caveat of achieving independent splits with confounders such as geographical region, see below).
Then the model was used to classify a new set of data (n=918) [...] but this does not have the labels
... which was probably the purpose of the model in the first place. After all we build predictive models in order to get predictions for cases where reference analyses (other ways of measuring the predicted outcome) are fundamentally or practically unavailable.
So, in that situation, taking other variables to do some plausibility checks (which is how I understand
classification was then evaluated based on another variable.
) may be the best that was possible in the given situation.
Contrary to your understanding, this may (should) not be intended to replace a proper validation but may be instead what is good prediction practice: to check alongside doing prediction that the cases to be predicted do fit into the calibrated model space.
That being said, such checks should also be part of a proper validation. But of course they cannot replace the comparison of predicitions with reference values.
Now, a crucial point you describe is that there are other known and potentially confounding factors in which the predicted data differs from the (whole) training set:
from a geographically different area,
There are different possibilities to deal with this:
- If the training data covers multiple geographic areas, the cross validation (or any single train/test split) should split so that predictions for unknown geographic areas are tested.
Iff that was done, the cross validation estimate is a legitimate approximation for prediction error when encountering data from unknown geographical regions and can directly be used for the predictions of the really unknown (unlabeled) data.
If the training data does not cover multiple geographic areas (or the splitting did not produce independence wrt. geographical region), only prediction error for the known regions can be estimated by cross valiation. But an additional train/tetst split would have the same problem here.
I.e., that estimate of generalization error has an optimistic bias of unknown size.
In this situation, any further information that helps judging whether crucial differences between the geographical regions have to be expected or may be expected to be negligible will be valuable even though they do not provide a proper method validation, and here the comparison of those other variables comes into play: if factors that are known to be correlated with the task are alike, using the model to predict the new data isn't as risky as if those factors show marked differences.
Whether this is appropriate for a scientific paper IMHO hinges crucially on whether the paper is clearly aware and openly discussing this limitation. Iff that's the case, I'd be much more relaxed. After all, science often evolves incrementally - the question then would be whether the paper has sufficient substance to merit publication without a validation for the new geographical region.
Two more points to keep in mind:
People do and publish preliminary studies in order to have a basis on which to ask for funding for the appropriate validation/extension of the studies.
Requiring that the paper may not be published without proper validation means that we restrict the possibility to present preliminary findings and conclusions for grant applications that did already undergo peer-review.
Splitting a given data set into training and testing will still be what is called internal validation in clinical diagnostics/analytical chemistry. In other words, such a split is in many cases not sufficient for a full method validation. Depending on the situation (and how important deep insight into the performance is, e.g. in terms of diagnosing patients and the dangers of misdiagnoses), validation of an already built model may require whole studies on its own, checking performance for many sub-populations (other regions, sex & gender, age, lifestyle, co-morbidities, ...) etc.
So testing the predictions on unknown regions is one crucial factor that is obvious from what you wrote, but it is probably only one of many such factors that need to be taken into account.
