1
$\begingroup$

I understand the situation where we perform cross validation. We keep a split of the data aside never to be touched during cross validation, and test the model obtained from cross validation on the test set. My understanding is that this is done to evaluate and test the generalizability of the model, but point to note is that the test set also has the labels.

I’m running into a scenario where someone created a model from dataset (n=1068) and performed cross validation without splitting the data into training and testing set. Then the model was used to classify a new set of data (n=918) from a geographically different area, but this does not have the labels. The classification was then evaluated based on another variable.

Is this okay to do even though this is validated on two geographically separate datasets (n=918; n=216), but there are no accuracy or performance measurements? We can only tell how well the model performs in classifying new dataset, and its really good.

This is an academic paper, so: 1) Should we not need the the test set to evaluate the model in this case? Validation is done through other sets without labels)

$\endgroup$
  • $\begingroup$ I am confused. How did this person create a model and then performed CV? And then tested said model on a separate test set which did not have labels? How did this person assess the performance then? What is this other variable and how does it fit in? $\endgroup$ – user2974951 Nov 26 '19 at 9:45
  • $\begingroup$ Exactly!! That's my problem too, and being a novice I'm asking to understand whether this method is actually valid. Now, there's definitely some "domain knowledge" involved in assessing the validity of the model in the external dataset, but I still struggle to understand how the model performance was evaluated!! It doesn't make sense to me and was wondering if somebody could help me understand. I need to first make sure that I'm correct in my thinking before raising question to the person who built the model. $\endgroup$ – user1828605 Nov 26 '19 at 13:33
2
$\begingroup$
  • You are right to be cautious in the described situation, and also that
  • proper validation is needed.
  • However, splitting off a test set in addition to the cross validation would not be sufficient to get such a proper validation in the described situation.

I understand the situation where we perform cross validation. We keep a split of the data aside never to be touched during cross validation, and test the model obtained from cross validation on the test set. My understanding is that this is done to evaluate and test the generalizability of the model, but point to note is that the test set also has the labels.

No: cross validation in itself doesn't give you a model. Cross validation does what the single split does (estimating generalization error), with two differences:

  1. it uses more than one split into training and test,
  2. cross validation error is used as approximation for generalization error of the model built on the whole available data, whereas for a single split often the model on the training subset is kept (but not necessarily, sometimes that error is used like cross validation error as approximation for the generalization error for the model built on the whole data set).

A second, independent estimate of generalization error is needed iff the first estimate is used during training (e.g. for hyperparameter tuning). And that 2nd estimate can again be e.g. cross validation (see nested aka double cross validation) or a single split (or out-of-bootstrap or a validation study, ...)
If no such hyperparameter tuning takes place, the additional single split does not give any advantage over the cross validation.


I’m running into a scenario where someone created a model from dataset (n=1068) and performed cross validation without splitting the data into training and testing set.

(as explained above, cross validation does split into training and testing sets, repeatedly)

So, unless there was data-driven model selection/hyperparameter tuning, this is as it is supposed to be (subject to the further caveat of achieving independent splits with confounders such as geographical region, see below).

Then the model was used to classify a new set of data (n=918) [...] but this does not have the labels

... which was probably the purpose of the model in the first place. After all we build predictive models in order to get predictions for cases where reference analyses (other ways of measuring the predicted outcome) are fundamentally or practically unavailable.

So, in that situation, taking other variables to do some plausibility checks (which is how I understand

classification was then evaluated based on another variable.

) may be the best that was possible in the given situation.

Contrary to your understanding, this may (should) not be intended to replace a proper validation but may be instead what is good prediction practice: to check alongside doing prediction that the cases to be predicted do fit into the calibrated model space.

That being said, such checks should also be part of a proper validation. But of course they cannot replace the comparison of predicitions with reference values.


Now, a crucial point you describe is that there are other known and potentially confounding factors in which the predicted data differs from the (whole) training set:

from a geographically different area,

There are different possibilities to deal with this:

  • If the training data covers multiple geographic areas, the cross validation (or any single train/test split) should split so that predictions for unknown geographic areas are tested.
    Iff that was done, the cross validation estimate is a legitimate approximation for prediction error when encountering data from unknown geographical regions and can directly be used for the predictions of the really unknown (unlabeled) data.
  • If the training data does not cover multiple geographic areas (or the splitting did not produce independence wrt. geographical region), only prediction error for the known regions can be estimated by cross valiation. But an additional train/tetst split would have the same problem here.

    I.e., that estimate of generalization error has an optimistic bias of unknown size.

    In this situation, any further information that helps judging whether crucial differences between the geographical regions have to be expected or may be expected to be negligible will be valuable even though they do not provide a proper method validation, and here the comparison of those other variables comes into play: if factors that are known to be correlated with the task are alike, using the model to predict the new data isn't as risky as if those factors show marked differences.

    Whether this is appropriate for a scientific paper IMHO hinges crucially on whether the paper is clearly aware and openly discussing this limitation. Iff that's the case, I'd be much more relaxed. After all, science often evolves incrementally - the question then would be whether the paper has sufficient substance to merit publication without a validation for the new geographical region.

Two more points to keep in mind:

  • People do and publish preliminary studies in order to have a basis on which to ask for funding for the appropriate validation/extension of the studies.
    Requiring that the paper may not be published without proper validation means that we restrict the possibility to present preliminary findings and conclusions for grant applications that did already undergo peer-review.

  • Splitting a given data set into training and testing will still be what is called internal validation in clinical diagnostics/analytical chemistry. In other words, such a split is in many cases not sufficient for a full method validation. Depending on the situation (and how important deep insight into the performance is, e.g. in terms of diagnosing patients and the dangers of misdiagnoses), validation of an already built model may require whole studies on its own, checking performance for many sub-populations (other regions, sex & gender, age, lifestyle, co-morbidities, ...) etc.
    So testing the predictions on unknown regions is one crucial factor that is obvious from what you wrote, but it is probably only one of many such factors that need to be taken into account.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Since there is an overlap between the training sets, the different folds of the cross-validation are not truly independent tests of the method. Nevertheless, sometimes that is the best you can do.

The difficulty here is that the training set is not very big. To get statistically robust estimate of the model's performance, you need a test set with hundreds of instances. Reserving such a big test set would reduce the training set, so reducing the quality of the model.

If it isn't possible to get more data, then it's a reasonable compromise to use cross-validation in such cases. If you then acquire some new data, and the model turns out to be good, then that's great.

In your case, the data from a different area might have had a different distribution. I'm not clear how you know "its really good" since the new dataset "does not have the labels".

Of course, how much data you need depends on the complexity of the model, roughly speaking the number of free parameters, and precisely speaking the capacity, as defining by Vapnik and Chernovenko. Except for neural nets, which seem to work better than this theory predicts ...

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Understood. This is a medical data and generally the data set are hard to come by. My question was somewhat different than what you explained. My concern was also the same about the different area dataset. That's why I was wondering if we needed to split the dataset into two - training and testing set. Or can we just leave the way it is with only cross-validation but without evaluating it with a test set and assess the performance? $\endgroup$ – user1828605 Nov 25 '19 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.