1
$\begingroup$

I have texts from the organisations A, B, C and D For A, B, C I also have the outcomes (0/1). My goal is to create a classification model from the texts of A, B, C, which is able to predict the outcomes of D.

In general, the texts are very similar in structure and the wording is also related. Nevertheless, I can not exclude that every organisation has its own peculiarities in wording.

Now I am considering whether I should apply a specific cross-validation strategy to avoid that my model uses organisation-specific characteristics of the texts for classification. Does it make sense to adapt the CV so that, for example, I form the folders in such a way that an organisation is always a block? Within a folder, you could use A and B for training and evaluate the data at C. Then A and C for training, etc...This way only models that do NOT use the specific elements of A and B should be "successful".

Is there a name for this strategy and can anyone recommend literature? Are there any potential risks with this approach?

Improve this question
$\endgroup$
7
  • $\begingroup$ By casting this as a classification task instead of a prediction task you are probably not meeting the study's goals. But validation is a poor way to underestanding organizational characteristics. It is better to model the effect of those characteristics then to understand their estimated effects with a whole-model fit. $\endgroup$
    – Frank Harrell
    Nov 14, 2020 at 12:43
  • $\begingroup$ Thank you very much for this thought-provoking impulse. Of course, it is a fundamental question whether an absolute binary decision makes sense. I am thinking about it myself and am still in the planning phase. Nevertheless my problem remains: How do I minimize the error probability of a model from A,B,C for a prediction of the outcomes of D (the outcomes of D are not known and not easily observable) ? Especially if I have a lot of possible explanatory variables? $\endgroup$
    – Martin
    Nov 14, 2020 at 13:43
  • 1
    $\begingroup$ The fact that a binary decision may ultimately be needed is not what drives the choice of analysis. Given a utility/cost/loss function you convert probabilities to optimum decisions (maximizing expected utility) and can also capitalize on gray zone probabilities (say 0.4-0.6) be used to indicate "no decision". Classifiers know nothing about optimum decisions. $\endgroup$
    – Frank Harrell
    Nov 14, 2020 at 17:11
  • 1
    $\begingroup$ Depending on the effective sample size and the number of candidate features, one good approach is to use the Efron-Gong optimism bootstrap to validate the whole-sample model, focusing on a proper probability accuracy scoring rule such as the Brier score. $\endgroup$
    – Frank Harrell
    Nov 14, 2020 at 22:18
  • 1
    $\begingroup$ @Frank Without knowing anything of their studies goals how can you assume they aren’t meeting them? $\endgroup$
    – astel
    Nov 15, 2020 at 19:46

1 Answer 1

Reset to default
2
$\begingroup$

Whatever kind of resampling (CV or bootstrap) verification/validation scheme you use, when you know or suspect that your data is clustered such as by organization in your case, you need to split at the cluster level, otherwise the clusters create data leaks/lead to dependence. More generally speaking, you need to split so that also the highest-level confounding factors are independent for the splits.

In your case this basically means leave one organization out cross validation (or a similar subset of bootstrapped models) due to there being only 3 organizations.

So splitting strategy and model setup both follow from the data structure (cl you have/set up.

Everything Frank Harrell comments about predicting probabilities and using a proper scoring rule applies - these are modeling decisions/choices which are completely independent of the resampling scheme/data plan for verification.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.