I have texts from the organisations A, B, C and D For A, B, C I also have the outcomes (0/1). My goal is to create a classification model from the texts of A, B, C, which is able to predict the outcomes of D.
In general, the texts are very similar in structure and the wording is also related. Nevertheless, I can not exclude that every organisation has its own peculiarities in wording.
Now I am considering whether I should apply a specific cross-validation strategy to avoid that my model uses organisation-specific characteristics of the texts for classification. Does it make sense to adapt the CV so that, for example, I form the folders in such a way that an organisation is always a block? Within a folder, you could use A and B for training and evaluate the data at C. Then A and C for training, etc...This way only models that do NOT use the specific elements of A and B should be "successful".
Is there a name for this strategy and can anyone recommend literature? Are there any potential risks with this approach?