Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Say we have n similar data sets, with the same variables, and outcome labels x and y. In these data sets, domains slightly differ as suggested by the proportion of the minority class x (ranging from 1%-15%).
First-off: I don't think that there is just one truth about your questions.
And (how) is this related to transfer learning, domain adaptation and multi-task learning?
Transfer learning is exactly what you describe. In your case, the problem domain differs, but the structure remains the same. The research area of transfer learning in general focuses on three questions: when, how and what to transfer. Overview: https://link.springer.com/article/10.1007/s10462-013-9406-y
How can we develop a robust ML algorithm that works out-of-the-box on new data sets with varying proportions of the classes (i.e. 1-99 or 15-85)? Which algorithms are a good-fit for this purpose?
Generally, if you apply ML, you strive to build models that represent concepts about your target, or "general knowledge". In your case, the goal is to not build models that "remember data", meaning, that overfit. If you try to predict the same thing on both data sets A and B: just build a good model on data set A and it should work properly on B. Problem: if your data is for instance a set of blog entries and you want to determine the sentiment, the domain might as well matter a lot. If A is about movies and B is about politics, general concepts about sentiments might be the same, but domain-specific "signaling" words might differ in their sentiment. Transferring a model without adjustment might therefore not be a good choice. Thus, starting of with a transferred model and adapt it somehow to the new data might work!
