If the test set is not located the data domain of the training set, then the prediction result for the test set should be worse than those in the train set domain. it is right to express this?
Yes and no: usually the if the test data is outside the training domain, then the results usually are worse. But keep in mind that this is a symptom of overfitting your model to the training domain. The thing that matters is the quality of the predictions for your application domain (it needs to be fit for purpose). So the testing needs to check the desired application domain, and it often makes sense to test various aspects of that in a more detailed manner.
You can then think and judge how closely the training domain needs to match the application domain.
In certain ways, predictive models are usually applied outside the training domain, while for other aspects the predictions are done inside the training domain.
Consider a model that predicts some analyte concentration of clinical relevance as a function of some kind of measurement (e.g. spectroscopic).
The application domain will be spectra of new (unknown and measured in the future) patients. For training I may get away with preparing calibration samples*. However, as the application domain are patients, I need to test also patient sample predictions against reference values. Even if I train on patient samples, I need to test for the application domain of unknown patients.
In addition, I need to establish how often the calibration needs to be redone. I.e. I need to test the application domain of unknown patient samples acquired and measured hours, days, weeks after the training data.
So, how can I quantify the applicability domain for a predictive model? that means, when I have the new dataset, how can I check if it located in the applicability domain of the model?
I may be able to specify from the training spectra limits of the distribution that I want to restrict my predictions on. This would be the training domain in the spectra ($\mathbf X$) space and often guided
- either rather closely by the training data ("do not extrapolate outside the calibrated signal range")
- or by external knowledge, e.g. requiring 0 $\leq$ absorbance $\leq$ 1 in a spectra quality test.
This is one way of specifying applicability.
For other aspects of applicability, the reasoning is usually the other way round: you specify how much deterioration of the predictive quality you are willing to tolerate, and then look e.g. for how long you can use the same calibration. So you end up with a specification "redo calibration weekly", or you specify that test samples should be run every so often.
These arguments work exatcly the same for a classification instead of regression/calibration. (Just that measuring the predictive ability of a classifier is more difficult.)
From your tags I assume you ask about QSAR. Also in that case, the model is typically built in order to predict the activity of compounds unknown to the model, i.e. outside the training data. Meaning whatever selection of compounds you train on, you need to test with compounds unknown to the model. You can then go on an slice the test results according e.g. to different groups of compounds. You may also exclude whole groups of compounds from training and check how that model does for this group.