1
$\begingroup$

I have a big dataset and I want to build a classification model (svm, rf, ann etc.). Then I split the original dataset into training set and test set. I build the model using training set. After it was done, I use the model to predict the test set. Here, the training set and test set should be similar in data structure, that means they are located in the same data domain, right?

If the test set is not located the data domain of the training set, then the prediction result for the test set should be worse than those in the train set domain. it is right to express this?

So, how can I quantify the applicability domain for a predictive model? that means, when I have the new dataset, how can I check if it located in the applicability domain of the model?

some links: http://en.wikipedia.org/wiki/Applicability_domain http://www.moleculardescriptors.eu/tutorials/T7_moleculardescriptors_ad.pdf

Any points are appreciated.

$\endgroup$
1
$\begingroup$

If the test set is not located the data domain of the training set, then the prediction result for the test set should be worse than those in the train set domain. it is right to express this?

Yes and no: usually the if the test data is outside the training domain, then the results usually are worse. But keep in mind that this is a symptom of overfitting your model to the training domain. The thing that matters is the quality of the predictions for your application domain (it needs to be fit for purpose). So the testing needs to check the desired application domain, and it often makes sense to test various aspects of that in a more detailed manner. You can then think and judge how closely the training domain needs to match the application domain.

In certain ways, predictive models are usually applied outside the training domain, while for other aspects the predictions are done inside the training domain.

Consider a model that predicts some analyte concentration of clinical relevance as a function of some kind of measurement (e.g. spectroscopic).
The application domain will be spectra of new (unknown and measured in the future) patients. For training I may get away with preparing calibration samples*. However, as the application domain are patients, I need to test also patient sample predictions against reference values. Even if I train on patient samples, I need to test for the application domain of unknown patients. In addition, I need to establish how often the calibration needs to be redone. I.e. I need to test the application domain of unknown patient samples acquired and measured hours, days, weeks after the training data.

So, how can I quantify the applicability domain for a predictive model? that means, when I have the new dataset, how can I check if it located in the applicability domain of the model?

I may be able to specify from the training spectra limits of the distribution that I want to restrict my predictions on. This would be the training domain in the spectra ($\mathbf X$) space and often guided - either rather closely by the training data ("do not extrapolate outside the calibrated signal range") - or by external knowledge, e.g. requiring 0 $\leq$ absorbance $\leq$ 1 in a spectra quality test.

This is one way of specifying applicability.

For other aspects of applicability, the reasoning is usually the other way round: you specify how much deterioration of the predictive quality you are willing to tolerate, and then look e.g. for how long you can use the same calibration. So you end up with a specification "redo calibration weekly", or you specify that test samples should be run every so often.

These arguments work exatcly the same for a classification instead of regression/calibration. (Just that measuring the predictive ability of a classifier is more difficult.)

From your tags I assume you ask about QSAR. Also in that case, the model is typically built in order to predict the activity of compounds unknown to the model, i.e. outside the training data. Meaning whatever selection of compounds you train on, you need to test with compounds unknown to the model. You can then go on an slice the test results according e.g. to different groups of compounds. You may also exclude whole groups of compounds from training and check how that model does for this group.

$\endgroup$
  • $\begingroup$ Thank @cbeleites. One more thing is that can you give an example to explain how to apply your anwser, best in R code? and can you do some work for finish the calculation of AD shown in the above url? since for people like me, it is difficult to change the statistical equation to R code, and for me I often apply the existing approach to solve my problem. so thank you for your help. and Yes, my question is about QSAR and AD in QSAR has some definition as I list in the url and some others. but the existing applications are scarce, especially in R. so I hope you and other statistician can help. $\endgroup$ – Kevin Apr 25 '14 at 13:06
  • 1
    $\begingroup$ @Kevin: this sounds to me as if you need to find proper consulting. Your links state that there is no procedure to define the applicability domain for QSAR models that is universally agreed upon. In other words, you need to think what properties your application needs and from that conclude how to verify that your QSAR model is fit for purpose (= valid) and which definition(s) of the applicability domain can be used. My guess is that this is much less a problem of coding than of thoroughly understanding the application. But that's my 2 ct, and I don't have any first hand experience in QSAR. $\endgroup$ – cbeleites supports Monica Apr 25 '14 at 15:32
  • $\begingroup$ thank you. You are right. For me, it is true to understand the application well. I am learning some literatures for this. $\endgroup$ – Kevin Apr 25 '14 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.