I am using Logistic Regression (LR) to obtain Coronary Artery Disease CAD probability equation. The data set has 16 candidate predictors, all continuous. There are two groups, CAD patient group (70 subjects), and an age-matched control group AMC (~ 70 subjects). I have a feeling that absence of young healthy control group YHC is impacting the learning process and thus the performance of the model negatively.

I found an interesting example in “Learning from Data” book, page 172, by Yasser Abu-Mostafa of Caltech. The example comes from finance, but there are similarities to my situation. Banks often build predictive models for credit approval based on historical data of their customers.

For instance, in our credit example of Chapter 1, the bank created a training set from a database of previous customers and how they performed for the bank. Such a set necessarily excludes those who applied to the bank for credit cards and were rejected, because the bank does not have data on how they would have performed, if they were accepted. Since future applicants will from a mixed population including those who would have been rejected in the past, the “test set” come from a different distribution than the training set, and we have a case of sampling bias … . In this particular case, if no data on the applicants that were rejected is available, nothing much can be done other than to acknowledge that there is a bias in the final predictor that learning will produce, since a representative training set is just not

In my case, the original population is patients presenting with chest pain (== database of previous customers), some of these will be diagnosed as patients ( == positive == no default history) and others healthy (== negative == credit default). The missing group is then the YHC who the hospital does not know how they would have performed, because they were considered from the beginning as healthy ( == negative == rejected for credit card).

Unlike the credit approval situation, future patients are likely to come from a population who probably does not include YHC. The inclusion of these in the training set seems to make sense, however, does their absence from the testing set cause bias?

  • 1
    $\begingroup$ Bias in what sense? If the model will be used to judge patients, not healthy people, then having a test set of patients should be OK. $\endgroup$ Sep 9, 2017 at 20:04


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.