I am using Logistic Regression (LR) to obtain Coronary Artery Disease CAD probability equation. The dataset has 16 candidate predictors, all continuous. There are two groups, CAD patient group (70 subjects), and an age-matched control group AMC (~ 70 subjects). Given that the control group is age-matched, I have a feeing that this is impacting the performance of the model negatively.

To test this, I have divided the original dataset randomly into 50% training set, and a testing set. I then included an additional group, Young Healthy Control (YHC) (who represent the healthy population more accurately) to the training set. Once a model has been selected, I applied this model to predict on the test set (which does not include YHC). The performance showed improvement compared with the original approach.

Is this approach acceptable, given that primary objective of the CAD equation is to predict CAD on a population similar to the original dataset?


So long as your model performance is being evaluated in a data set that is representative of the data you want to apply the model to, yes, that experimental design is fine. It may be that the YHC is giving the model more clear examples of what a healthy heart looks like. Depending on how you designed you experiment, it could also just be that getting more data is the reason for improved performance (it is unclear from your description if the new training set is CAD+AMC+YHC or CAD+YHC).

The only thing I would caution is since the new data may be introducing artifacts, go back and double check all the rest of your experimental design to be 100% certain that the improvement is not a mistake. I would start by making sure you are using the exact same patients as your testing set for your two models (and that the model never sees those data of course). Check multiple statistics for your model (Accuracy, ROC, Kappa, etc.) to be sure the improvement is robust to evaluation metrics. One thing you may even do is plot a histogram of the class probabilities for your testing set for both models. If the model is good, each class should have nice separate peaks on either side of your threshold. If the values are either scrunched up as one peak in the middle that probably means the model isn't great, but you should have seen that in the other metrics. If they are way spread out at the extreme ends of the distribution, I would be skeptical that it might be too good to be true and there may be some data leakage or artifacts that are influencing your results.

  • $\begingroup$ Thanks for the comments. The new training set is CAD+AMC+YHC. An experiment in which I fix the patient group and and compare the two model trained on sets with and without YHC is a good idea, although this will not rule out the possibility of improved performance due to simply more data. The checks you suggest are also useful. $\endgroup$ – Abbas Dec 22 '16 at 10:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.