1
$\begingroup$

I'm trying to build/train model that depends on many attributes where age is the most important one (it has significant impact on AUC).

Overall target class count is quite balanced (+40% vs. -60%) and whole dataset is small (~150 samples).

This is a plot of target class (diagnosis) distribution for age: enter image description here

Maximum frequency/count peaks are shifted in opposite directions for each target class (diagnosis) I would like to increase general accuracy of model (e.g. GBM or GLM).

I wonder what I can do to minimalize impact of such distribution for predictions (false negative for young and false positive for old patients).

Are there specific methods for age adjustment, sampling or cost/penalty functions that increase performance (e.g. AUC) of model (especially for GBM or GLM)?

Improve this question
$\endgroup$
3
  • $\begingroup$ What exactly are you concerned about here? The class conditional distributions of age are different, but in their sample volume they are not too skewed to cause problems imo. $\endgroup$
    – deemel
    Commented May 6, 2019 at 14:23
  • $\begingroup$ I agree with @Rickyfox on this; I think you are doing well as it is really! Please see my answer below for some additional points. $\endgroup$
    – usεr11852
    Commented May 6, 2019 at 19:32
  • $\begingroup$ I suppose that age is a special attribute and it has substantial influence on predictions. My concern is that age can be overestimated in model training and the other attributes could finally enable more precise classification since age is rather rough attribute (in medical data analysis) $\endgroup$
    – MLearner
    Commented May 7, 2019 at 10:30

1 Answer 1

Reset to default
4
$\begingroup$

Do not do anything to your sampling distribution. This is not a situation where we have strong class imbalance; there is no reason to perplex things.

Do assess the classifier performance based on calibration plots. In general what we want is a well-calibrated model. I would suggest trying a Generalised Additive Model (GAM) so non-linear relations between the predictor variables and the response can be taken into account. Aside GAMs using a penalisation approach like elastic net, might be a good idea especially given the relatively small size available. To that extent, 150 samples do not offer a lot room of generalisation so it would be essentially to cross-validate your results. The methodology presented in Beleites et al. (2013) Sample size planning for classification models is a good starting point.

Side-note for GBMs: A GBM while great, usually does not offer well-calibrated probabilities out of the box. The extra calibration step (i.e. Platt scaling, isotonic regression, beta calibration, etc.) requires a hold-out sample and with 150 samples to begin with, this is too expensive at this point. It is more prudent to focus on learners like GLM/GAM that are well-calibrated out-of-the-box (see the CV.SE thread on Why is logistic regression well calibrated, and how to ruin its calibration?, for more details).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.