1
$\begingroup$

Say one continuous variable differentiates between disease and nondisease quite accurately, but as people progress in age, this variable becomes less accurate. Is there a way to determine the accuracy of this variable on the continuous scale of age?

This is for example the case considering the BMI in predicting the cardio-metabolic health; an increased BMI is less reflective for the cardio-metabolic health as people progress in age.

I have no dataset at the current moment, but if strictly necessary according to the rules of the forum, I can try to simlate a dataset.

$\endgroup$
2
  • $\begingroup$ Do you have some plots? $\endgroup$ Commented Aug 6 at 7:57
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Commented Aug 6 at 7:58

1 Answer 1

0
$\begingroup$

Accuracy is a highly problematic KPI, because it presupposes one very specific cost structure without making this explicit. This is most problematic for "unbalanced" data, but pertains also to "balanced" data. Why is accuracy not the best measure for assessing classification models?

Instead, I recommend you work with probabilistic classifications, and assess these using a proper scoring rule, such as the Brier or the log score.

Now, what you can do is to build your probabilistic classifier and assess its performance on test data, using a proper scoring rule. And of course you can plot the age variable of your test dataset against its Brier or log score contributions, and if you want to, analyze it using standard tools like regression. Since the predictive performance may depend on more factors than just age, it may make sense to run a multivariate model. And age may well have a nonlinear influence, so it might make sense to use a spline transform.

Each contribution to the Brier score is either $\hat{p}_i^2$ or $(1-\hat{p}_i)^2$, i.e., they are all bounded between zero and one, so it may make sense to do a beta regression. However, if you have lots of data, a straightforward OLS might be fine. The log score is unbounded below (and bounded above by zero), so an OLS would make sense.

You may need a lot of test data to actually detect a signal, simply because we can't observe the underlying true probability of being of the target class, only the final outcome.

Finally, yes, you could also use accuracy as a KPI: for each instance in the test sample, encode whether it was classified correctly or not, and then run a logistic regression on this outcome against age or any other variable. But per above, optimizing accuracy will mislead your original model in the first place, so I very much recommend not to do this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.