3
$\begingroup$

I'm working on the SAMHSA Mental Health Client-Level Dataset. I'm trying to train classifiers to predict the disorder given the rest of the columns. There are 13 binary disorder columns (bipolar, schizophrenia, ADHD, etc.) based on diagnoses.

Code: https://github.com/jacksonwalters/ml-examples/tree/main/mental_health_client-level_data

I've trained a RandomForestClassifier and multi-class LogisticRegression. They are 36% accuracy, 30% precision, 36% recall. I binned the disorders [0 disorders, 1 disorder, > 1 disorder]. I also tried binary encoding the 2^13=8192 combinations of disorders, which had similar accuracy but 17% precision. A random guess in the former case should be ~7%.

If I predict the k-means labels, I get ~92% accuracy, precision, recall.

For the disorder labeling, should I use LogisticRegression and drop all other columns, and perform the classification on the remaining columns? I'd do this for all 13 columns, then just divide by the sum to get 13d vector probability outputs.

The output/labels are really Boolean vectors. If someone is diagnosed with ADHD by two different doctors, they do not have 2*ADHD, they just have ADHD, so ADHD + ADHD = ADHD, i.e. idempotent.

confusion matrix logistic regression

Improve this question
$\endgroup$
12
  • 1
    $\begingroup$ Welcome to Cross Validated! What is the outcome, a list of the disorders, something like “patient 1 has ADHD, patient 2 has schizophrenia and is bipolar, patient 3 has none of the disorders, etc”? $\endgroup$
    – Dave
    Commented Mar 6 at 14:59
  • 1
    $\begingroup$ And a patient can have one, multiple, or zero of the disorders, yes? $\endgroup$
    – Dave
    Commented Mar 6 at 15:33
  • 1
    $\begingroup$ Then it seems that you have a multilabel problem. Perhaps do some reading on that subject and see if it makes sense in the context of your problem. $\endgroup$
    – Dave
    Commented Mar 6 at 15:47
  • 1
    $\begingroup$ I think it would be very reasonable to model the (possible dependent) probabilities of thirteen binary variables. Multivariate probit regression might be a reasonable starting point. $\endgroup$
    – Dave
    Commented Mar 6 at 16:19
  • 1
    $\begingroup$ What is “drop all but one”? $\endgroup$
    – Dave
    Commented Mar 6 at 16:25

1 Answer 1

Reset to default
2
$\begingroup$

Binary problems use features to make predictions about a binary outcome: yes/no, alive/dead, dog/cat, etc.

A problem uses features to make predictions about an outcome that can be one of multiple categories: dog/cat/alligator, 0/1/2/3/4/5/6/7/8/9, shirt/boots/dress, etc.

A problem uses features to make predictions about an outcome than can be one, several, or none or multiple categories.

This is an archetypal problem.

  1. There are thirteen categories, so the problem cannot be binary.

  2. The categories are not mutually exclusive, so the problem cannot be .

Therefore, a reasonable task is to predict the probability of having each disorder vs not having the disorder, exactly a problem. One of the more basic models that still treats this as more complex than just thirteen binary problems (e.g., bipolar vs not, ADHD vs not, etc) is a multivariate probit model, which allows for correlations between outcomes. For instance, if ADHD and schizophrenia tend to go together, the model can make a point to assign high probabilities to one when the other has a high predicted probability.

Sophisticated models of such an outcome could include neural networks. These multilabel problems are far from obscure, so there should be plenty of tutorials available about the coding.

I will close with links to some posts on here about multilabel problems.

Ideal loss for a multi-label problem with soft targets

How robust is multinomial logistic regression to having a multi-label problem shoehorned into it?

Predictions when multiple outcomes

What is the statistical model for a multi-label problem?

Duplicates with different labels in multi-label classification

What is the difference between a multiclass and a multilabel problem?

What are the measure for accuracy of multilabel data?

Multilabel classification metrics on scikit

How to apply neural networks on multi-label classification problems?

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ Excellent answer, thank you for the disambiguation. I've implemented both binary predictions for all 13 classes, and I'm using MultiOutputClassifier to do multi-label predictions. $\endgroup$
    – Jackson Walters
    Commented Mar 6 at 18:27
  • 1
    $\begingroup$ @JacksonWalters Fitting thirteen binary models (e.g., logistic regression) is a way to solve a multi-label problem. Each of those thirteen models gives you the (predicted) probability of a disorder, so you wind up with the probability of each disorder. However, such an approach makes an assumption that the disorders are independent, which can be relaxed when you explicitly model the disorders jointly. $\endgroup$
    – Dave
    Commented Mar 6 at 20:39
  • $\begingroup$ I haven't looked at the error metric for the 13 binary models yet, but for the MultiOutputClassifier(LogisticRegression(...)) I'm using hamming_loss and getting .1. I think this is consistent, in that actually predicting the diagnostic multi-label from demographic info plus some life factors should be pretty hard without any symptom info. Does using MultiOutputClassifier count as modeling them jointly? That's what I want to do, because they're not independent. $\endgroup$
    – Jackson Walters
    Commented Mar 6 at 21:51
  • 1
    $\begingroup$ @JacksonWalters The documentation suggests to me that your function assumes independence. Software suggestions to implement a multivariate probit or other multilabel model are considered off-topic here but might be on-topic elsewhere. $\endgroup$
    – Dave
    Commented Mar 6 at 21:57
  • $\begingroup$ Ah, okay. It appears multi-output != multi-label, I got confused. I did run the hamming_loss on the 13-binary-models, and got .12. I will seek elsewhere for a multilabel implementation. OneVsRestClassifier looks good. $\endgroup$
    – Jackson Walters
    Commented Mar 6 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.