0
$\begingroup$

I have been given a task to solve using a neural network and I think that using a neural network is the wrong algorithm to use.

The idea is simple: given at most 3 diagnoses, predict which medication(s) a health professional will prescribe.

The main issue is that each diagnosis can have many valid medications prescribed. A doctor could prescribe two medications for high blood pressure, while another may prescribe only one. And another may prescribe none. They are all right. We are not interested in what is 'right' from a scientific point of view; we will consider it correct if say, arbitrarily, at least 1% of doctors would have prescribed that particular drug or combination of drugs.

To me this sounds like we need a neural network that outputs multiple correct answers. How would this be done? I understand that each my 1000 output nodes will receive a probability, with the network predicting the medication with the highest probability. I could consider the top 3 to 5 highest probability medications as correct. But then how would I produce an ROC curve? This would then rewriting many functions, such as the def of accuracy, true positives, sensitivity, auc, etc, in scikit-learn from scratch. And I get suspicious when many things have to be adapted. Our use case is not that exceptional and should be solvable with something off the shelf is how I feel without being able to back it up. It seems that we are really conflating two problems: can we predict what a doctor would prescribe, and is that prescription inappropriate?

1) I would like confirmation that a neural network is not typically used for this task and why. I need talking points to discuss with my superiors. 2) What level of data science practitioner would be required to solve this problem? Is this appropriate for a beginner? Because it seems to me that producing multi-class ROC curves in which prediction is not just one of many classes but anywhere between 1-10 classes will require implementing techniques from scratch and customizing parts of scikit-learn. 2. What algorithm is best for my task? Anomaly detection, a recommendation system, association rules? And why?

Many thanks!

$\endgroup$
8
  • $\begingroup$ Maybe a stupid question but why not just build a different predictor for each of your 1000 possible medications? This can be done with neural networks or any other model. $\endgroup$ May 18, 2017 at 4:43
  • $\begingroup$ Not stupid at all but could you elaborate? So for each individual medication, have a binary neural network built for it? That is, an aspirin neural network would have aspirin as one output class and 'other' as the remaining possibility? That would lead to a series of one vs the rest scenarios. Is that not what a traditional multi class networks do already? $\endgroup$
    – user798719
    May 18, 2017 at 4:51
  • $\begingroup$ Good point. I guess either would work. With either a single predictor or multiple predictors, what you need is access to the activation levels of the output nodes. I would think any standard implementation of neural networks would give you this. That being said, I would say that going straight to neural networks seems like overkill. Am I understanding correctly that the input is just up to three diagnosis? There doesn't seem to be any reason to jump to such a complex method. $\endgroup$ May 18, 2017 at 5:01
  • 1
    $\begingroup$ Neural networks or really any ML that can do multiclass (think Random Forest) sounds pretty standard to me. Any introductory ML textbook should be able to walk you through. $\endgroup$
    – Calimo
    May 18, 2017 at 7:10
  • 1
    $\begingroup$ It seems a stupid task. Why choose the method and then start looking for answers.? Perhaps you could look into bayesian networks.. They are better suited to a more customised output. $\endgroup$
    – seanv507
    May 18, 2017 at 20:24

2 Answers 2

3
$\begingroup$

Check into multi-label classification. In this paradigm, an input is allowed to have multiple categories. In your case, the diagnoses are the input, and the prescriptions are the outputs. Multi-label classification could be a good choice for you since you seem concerned about using nonstandard evaluation metrics—the community has already taken care of this for you (see the corresponding section in the wiki link above). Added bonus: scikit-learn can be used to train and evaluate multi-label classifiers!

$\endgroup$
6
  • $\begingroup$ Thanks for this. It looks like neural networks are uncommonly used for multi label classification while random forests and nearest neighbor already support this. The bigger problem to me is model evaluation. It seems that I may have to write my own multi label auc and roc functions which I have no confidence in doing. $\endgroup$
    – user798719
    May 19, 2017 at 3:44
  • $\begingroup$ scikit-learn extends some binary metrics such as AUC and F1 score to the multi-label case: link. However, it does not implement a multi-label roc curve. I'm having a hard time imagining what such a curve would even look like and how it could be interpreted, given that you have 1000+ possible labels! You might want to stick with simpler metrics for this problem. $\endgroup$
    – scherm
    May 22, 2017 at 13:56
  • $\begingroup$ Thanks. But from what I read, the metrics are all or nothing. Meaning if your label is [1,2,3] and your prediction is [1,2] then your accuracy would be 0.0. I would want this sample to be counted as correct because at least one member of the true label ,set [1,2,3], was predicted. (In this case two items [1,2] were predicted ) $\endgroup$
    – user798719
    May 23, 2017 at 5:27
  • $\begingroup$ "Accuracy classification score. In multilabel classification, this function computes subset accuracy: the set of labels predicted for a sample must exactly match the corresponding set of labels in y_true. " Just want confirmation that my use case is nonstandard enough such that I will have to implement my own model performance metrics from scratch. $\endgroup$
    – user798719
    May 23, 2017 at 5:36
  • $\begingroup$ You're right that sklearn.metrics.accuracy_score is a harsh metric. However, there are other options such as Hamming loss (lower is better) or the related Hamming score (higher is better) which allow for imperfect matching between predicted labels and true labels. An implementation of Hamming score can be found here. $\endgroup$
    – scherm
    May 23, 2017 at 13:24
1
$\begingroup$

Maybe you can use Support Vector machines for this kind of classifications.

You can implement a neural network to extract relevant information from the input, convert to a lower-dimensional vector, and then pass it to an SVM. This might work because for an SVM to predict a class, the penultimate layer input just has to be over a particular threshold. So, if two of such nodes are over their threshold values as calculated during training, they both will be labelled as ones, whereas the rest would be zeros.

import torch
import torch.nn as nn

class SVM(nn.Module):
    def __init__(self, input_nodes, output_nodes):
        super().__init__()
        self.fully_connected = nn.Linear(input_nodes, output_nodes)
        
    def forward(self, x):
        fwd = self.fully_connected(x)  # Forward pass
        return fwd

This is a starter code for implementing SVMs in PyTorch, which you train like any other network in PyTorch in a one v/s all manner. The loss function to be used here is hinge loss.

This is an implementation of a linear kernel. You may experiment with different types of kernels available to optimize the performance of your model.

And as far as calculating ROC scores, you can also calculate it in a one v/s all way

$\endgroup$
1
  • $\begingroup$ Hi Ayush Nath, this is a nicely written answer, welcome to Stats SE! $\endgroup$
    – Fato39
    Dec 17, 2020 at 10:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.