Neural networks that predict more than one class as being correct

Question

I have been given a task to solve using a neural network and I think that using a neural network is the wrong algorithm to use.

The idea is simple: given at most 3 diagnoses, predict which medication(s) a health professional will prescribe.

The main issue is that each diagnosis can have many valid medications prescribed. A doctor could prescribe two medications for high blood pressure, while another may prescribe only one. And another may prescribe none. They are all right. We are not interested in what is 'right' from a scientific point of view; we will consider it correct if say, arbitrarily, at least 1% of doctors would have prescribed that particular drug or combination of drugs.

To me this sounds like we need a neural network that outputs multiple correct answers. How would this be done? I understand that each my 1000 output nodes will receive a probability, with the network predicting the medication with the highest probability. I could consider the top 3 to 5 highest probability medications as correct. But then how would I produce an ROC curve? This would then rewriting many functions, such as the def of accuracy, true positives, sensitivity, auc, etc, in scikit-learn from scratch. And I get suspicious when many things have to be adapted. Our use case is not that exceptional and should be solvable with something off the shelf is how I feel without being able to back it up. It seems that we are really conflating two problems: can we predict what a doctor would prescribe, and is that prescription inappropriate?

1) I would like confirmation that a neural network is not typically used for this task and why. I need talking points to discuss with my superiors. 2) What level of data science practitioner would be required to solve this problem? Is this appropriate for a beginner? Because it seems to me that producing multi-class ROC curves in which prediction is not just one of many classes but anywhere between 1-10 classes will require implementing techniques from scratch and customizing parts of scikit-learn. 2. What algorithm is best for my task? Anomaly detection, a recommendation system, association rules? And why?

Many thanks!

Answer 1

Check into multi-label classification. In this paradigm, an input is allowed to have multiple categories. In your case, the diagnoses are the input, and the prescriptions are the outputs. Multi-label classification could be a good choice for you since you seem concerned about using nonstandard evaluation metrics—the community has already taken care of this for you (see the corresponding section in the wiki link above). Added bonus: scikit-learn can be used to train and evaluate multi-label classifiers!

Answer 2

Maybe you can use Support Vector machines for this kind of classifications.

You can implement a neural network to extract relevant information from the input, convert to a lower-dimensional vector, and then pass it to an SVM. This might work because for an SVM to predict a class, the penultimate layer input just has to be over a particular threshold. So, if two of such nodes are over their threshold values as calculated during training, they both will be labelled as ones, whereas the rest would be zeros.

import torch
import torch.nn as nn

class SVM(nn.Module):
    def __init__(self, input_nodes, output_nodes):
        super().__init__()
        self.fully_connected = nn.Linear(input_nodes, output_nodes)
        
    def forward(self, x):
        fwd = self.fully_connected(x)  # Forward pass
        return fwd

This is a starter code for implementing SVMs in PyTorch, which you train like any other network in PyTorch in a one v/s all manner. The loss function to be used here is hinge loss.

This is an implementation of a linear kernel. You may experiment with different types of kernels available to optimize the performance of your model.

And as far as calculating ROC scores, you can also calculate it in a one v/s all way