I have been given a task to solve using a neural network and I think that using a neural network is the wrong algorithm to use.

The idea is simple: given a of at most 3 diagnoses, predict which medication(s) a health professional will prescribe.

The main issue is that each diagnosis can have many valid medications prescribed. A doctor could prescribe two medications for high blood pressure, while another may prescribe only one. And another may prescribe none. They are all right. We are not interested in what is 'right' from a scientific point of view; we will consider it correct if say, arbitrarily, at least 1% of doctors would have prescribed that particular drug or combination of drugs.

To me this sounds like we need a neural network that outputs multiple correct answers. How would this be done? I understand that each my 1000 output nodes will receive a probability, with the network predicting the medication with the highest probability. I could consider the top 3 to 5 highest probability medications as correct. But then how would I produce an ROC curve? This would then rewriting many functions, such as the def of accuracy, true positives, sensitivity, auc, etc, in scikit-learn from scratch. And I get suspicious when many things have to be adapted. Our use case is not that exceptional and should be solvable with something off the shelf is how I feel without being able to back it up.

1. I would like confirmation that a neural network is not typically used for this task and why. I need talking points to discuss with my superiors.
2. What level of data science practitioner would be required to solve this problem? Is this appropriate for a beginner? Because it seems to me that producing multi-class ROC curves in which prediction is not just one of many classes but anywhere between 1-10 classes will require implementing techniques from scratch and customizing parts of scikit-learn.

2. What algorithm is best for my task? Anomaly detection, a recommendation system, association rules? And why?

Many thanks!

I have been given a task to solve using a neural network and I think that using a neural network is the wrong algorithm to use.

The idea is simple: given at most 3 diagnoses, predict which medication(s) a health professional will prescribe.

The main issue is that each diagnosis can have many valid medications prescribed. A doctor could prescribe two medications for high blood pressure, while another may prescribe only one. And another may prescribe none. They are all right. We are not interested in what is 'right' from a scientific point of view; we will consider it correct if say, arbitrarily, at least 1% of doctors would have prescribed that particular drug or combination of drugs.

To me this sounds like we need a neural network that outputs multiple correct answers. How would this be done? I understand that each my 1000 output nodes will receive a probability, with the network predicting the medication with the highest probability. I could consider the top 3 to 5 highest probability medications as correct. But then how would I produce an ROC curve? This would then rewriting many functions, such as the def of accuracy, true positives, sensitivity, auc, etc, in scikit-learn from scratch. And I get suspicious when many things have to be adapted. Our use case is not that exceptional and should be solvable with something off the shelf is how I feel without being able to back it up. It seems that we are really conflating two problems: can we predict what a doctor would prescribe, and is that prescription inappropriate?

1. I would like confirmation that a neural network is not typically used for this task and why. I need talking points to discuss with my superiors.
2. What level of data science practitioner would be required to solve this problem? Is this appropriate for a beginner? Because it seems to me that producing multi-class ROC curves in which prediction is not just one of many classes but anywhere between 1-10 classes will require implementing techniques from scratch and customizing parts of scikit-learn.

2. What algorithm is best for my task? Anomaly detection, a recommendation system, association rules? And why?

Many thanks!

I have been given a task and I think that using a neural network is the wrong algorithm to use.

The idea is simple: given a of at most 3 diagnoses, predict which medication(s) a health professional will prescribe.

The main issue is that each diagnosis can have many valid medications prescribed. A doctor could prescribe two medications for high blood pressure, while another may prescribe only one. And another may prescribe none. They are all right. We are not interested in what is 'right' from a scientific point of view; we will consider it correct if say, arbitrarily, at least 1% of doctors would have prescribed that particular drug or combination of drugs.

To me this sounds like we need a neural network that outputs multiple correct answers. How would this be done? I understand that each my 1000 output nodes will receive a probability, with the network predicting the medication with the highest probability. I could consider the top 3 to 5 highest probability medications as correct. But then how would I produce an ROC curve? This would then rewriting many functions, such as the def of accuracy, true positives, sensitivity, auc, etc, in scikit-learn from scratch. And I get suspicious when many things have to be adapted. Our use case is not that exceptional and should be solvable with something off the shelf is how I feel without being able to back it up.

1. I would like confirmation that a neural network is not typically used for this task and why. I need talking points to discuss with my superiors.
2. What level of data science practitioner would be required to solve this problem? Is this appropriate for a beginner? Because it seems to me that producing multi-class ROC curves in which prediction is not just one of many classes but anywhere between 1-10 classes will require implementing techniques from scratch and customizing parts of scikit-learn.

2. What algorithm is best for my task? Anomaly detection, a recommendation system, association rules? And why?

Many thanks!

I have been given a task to solve using a neural network and I think that using a neural network is the wrong algorithm to use.

The idea is simple: given a of at most 3 diagnoses, predict which medication(s) a health professional will prescribe.

The main issue is that each diagnosis can have many valid medications prescribed. A doctor could prescribe two medications for high blood pressure, while another may prescribe only one. And another may prescribe none. They are all right. We are not interested in what is 'right' from a scientific point of view; we will consider it correct if say, arbitrarily, at least 1% of doctors would have prescribed that particular drug or combination of drugs.

To me this sounds like we need a neural network that outputs multiple correct answers. How would this be done? I understand that each my 1000 output nodes will receive a probability, with the network predicting the medication with the highest probability. I could consider the top 3 to 5 highest probability medications as correct. But then how would I produce an ROC curve? This would then rewriting many functions, such as the def of accuracy, true positives, sensitivity, auc, etc, in scikit-learn from scratch. And I get suspicious when many things have to be adapted. Our use case is not that exceptional and should be solvable with something off the shelf is how I feel without being able to back it up.

1. I would like confirmation that a neural network is not typically used for this task and why. I need talking points to discuss with my superiors.
2. What level of data science practitioner would be required to solve this problem? Is this appropriate for a beginner? Because it seems to me that producing multi-class ROC curves in which prediction is not just one of many classes but anywhere between 1-10 classes will require implementing techniques from scratch and customizing parts of scikit-learn.

2. What algorithm is best for my task? Anomaly detection, a recommendation system, association rules? And why?

Many thanks!