I have a task where a specific suitable action needs to be taken for each event. For instance in case we see event1, the good action to take is to do action1, and for event2, the good action is to do action2, and so on. The good action is decided based on the outcome of the event. This outcome can be positive or negative. So if we take an action for an event, and it's outcome turns out to be positive outcome, we say the action we took was a good action. Otherwise it was not a suitable action (we should have taken another action, and we can only take at most one action for each event). We also have the option of doing no action, as sometimes some events can end up having positive outcome without doing any action from us. So the task is a multi-classification problem where given a set of events we need to find the best action (including no-action) that has the highest probability of making that event to have the positive outcome.
What I have tried so far
At first, I trained a multi-class classifier (i.e. Multi-layer perceptron network) with the set of possible actions and no-action as it's classes. I trained this network only on those events that had positive outcome, and did not considered events with negative outcomes at all. I hoped the model could learn the suitable action for each event based on these positive examples (events with positive outcome).
But, there are the two things I think need improvement with this paradigm:
when the model is trained, I get the predicted action for a test event using a softmax approach where the action with highest probability is chosen as the suitable action for that event. However, I think just getting the maximum probability might not be the best option because what if all actions have very close probabilities and the winner is chosen by a subtle difference with the next best one (specifically, no-action can be taken when in fact the next actual action could be good enough as well, and if we prefer to take an action if it does not hurt). Also, the output corresponding to each label in this multi-class classification I believe are only scores and not probabilities precisely. So, the question is what is best way of picking one label out of the predicted labels of this multi-class classifier?
I was thinking about somehow involving negative outcome events in my training as well because we have a very large number of them in our training set and maybe the model can learn better which actions it should pick if we know unsuitable actions as well. So, there are two options here: 1) we can have a set of multiple binary classifiers for each action (including no-action one) where each trained on the set of events (positive or negative outcome) where that action has been chosen. 2) we can have two multi-class classifiers one as before, trained only on positive outcome events, and a new one, trained only on negative outcome events.
So I wonder what could be the pros/cons of the two above approaches. We prefer the second option as it imposes less changes in our current architecture. But, I have a hard time how we should combine the output of the two classifiers in this case. For instance, let's say classifier one selects action1, and classifier2 selects action1' (not choose action1). Then, which action we should pick at the end if action1' has higher score than action1? pick no-action instead?