custom loss function to optimize payoff via binary decision

I have written a custom loss function that is supposed to optimize a payoff via a binary decision. However, the neural networks is struggling to convert, and I'm suspecting that there's something wrong with this function.

The payoff looks as follows. Imagine for each sample the neural network has to decide whether to engage or not (True/False). Each sample has some explanatory variables and an individual payoff (p) that is known. The reward or loss is then as follows:

• true positive: payoff (as passed in via variable over a second dimension in y_true). Payoff is a vector and is different for every sample.
• true negative: 0
• false positive: -1
• false negative: 0 (but has an opportunity cost of payoff)

The custom keras loss function looks as follows (I'm passing in payoffs via y_true, but split it out at the beginning. This is necessary as the keras payoff function only allows two parameters):

def custom_cross_entropy(y_true, y_pred):
# y_true has the payoffs in the last dimension
y_true, payoffs = splitter(y_true)

tp_weight = K.abs(payoffs)  # true positive has a positive payoff as described in the payoff vector, it's different for every item
tn_weight = 0  # true negative has a payoff of 0
fp_weight = 1  # cost of 1 for false positive
fn_weight = K.abs(payoffs)  # false negative has an opportunity cost of payoff (vector)

loss = -K.mean(fn_weight * y_true * K.log(y_pred + 1e-7) +  # false negative
fp_weight * (1 - y_true) * K.log(1 - y_pred + 1e-7)  # false positive

def splitter(y_true):
payoffs = y_true[:, 1]
payoffs = K.expand_dims(payoffs, 1)
y_true = y_true[:, 0]
y_true = K.expand_dims(y_true, 1)
return y_true, payoffs


What could be wrong with this payoff function? I only use the false negative and false positives. Is that correct? What else could be the problem? How would the loss function differ, if true negatives had a payoff of x? It seems that only using the false true and negatives is not enough. Any suggestions are appreciated.