I have a trading system where the model receives 9 time-series and predict :

A - strong down
B - week down
C - neutral 
D - week up
E - strong up

(these classes are generated from an histogram to have a balanced training dataset ... the histogram is separated in 20% parts of examples centered in 0... accuracy of 20% is the baseline)

For each class I activate a different parametrized trading mechanism...

My model is giving acceptable results. Here is the resulting confusion matrix for val data (28.99% acc):

[[32 20  3  6  8]
 [35 19  9  7 16]
 [30  9  6 14 24]
 [21 14  9 18 29]
 [ 9 14  3 14 45]]

My question starts here :

I.e. If the model predicts B-"week down" but in reality is A-"strong down" is a miss, but in reality, it will make money...

So in this confusion matrix we can see that it happens 20 times (cell[0,1]) ... also if it is B but the model says A it will make money in 35 trades (cell[1,0])...

And also the same for the UP cases..

All together (from the confusion matrix) : 32+20+35+19 + 18+29+14+45 = 212 winning trades 21+14+9+14 + 6+8+7+16 = 95 losing trades

Assuming negative trades cancel in equal (in reality will not be equal..) positive trades, the total is = 117 winning trades.

What I want is to create a loss function based on categorical_crossentropy but somehow consider:

  • pred A real B - half miss
  • pred B real A - half miss
  • pred D real E - half miss
  • pred E real D - half miss

Do not penalize too much this cases. I think this will increase a bit the total number of positive trades. It will guide the learning a litle bit better (maybe not for accuracy but for a better loss that generates a better confusion matrix for profit )...

I have created a custom loss function that reduces 3% the loss for these cases:

def my_loss(y_pred, y_true):

    y_pre_indexes = K.argmax(y_pred, axis=1) 
    y_test_indexes= K.argmax(y_true, axis=1)

    TN = K.tf.logical_or( K.tf.logical_or (K.tf.logical_and(K.equal(y_pre_indexes,0),K.equal(y_test_indexes,1)),
                          K.tf.logical_or (K.tf.logical_and(K.equal(y_pre_indexes,3),K.equal(y_test_indexes,4)),

    pos_neg = K.cast(TN, K.floatx()) *(-0.03) + 1 

    return K.categorical_crossentropy(y_pred, y_true)*pos_neg 

(in the code the classes are : 0-A 1-B 3-D 4-E. 2-C is predicting neutral - ignore..)

but fixing to a fixed number of 3% to reduce loss for these cases seems a litle bit hard coded.... Something better inside the categorical_crossentropy math philosophy should be better.

Any sugestions?

Thanks in advance!

  • $\begingroup$ in reinforcement learning, there is a concept called "reward shaping" which seems worthwhile to consider, since what you really care about here is profit, not whether or not the network predicted A or B $\endgroup$ – Sycorax Jan 30 at 19:35
  • $\begingroup$ I am looking at that concept. For this particular case, I would just want to adapt categorical_crossentropy to include "half miss" (or "half right") :-) in some cases of y_test vs y_pred. The identification of these cases in my custom loss (receiving the batch to test...), shown in the post, is already implemented I think it would be just some simple adaptation to the categorical_crossentropy function... For sure some mathematician would make this adjustment with a simple alteration in the categorical_crossentropy function. I think. $\endgroup$ – rjpg Jan 31 at 0:20
  • $\begingroup$ Could you turn the problem in a classification problem with 5 classes like here cs.waikato.ac.nz/~eibe/pubs/ordinal_tech_report.pdf $\endgroup$ – Robin Nicole Jan 31 at 0:58
  • $\begingroup$ that paper seems to be very related to my problem I will have to see in detail! Meanwhile, I was looking to the implementation of cross-entropy in keras and I don't see any harm, to the cross-entropy logic, if I use for the target/real values probabilities. For example, for class A instead of use [1,0,0,0,0] on the y_test use [0.8,0.2,0,0,0]. This way it will subtract the log of two probabilities, the correct one in 80% and the "okay" one with 20%. Does anyone think this solution has some problem? or in cross-entropy only one real class can have the value 1 ? $\endgroup$ – rjpg Jan 31 at 2:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.