Neural Network : Counter-intuitive result from a binary classification, perhaps due to experimental design?

Question

I am setting up a binary classification problem where if a trade is profitable it gets a 1, and if it is unprofitable, it gets a 0.

I have some independent variables, one of which is price_of_security. I have the dependent variable of trade_profit. I have data on about 20,000 such trades. I train a neural network on this data and then save the model. Next, I take 1 data point and make ten copies of it with increments of the price going down, and increments of the price going up, I hold all of the other independent variables constant. Like so:

|ticker|  price|
|------|-------|
|   WTO| 216.94|
|   WTO| 217.94|
|   WTO| 218.94|
|   WTO| 219.94|
|   WTO| 220.94| 
|   WTO| 221.94| # This is the original price
|   WTO| 222.94|
|   WTO| 223.94|
|   WTO| 224.94|
|   WTO| 225.94|

I then put this data into my saved neural network and see if the neural network associated a lower price with a higher chance of it being a successful trade. I believe that it intuitively should associate a cheaper price with a higher chance of success and also associate higher prices with lower chances of success. My reasoning is that if the fair value of a security is 215, then my model should not recommend a buy at 220 and it should recommend a buy at 210. However, this is what the model predicts:

|ticker|  price| probability_1| label|
|------|-------|--------------|------|
|   WTO| 216.94|          0.32|     0|
|   WTO| 217.94|          0.37|     0|
|   WTO| 218.94|          0.42|     0|
|   WTO| 219.94|          0.44|     0|
|   WTO| 220.94|          0.49|     0|
|   WTO| 221.94|          0.51|     1|
|   WTO| 222.94|          0.56|     1|
|   WTO| 223.94|          0.57|     1|
|   WTO| 224.94|          0.58|     1|
|   WTO| 225.94|          0.60|     1|

The model, however, finds that higher prices are associated with a higher chance of the trade being a 1 (aka a successful trade). I am trying to understand what is going on so I did a table of all the data I have to see what the model is learning. This is the table:

|    bin    |   num trades | ratio succes|
|-----------|--------------|-------------|
|   200-205 |           568|         0.85|
|   205-210 |           754|         0.79|
|   210-215 |          1200|         0.78|
|   215-220 |          4452|         0.55|
|   220-225 |          6783|         0.59|
|   225-230 |          8197|         0.68|

After a lot of thinking, I think the reason that I'm seeing the model associate a higher chance of success with a higher price is because there are so more (in total) successful trades in the higher-prices. Perhaps those trades make less money per trade, but the trained model only cares about the 0 or the 1, not about the magnitude of the profit. I think that is why I'm seeing this counter-intuitive results.

Should this experiment be set up as a regression problem rather than a classification problem?

Answer 1

You are probably picking up a spurious correlation because your input variable is not stationary. One common way of getting a (more) stationary series is to take the first differences.

Some leads: https://www.researchgate.net/post/Is_it_necessary_to_ensure_stationarity_of_all_time_series_variables_when_you_run_a_Vector_Autoregressive_VAR_Model

In this particular instance, one would expect such results if the price was stagnant for a long period of time below 221.94 and then rallied with minimal pullbacks after breaking that level.