A reward becomes a penalty if

Question

I am working to build a reinforcement agent with DQN. The agent would be able to place buy and sell orders for a day trading purpose. I am facing a little problem with that project. The question is "how to tell the agent to maximize the profit and avoid the transaction where the profit is less than 100$".

I want to maximize the profit inside a trading day and avoid to place the pair (limit buy order, limit sell order) if the profit on that transaction is less than 100$. The idea here is to avoid the little noisy movements. Instead, I prefer long beautiful profitable movements. Be aware that I thought using the "Profit & Loss" as the reward.

"I want the minimal profit per transaction to be 100$" ==> It seems this is not something that is enforceable. I can train the agent to maximize profit per transaction, but how that profit is cannot be ensured.

At the beginning, I wanted to tell the agent, if the profit of a transaction is 50 dollars, I will remove 100 dollars, then It becomes a penalty of 50 dollars for the agent. I thought it was a great way to tell the agent to not place a limit buy order if you are not sure it will give us a minimal profit of 100$. It seems that all I would be doing there is simply shifting the value of the reward. The agent only cares about maximizing the sum of rewards and not taking care of individual transactions.

How to tell the agent to maximize the profit and avoid the transaction where the profit is less than 100$? With that strategy, what guarantee that the agent will never make a buy/sell decision that results in less than 100 dollars profit? Does the sum of reward - # transaction * 100 can be a solution?

Answer 1

Your utility function is basically

$$ U(x) = \max(\$100, x) $$

so all the profits below \$100 are equally bad. Above this, the more profit, the better. The problem is that the function is flat below \$100, so the optimizer can get stuck in such region. To avoid this, you would need to use some kind of optimizer that is able to make "jumps" outside such region, rather then something that only makes incremental improvements (like gradient descent). This would possibly depend on initialization. I am not an expert in reinforced learning, so I don't feel I could give you more detailed hints.

With that strategy, what guarantee that the agent will never make a buy/sell decision that results in less than $100 profit?

Nothing would give you such guarantees. What you are describing is simply an if (profit <= 100) ... else ... block of code inside your agent, that reacts on profits below \$100 (e.g. fails and restarts).