0
$\begingroup$

I am working to build a reinforcement agent with DQN. The agent would be able to place buy and sell orders for a day trading purpose. I am facing a little problem with that project. The question is "how to tell the agent to maximize the profit and avoid the transaction where the profit is less than 100$".

I want to maximize the profit inside a trading day and avoid to place the pair (limit buy order, limit sell order) if the profit on that transaction is less than 100$. The idea here is to avoid the little noisy movements. Instead, I prefer long beautiful profitable movements. Be aware that I thought using the "Profit & Loss" as the reward.

"I want the minimal profit per transaction to be 100$" ==> It seems this is not something that is enforceable. I can train the agent to maximize profit per transaction, but how that profit is cannot be ensured.

At the beginning, I wanted to tell the agent, if the profit of a transaction is 50 dollars, I will remove 100 dollars, then It becomes a penalty of 50 dollars for the agent. I thought it was a great way to tell the agent to not place a limit buy order if you are not sure it will give us a minimal profit of 100$. It seems that all I would be doing there is simply shifting the value of the reward. The agent only cares about maximizing the sum of rewards and not taking care of individual transactions.

How to tell the agent to maximize the profit and avoid the transaction where the profit is less than 100$? With that strategy, what guarantee that the agent will never make a buy/sell decision that results in less than 100 dollars profit? Does the sum of reward - # transaction * 100 can be a solution?

$\endgroup$

1 Answer 1

1
$\begingroup$

Your utility function is basically

$$ U(x) = \max(\$100, x) $$

so all the profits below \$100 are equally bad. Above this, the more profit, the better. The problem is that the function is flat below \$100, so the optimizer can get stuck in such region. To avoid this, you would need to use some kind of optimizer that is able to make "jumps" outside such region, rather then something that only makes incremental improvements (like gradient descent). This would possibly depend on initialization. I am not an expert in reinforced learning, so I don't feel I could give you more detailed hints.

With that strategy, what guarantee that the agent will never make a buy/sell decision that results in less than $100 profit?

Nothing would give you such guarantees. What you are describing is simply an if (profit <= 100) ... else ... block of code inside your agent, that reacts on profits below \$100 (e.g. fails and restarts).

$\endgroup$
4
  • $\begingroup$ With that strategy, what guarantee that the agent will never make a buy/sell decision that results in less than $100 profit? $\endgroup$
    – fgauth
    Jan 18, 2019 at 17:59
  • 1
    $\begingroup$ @fgauth of course you can never guarantee that you won't lose money / make less than $100 on a trade. If you could, you'd be rich. $\endgroup$
    – shimao
    Jan 19, 2019 at 15:11
  • $\begingroup$ You answer is well, but I am not a fan. If the transaction gives a negative profit, then the reward function gives a minimal reward of $100. We need to punish the agent if he placed a pair (limit buy order, limit sell order) that occurred a negative profit. $\endgroup$
    – fgauth
    Jan 19, 2019 at 21:43
  • $\begingroup$ @fgauth so the profit that is greater then zero, but <\$100 is acceptable? You said that it is unacceptable for the profit to go below \$100. $\endgroup$
    – Tim
    Jan 20, 2019 at 7:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.