1
$\begingroup$

The RL model I am building is one form of DQN. Its internal network is a regular deep NN.

In the application I am looking at, there is a cost for taking a different action (compared to the previous action).

That means, even when the model achieves a positive reward r when going from t to t+1, this r would be reduced by the cost if the action taken is different from the last action. Currently the model inputs are the environment data.

Is there any way to penalise the model for choosing to take different actions often? Or any other form of constraints that I could apply to the model?

$\endgroup$

2 Answers 2

1
$\begingroup$

As I understand your description, the previous action taken influences the distribution of reward values possible from the current time step. That means effectively the previous action should be part of the current state in order to maintain the Markov property - which is theoretically required so that the agent can correctly assign expected returns to action values.

Add the previous action taken to the state features - for a neural network approximator in DQN, I would expect that to be a one-hot-encoded feature, i.e. multiple new input fields.

This will correctly model the MDP as you need it to describe your problem. Everything else should then fall into place then with no need to change how the DQN solver works.

$\endgroup$
2
  • $\begingroup$ Hello Neil, if I understand you correctly, are you suggesting that the only thing I should do is make the last action (in one hot encoding format) an input to the current neural network in DQN? $\endgroup$
    – ZXY
    Jul 18, 2020 at 22:27
  • $\begingroup$ @ZXY: Yes. More formally it should be considered part of the state. $\endgroup$ Jul 19, 2020 at 7:50
0
$\begingroup$

If your aim is to "penalize the model for choosing to take different actions often", one possibility is to penalize the entropy of the policy on top of the usual expected future reward with some tradeoff coefficient $\lambda > 0$:

$$ E_{\pi}\left[ \sum_{t=0}^T \mathcal{R}(s_t,s_{t+1},a_t) - \lambda H(\pi(a \vert s_t)\right]$$

Note this is the opposite of what is commonly done to avoid collapse of the policy to greedy behavior, and you should be very careful with this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.