1
$\begingroup$

What are the methods of handling constrains in RL? As I understand there are two types of environments: constrained (doesn't allow some actions and states) and unconstrained (allows any action and state). 1. Is adding penalty to reward the only way to handle undesirable outcomes if the environment is unconstrained? 2. How to handle situation when environment is constrained and agent takes unfeasible action (i.e. tries to insert more items into knapsack than it can actually hold)? Terminate episode early? Add penalty to reward? Anything else?

$\endgroup$

1 Answer 1

2
$\begingroup$
  1. Is adding penalty to reward the only way to handle undesirable outcomes if the environment is unconstrained?

Yes, pretty much this. The reward function is the feedback mechanism that the agent will use to learn what is marked as undesirable. To avoid catastrophic end results, then pre-training in simulation, and planning phases (also a form of simulation) before acting in the real environment may help, but these will refer to the reward function to discover and avoid any low reward outcome.

  1. How to handle situation when environment is constrained and agent takes unfeasible action (i.e. tries to insert more items into knapsack than it can actually hold)?

That is not possible. When an environment is constrained, it means that the action cannot be taken.

If you mean that the agent might select an impossible action due to it being the highest predicted value, or as direct output of a policy function, then there are a few different approaches you can take:

  • Use information from the environment to filter the agent to only select from allowed actions. If possible this is simple and clean. It is how Alpha Go selects game moves, for example. This does require that the environment provides some way to enumerate or set bounds on the allowed actions.

  • Let the environment respond by not changing state and returning some small reward penalty for the time step when an impossible action is selected. This works in environments where it is not possible to tell in advance that an action was not possible, and there is no consequence other than lost time for attempting the impossible action*. This is usually less preferable to filtering the actions, because the agent must use resources to learn the disallowed actions in addition to the rest of the optimisation problem.

  • If you are thinking of ending an episode, perhaps with a penalty, then this has effectively turned your constrained environment into an unconstrained one where the agent can fail overall at the task. This might be OK in a training environment with safety features e.g. to stop a robot walking off a table, when in a production environment it may not have the same safety features. Efectively you would simulate just the catastrophic parts of the environment in your constrained training setup, so that the agent learns to avoid undesirable outcomes in an unconstrained environment.


* This is also quite a common scenario described in toy grid world problems, where the agent is allowed to bump into obstacles whilst solving a simple maze. However, those problems are to demonstrate learning algorithms, not necessarily best practice in designing an agent to solve a more complex problem.

$\endgroup$
4
  • $\begingroup$ Just need some clarification on 2. Autonomous aircraft wants to make a sharp turn of 45 degrees. Because of laws of physics it is possible to turn only 5 so the environment turns the aircraft by 5 and provides feedback via state that aircraft turned only by 5. Then the agent uses 5 (instead of 45) when updating his neural net (DDQN). Or should it use penalize reward? I see several messages: for a given state action 45 is not feasible while action 5 is. How to communicate this to value (DDQN), policy(DSPG), or both(A2C) networks? $\endgroup$
    – TFbie
    Commented Nov 15, 2019 at 15:16
  • $\begingroup$ @TFbie: The agent should be updated based on the output that was used. If you constrain after accepting the output (e.g. by servo motors refusing to use the whole signal), then that is a property of the environment, and there is no consequence to the agent. I would not even penalise in that scenario, just take the action as whatever the agent sends to the control box. $\endgroup$ Commented Nov 15, 2019 at 15:19
  • $\begingroup$ If I understand correctly agent should be updated on the action he has taken. In servo motor case it is the whole signal. It doesn't matter that servo motor refused the whole signal. In real world environment may be constrained by physical laws and by undesirable outcomes (i.e. unmanned aircraft flying over a country which doesn't allow this). So in the first case agent is updated based on action he has taken - not the one which was realized by environment - while in the second agent is penalized. Correct? $\endgroup$
    – TFbie
    Commented Nov 16, 2019 at 0:53
  • $\begingroup$ @TFbie: Yes. In your example case, you may need to adjust what you think of as the boundaries of the agent. The agent is only the decision-making component. The servo motor in your example is not part of the agent, even though it directly converts the agent's decision into movement. In the same way you might think of the "agent" of an animal being its nervous system, and not include its muscles or body. $\endgroup$ Commented Nov 16, 2019 at 8:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.