How to handle constraints/undesirable outcomes in Reinforcement Learning What are the methods of handling constrains in RL? As I understand there are two types of environments: constrained (doesn't allow some actions and states) and unconstrained (allows any action and state). 
1. Is adding penalty to reward the only way to handle undesirable outcomes if the environment is unconstrained? 
2. How to handle situation when environment is constrained and agent takes unfeasible action (i.e. tries to insert more items into knapsack than it can actually hold)? Terminate episode early? Add penalty to reward? Anything else?
 A: 
  
*
  
*Is adding penalty to reward the only way to handle undesirable outcomes if the environment is unconstrained? 
  

Yes, pretty much this. The reward function is the feedback mechanism that the agent will use to learn what is marked as undesirable. To avoid catastrophic end results, then pre-training in simulation, and planning phases (also a form of simulation) before acting in the real environment may help, but these will refer to the reward function to discover and avoid any low reward outcome.


  
*How to handle situation when environment is constrained and agent takes unfeasible action (i.e. tries to insert more items into knapsack than it can actually hold)?
  

That is not possible. When an environment is constrained, it means that the action cannot be taken.
If you mean that the agent might select an impossible action due to it being the highest predicted value, or as direct output of a policy function, then there are a few different approaches you can take:


*

*Use information from the environment to filter the agent to only select from allowed actions. If possible this is simple and clean. It is how Alpha Go selects game moves, for example. This does require that the environment provides some way to enumerate or set bounds on the allowed actions.

*Let the environment respond by not changing state and returning some small reward penalty for the time step when an impossible action is selected. This works in environments where it is not possible to tell in advance that an action was not possible, and there is no consequence other than lost time for attempting the impossible action*. This is usually less preferable to filtering the actions, because the agent must use resources to learn the disallowed actions in addition to the rest of the optimisation problem.

*If you are thinking of ending an episode, perhaps with a penalty, then this has effectively turned your constrained environment into an unconstrained one where the agent can fail overall at the task. This might be OK in a training environment with safety features e.g. to stop a robot walking off a table, when in a production environment it may not have the same safety features. Efectively you would simulate just the catastrophic parts of the environment in your constrained training setup, so that the agent learns to avoid undesirable outcomes in an unconstrained environment.

* This is also quite a common scenario described in toy grid world problems, where the agent is allowed to bump into obstacles whilst solving a simple maze. However, those problems are to demonstrate learning algorithms, not necessarily best practice in designing an agent to solve a more complex problem.
