Consider a reinforcement learning setup where some parts of the observation space are only partially available, i.e. the corresponding information is sometimes not available. For example consider a setup of four mirrors which are used to guide a laser beam in order to focus it on some target. The objective is to adjust the mirrors in order to focus the beam and keep it focused on the target (the environment might be dynamic, for example the target could slowly move around). The mirrors are equipped with position sensors and hence the observation space consists of the mirror angles as well as the laser beam position at each mirror (and the target).

Example 1

Now in case one of the upstream mirrors is badly adjusted the laser beam won't even reach subsequent mirrors and hence no position information is available there:

Example 2

My question is how to handle this missing information from the observation point of view? In case the beam hits each of the mirrors as well as the target my observation would look like:

$$ \mathrm{obs} = \left(\alpha_1, \alpha_2, \alpha_3, \alpha_4, x_1, x_2, x_3, x_4, x_{target}\right) $$

where $\alpha_i, x_i$ is the angle and laser beam position of mirror $i$.

Now for the example of the badly adjusted mirror, the information $x_3, x_4, x_{target}$ is not available and I'm unsure what to return in such cases? Since the reinforcement learning agent is computing actions from the observations I can't use numbers such as Inf or NaN because they are not compatible with the computational model. I thought about extending the observation space corresponding to the positions and using the new high end for "misses the mirror". E.g. if the mirrors have length 5 cm then the observation space for the position is $[0, 5]$ and I could extend it to $[0, a]$ with $a > 5$ and then assign $x_i = a$ in case the laser beam doesn't reach mirror $i$. I am however worried because the agent probably uses some continuous function of the observations to compute the actions and then the choice of $a$ will introduce a discontinuity. Also if $a$ is close to $5$ it looks like the beam is close to the mirror while in fact it could be anywhere. So the agent's underlying model would probably not differentiate between the ranges $[0, 5]$ and $[5, a]$ and instead assume / infer something wrong.

Another option I have thought about is to use five different agents, the first one being active if only the first mirror is hit, the second agent being active if they beam reaches mirror 2 but not mirror 3 and so on. The fifth agent finally becomes active if the laser beam makes it all the way through until the target. Then however these different agents are somehow disconnected and hence upstream agents never profit from downstream observations. E.g. the first agent might never learn to center the laser beam on the second mirror because once the second mirror is reached (anywhere) not updates to the first agent are performed anymore (though the second agent could perform this "fine-tuning").

Does anyone have an idea how to handle this kind of situation and how to deal with observations that are sometimes not available? Any help is greatly appreciated, thanks in advance.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.