0
$\begingroup$

I am stuck with an RL problem where the state space is continuous and the action space is discrete. The problem is a continuing task problem; there are no episode boundaries.

So far, I have tried to solve it using a policy gradient algorithm, considering that episode length equals 1, but the results are pretty bad. I guess the gradient estimation is quite bad with just one sample.

I have been trying to find different RL algorithms for a continuing task problem, but almost everything that I found is suitable for an episodic task. The only algorithm that I have been able to find for a continuing task problem is on the book "RL: an introduction" from Sutton and Barto on section 13.6, which is called "Actor–Critic with Eligibility Traces".

Is it acceptable to use episodic task algorithms with length 1 for a continuing task problem? Is it possible to modify them to handle a continuing task?

Any other algorithms I could use for an RL continuing task?

Update:

The problem I was trying to tackle is a contextual multi-armed bandit problem. The context is a vector on $\mathbb{R}^6$ and I have 2 different actions (either 0 or 1). Depending on the context and the action taken, the reward can be positive or negative. The main goal is to maximize the reward.

I was thinking of this problem as having an agent who takes actions in discrete time intervals. For a time interval $i$, the agent gets a new context and takes action. At the beginning of the next time interval, it receives a reward. The agent can get any context randomly.

As contexts do not follow any sequence, I was thinking of this as a continuing task. I tried implementing a policy gradient, similar to https://gist.github.com/xkrishnam/d9a62d52d28eb943c3965c6cf631ad30#file-contextualpolicy-n-arm-bandit-ipynb

I have also tried UCB and TS and the results were also quite bad

$\endgroup$
4
  • 1
    $\begingroup$ You don't need an episodic task to solve it with standard RL approaches, at least not if you are using Temporal Difference (learning, TD). Are you using TD or are you using a Monte Carlo approach? $\endgroup$ Commented Nov 4, 2021 at 14:11
  • $\begingroup$ @user2974951: OP has apparrently tried to adapt a Monte Carlo approach (some variant of REINFORCE I suspect from the description). Actor-Critic would typically count as a TD approach. $\endgroup$ Commented Nov 4, 2021 at 15:51
  • $\begingroup$ Updated the post $\endgroup$
    – herox
    Commented Nov 4, 2021 at 18:13
  • 2
    $\begingroup$ Thanks for the update. AFAICS you don't have a continuing MDP environment as you suggest in the question. Given that you don't have an RL problem, but a contextual bandit problem, why do you want to use RL methods and treat it like an RL problem? Could you explain how you know that the results are bad? E.g. is there a known policy for the problem that your agent cannot beat? Do the policies that the agents learn you have made so far beat the random policy, and both of the two fixed policies? $\endgroup$ Commented Nov 4, 2021 at 20:15

1 Answer 1

0
$\begingroup$

In practice RL is going to be fit with episodes of finite duration.

If your problem genuinely involves episodes of infinite duration where any reward is only known at the end then nothing will work.

So, you must cast your problem as multiple finite length episodes. You may need to set up boundary information at the start of an episode. You may also need to work out intermediate reward signals.

$\endgroup$
3
  • 1
    $\begingroup$ OP has not said anything about the reward structure. Having a reward "at the end" in a continuing problem would be a poor problem description, so hopefully the OP has not done that. $\endgroup$ Commented Nov 4, 2021 at 15:49
  • $\begingroup$ Just updated the post $\endgroup$
    – herox
    Commented Nov 4, 2021 at 18:13
  • $\begingroup$ Neil, I'm not sure what the source of your confusion is, but if the reward structure is not tied to episode boundaries then the 'problem' is less problematic. For instance an independent per-timestep reward on a problem with no episode boundary is trivial to cut up. $\endgroup$ Commented Nov 5, 2021 at 9:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.