# RL algorithms for continuing task problems

I am stuck with an RL problem where the state space is continuous and the action space is discrete. The problem is a continuing task problem; there are no episode boundaries.

So far, I have tried to solve it using a policy gradient algorithm, considering that episode length equals 1, but the results are pretty bad. I guess the gradient estimation is quite bad with just one sample.

I have been trying to find different RL algorithms for a continuing task problem, but almost everything that I found is suitable for an episodic task. The only algorithm that I have been able to find for a continuing task problem is on the book "RL: an introduction" from Sutton and Barto on section 13.6, which is called "Actor–Critic with Eligibility Traces".

Is it acceptable to use episodic task algorithms with length 1 for a continuing task problem? Is it possible to modify them to handle a continuing task?

Any other algorithms I could use for an RL continuing task?

Update:

The problem I was trying to tackle is a contextual multi-armed bandit problem. The context is a vector on $$\mathbb{R}^6$$ and I have 2 different actions (either 0 or 1). Depending on the context and the action taken, the reward can be positive or negative. The main goal is to maximize the reward.

I was thinking of this problem as having an agent who takes actions in discrete time intervals. For a time interval $$i$$, the agent gets a new context and takes action. At the beginning of the next time interval, it receives a reward. The agent can get any context randomly.

As contexts do not follow any sequence, I was thinking of this as a continuing task. I tried implementing a policy gradient, similar to https://gist.github.com/xkrishnam/d9a62d52d28eb943c3965c6cf631ad30#file-contextualpolicy-n-arm-bandit-ipynb

I have also tried UCB and TS and the results were also quite bad

• You don't need an episodic task to solve it with standard RL approaches, at least not if you are using Temporal Difference (learning, TD). Are you using TD or are you using a Monte Carlo approach? Nov 4, 2021 at 14:11
• @user2974951: OP has apparrently tried to adapt a Monte Carlo approach (some variant of REINFORCE I suspect from the description). Actor-Critic would typically count as a TD approach. Nov 4, 2021 at 15:51
• Updated the post Nov 4, 2021 at 18:13
• Thanks for the update. AFAICS you don't have a continuing MDP environment as you suggest in the question. Given that you don't have an RL problem, but a contextual bandit problem, why do you want to use RL methods and treat it like an RL problem? Could you explain how you know that the results are bad? E.g. is there a known policy for the problem that your agent cannot beat? Do the policies that the agents learn you have made so far beat the random policy, and both of the two fixed policies? Nov 4, 2021 at 20:15