Questions tagged [temporal-difference]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
97 views

How is TD(1) of TD(lambda) equivalent to Monte Carlo?

In Sutton and Barto's book about RL they say that the TD($\lambda$) algorithm is equivalent to Monte Carlo when $\lambda = 1$. I don't see how that is the case. They define the lambda return as: $$...
1
vote
2answers
85 views

Compare means between samples, while controlling for sampling differences - valid to use regression this way?

There are two independent samples of people, drawn from a population of a city at times $t_1$ and $t_2$, a decade apart*. The people were asked rate their preference regarding some question $Q$ on a ...
0
votes
1answer
18 views

Confusion about the derivation of the TD-Learning update rule

I am currently trying to understand the paper "Learning to Predict by the Methods of Temporal Differences" by Sutton. I am stuck with the following step: (From "Learning to Predict by the Methods of ...
0
votes
0answers
52 views

Adding additional constrains to OpenAi Gym

I'm currently working trough some examples which should finally end in a DQN Reinforcement Learning for the CartPole example in the openAI-Gym. Copied some code from GitHub which isn't deep yet: <...
8
votes
1answer
1k views

Why do temporal difference (TD) methods have lower variance than Monte Carlo methods?

In many reinforcement learning papers, it is stated that for estimating the value function, one of the advantages of using temporal difference methods over the Monte Carlo methods is that they have a ...
10
votes
2answers
6k views

When are Monte Carlo methods preferred over temporal difference ones?

I've been doing a lot of research about Reinforcement Learning lately. I followed Sutton & Barto's Reinforcement Learning: An Introduction for most of this. I know what Markov Decision Processes ...
1
vote
0answers
519 views

In $TD(\lambda)$, is there a way to choose an effective $\lambda$ based on the environment?

In $TD(\lambda)$, is there a way to choose an effective $\lambda$ based on the environment?