Questions tagged [temporal-difference]
The temporal-difference tag has no usage guidance.
21
questions
24
votes
2
answers
23k
views
When are Monte Carlo methods preferred over temporal difference ones?
I've been doing a lot of research about Reinforcement Learning lately. I followed Sutton & Barto's Reinforcement Learning: An Introduction for most of this.
I know what Markov Decision Processes ...
- 343
20
votes
1
answer
9k
views
Why do temporal difference (TD) methods have lower variance than Monte Carlo methods?
In many reinforcement learning papers, it is stated that for estimating the value function, one of the advantages of using temporal difference methods over the Monte Carlo methods is that they have a ...
- 320
10
votes
1
answer
3k
views
How is TD(1) of TD(lambda) equivalent to Monte Carlo?
In Sutton and Barto's book about RL they say that the TD($\lambda$) algorithm is equivalent to Monte Carlo when $\lambda = 1$. I don't see how that is the case.
They define the lambda return as:
$$...
- 307
4
votes
1
answer
1k
views
Why is temporal difference learning biased in reinforcement learning?
When I learn reinforcement learning from David Silver's online video, I saw "the objective of TD learning, $r_t + \gamma V(s_{t+1})$ is a biased target for learning value function. " I know the ...
- 2,043
2
votes
1
answer
614
views
How to define number of states in reinforcement learning
I'm a robotic engineer who's relatively new to reinforcement learning and I want to try to do simple reinforcement learning on a robot to optimize its velocity. I am however having trouble with ...
- 21
2
votes
1
answer
42
views
What are suitable statistical approaches to examine temporal variation of species abundance/ community composition across multiple sites?
I'm looking for appropriate types of analysis for a data set that contains counts of different crab species across 4 sites with 3 replicates per site (12 in total) over a time period of 1.5 years - 5 ...
- 21
2
votes
0
answers
805
views
In $TD(\lambda)$, is there a way to choose an effective $\lambda$ based on the environment?
In $TD(\lambda)$, is there a way to choose an effective $\lambda$ based on the environment?
- 21
1
vote
1
answer
197
views
Understanding distributional Temporal Difference Learning
I am trying to understand a recently published Deepmind Paper. In the supplementary information of the paper (accessible here), the authors explain distributional temporal difference learning.
In ...
- 145
1
vote
1
answer
2k
views
How to normalise changes that occur over time spans of different duration?
I want to compare changes in a variable that occur over time spans of different duration.
Here is a hypothetical example:
Precipitation in Region A decreased by 300 mm (from 1000 mm to 700 mm) over a ...
- 111
1
vote
2
answers
1k
views
Compare means between samples, while controlling for sampling differences - valid to use regression this way?
There are two independent samples of people, drawn from a population of a city at times $t_1$ and $t_2$, a decade apart*. The people were asked rate their preference regarding some question $Q$ on a ...
- 63
1
vote
0
answers
25
views
investigating and testing temporally overlapping data points in R
This is a two-part question that first asks how to query some data I have in R, and secondly, asks what might be the appropriate statistical operations to test any perceived relationships between ...
- 33
1
vote
0
answers
741
views
Why is least squares temporal difference (LSTD) method more sample efficient compared to Temporal difference (TD) for value function approximation
I am trying to understand how LSTD works in value function approximation. I am reading the preliminaries of this paper. I sort of understand how the LSTD method differs from TD learning.
In TD ...
- 1,126
1
vote
0
answers
82
views
How to analyse temporal variation in R---vigilance of birds
I am currently doing a project for my Bachelor degree, for which I study vigilance in birds. Obviously, I have to analyse the data statistically. However, I do not know where to begin since I do not ...
- 11
0
votes
1
answer
2k
views
What is temporal leakage?
So I've been trying to work out exactly what temporal leakage is for a while now and I'm getting nowhere.
I'm not necessarily looking to code or anything, I'm more so interested in what it actually is ...
0
votes
0
answers
37
views
Would temporal difference learning be unbiased if the value function was initialized to zero?
I was reading this answer to a question that asked why TD learning was biased.
The answer showed that TD learning was biased due to the starting value of the value function:
\begin{align}
V_{n+1}(s) &...
- 1
0
votes
0
answers
16
views
In Sutton & Barto, what is meaning of weight given to actual final return after termination in the Delta return? (equation 12.3)
The equation 12.3, in the book RL An Intro by Sutton & Barto, is shown below with its "diagramatic" explanation in figure 12.2:
Once the episode has terminated at time T, what's the ...
- 3
0
votes
1
answer
65
views
Tsitsiklis and Van Roy’s Counterexample - Reinforcement Learning Understanding Math Derivations
I have been going through "Sutton & Barto Book: Reinforcement Learning: An Introduction", and in "Chapter 11: Off-policy Methods with Approximation", Example 11.1 briefly ...
- 1
0
votes
0
answers
29
views
Modeling Temporal-Interval Distance
I'm trying to define a model for comparing temporal intervals, and can't find an appropriate distance function that incorporates the different relations two temporal intervals might have, e.g. overlap,...
0
votes
1
answer
95
views
Q-Learning [Sutton]: why random variable in formula
Sutton et al. use throughout their book Reinforcement Learning capital letters to describe random variables. At page 131 they introduce Q-Learning.
$Q(S_t,A_t)\leftarrow Q(S_t,A_t) + \alpha [R_{t+1} ...
- 3
0
votes
1
answer
80
views
Confusion about the derivation of the TD-Learning update rule
I am currently trying to understand the paper "Learning to Predict by the Methods of Temporal Differences" by Sutton.
I am stuck with the following step:
(From "Learning to Predict by the Methods of ...
- 111
0
votes
0
answers
125
views
Adding additional constrains to OpenAi Gym
I'm currently working trough some examples which should finally end in a DQN Reinforcement Learning for the CartPole example in the openAI-Gym.
Copied some code from GitHub which isn't deep yet:
<...
- 681