Questions tagged [temporal-difference]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
37 views

Would temporal difference learning be unbiased if the value function was initialized to zero?

I was reading this answer to a question that asked why TD learning was biased. The answer showed that TD learning was biased due to the starting value of the value function: \begin{align} V_{n+1}(s) &...
hk1510's user avatar
  • 1
0 votes
0 answers
16 views

In Sutton & Barto, what is meaning of weight given to actual final return after termination in the Delta return? (equation 12.3)

The equation 12.3, in the book RL An Intro by Sutton & Barto, is shown below with its "diagramatic" explanation in figure 12.2: Once the episode has terminated at time T, what's the ...
Kanishk's user avatar
0 votes
1 answer
65 views

Tsitsiklis and Van Roy’s Counterexample - Reinforcement Learning Understanding Math Derivations

I have been going through "Sutton & Barto Book: Reinforcement Learning: An Introduction", and in "Chapter 11: Off-policy Methods with Approximation", Example 11.1 briefly ...
Emre Y.'s user avatar
2 votes
1 answer
42 views

What are suitable statistical approaches to examine temporal variation of species abundance/ community composition across multiple sites?

I'm looking for appropriate types of analysis for a data set that contains counts of different crab species across 4 sites with 3 replicates per site (12 in total) over a time period of 1.5 years - 5 ...
Susanne Bähr's user avatar
1 vote
0 answers
25 views

investigating and testing temporally overlapping data points in R

This is a two-part question that first asks how to query some data I have in R, and secondly, asks what might be the appropriate statistical operations to test any perceived relationships between ...
Wangana's user avatar
  • 33
0 votes
0 answers
29 views

Modeling Temporal-Interval Distance

I'm trying to define a model for comparing temporal intervals, and can't find an appropriate distance function that incorporates the different relations two temporal intervals might have, e.g. overlap,...
Qais Abou Housien's user avatar
1 vote
0 answers
742 views

Why is least squares temporal difference (LSTD) method more sample efficient compared to Temporal difference (TD) for value function approximation

I am trying to understand how LSTD works in value function approximation. I am reading the preliminaries of this paper. I sort of understand how the LSTD method differs from TD learning. In TD ...
calveeen's user avatar
  • 1,126
0 votes
1 answer
2k views

What is temporal leakage?

So I've been trying to work out exactly what temporal leakage is for a while now and I'm getting nowhere. I'm not necessarily looking to code or anything, I'm more so interested in what it actually is ...
Fluffyrox4's user avatar
1 vote
1 answer
2k views

How to normalise changes that occur over time spans of different duration?

I want to compare changes in a variable that occur over time spans of different duration. Here is a hypothetical example: Precipitation in Region A decreased by 300 mm (from 1000 mm to 700 mm) over a ...
M_S's user avatar
  • 111
2 votes
1 answer
615 views

How to define number of states in reinforcement learning

I'm a robotic engineer who's relatively new to reinforcement learning and I want to try to do simple reinforcement learning on a robot to optimize its velocity. I am however having trouble with ...
Mr_Melon's user avatar
4 votes
1 answer
1k views

Why is temporal difference learning biased in reinforcement learning?

When I learn reinforcement learning from David Silver's online video, I saw "the objective of TD learning, $r_t + \gamma V(s_{t+1})$ is a biased target for learning value function. " I know the ...
DiveIntoML's user avatar
  • 2,043
1 vote
0 answers
82 views

How to analyse temporal variation in R---vigilance of birds

I am currently doing a project for my Bachelor degree, for which I study vigilance in birds. Obviously, I have to analyse the data statistically. However, I do not know where to begin since I do not ...
Kalia's user avatar
  • 11
0 votes
1 answer
96 views

Q-Learning [Sutton]: why random variable in formula

Sutton et al. use throughout their book Reinforcement Learning capital letters to describe random variables. At page 131 they introduce Q-Learning. $Q(S_t,A_t)\leftarrow Q(S_t,A_t) + \alpha [R_{t+1} ...
GeoRie's user avatar
  • 3
1 vote
1 answer
197 views

Understanding distributional Temporal Difference Learning

I am trying to understand a recently published Deepmind Paper. In the supplementary information of the paper (accessible here), the authors explain distributional temporal difference learning. In ...
Doc's user avatar
  • 145
10 votes
1 answer
3k views

How is TD(1) of TD(lambda) equivalent to Monte Carlo?

In Sutton and Barto's book about RL they say that the TD($\lambda$) algorithm is equivalent to Monte Carlo when $\lambda = 1$. I don't see how that is the case. They define the lambda return as: $$...
Stefan Dimeski's user avatar
1 vote
2 answers
1k views

Compare means between samples, while controlling for sampling differences - valid to use regression this way?

There are two independent samples of people, drawn from a population of a city at times $t_1$ and $t_2$, a decade apart*. The people were asked rate their preference regarding some question $Q$ on a ...
user3554004's user avatar
0 votes
1 answer
80 views

Confusion about the derivation of the TD-Learning update rule

I am currently trying to understand the paper "Learning to Predict by the Methods of Temporal Differences" by Sutton. I am stuck with the following step: (From "Learning to Predict by the Methods of ...
Jonas Eschmann's user avatar
0 votes
0 answers
125 views

Adding additional constrains to OpenAi Gym

I'm currently working trough some examples which should finally end in a DQN Reinforcement Learning for the CartPole example in the openAI-Gym. Copied some code from GitHub which isn't deep yet: <...
Mr.Sh4nnon's user avatar
20 votes
1 answer
9k views

Why do temporal difference (TD) methods have lower variance than Monte Carlo methods?

In many reinforcement learning papers, it is stated that for estimating the value function, one of the advantages of using temporal difference methods over the Monte Carlo methods is that they have a ...
Infintyyy's user avatar
  • 320
24 votes
2 answers
23k views

When are Monte Carlo methods preferred over temporal difference ones?

I've been doing a lot of research about Reinforcement Learning lately. I followed Sutton & Barto's Reinforcement Learning: An Introduction for most of this. I know what Markov Decision Processes ...
Anne-dirk's user avatar
  • 343
2 votes
0 answers
805 views

In $TD(\lambda)$, is there a way to choose an effective $\lambda$ based on the environment?

In $TD(\lambda)$, is there a way to choose an effective $\lambda$ based on the environment?
HenrySky's user avatar