# Questions tagged [temporal-difference]

The tag has no usage guidance.

21 questions
Filter by
Sorted by
Tagged with
37 views

### Would temporal difference learning be unbiased if the value function was initialized to zero?

I was reading this answer to a question that asked why TD learning was biased. The answer showed that TD learning was biased due to the starting value of the value function: \begin{align} V_{n+1}(s) &...
16 views

### In Sutton & Barto, what is meaning of weight given to actual final return after termination in the Delta return? (equation 12.3)

The equation 12.3, in the book RL An Intro by Sutton & Barto, is shown below with its "diagramatic" explanation in figure 12.2: Once the episode has terminated at time T, what's the ...
65 views

### Tsitsiklis and Van Roy’s Counterexample - Reinforcement Learning Understanding Math Derivations

I have been going through "Sutton & Barto Book: Reinforcement Learning: An Introduction", and in "Chapter 11: Off-policy Methods with Approximation", Example 11.1 briefly ...
42 views

### What are suitable statistical approaches to examine temporal variation of species abundance/ community composition across multiple sites?

I'm looking for appropriate types of analysis for a data set that contains counts of different crab species across 4 sites with 3 replicates per site (12 in total) over a time period of 1.5 years - 5 ...
1 vote
25 views

### investigating and testing temporally overlapping data points in R

This is a two-part question that first asks how to query some data I have in R, and secondly, asks what might be the appropriate statistical operations to test any perceived relationships between ...
• 33
29 views

### Modeling Temporal-Interval Distance

I'm trying to define a model for comparing temporal intervals, and can't find an appropriate distance function that incorporates the different relations two temporal intervals might have, e.g. overlap,...
1 vote
742 views

### Why is least squares temporal difference (LSTD) method more sample efficient compared to Temporal difference (TD) for value function approximation

I am trying to understand how LSTD works in value function approximation. I am reading the preliminaries of this paper. I sort of understand how the LSTD method differs from TD learning. In TD ...
• 1,126
2k views

### What is temporal leakage?

So I've been trying to work out exactly what temporal leakage is for a while now and I'm getting nowhere. I'm not necessarily looking to code or anything, I'm more so interested in what it actually is ...
1 vote
2k views

### How to normalise changes that occur over time spans of different duration?

I want to compare changes in a variable that occur over time spans of different duration. Here is a hypothetical example: Precipitation in Region A decreased by 300 mm (from 1000 mm to 700 mm) over a ...
• 111
615 views

### How to define number of states in reinforcement learning

I'm a robotic engineer who's relatively new to reinforcement learning and I want to try to do simple reinforcement learning on a robot to optimize its velocity. I am however having trouble with ...
• 21
1k views

### Why is temporal difference learning biased in reinforcement learning?

When I learn reinforcement learning from David Silver's online video, I saw "the objective of TD learning, $r_t + \gamma V(s_{t+1})$ is a biased target for learning value function. " I know the ...
• 2,043
1 vote
82 views

### How to analyse temporal variation in R---vigilance of birds

I am currently doing a project for my Bachelor degree, for which I study vigilance in birds. Obviously, I have to analyse the data statistically. However, I do not know where to begin since I do not ...
• 11