Questions tagged [temporal-difference]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
14 views

How can I quantify temporal variation into a single value without losing magnitude?

For my ecological research on the impacts of intensity of space-use on vegetation structure I need to quantify temporal variation in the intensity of space-use across years (resulting in a single ...
user avatar
  • 1
0 votes
0 answers
27 views

Modeling Temporal-Interval Distance

I'm trying to define a model for comparing temporal intervals, and can't find an appropriate distance function that incorporates the different relations two temporal intervals might have, e.g. overlap,...
user avatar
0 votes
0 answers
30 views

Stationarizing Time-Series

This is a general question about stationary time series. Consider a time-series t with frequency k (e.g., 52 for weekly data). If this time series was non-stationary such that a single difference (i.e....
user avatar
1 vote
0 answers
102 views

Why is least squares temporal difference (LSTD) method more sample efficient compared to Temporal difference (TD) for value function approximation

I am trying to understand how LSTD works in value function approximation. I am reading the preliminaries of this paper. I sort of understand how the LSTD method differs from TD learning. In TD ...
user avatar
  • 876
0 votes
1 answer
783 views

What is temporal leakage?

So I've been trying to work out exactly what temporal leakage is for a while now and I'm getting nowhere. I'm not necessarily looking to code or anything, I'm more so interested in what it actually is ...
user avatar
0 votes
1 answer
355 views

How to normalise changes that occur over time spans of different duration?

I want to compare changes in a variable that occur over time spans of different duration. Here is a hypothetical example: Precipitation in Region A decreased by 300 mm (from 1000 mm to 700 mm) over a ...
user avatar
  • 101
2 votes
1 answer
312 views

How to define number of states in reinforcement learning

I'm a robotic engineer who's relatively new to reinforcement learning and I want to try to do simple reinforcement learning on a robot to optimize its velocity. I am however having trouble with ...
user avatar
3 votes
1 answer
468 views

Why is temporal difference learning biased in reinforcement learning?

When I learn reinforcement learning from David Silver's online video, I saw "the objective of TD learning, $r_t + \gamma V(s_{t+1})$ is a biased target for learning value function. " I know the ...
user avatar
  • 1,643
1 vote
0 answers
50 views

How to analyse temporal variation in R---vigilance of birds

I am currently doing a project for my Bachelor degree, for which I study vigilance in birds. Obviously, I have to analyse the data statistically. However, I do not know where to begin since I do not ...
user avatar
  • 11
0 votes
1 answer
59 views

Q-Learning [Sutton]: why random variable in formula

Sutton et al. use throughout their book Reinforcement Learning capital letters to describe random variables. At page 131 they introduce Q-Learning. $Q(S_t,A_t)\leftarrow Q(S_t,A_t) + \alpha [R_{t+1} ...
user avatar
  • 3
1 vote
1 answer
161 views

Understanding distributional Temporal Difference Learning

I am trying to understand a recently published Deepmind Paper. In the supplementary information of the paper (accessible here), the authors explain distributional temporal difference learning. In ...
user avatar
  • 145
6 votes
1 answer
2k views

How is TD(1) of TD(lambda) equivalent to Monte Carlo?

In Sutton and Barto's book about RL they say that the TD($\lambda$) algorithm is equivalent to Monte Carlo when $\lambda = 1$. I don't see how that is the case. They define the lambda return as: $$...
user avatar
1 vote
2 answers
565 views

Compare means between samples, while controlling for sampling differences - valid to use regression this way?

There are two independent samples of people, drawn from a population of a city at times $t_1$ and $t_2$, a decade apart*. The people were asked rate their preference regarding some question $Q$ on a ...
user avatar
0 votes
1 answer
51 views

Confusion about the derivation of the TD-Learning update rule

I am currently trying to understand the paper "Learning to Predict by the Methods of Temporal Differences" by Sutton. I am stuck with the following step: (From "Learning to Predict by the Methods of ...
user avatar
0 votes
0 answers
96 views

Adding additional constrains to OpenAi Gym

I'm currently working trough some examples which should finally end in a DQN Reinforcement Learning for the CartPole example in the openAI-Gym. Copied some code from GitHub which isn't deep yet: <...
user avatar
16 votes
1 answer
5k views

Why do temporal difference (TD) methods have lower variance than Monte Carlo methods?

In many reinforcement learning papers, it is stated that for estimating the value function, one of the advantages of using temporal difference methods over the Monte Carlo methods is that they have a ...
user avatar
  • 280
17 votes
2 answers
16k views

When are Monte Carlo methods preferred over temporal difference ones?

I've been doing a lot of research about Reinforcement Learning lately. I followed Sutton & Barto's Reinforcement Learning: An Introduction for most of this. I know what Markov Decision Processes ...
user avatar
  • 173
1 vote
0 answers
667 views

In $TD(\lambda)$, is there a way to choose an effective $\lambda$ based on the environment?

In $TD(\lambda)$, is there a way to choose an effective $\lambda$ based on the environment?
user avatar