Questions tagged [temporal-difference]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
22 views

How to analyse temporal variation in R---vigilance of birds

I am currently doing a project for my Bachelor degree, for which I study vigilance in birds. Obviously, I have to analyse the data statistically. However, I do not know where to begin since I do not ...
0
votes
0answers
9 views

Does varying maximum achievable reward influence learning for policy gradient methods?

An agent has to go around a map by following a path composed of N checkpoints. The state is the relative position of the next 4 checkpoints. The agent receives +1 every time it takes a checkpoint, and ...
0
votes
0answers
20 views

Model Checking : Safety and Liveness properties [closed]

I know what Safety and Liveness properties are and the relation between Safety and Badprefixes of a LT property. I wanted to understand about the closure properties and why closure of a safety ...
0
votes
1answer
22 views

Q-Learning [Sutton]: why random variable in formula

Sutton et al. use throughout their book Reinforcement Learning capital letters to describe random variables. At page 131 they introduce Q-Learning. $Q(S_t,A_t)\leftarrow Q(S_t,A_t) + \alpha [R_{t+1} ...
0
votes
0answers
17 views

Reinforcement Learning by Sutton: Episodic and Continuing Tasks

Referred to Reinforcement Learning by Sutton: What are Episodic and Continuing Tasks? In my opinion: $\textbf{Episodic Tasks}$ A task is episodic, if there exists a final time step $T$ so that the ...
0
votes
0answers
18 views

Understanding distributional Temporal Difference Learning

I am trying to understand a recently published Deepmind Paper. In the supplementary information of the paper (accessible here), the authors explain distributional temporal difference learning. In ...
1
vote
1answer
169 views

How is TD(1) of TD(lambda) equivalent to Monte Carlo?

In Sutton and Barto's book about RL they say that the TD($\lambda$) algorithm is equivalent to Monte Carlo when $\lambda = 1$. I don't see how that is the case. They define the lambda return as: $$...
1
vote
2answers
89 views

Compare means between samples, while controlling for sampling differences - valid to use regression this way?

There are two independent samples of people, drawn from a population of a city at times $t_1$ and $t_2$, a decade apart*. The people were asked rate their preference regarding some question $Q$ on a ...
0
votes
1answer
21 views

Confusion about the derivation of the TD-Learning update rule

I am currently trying to understand the paper "Learning to Predict by the Methods of Temporal Differences" by Sutton. I am stuck with the following step: (From "Learning to Predict by the Methods of ...
0
votes
0answers
52 views

Adding additional constrains to OpenAi Gym

I'm currently working trough some examples which should finally end in a DQN Reinforcement Learning for the CartPole example in the openAI-Gym. Copied some code from GitHub which isn't deep yet: <...
8
votes
1answer
2k views

Why do temporal difference (TD) methods have lower variance than Monte Carlo methods?

In many reinforcement learning papers, it is stated that for estimating the value function, one of the advantages of using temporal difference methods over the Monte Carlo methods is that they have a ...
11
votes
2answers
7k views

When are Monte Carlo methods preferred over temporal difference ones?

I've been doing a lot of research about Reinforcement Learning lately. I followed Sutton & Barto's Reinforcement Learning: An Introduction for most of this. I know what Markov Decision Processes ...
1
vote
0answers
528 views

In $TD(\lambda)$, is there a way to choose an effective $\lambda$ based on the environment?

In $TD(\lambda)$, is there a way to choose an effective $\lambda$ based on the environment?