Questions tagged [q-learning]

A popular reinforcement learning algorithm, an instance of TD (temporal difference) learning.

Filter by
Sorted by
Tagged with
2
votes
1answer
18 views

SARSA when policy is not epilon greedy

I would like to clarify a doubt that I have regarding SARSA. SARSA can be used for optimal control when the policy to take action $a$ is epsilon greedy. Suppose that the policy to take action $a$ is ...
3
votes
1answer
47 views

Association between Current state/action and the far future reward

How the Agent make the association between the current $Q_t(s_t,a_t)$ and a far future reward that by nature of my environment we get reward at least after 10-15 time steps from the action taken. If ...
0
votes
1answer
23 views

Understanding the Q-learning loss function?

Perhaps this can be explained a little more to me. I understand what's in literature but I'm struggling to understand why this is the preferred loss. If we have an agent that can move ↑↓→← and for ...
1
vote
1answer
21 views

Learning in the bit flipping environment

I'm looking at the Andrychowicz et al. paper and running through the gauntlet of implementing a DQN and then implementing a DQN with HER. In this paper, they mention a bit-flipping environment: a ...
0
votes
0answers
55 views

As epsilon decays, rewards gets worse during exploitation than exploration

I am currently trying to write learning agent from the "Human Level Control in DRL" Paper in Tensorflow 2.0. I've copied the recommended hyperparameters and picked the easiest environment ...
0
votes
0answers
23 views

sarsa, epxected sarsa, and qlearning convergence

After implementing all 3 algorithms for problem X, it turns out that sarsa converges significantly faster than both expected sarsa and qlearning. What causes this? Is this affected by the problem? or ...
0
votes
0answers
8 views

Q-learning with evaluation more the 1 step ahead

I am trying to make an AI play an arcade game. The game requires some complex decisions and the agent is not rewarded instantly after performing an action. His projectiles take time to travel and some ...
1
vote
0answers
23 views

Why and under what conditions does Q learning converge?

I am looking for a modern proof on why Q learning converges in the tabular setting. I've skimmed the original proof by Dayan and Watkins and I have to say that the terminology and approach are a bit ...
1
vote
1answer
13 views

Does gradient descent work for tabular Q learning?

Suppose I have a tabular Q learning problem such as grid-world. Let our loss be defined as, $$\hat{L}(Q)=0.5(Q(s,a)-(r+\gamma\max_{a'}{Q(s',a')}))^2$$ Then $Q_{k+1}(s,a) = Q_k(s,a) - \eta \nabla \hat {...
1
vote
2answers
23 views

In Reinforcement Learning (DQN), is there a way to constrain/penalise the model so that it doesn't take a different action very often?

The RL model I am building is one form of DQN. Its internal network is a regular deep NN. In the application I am looking at, there is a cost for taking a different action (compared to the previous ...
1
vote
0answers
17 views

How to define a Reinforcement Learning episode when dealing with dataset of customer's features?

so one of friends sent me some problem he was working on lately and I couldn't help but to wonder how could it be solved using Q-learning. The statement is as follows : Given the following datasets, ...
1
vote
0answers
17 views

Equivalence of state-action value and value function?

In Reinforcement Learning (RL) two of the most widespread measures of performance for a policy $\pi$ are the value function $$V^\pi\left(s\right)=\mathbb{E}_\pi\left[\sum_{i=0}^{\infty}{\gamma^ir_{t+i}...
0
votes
0answers
22 views

Dueling DQN - solving the identifiability problem

I want to implement a Dueling DQN. I know that the idea is to estimate V (scalar), the value of a given state, and A (vector) - the advantage, separately. A is the difference between V and the Q ...
2
votes
0answers
42 views

Preprocessing data for the learning step

I am currently reading "Human level control through deep reinforcement learning" and I came across the algorithm in the paper. I am confused because the algorithm uses a different notation ...
0
votes
0answers
21 views

Deep Q learning

Ok so I have two questions When we back propagate and update our weights, does it immediately update our Q values and Target Q values(If using just one neural network) is the updating process a slow ...
0
votes
0answers
8 views

My Double DQN with Experience Replay produces a no-action decision most of the time. Why?

I've written a Double DQN-based stock trading bot using mainly time series stock data. The internal network of the Double DQN is a LSTM which handles the time series data. An Experience Replay buffer ...
-1
votes
1answer
48 views

Deep Q learning algorithm

Ok I’m a bit confused with this code, what exactly is a time step, isn’t it like when an action is performed,it goes to the next time step, and also, the gradient descent steps is a repeat until ...
2
votes
1answer
43 views

In a DQN, can Prioritized Experience Replay actually perform worse than a regular Experience Replay?

I've written a Double DQN-based stock trading bot using mainly time series stock data. I've recently upgraded my Experience Replay(ER) code with a version of Prioritized Experience Replay (PER) ...
0
votes
0answers
19 views

Instability in Multi-Agent Reinforcement Learning

I've got a program tackling a Multi-Agent Markov Decision Process using Deep Reinforcement Learning. To solve the problem I've tried a few algorithms broadly belonging to these two classes: A2C (...
2
votes
2answers
48 views

Can an agent be trained in a completely random environment (the rules and actions stay the same)

I am specifically talking about the FrozenLake example from openAI gym to illustrate my question. https://gym.openai.com/envs/FrozenLake-v0/ The way I understand how Q-Learning or DQL works on the ...
0
votes
1answer
36 views

Consider more than just a single state-action pair before giving a reward in deep q learning?

TLDR If the environment is rather complex (momentum, velocity, ...), but the action space is rather easy (left, right), is it not better to take multiple decisions into account before giving a reward ...
2
votes
1answer
45 views

Q-learning update formula

I am a beginner and it is my first question. I know that Q-learning update equation is: $Q(s_t, a_t) = Q(s_t, a_t)+α(r_{t+1} +γ·max_AQ(s_{t+1}, a_t)−Q(s_t, a_t))$ But in some of the researches it is ...
0
votes
1answer
38 views

Q-Learning [Sutton]: why random variable in formula

Sutton et al. use throughout their book Reinforcement Learning capital letters to describe random variables. At page 131 they introduce Q-Learning. $Q(S_t,A_t)\leftarrow Q(S_t,A_t) + \alpha [R_{t+1} ...
0
votes
0answers
140 views

Reinforcement Learning by Sutton: Episodic and Continuing Tasks

Referred to Reinforcement Learning by Sutton: What are Episodic and Continuing Tasks? In my opinion: $\textbf{Episodic Tasks}$ A task is episodic, if there exists a final time step $T$ so that the ...
0
votes
0answers
30 views

Model-based Reinforcement Learning: Calculate expected Reward

I'm very new to reinforcement learning and I try to understand the concept of it. In modelbased learning, there is a known probability distribution of the rewards and of the transitions. So let's say ...
1
vote
0answers
104 views

Expected SARSA, SARSA and Q-learning

I would much appreciate if you could point me in the right direction regarding this question about targets for approximate ...
1
vote
1answer
85 views

Deep Q Learning best practice

I'm new in deep q-learning and I have understood the main concepts of it and I'm trying to solve problems with DQL. The problem is that I don't know how to initialize some key values of the algorithm ...
0
votes
1answer
16 views

Small difference of q-function between different actions for the same state

I am trying out reinforcement learning using Q-learning. The data come from some made-up equations so I have infinite number of data. One thing that troubles me is after I learn the Q-function, I use ...
3
votes
1answer
168 views

DQN - agent doesn't improve policy

I have a simple grid environment. The player is in the upper left corner and it's goal is to get to lower right corner. The player receives +0.2 points for moving in the direction of the goal, -0.2 ...
1
vote
1answer
352 views

Delayed Rewards in Reinforcement Learning

I have an MDP where the rewards are delayed for six steps as follows: The reward from action at time t is received when the action at time t+6 is taken. The reward from action at time t+1 is ...
0
votes
0answers
35 views

How to use data from other policies in order to find optimal policy in model-free rl?

I am struggling to understand whether experience from one policy can be used to find optimal policy. Suppose that I have gathered many data (state, action, reward, next_state) by following random ...
0
votes
1answer
313 views

Q-learning agent stucks in an infinite loop

I am simulating a mouse to find a cheese on an empty table. I randomly put a cheese on the table and let the mouse find the cheese without falling off the table. The problem is, in test part, agent ...
2
votes
1answer
37 views

Rationale behind Q-learning

I am reading Sutton Barto on Reinforcement Learning. I understand that $TD(\lambda)$ methods propose better performance than Monte Carlo methods, with TD methods combining advantages of Dynamic ...
2
votes
1answer
234 views

Reward attribution in deep q learning and texas holdem poker

I’m having issues with reward attribution in poker using deep q learning. Multiple actions will yield one reward, but the reward is only known at the end of the hand, not before. I have built a gym ...
1
vote
1answer
28 views

Why not sample action from Q values?

When collecting experience from which to estimate a Q(s,a) function, one common technique in the literature is to follow an epsilon greedy-strategy. In this strategy, the agent selects a random action ...
1
vote
1answer
224 views

How does the Dyna Q algorithm works?

I'm having a hard time trying to understand how the dyna Q algorithm works. I put the picture which helps me to understand. My questions are: What planning really means? (it's the (f) in this picture)...
0
votes
1answer
332 views

Episodic Semi-gradient Q-learning for Estimating approximation of optimal action-value function

at page 244 of Sutton and Barto book on Reinforcement Learning (book) is described the pseudocode for episodic semi-gradient Sarsa, while it is never given a pseudocode for the corresponding episodic ...
1
vote
1answer
431 views

DQN - How to feed the input of 4 still frames from a game as one single state input

I was reading this blog about Deep Q-Learning. 1- In the The input section of the blog, I wanted to know how do we feed the 4 still-frames/screenshots from the game, that represent the input state, ...
0
votes
2answers
178 views

DQN - breaking correlation between consecutive samples and random sampling

I was reading through some blogs about Deep Q-Learning (DQN), and I have 2 questions: 1- I didn't understand how breaking the correlation between consecutive samples (i.e. train the network with ...
0
votes
0answers
36 views

Is there a branch of machine learning that can deal with near infinite state spaces

So I have a game type problem defined as follows; Up to 10 players Each player has: 64 tiles 200 piece types Up to 20 pieces in play at any time There's a random ...
0
votes
1answer
58 views

How do Tile Coding offsets still cover full state space / affect edge cases?

Reading Sutton & Barto I’m having a hard time visualizing the implementation of the tile coding discretization of states. Specifically, if tilings are offset, how does this effect edge cases? For ...
0
votes
2answers
46 views

Different algorithms categorized in reinforcement learning

For some time I am going through reinforcement learning, and have found a lot of diverse information specially in area of Policies (algorithms). I figured out that policies can be classified in On ...
1
vote
1answer
423 views

Differences between Sarsa and Q-learning control procedural algorithms

I am referring to pages 130-131 of Sutton and Barto book on Reinforcement Learning available here: book I don't understand the slight difference that there is between the two procedural algorithms ...
1
vote
1answer
625 views

Limits and constraints for Q-learning

I have simple implementation of Q-learning algorithm and I'm trying to run it on States space size = 36865 Actions space size = 25 So my resulting Q-table is ...
1
vote
1answer
389 views

Q-learning shows worse results than value iteration

I'm trying to solve the same problem with different algorithms (Travel max possible distance with a car). While using value iteration and policy iteration I was able to get the best results possible ...
0
votes
1answer
36 views

Reinforcement learning based Q-learning for wireless routing

In the Q-learning method to get the optimal strategy, the update method is like the following: \begin{equation} Q(S,A) \leftarrow \ Q(S,A) + \alpha [R+\gamma~max_a(Q(s',a)) -Q(S,A)] \end{equation} If ...
1
vote
0answers
35 views

Vanishing reward function in Q-Learning

Imagine that the agent receives a positive reward upon reaching a state $s$. Once the state $s$ has been reached the positive reward associated with it vanishes and appears somewhere else in the state ...
0
votes
1answer
565 views

How to explain and visualize a Q Learning Agent?

What are some common visualization approaches used in explaining the behavior of a Q-Learning agent? Here is an excerpt of some example Q values for 5 actions serialized to json: ...
0
votes
1answer
159 views

How does DQN parameter updates work in simulation?

I've already read almost every Questions-answers and material related to DQN, deep reinforcement learning, but I'm struggling to start working on simulation. First of all, I'm trying to code using ...
2
votes
0answers
47 views

Deep Q - Learning Exploration - BestQ Value

I am trying to implement a Deep Q - Network to play Asteroids. Unfortunately, I am not sure how to calculate the Q Value exactly if I am exploring. For example, the Agent is exploring for 1 second (...