# Tag Info

Accepted

### What happens when DQN gradients become too big?

It's hard to say for sure, since the slide is so sparse, but I think the implication is that squaring a number with absolute value larger than 1 can become very large very quickly, and likewise the ...
• 80.4k
Accepted

### Rationale for the objective function in a flow network based generative model by Yoshua Bengio?

If the state-space is small, you could indeed solve by dynamic programming (which would require visiting all the states, though). In exponentially large state-spaces (which are the ones you care about ...
Accepted

### Proof of Lemma 3 in TRPO

I believe this is just the law of total probability; \begin{align} \Pr(A_t \neq \tilde A_t) = \int \Pr(A_t \neq \tilde A_t \mid S_t = s) \Pr(S_t = s) \mathrm d s \leq \alpha \int \Pr(S_t = s) \mathrm ...
• 31
Accepted

### Multi-armed bandit - how does the gambler choose what's the best strategy?

You mention two options, $\epsilon=0$ and $\epsilon>0$. For $\epsilon=0$ there is no exploration, it's all exploitation, so your agent will always select the arm with the maximal reward. If you ...
• 6,495
Accepted

### I would like to ask: the setting of the reward function in the reinforcement learning

The reward is a unitless quantity you want to maximize. So if $R_v$ is in units of $m/s$, then $w_2$ is in units of $s/m$. (Alternatively, if you prefer, the reward is in units of "utilons" ...
• 23.4k
Accepted

### Choosing "Target Entropy" for Soft-Actor-Critic (SAC) algorithm

I can't answer on behalf of the authors, but it makes sense to me that they would choose a default value $\propto \text{dim}(\mathcal{A})$. If they instead chose it at some fixed (i.e. not dependent ...
Accepted

### Estimated Optimal Policy vs True Optimal Policy

The optimal policy refers to the actual value, not the estimated value -- in fact, some reinforcement learning agents might not even attempt to estimate the value of a given state. do we still call ...
• 23.4k

### About "Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation"？

As in deep reinforcement learning, one does not need to literally hold the state space in memory explicitly (i.e. we don't need to "know all the states"). Rather, what is used is a function ...
• 131
Accepted

### In a sparse reward problem, is it possible to remove reward shaping once the RL agent trains long enough to consistently reach the final reward?

It seems that the hyperlink is broken, so for now, I will attempt to clarify some details without having read the paper and can answer the specifics later. I suspect that even though the agent has ...

### How to find the gradient when a black box I/O function is involved in evaluation of the loss?

It's not possible to learn $NN_{pi}$ without (at least approx.) knowing $\mathbf F$. You can approximate the derivative of $\mathbf F$ wrt the output of the neural network, say $o$, numerically. ...
• 52.6k
1 vote

### How to learn a filter on a dataset?

This depends a lot on the range of possible objective functions. For arbitrary objective functions, there is no method better than enumeration of all the $2^{len(X)}$ possible filters. For well-...
• 995
1 vote

### How to compute the probability of trajectories term in Stochastic Gradient Meta Reinforcement Learning

The probability of the trajectory should be $q(\tau, \theta) = Pi_{i=0}^H \pi(s_i, a_i | \theta)$. That’s at least how we calculate the importance sampling correction factor in off-policy learning. I’...
1 vote

### Question about the "sample" in "Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation"

In the paper, the state space represents the space of molecules, but the only movement directions allowed are additive edits. This creates a Directed Acyclic Graph, since after doing an edit the "...
• 131
1 vote
Accepted

### Designing a support reward in reinforcement learning to help the agent reaching the goal when there is an obstacle between them

It sounds like you should compute the shortest path between any two points $x, y$ in your environment, and use that as your distance function, rather than euclidean distance. Even if an exact shortest ...
• 23.4k
1 vote
Accepted

### Sigmoid equivalent to Softmax exercise 2

After slightly simplifying the notation, by the properties of the exponential function: $$\frac{e^{a-b}}{e^{a-b} + e^{b-b}} = \frac{e^a e^{-b}}{e^a e^{-b} + e^b e^{-b}} = \frac{e^a}{e^a + e^b}$$ ...
• 117k
1 vote

### In reinforcement learning/multi-armed bandits, why do we look at expected reward and not the most likely reward?

These are averages... so you should interpret them as such. In the limit, the second option is better as it yields higher returns. However, if you were to repeat this experiment only once then the ...
• 6,495
1 vote
Accepted

### actor update in DDPG algorithm (and in general actor-critic algorithms)

The second equation is minimizing the TD-error of the critic -- if $\delta_t$ is positive, then that means our estimate of the value of $x_t, a_t$ was too pessimistic, so we want to update $\omega$ in ...
• 23.4k
1 vote
Accepted

### does Thompson sampling for price optimisation require discriminative pricing

If by discriminative pricing you mean that n customers could encounter m different prices at a/the same point in time then it's not a problem. You would sample the price for a batch of customers and ...
• 117k
1 vote

### how to proof decaying ε-greedy algorithm has a s logarithmic asymptotic total regret?

Here's an informal proof sketch; On non-explore round $t \leq T$, what is the probability we take a suboptimal action $a \neq a^*$? Let $\hat \mu(a; n)$ denote the random empirical average of $n$ ...
• 31
1 vote
Accepted

### Why the $\gamma^t$ is needed here in REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for $\pi_{*}$?

The intuitive, short version is that policy gradient methods directly try to find a policy that maximizes the agent's rewards conditioned on the starting state $S_0$ (or in general, the starting state ...
1 vote

### Can reinforcement learning directly work with advantage function?

When we are at a state s, we only need to determine the relative performance among different actions in order to choose the optimal action. In other words, we only ...
• 6,284
1 vote

### Risk-averse multi-armed bandits

Yes, you can! One way of taking into account the risk is to use distributional reinforcement learning, where you learn the entire distribution of the rewards (in each state) rather than just keeping ...
1 vote

### Derivation of expected value in REINFORCE policy gradient

I came across this very interesting website with lots of properties for conditional expected values: https://randomservices.org/random/expect/Conditional.html One of the formulas is this: r(X)\...
• 151
1 vote

### Bandit-like setup but taking max reward over sequential choices

I don't know any relevant paper discussing a similar scenario, but this is also not my main area of expertise. If I understand you correctly, you have $n$ arms and can pull only $k \ll n$ of them at a ...
• 117k
1 vote

### Bandit-like setting with maximum reward over multiple arms?

I believe this setting falls under the category of combinatorial bandits. In such setting, the agent can select a subsets of all arms, i.e., the action set $\mathcal{A}$ is a subset of the power set \$\...
• 791
1 vote

### Why are the value and policy iteration dynamic programming algorithms?

Have you seen Silver's lecture? Did you know Bellman coined the dynamic programming term, his first book was called "Dynamic programming" in 1957, see Wikipedia? DP is an algorithm ...

Only top scored, non community-wiki answers of a minimum length are eligible