7 votes
Accepted

What happens when DQN gradients become too big?

It's hard to say for sure, since the slide is so sparse, but I think the implication is that squaring a number with absolute value larger than 1 can become very large very quickly, and likewise the ...
user avatar
  • 80.4k
3 votes
Accepted

Rationale for the objective function in a flow network based generative model by Yoshua Bengio?

If the state-space is small, you could indeed solve by dynamic programming (which would require visiting all the states, though). In exponentially large state-spaces (which are the ones you care about ...
user avatar
3 votes
Accepted

How can you combine control variates with antithetic variates

Using antithetic variates to improve the Monte Carlo approximation of $\mathbb E^F[h(X)]$ mean generating correlated realisations from $F$, $X_1,\ldots,X_n$ such that$$\text{var}(h(X_1)+\cdots+h(X_n))&...
user avatar
  • 93.6k
3 votes
Accepted

How do find the best arm in a multi-armed bandit when exploitation is unimportant?

This answer is relevant to your question. If you are interested in minimizing the number of pulls to identify the best arm, the setting you want to use is Best Arm Identification. In this setting, you ...
user avatar
2 votes
Accepted

Why Does Double-Q Learning Converge So Slowly?

Overestimation occurs when rewards are awarded stochastically. The larger the variance of these stochastic rewards, the more severe the overestimation. So when there is a specific problem where good ...
user avatar
2 votes

Why does CLT converge quadratically?

Let us define $$ S = \sqrt{n} \left( \frac{X - \mu}{\sigma} \right) \xrightarrow{d} N(0,1) $$ We can use Edgeworth series to approximate the probability distribution in terms of cumulants: $$ G(x) = P(...
user avatar
2 votes

Proof of Lemma 3 in TRPO

I believe this is just the law of total probability; \begin{align} \Pr(A_t \neq \tilde A_t) = \int \Pr(A_t \neq \tilde A_t \mid S_t = s) \Pr(S_t = s) \mathrm d s \leq \alpha \int \Pr(S_t = s) \mathrm ...
user avatar
  • 31
2 votes
Accepted

Multi-armed bandit - how does the gambler choose what's the best strategy?

You mention two options, $\epsilon=0$ and $\epsilon>0$. For $\epsilon=0$ there is no exploration, it's all exploitation, so your agent will always select the arm with the maximal reward. If you ...
user avatar
  • 6,495
2 votes
Accepted

I would like to ask: the setting of the reward function in the reinforcement learning

The reward is a unitless quantity you want to maximize. So if $R_v$ is in units of $m/s$, then $w_2$ is in units of $s/m$. (Alternatively, if you prefer, the reward is in units of "utilons" ...
user avatar
  • 23.4k
2 votes
Accepted

Choosing "Target Entropy" for Soft-Actor-Critic (SAC) algorithm

I can't answer on behalf of the authors, but it makes sense to me that they would choose a default value $\propto \text{dim}(\mathcal{A})$. If they instead chose it at some fixed (i.e. not dependent ...
user avatar
2 votes
Accepted

Estimated Optimal Policy vs True Optimal Policy

The optimal policy refers to the actual value, not the estimated value -- in fact, some reinforcement learning agents might not even attempt to estimate the value of a given state. do we still call ...
user avatar
  • 23.4k
2 votes

About "Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation"?

As in deep reinforcement learning, one does not need to literally hold the state space in memory explicitly (i.e. we don't need to "know all the states"). Rather, what is used is a function ...
user avatar
  • 131
2 votes
Accepted

In a sparse reward problem, is it possible to remove reward shaping once the RL agent trains long enough to consistently reach the final reward?

It seems that the hyperlink is broken, so for now, I will attempt to clarify some details without having read the paper and can answer the specifics later. I suspect that even though the agent has ...
user avatar
2 votes

How to find the gradient when a black box I/O function is involved in evaluation of the loss?

It's not possible to learn $NN_{pi}$ without (at least approx.) knowing $\mathbf F$. You can approximate the derivative of $\mathbf F$ wrt the output of the neural network, say $o$, numerically. ...
user avatar
  • 52.6k
1 vote

How to learn a filter on a dataset?

This depends a lot on the range of possible objective functions. For arbitrary objective functions, there is no method better than enumeration of all the $2^{len(X)}$ possible filters. For well-...
user avatar
1 vote

How to compute the probability of trajectories term in Stochastic Gradient Meta Reinforcement Learning

The probability of the trajectory should be $q(\tau, \theta) = Pi_{i=0}^H \pi(s_i, a_i | \theta)$. That’s at least how we calculate the importance sampling correction factor in off-policy learning. I’...
user avatar
1 vote

Question about the "sample" in "Flow Network based Generative Models for Non-Iterative Diverse Candidate Generation"

In the paper, the state space represents the space of molecules, but the only movement directions allowed are additive edits. This creates a Directed Acyclic Graph, since after doing an edit the "...
user avatar
  • 131
1 vote
Accepted

Designing a support reward in reinforcement learning to help the agent reaching the goal when there is an obstacle between them

It sounds like you should compute the shortest path between any two points $x, y$ in your environment, and use that as your distance function, rather than euclidean distance. Even if an exact shortest ...
user avatar
  • 23.4k
1 vote
Accepted

Sigmoid equivalent to Softmax exercise 2

After slightly simplifying the notation, by the properties of the exponential function: $$ \frac{e^{a-b}}{e^{a-b} + e^{b-b}} = \frac{e^a e^{-b}}{e^a e^{-b} + e^b e^{-b}} = \frac{e^a}{e^a + e^b} $$ ...
user avatar
  • 117k
1 vote

In reinforcement learning/multi-armed bandits, why do we look at expected reward and not the most likely reward?

These are averages... so you should interpret them as such. In the limit, the second option is better as it yields higher returns. However, if you were to repeat this experiment only once then the ...
user avatar
  • 6,495
1 vote
Accepted

actor update in DDPG algorithm (and in general actor-critic algorithms)

The second equation is minimizing the TD-error of the critic -- if $\delta_t$ is positive, then that means our estimate of the value of $x_t, a_t$ was too pessimistic, so we want to update $\omega$ in ...
user avatar
  • 23.4k
1 vote
Accepted

does Thompson sampling for price optimisation require discriminative pricing

If by discriminative pricing you mean that n customers could encounter m different prices at a/the same point in time then it's not a problem. You would sample the price for a batch of customers and ...
user avatar
  • 117k
1 vote

how to proof decaying ε-greedy algorithm has a s logarithmic asymptotic total regret?

Here's an informal proof sketch; On non-explore round $t \leq T$, what is the probability we take a suboptimal action $a \neq a^*$? Let $\hat \mu(a; n)$ denote the random empirical average of $n$ ...
user avatar
  • 31
1 vote
Accepted

Why the $\gamma^t$ is needed here in REINFORCE: Monte-Carlo Policy-Gradient Control (episodic) for $\pi_{*}$?

The intuitive, short version is that policy gradient methods directly try to find a policy that maximizes the agent's rewards conditioned on the starting state $S_0$ (or in general, the starting state ...
user avatar
1 vote

Can reinforcement learning directly work with advantage function?

When we are at a state s, we only need to determine the relative performance among different actions in order to choose the optimal action. In other words, we only ...
user avatar
  • 6,284
1 vote

Risk-averse multi-armed bandits

Yes, you can! One way of taking into account the risk is to use distributional reinforcement learning, where you learn the entire distribution of the rewards (in each state) rather than just keeping ...
user avatar
1 vote

Derivation of expected value in REINFORCE policy gradient

I came across this very interesting website with lots of properties for conditional expected values: https://randomservices.org/random/expect/Conditional.html One of the formulas is this: $$r(X)\...
user avatar
1 vote

Bandit-like setup but taking max reward over sequential choices

I don't know any relevant paper discussing a similar scenario, but this is also not my main area of expertise. If I understand you correctly, you have $n$ arms and can pull only $k \ll n$ of them at a ...
user avatar
  • 117k
1 vote

Bandit-like setting with maximum reward over multiple arms?

I believe this setting falls under the category of combinatorial bandits. In such setting, the agent can select a subsets of all arms, i.e., the action set $\mathcal{A}$ is a subset of the power set $\...
user avatar
1 vote

Why are the value and policy iteration dynamic programming algorithms?

Have you seen Silver's lecture? Did you know Bellman coined the dynamic programming term, his first book was called "Dynamic programming" in 1957, see Wikipedia? DP is an algorithm ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible