Consider a simple regression or classification problem using a typical squared loss or cross-entropy loss respectively.

The supervised setup uses the gradient as estimated by:

$$\nabla_W J \approx \nabla_W l(f(x), y)$$

where $x$ is the input (or batch of inputs), sampled from the dataset $X$, $f$ is the network, $y$ is the target, and $l$ is the loss. From the MLE perspective, $f(x)$ parameterizes a probability distribution $\pi_f$ and we are actually computing

$$\nabla_W J \approx -\nabla_W \log \pi_{f(x)}(y)$$

where $\pi$ is either a categorial distribution in the case of classification or a gaussian in the case of regression.

On the other hand, we can frame this problem as a one-step MDP where the state is the input data $x$, and the action space is either the real line (regression) or a fixed set (classification). We take a single action $\hat y$, and then the episode ends. The REINFORCE gradient estimator says

$$\nabla_W J \approx E_{\hat{y} \sim \pi_{f(x)}} \left[R(\hat y) \nabla_W \log \pi_{f(x)}(\hat y) \right]$$

The obvious reward would just be the negative loss:

$$\nabla_W J \approx E_{\hat{y} \sim \pi_{f(x)}} \left[\log \pi_{f(x)}(\hat y)\cdot \nabla_W \log \pi_{f(x)}(\hat y) \right]$$

Where in practice the expectation is estimated by sampling. However, since we are only taking a single step, it would be silly not to directly evaluate the expectation if it is possible to do so.

This expression looks very similar to our supervised setup, but it is not quite identical.

It is possible to manipulate one expression the other to show that they are equivalent or somehow related? Or are the gradient estimates fundamentally different?

Clarifications: In both cases, the gradient estimate is computed by sampling $x \sim X$, so the exact gradients should be:

$$\nabla_W J = E_{x\sim X}\left[-\nabla_W \log \pi_{f(x)}(y) \right]$$


$$\nabla_W J = E_{x\sim X}\left[ E_{\hat{y} \sim \pi_{f(x)}} \left[\log \pi_{f(x)}(\hat y)\cdot \nabla_W \log \pi_{f(x)}(\hat y) \right]\right]$$

  • 1
    $\begingroup$ Is this homework? For the supervised setup, it's kind of a detail, but I think details are important perhaps, I think the expressions should be summed over multiple points. I think this is useful to do, since it will then match up with a monte-carlo sampling form of the expectation in the second part. ie the expectation is approximated by sampling a few points. $\endgroup$ – Hugh Perkins Mar 24 '18 at 0:55
  • 1
    $\begingroup$ In addition, I guess what you really want to do is show they converge to the same thing? So, set the gradients both equal to zero, and compare perhaps? $\endgroup$ – Hugh Perkins Mar 24 '18 at 0:56
  • 1
    $\begingroup$ Finally, I'm not sure that log loss is the 'obvious' reward. Perhaps you can hack around with different rewards, till you find one that makes the expressions equivalent? eg what about using non-log loss, or similar? $\endgroup$ – Hugh Perkins Mar 24 '18 at 0:57
  • 1
    $\begingroup$ No, it's not homework. Also there's a difference between converging to the same value and the gradients being the same -- REINFORCE provides notoriously noisy gradients in real RL problems, so I'm wondering how it would do in an much easier supervised setup. It's possible another loss other than negative log likelihood may work, but it seems like a reasonable place to start considering that there's a built-in log in the REINFORCE estimate. $\endgroup$ – shimao Mar 24 '18 at 1:05
  • 1
    $\begingroup$ Ok. It occurs to me that you could use a reward of 1, which might give a simpler expression? Of course, technically, giving the same reward each time, wont cause any learning to take place, but it might be an interesting baseline? For the MLE case, you're assuming that $y$ is given/fixed (I think?), but you could also draw $y$ from a probability distribution, eg a gaussian centred on $f(x)$ perhaps? $\endgroup$ – Hugh Perkins Mar 24 '18 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.