2
$\begingroup$

Given a softmax output layer, what does it mean to "follow the gradient"? Usually that would consist in "increasing the output" but obviously the softmax has no notion of "increasing the output" since it's normalized

So, I was wondering what actually happens following the gradient of a Softmax layer, and my take is that the output stays the same, but the pre-activations increases, thus having $$ pre = W*x+b\\ o = softmax(pre) $$ will increase the values of $pre$

Improve this question
$\endgroup$
4
  • $\begingroup$ what do you mean by "Usually that would consist in "increasing the output""? $\endgroup$
    – gunes
    Commented Dec 19, 2022 at 23:26
  • $\begingroup$ @gunes I mean that given a function, doing a step of gradient ascent would "increase" the output... for example $f(x)=x^2$, applying $x = x + \eta \nabla_x f$ would increase the final value of $f$ evaluated in the new $x$ $\endgroup$
    – Alberto
    Commented Dec 20, 2022 at 0:10
  • $\begingroup$ I'm not too sure about this, but in the $f(x)=x^2$ example you gave, the input $x$ gets updated with gradient ascent, but for the softmax function we never update $\text{pre}$ (i.e., the input). In training neural networks, we update the parameters, and since softmax layers don't have trainable parameters, I'm not sure the comparison with $f$ works. $\endgroup$
    – kmkurn
    Commented Dec 20, 2022 at 5:47
  • $\begingroup$ @kmkurn take $f(x) = ax^2$, so a gradient step on $a$ and the output will increase, same thing $\endgroup$
    – Alberto
    Commented Dec 20, 2022 at 9:32

1 Answer 1

Reset to default
2
$\begingroup$

Usually, neural networks use gradient descent, not ascent, because there is a loss function to be minimized. But, this doesn't matter for the discussion, as the same problem can be formulated as maximization of negative loss, or likelihood. So, let's assume we use gradient ascent, call the loss function as the optimization function.

The final gradients we care are always the optimization function's gradients with respect to the network parameters'. Here, $f$, is the optimization function. With gradient ascent, we expect to increase the value of $f$. But, that doesn't mean we increase the intermediate layers' outputs inside it. It's the final value of the loss/optimization function that's being affected from the ascent or descent strategy.

For example, if the output label is $y=1$, and the loss function is $f=(y-o)^2$, and we use gradient descent, the output, $o$, should get close to $1$ to decrease the loss. This means, with gradient descent, we decrease the loss but increase the network's output for this test case.

Improve this answer
$\endgroup$
3
  • $\begingroup$ there is no loss involved here, I'm taking the gradient of the output (of the softmax layer) with respect to the parameters, and I'm asking what would happen to the output at next iteration, if i take a gradient step wrt to that gradient $\endgroup$
    – Alberto
    Commented Dec 21, 2022 at 9:55
  • $\begingroup$ I assumed there would be a loss (or optimization goal) since you gave your example with neural network layers. But, if there isn't, then the last layer is a muti output function, which would make the discussion harder because let alone softmax, what does increasing a vector of outputs mean? This is not a case in neural nets. We have single value to optimize at the end (even if it can consist multiple summands). If it was a single output one, e.g. sigmoid, then yes, gradient ascent will try to increase the function value just as gradient descent would decrease it. $\endgroup$
    – gunes
    Commented Dec 21, 2022 at 13:03
  • $\begingroup$ but also 10 sigmoidal output would work, and doing multiple gradient step WRT the output would just cause those sigmoidal to output 1... the point is that softmax is normalized, thus my though is that the output would stay the same, as the gradients are applied with the same stepsize, and thus just the preactivations increase, without any change in the softmax due to the normalization $\endgroup$
    – Alberto
    Commented Dec 21, 2022 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.