0
$\begingroup$

I just found out about the softmax function in machine learning. It creates even probabilistic distribution out of a vector of numbers, which means that all numbers up to 1.

It sounds a lot like the vector normalization that gives a vector of length 1 in 3D graphics (linear algebra).

Can anyone explain the difference between those two ? I hope this is not a silly question and I get the answer.

$\endgroup$
2
  • $\begingroup$ It's more like the $\frac1{\sqrt{2\pi}}$ normalization term in the Gaussian distribution rather than vector normalization. $\endgroup$
    – CyclotomicField
    Commented May 4 at 1:49
  • 2
    $\begingroup$ Try applying softmax to the vector $v = (2,2,1)$, then try normalizing $v$ and compare the results. These are just two different things. $\endgroup$
    – littleO
    Commented May 4 at 6:24

1 Answer 1

1
$\begingroup$

Probability & SoftMax :

Positive numbers , Zeros , negative numbers , large numbers , small numbers can all occur in a vector.
When the numbers represent various Possibilities of some Entity , we want to know which Possibility is most likely for that Entity.

Heuristically , we want to choose the largest number out of those various Possibilities.
Hence , we hand-wave & say these are Probabilities.
BUT , the numbers might be Negative & might add up to some invalid total , which is a requirement for Probabilities.
SOLUTION : Convert to "SoftMax" , where the new numbers are Positive & will add up to $1$ total , looking like Pseudo-Probabilities.

Vector Normalization :

Positive numbers , Zeros , negative numbers , large numbers , small numbers can all occur in a vector.
We want a new vector which fits in the Unit N-Dimensional Ball.
Divide though-out [[ Shrinking/Scaling Operation ]] by the Norm to get that.
The new Unit Vector has Norm $1$.
Sum of Components will not be $1$.
There might still be Negative Components & Zero Components.
Hence it will generally not match SoftMax.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.