I am repeatedly surprised by how often these three things appear while any ML discussion is there:
Please help me understand/direct me to resources which provided an intuitive + mathematical (rigorous) validations of these observations. Thanks a lot.
Transforming the likelihood with log, thus giving log-likelihood often makes things easier to handle. A well known case is estimating with maximum likelihood parameters of a Gaussian population from random samples. Working directly with likelihood is hard, however taking logs it will separate the components of the distribution and since they are independent they can be fit separately. This is possible obviously because log is a monotonic transformation over positive values and thus maintain the solution. An example can be seen here: Why minimize negative log likelihood?
Softmax is a way to transform distances from $(-\infty,\infty)$ interval to $(0,1)$ interval. This kind of mapping is useful mostly when you want to map distances into probability. For example consider you have a categorical distribution, like a variable which can take $k$ possible values, which can be represented as a vector of zeros with 1 on the selected category. However instead of binary variables you have a distance for each category, aka a vector of distances? How can you transform that into the desired space? One solution in softmax. Note that it is not the only one and there is not always a connection with log likelihood. Wikipedia page should give you clear details.
Sigmoid is a particular case of softmax. Also it is not used only in NNs. Often is seen in logistic regression. Basically often logistic function is called sigmoid for binary case and softmax for cases when you have more than 2 categories.
