If I have data $\{(x_i, y_i)\}_{i=1}^n$ where the dependent variable is binary $(y_i = 0,1)$ I can model it using a logistic function:

$$f(x; \alpha, \beta) = \frac{1}{1 + e^{-(\alpha + \beta x)}}$$

One way to estimate the parameters is by using the likelihood function,

$$L(\alpha,\beta) = \prod_{i=1}^n{f(x_i;\alpha,\beta)^{y_i}(1 - f(x_i;\alpha,\beta))^{(1-y_i)}}$$

and maximizing it using gradient ascent over $\alpha$ and $\beta$, since no closed form for the critical point exists.

I've been reading a bit on information geometry and have seen that for probability distributions it is more effective (in terms of the number of steps to converge) to use the natural gradient given by the KL-divergence and Fisher Information matrix, since it better captures the notion of distance between two distributions. Is there a similar notion of divergence between two logistic functions that could be leveraged to define a more efficient gradient? Is the Euclidean distance in $\mathbb{R}^2$ between the parameters $\alpha$ and $\beta$ the best way to traverse the parameter space?


Even though this question isn't all that new, I thought I would provide some information. I came across this question when I had a somewhat similar question in mind. I think this is a great resource for your question: https://www.eecs189.org/static/notes/n17.pdf

The example uses both likelihood and K-L divergence. The author uses cross-entropy as the loss function and K-L divergence to estimate the parameters.

This might provide some insight, as well: https://math.stackexchange.com/questions/1074276/how-is-logistic-loss-and-cross-entropy-related


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.