I have read (e.g. here) that a (multinomial) logistic regressor corresponds to a maximum entropy classifier.
My question is, how does one end up with the formula for logistic regression starting with the maximum entropy principle?
$\begingroup$
$\endgroup$
-
1$\begingroup$ One way is by considering the dual convex program. $\endgroup$ – cardinal Nov 22 '13 at 17:24
-
$\begingroup$ Thanks @cardinal - Do you know of any sources that I can look up to learn more about the dual convex derivation? $\endgroup$ – Josh Nov 22 '13 at 18:51
-
$\begingroup$ Hi Josh. I'm not entirely sure where to best direct you off-hand. You might look in Boyd & Vandenberghe (available for free online) to start. I will hunt around here in a bit and see if I can give you a more definitive pointer. Are you familiar with convex programming? $\endgroup$ – cardinal Nov 22 '13 at 19:20
-
1$\begingroup$ Thanks @cardinal. I am familiar with convex programming and duality. I guess what I am looking for is: (1) the problem formulation (2) the formulation of maximum entropy (I presume as constraints), and somehow a proof that logistic regressor emerges as the optimal solution. $\endgroup$ – Josh Nov 22 '13 at 19:24
-
1$\begingroup$ See win-vector.com/dfiles/LogisticRegressionMaxEnt.pdf for an explanation $\endgroup$ – kjetil b halvorsen Apr 12 '18 at 8:33
