I have read (e.g. here) that a (multinomial) logistic regressor corresponds to a maximum entropy classifier.

My question is, how does one end up with the formula for logistic regression starting with the maximum entropy principle?

  • 1
    $\begingroup$ One way is by considering the dual convex program. $\endgroup$ – cardinal Nov 22 '13 at 17:24
  • $\begingroup$ Thanks @cardinal - Do you know of any sources that I can look up to learn more about the dual convex derivation? $\endgroup$ – Josh Nov 22 '13 at 18:51
  • $\begingroup$ Hi Josh. I'm not entirely sure where to best direct you off-hand. You might look in Boyd & Vandenberghe (available for free online) to start. I will hunt around here in a bit and see if I can give you a more definitive pointer. Are you familiar with convex programming? $\endgroup$ – cardinal Nov 22 '13 at 19:20
  • 1
    $\begingroup$ Thanks @cardinal. I am familiar with convex programming and duality. I guess what I am looking for is: (1) the problem formulation (2) the formulation of maximum entropy (I presume as constraints), and somehow a proof that logistic regressor emerges as the optimal solution. $\endgroup$ – Josh Nov 22 '13 at 19:24
  • 1
    $\begingroup$ See win-vector.com/dfiles/LogisticRegressionMaxEnt.pdf for an explanation $\endgroup$ – kjetil b halvorsen Apr 12 '18 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.