I have been trying to figure out the implementation in knime. The tool says it uses Fisher scoring (FS). I understand Newton Raphson-method from http://www.win-vector.com/blog/2011/09/the-simpler-derivation-of-logistic-regression/ but can someone explain what is the exact difference between Fisher scoring and the Newton-Raphson method.

This is my knowledge of FS:

1) There is a score function $V(\theta)$ which is the gradient(derivative) of the log-likelihood function. //reference wikipedia

2) For weight updates the Hessian of log-likelihood is used.

Both of the above steps are done in Newton-Raphson method also but there isn't any mention of the score function but it does take first derivative and obtain Hessian.

It is mentioned that Fisher information as the variance of score which is Jacobian, or expected value of observed information which is Hessian. So in the final equation for weight update using Fisher information I don't understand how to take expected value using Hessian. Is it something like subtracting each field with its column mean and so obtain a final matrix which multiplied with score to obtain the second part of RHS in weight update equation?

I know my understanding of the algorithm is cluttered...Can someone detail the step by step procedure for calculating the Fisher information.

  • $\begingroup$ stats.stackexchange.com/questions/130110/… ... (in the question) or stats.stackexchange.com/questions/235514/… (ditto); the log-likelihood is in several other questions. It's not clear what the point of posting a slab of code is; we're not a "translate code into mathematics" site -- and what if the code is wrong? Better to ask what the log-likelihood is (and then you can try to see if it's correctly implemented) $\endgroup$ – Glen_b May 29 '17 at 9:32
  • $\begingroup$ Also see Elements of Statistical Learning, Eqn 4.20 or stat.cmu.edu/~cshalizi/uADA/12/lectures/ch12.pdf eq 12.7 to 12.10 $\endgroup$ – Glen_b May 29 '17 at 9:37
  • $\begingroup$ @Glen_b The code is implemented in knime tool. See the link posted along. I have gone through many sites but the formula here I couldn't connect with any and so is the question. $\endgroup$ – Devi May 29 '17 at 9:45
  • $\begingroup$ This doesn't seem to be a suitable question here. We're not a code review site or a code-translator site. "What is the log-likelihood for logistic regression" is on topic (but covered). "What is this code doing?" is off topic. What it's implemented in and what you link to doesn't alter that. $\endgroup$ – Glen_b May 29 '17 at 9:47
  • $\begingroup$ If you edit your question to ask something that doesn't appear to be about translating code, it may do better. $\endgroup$ – Glen_b May 29 '17 at 11:00

The logistic regression is a generalized linear model with canonical link which means the expected information matrix (EIM) or Fisher Information is the same as the observed information matrix (OIM). The way to compute the information matrix is the inverse of the negative of the Hessian evaluated at the parameter estimates.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.