Why should we use IRLS in logistic regression? I am really confused on why should we take IRLS as it seems that using gradient that with one derivatives at a time would solve the problem, what is the meaning of introducing Hessian matrix? Or did I understand anywhere wrong with that algorithm?
 A: (It's a little hard to guess what your understanding of the algorithm is, so judging if you misunderstood something is tricky)
If "solving the problem" was the only criterion, you could ask the same of any algorithm that converged to a local minimum.
So rather than look at why use one algorithm when another can also find an optimum, let's consider "what are some potential advantages of using one algorithm over another?"


*

*In this case, because it provides second derivative information, if the problem is sufficiently "nice" and we start sufficiently near the optimum, IRLS (along with a number of related methods) may in some cases be quite fast.

*Secondly, by having second-derivative information "built in", one can readily obtain information about the uncertainty in parameters estimates (estimates of the standard errors and covariances of parameter estimates). 

*Thirdly, there's a few pieces of other useful information that may be obtained reasonably easily from the final least-squares estimate (like the hat-matrix, for one example).
[This is not to suggest we should always use IRLS; it depends on the circumstances we find ourselves in -- but sometimes it may be a sensible and convenient choice. We have to weigh up the various costs and benefits.]
