# Questions tagged [hessian]

For on-topic questions involving the Hessian matrix, a square matrix generalizing the second derivative. Please include also a statistical methods tag. For purely mathemathical questions about the Hessian it is better to ask on math.SE at https://math.stackexchange.com/.

58 questions
360 views

### SPSS: GLMM and(adjusted) odds ratio

I am performing a retrospective study and the relative statistic analysis. I am studying the the risk factors for the occurrence of complications during medical procedures. I have 50 subjects ...
73 views

### Observed information matrix with multivariate normal distribution

$$\DeclareMathOperator\tr{tr} \DeclareMathOperator\vecOP{vec} \newcommand\di{\mathrm{d}} \newcommand\D{\mathrm{D}} \newcommand\Hess{\mathrm{H}}$$ I do not have much experience with matrix ...
2k views

### Why not use the third derivative for numerical optimization?

If Hessians are so good for optimization (see e.g. Newton's method), why stop there? Let's use the third, fourth, fifth, and sixth derivatives? Why not?
47k views

### Why is Newton's method not widely used in machine learning?

This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes: After reviewing a set of lectures on convex optimization, Newton's method ...
8k views

### Explanation of min_child_weight in xgboost algorithm

The definition of the min_child_weight parameter in xgboost is given as the: minimum sum of instance weight (hessian) needed in a child. If the tree partition step results in a leaf node with the ...
35 views

### Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
95 views

### Hessian of logistic loss - when $y \in \{-1, 1\}$

Logistic Regression has two possible formulations depending on how we select the target variable: $y \in \{0,1\}$ or $y \in \{-1,1\}$. This question discusses the derivation of Hessian of the loss ...
25 views

108 views

### Explanation of generalization of Newton's Method for multiple dimensions

I've been following the CS 229 lecture videos for machine learning, and in lecture 4 (~14:00), Ng explains Newton's Method for optimization to maximize an objective function ($f$), but doesn't clearly ...
700 views

### How the Hessian matrix is used in optimization if you can't invert it

I've seen quite a lot of work to do with approximating the Hessian such as the Hessian Vector Product but I'm not entirely sure how knowing the Hessian helps us evaluate the gradient step to take. ...
1k views

### How does the second derivative inform an update step in Gradient Descent?

I was reading the deep learning book by Begnio, Goodfellow and Courville and there was one section where they explain the second derivative that I don't understand (section 4.31): The second ...
117 views

### Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network

I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem ...
59 views

48 views

### Newton-Raphson Error

According to Agresti(2013) pg 364-365, iterative methods such as Newton-Raphson methods, \begin{aligned} \beta^\text{new} &= \beta^\text{old} + (X^{T}WX)^{-1}X^{T}(V) \end{aligned} help to ...
88 views

### Should Bayesian estimated error smaller than MLE?

I am dealing with a fitting problem. Specifically, I am fitting a Lorentzian profile to the power spectrum of an solar-like oscillating star. Three parameters in the Lorentzian profile characterize ...
274 views

### Multiclass: I want to develop a customized objective function with weights given by both label and prediction, for Xgboost

I want to develop a customized objective function with weights given by both label and prediction, for Xgboost. Example, let's say you have 2 classes I want to assign a penalties according to this ...
140 views

### Cramér–Rao bound to multiple parameters

I was reading Cramér–Rao bound to multiple parameters from Wikipedia page, but I could not follow this line in the article: Let $\displaystyle {\boldsymbol {T}}(X)$ be an estimator of any ...
1k views

### Poisson Regression and Hessian

I have been trying to estimate parameters of a poisson regression. I am using Newton Raphson method. This method requires that the inverse of Hessian be computed to obtain, updates to beta vector. The ...
894 views

### Gradient and hessian of the MAPE

I want to use MAPE(Mean Absolute Percentage Error) as my loss function. ...
3k views

### Intuition for the “information matrix equality” result?

I am trying to understand the intuition behind the "information matrix equality" condition in the Maximum Likelihood context (perhaps this is the only context?):  -E[H(\theta)] = E[s(\theta) s(\...
38 views

### Non-linear Likelihood function, large estimated standard errors

I have a highly non-linear (lots of jumps) likelihood function with K parameters (For example, a marked Hawkes Process used in seismology study). I implemented the L-BFGS-B optimization routine and it ...
67 views

### Pearlmutter's method for Hessian multiplication

I am trying to understand the abstract below from Pearlmutter's paper. Can someone clarify to me why $R_{\bf{v}}\{\bf{w}\}=\bf{v}$? Thanks a lot!
68 views

### Fast multiplication by the Hessian in Neural networks

I have question about the $R\{.\}$ function in Bishop's book on page 254 (see snippet below). My questions are as follows: I assume $R\{\bf w\}$ in (5.97) is the premultiplication of $\bf{v}^{T}$ ...