# Questions tagged [hessian]

For on-topic questions involving the Hessian matrix, a square matrix generalizing the second derivative. Please include also a statistical methods tag. For purely mathemathical questions about the Hessian it is better to ask on math.SE at https://math.stackexchange.com/.

110 questions
Filter by
Sorted by
Tagged with
1 vote
19 views

### Metropolis Hastings Proposal with Gradient and Hessian Information

I need to sample a high-dimensional parameter vector from a distribution where the gradient, the Hessian and the inverse of the Hessian of the log-likelihood are very cheap to compute. Are there any ...
113 views

### How to calculate covariance matrix in nonlinear least squares

I am fitting a nonlinear model to observations by using least squares to estimated the model parameters. Theoretically, the covariance matrix of the parameters can be estimated by inverting the ...
25 views

### Computational instability when computing covariance in glm or optim

When we estimate parameters with the maximum likelihood method, then we can estimate the standard error with the Hessian. The code below tries to estimate this for the case of the question: How to ...
434 views

### scipy minimize gives a hess_inv that is completely different from inv(statmodel.approx_hess)

I'm fitting a model with MLE using scipy.minimize (method BFGS). I want to have the hessian to compute its inverse and retrieve the standard error of each parameter....
26 views

### Coefficient standard error for "GLM" not in exponential family

For GLMs in the exponential family, we can obtain the standard errors for the regression coefficients as a function of the diagonal of the fisher information matrix. Does this still hold if the ...
52 views

### Large values in the inverse of a Hessian

I'm implementing Newton's method for a simple logistic regression model but I keep getting very large values for the inverse of the Hessian matrix. I am using the standard formulas found in books... I'...
1 vote
51 views

2k views

### How to Determine Gradient and Hessian for Custom Xgboost Functions?

I'm trying to tackle a regression problem in which I want to predict data that sometimes has extreme values. The current machine learning algorithm I'm using is xgboost, specifically the python ...
184 views

### How does the approximate Hessian update in LBFGS work?

Looking at the wikipedia page for BFGS... Wikipedia It looks like a rearranging of Newton's method, but I can't really explain why the update to the approximate Hessian would be given by the following ...
75 views

### What can a p-value (& sign) tell me about the marginal posterior distribution of a model parameter, and when?

EDIT: The tl;dr here would broadly be: given that both frequentist standard errors and a quadratic approximation of a Bayesian joint posterior can be obtained from the square root of the diagonal ...
235 views

53 views

### Is it possible to estimate the Hessian as the covariance of primal and cotangent?

Let's say we have a function $$f: \mathbb R^n \to \mathbb R.$$ Can we numerically approximate the Hessian $f''(x)$ as $$\textrm{Var}(a)^{-1} \textrm{Cov}(a, f'(a))$$ where $$E(a) = x?$$
4k views

### Derivation of Hessian for multinomial logistic regression in Böhning (1992)

This question is basically about row/column notation of derivatives and some basic rules. However, I couldn't figure out where I'm wrong. For multinomial logistic regression, I'm trying to get the ...
397 views

### Information Matrix for Conditional Likelihood

I am studying the MLE theory on my own and I am confused by the difference between the fisher information matrix for the full sample and for one observation, when it comes to conditional likelihood. ...
118 views

### Interpolation of Hessian matrix

I have a model where hessian matrices are calculated along a path. Since the calculation is done using finite differences, this is very time consuming. I have tried to calculate only every second ...
116 views

1k views

### Hessian of Ridge Regression

I have a ridge regression problem $$f(W) = \frac{1}{2} ||XW - Y||_2^2 + \lambda W^TW$$ I want to find the smoothness parameter of the function (the $L$ such that $f$ is $L$-smooth). For quadratic ...
1 vote
553 views

### When do we need Hessian Vector products vs Hessians (in meta-learning or Deep Learning)?

I was reading the paper MAML and they say: This approximation removes the need for computing Hessian-vector products in an additional backward pass So when doing the derivative of the derivative ...
137 views

### What is the difference between standard errors using the inverse of hessian and calculated using the the inverse of hessian and Fisher information?

In one of R packages for advanced survival analysis, the frailtypack, the output contains standard errors calculated in two ways, named: H (using the inverse of Hessian) and HIH (...
95 views

### Can the likelihood ratio estimate multivariate confidence levels?

Wilks' theorem describes the log-ratio between the highest likelihood of a distribution $\mathcal{L}$ (aka the dominant mode, given at $\vec{x}_{m}$) and the likelihood of a distribution at a given ...
1 vote