# Questions tagged [hessian]

For on-topic questions involving the Hessian matrix, a square matrix generalizing the second derivative. Please include also a statistical methods tag. For purely mathemathical questions about the Hessian it is better to ask on math.SE at https://math.stackexchange.com/.

76 questions
Filter by
Sorted by
Tagged with
11 views

### How can I obtain group (factor) level Hessian and gradient from GLMM in lme4? [closed]

Summary I am working on a problem where I want to determine group level contributions to the maximum likelihood in a GLMM. To do so, I need to access the Hessian and gradients at the group level. This ...
84 views

### Hessian optimization (Newton method) using the direction given by the gradient to make the next iteration step of the parameters

Reading Deep Learning Book (page 86) I am having trouble understanding the reasons behind using the gradient ($g$) as the direction of the step of the parameters ($x$). I understand that the Newton ...
38 views

### Laplace approximation in high-dimensions

Obviously computing the inverse Hessian is hard when a probability distribution is fitted on high-dimensional datapoints. One idea to reduce computational cost would be to approximate the distribution ...
60 views

### MLE: Standard error of function of parameter

Let's assume I'm fitting some arbitrary model via maximum likelihood. For simplicity let's assume I have only one parameter of interest, $\beta$. Let's choose a probit model to illustrate, with log-...
19 views

### Simplification of expression involving least squares estimators

Consider the sum of squared residuals of a linear regression given by $$S_i(a,b) = \sum_{i=1}^n (y_i-a-bx_i)^2$$ I have to show that the optimal values of $a$ and $b$ found using the first-order ...
47 views

75 views

### Custom objective function for classification in XGBoost using score values rather than probabilities

Is it possible to let customized objective functions in python-based XGBoost (classification setup) accept leaf scores as arguments (i.e. predicted values), rather than class probabilities? More ...
32 views

### How Hessian matrix is helping in taking big step towards minimization and how is it better than usual Gradient Descent? [duplicate]

I know what Hessian is and $θ:=θ−H^{-1}f′(θ)$ this relation too from Newton Raphson but what i dont understand is how Hessian is really helping with big step and also how is this efficient in ...
107 views

### Hessian of Ridge Regression

I have a ridge regression problem $$f(W) = \frac{1}{2} ||XW - Y||_2^2 + \lambda W^TW$$ I want to find the smoothness parameter of the function (the $L$ such that $f$ is $L$-smooth). For quadratic ...
39 views

### Hessian matrix of parameters for hill equation least squares approximation

Im trying to implement a function in java to get confidence interval of one of the parameters for a non linear regression fitting curve. The model is the hill equation (or four parameters logistic ...
91 views

### When do we need Hessian Vector products vs Hessians (in meta-learning or Deep Learning)?

I was reading the paper MAML and they say: This approximation removes the need for computing Hessian-vector products in an additional backward pass So when doing the derivative of the derivative ...
20 views

### What is the difference between standard errors using the inverse of hessian and calculated using the the inverse of hessian and Fisher information?

In one of R packages for advanced survival analysis, the frailtypack, the output contains standard errors calculated in two ways, named: H (using the inverse of Hessian) and HIH (...
60 views

### Can the likelihood ratio estimate multivariate confidence levels?

Wilks' theorem describes the log-ratio between the highest likelihood of a distribution $\mathcal{L}$ (aka the dominant mode, given at $\vec{x}_{m}$) and the likelihood of a distribution at a given ...
36 views

### Standard errors for Composite Marginal Likelihood

I am estimating a multivariate ordered probit model using a composite marginal likelihood (CML) approach. In other words, I replace the full likelihood function by a surrogate likelihood constructed ...
87 views

### Square root of an almost diagonal matrix

Is there an efficient way to compute square root of an almost diagonal symmetric Hessian matrix, which is diagonal with the exception of the last two columns and last two rows? Could the efficient ...
128 views

### AutoGrad Hessian in custom loss function in XGboost takes very long time

I am using the autograd package and generating the hessian of my loss function using that package as part of my custom loss function XGB model. However, it takes an extremely long time to iterate ...
281 views

### Calculate the uncertainty of a MLE

I have minimized the negative LL of a Poisson distribution to get an MLE of three parameters using scipy.minimize w/ Nelder-Mead. I want to calculate the uncertainty of the MLE. From reading, I ...
48 views

### Neural networks: why don't we use a multi-dimensional learning rate

I've searched a bit on the internet a have found the answer nowhere so I decided to post here. When confronted to an optimization problem, we know that the sanity of the problem can be characterized ...
234 views

### Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
618 views

### Hessian of logistic loss - when $y \in \{-1, 1\}$

Logistic Regression has two possible formulations depending on how we select the target variable: $y \in \{0,1\}$ or $y \in \{-1,1\}$. This question discusses the derivation of Hessian of the loss ...
38 views

465 views

### Multiclass: I want to develop a customized objective function with weights given by both label and prediction, for Xgboost

I want to develop a customized objective function with weights given by both label and prediction, for Xgboost. Example, let's say you have 2 classes I want to assign a penalties according to this ...
192 views

### Cramér–Rao bound to multiple parameters

I was reading Cramér–Rao bound to multiple parameters from Wikipedia page, but I could not follow this line in the article: Let $\displaystyle {\boldsymbol {T}}(X)$ be an estimator of any ...
3k views

### Why not use the third derivative for numerical optimization?

If Hessians are so good for optimization (see e.g. Newton's method), why stop there? Let's use the third, fourth, fifth, and sixth derivatives? Why not?
22k views

### Explanation of min_child_weight in xgboost algorithm

The definition of the min_child_weight parameter in xgboost is given as the: minimum sum of instance weight (hessian) needed in a child. If the tree partition step results in a leaf node with the ...
2k views

### Gradient and hessian of the MAPE

I want to use MAPE(Mean Absolute Percentage Error) as my loss function. ...
2k views

### How does the second derivative inform an update step in Gradient Descent?

I was reading the deep learning book by Begnio, Goodfellow and Courville and there was one section where they explain the second derivative that I don't understand (section 4.31): The second ...
56 views

### Non-linear Likelihood function, large estimated standard errors

I have a highly non-linear (lots of jumps) likelihood function with K parameters (For example, a marked Hawkes Process used in seismology study). I implemented the L-BFGS-B optimization routine and it ...
85 views

### Pearlmutter's method for Hessian multiplication

I am trying to understand the abstract below from Pearlmutter's paper. Can someone clarify to me why $R_{\bf{v}}\{\bf{w}\}=\bf{v}$? Thanks a lot!
101 views

### Fast multiplication by the Hessian in Neural networks

I have question about the $R\{.\}$ function in Bishop's book on page 254 (see snippet below). My questions are as follows: I assume $R\{\bf w\}$ in (5.97) is the premultiplication of $\bf{v}^{T}$ ...
Suppose I have the likelihood $f(X|\theta)$ of some rich model, where $\theta\in\mathbb{R}^n$, and I have been able to find its maximum, $\hat\theta$. Suppose further that for some $i$, the plot of \$...