Questions tagged [hessian]

For on-topic questions involving the Hessian matrix, a square matrix generalizing the second derivative. Please include also a statistical methods tag. For purely mathemathical questions about the Hessian it is better to ask on math.SE at https://math.stackexchange.com/.

Filter by
Sorted by
Tagged with
1
vote
0answers
11 views

How can I obtain group (factor) level Hessian and gradient from GLMM in lme4? [closed]

Summary I am working on a problem where I want to determine group level contributions to the maximum likelihood in a GLMM. To do so, I need to access the Hessian and gradients at the group level. This ...
1
vote
1answer
84 views

Hessian optimization (Newton method) using the direction given by the gradient to make the next iteration step of the parameters

Reading Deep Learning Book (page 86) I am having trouble understanding the reasons behind using the gradient ($g$) as the direction of the step of the parameters ($x$). I understand that the Newton ...
2
votes
2answers
38 views

Laplace approximation in high-dimensions

Obviously computing the inverse Hessian is hard when a probability distribution is fitted on high-dimensional datapoints. One idea to reduce computational cost would be to approximate the distribution ...
1
vote
1answer
60 views

MLE: Standard error of function of parameter

Let's assume I'm fitting some arbitrary model via maximum likelihood. For simplicity let's assume I have only one parameter of interest, $\beta$. Let's choose a probit model to illustrate, with log-...
0
votes
1answer
19 views

Simplification of expression involving least squares estimators

Consider the sum of squared residuals of a linear regression given by $$ S_i(a,b) = \sum_{i=1}^n (y_i-a-bx_i)^2$$ I have to show that the optimal values of $a$ and $b$ found using the first-order ...
0
votes
0answers
47 views

What is the second derivative (Hessian) of normalization function?

The normalization function over a $n$ elements column vector $\mathbf{x} $ is described as: $\frac{\mathbf{x}}{\|\mathbf{x}\|_2}$。(x divide by its L2-norm) The gradient of x is written as : $$\frac{\...
2
votes
1answer
119 views

Negative entries on the diagonal of the variance covariance matrix after MLE estimation of a Pearson Type 4 distribution

I am trying to estimate the parameters of a Pearson Type 4 distribution using maximum likelihood. At the estimated values some of the diagonal entries of the variance-covariance matrix are not ...
2
votes
0answers
32 views

Why does the dimension of gradient and Hessian matrix not conform for this function?

The function is $f(\mathbf{x}) = e^{-\frac{1}{2}\mathbf{x^TAx}}$, where $\mathbf{A}$ is a square symmetric matrix, and $\mathbf{x}$ is an n-vector. What I found were: $$ \begin{align*} \nabla f ...
0
votes
0answers
75 views

Custom objective function for classification in XGBoost using score values rather than probabilities

Is it possible to let customized objective functions in python-based XGBoost (classification setup) accept leaf scores as arguments (i.e. predicted values), rather than class probabilities? More ...
0
votes
0answers
32 views

How Hessian matrix is helping in taking big step towards minimization and how is it better than usual Gradient Descent? [duplicate]

I know what Hessian is and $θ:=θ−H^{-1}f′(θ)$ this relation too from Newton Raphson but what i dont understand is how Hessian is really helping with big step and also how is this efficient in ...
1
vote
1answer
107 views

Hessian of Ridge Regression

I have a ridge regression problem $$f(W) = \frac{1}{2} ||XW - Y||_2^2 + \lambda W^TW $$ I want to find the smoothness parameter of the function (the $L$ such that $f$ is $L$-smooth). For quadratic ...
0
votes
0answers
39 views

Hessian matrix of parameters for hill equation least squares approximation

Im trying to implement a function in java to get confidence interval of one of the parameters for a non linear regression fitting curve. The model is the hill equation (or four parameters logistic ...
0
votes
1answer
91 views

When do we need Hessian Vector products vs Hessians (in meta-learning or Deep Learning)?

I was reading the paper MAML and they say: This approximation removes the need for computing Hessian-vector products in an additional backward pass So when doing the derivative of the derivative ...
0
votes
0answers
20 views

What is the difference between standard errors using the inverse of hessian and calculated using the the inverse of hessian and Fisher information?

In one of R packages for advanced survival analysis, the frailtypack, the output contains standard errors calculated in two ways, named: H (using the inverse of Hessian) and HIH (...
2
votes
0answers
60 views

Can the likelihood ratio estimate multivariate confidence levels?

Wilks' theorem describes the log-ratio between the highest likelihood of a distribution $\mathcal{L}$ (aka the dominant mode, given at $\vec{x}_{m}$) and the likelihood of a distribution at a given ...
1
vote
0answers
36 views

Standard errors for Composite Marginal Likelihood

I am estimating a multivariate ordered probit model using a composite marginal likelihood (CML) approach. In other words, I replace the full likelihood function by a surrogate likelihood constructed ...
2
votes
0answers
87 views

Square root of an almost diagonal matrix

Is there an efficient way to compute square root of an almost diagonal symmetric Hessian matrix, which is diagonal with the exception of the last two columns and last two rows? Could the efficient ...
0
votes
0answers
128 views

AutoGrad Hessian in custom loss function in XGboost takes very long time

I am using the autograd package and generating the hessian of my loss function using that package as part of my custom loss function XGB model. However, it takes an extremely long time to iterate ...
0
votes
1answer
281 views

Calculate the uncertainty of a MLE

I have minimized the negative LL of a Poisson distribution to get an MLE of three parameters using scipy.minimize w/ Nelder-Mead. I want to calculate the uncertainty of the MLE. From reading, I ...
1
vote
2answers
48 views

Neural networks: why don't we use a multi-dimensional learning rate

I've searched a bit on the internet a have found the answer nowhere so I decided to post here. When confronted to an optimization problem, we know that the sanity of the problem can be characterized ...
1
vote
1answer
234 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1
vote
0answers
618 views

Hessian of logistic loss - when $y \in \{-1, 1\}$

Logistic Regression has two possible formulations depending on how we select the target variable: $y \in \{0,1\}$ or $y \in \{-1,1\}$. This question discusses the derivation of Hessian of the loss ...
0
votes
1answer
38 views

Hessian Matrix Values

Its an easy question but still i cant seem to find the hessian matrix. I have the following function : $$-2x^2 + \sqrt{2}xy - \frac52y^2$$ Find the hessian matrix for this function. $$f_{11} = -4 \...
1
vote
0answers
147 views

How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles?

For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension. I am trying to ...
2
votes
1answer
190 views

Observed information matrix with multivariate normal distribution

$$ \DeclareMathOperator\tr{tr} \DeclareMathOperator\vecOP{vec} \newcommand\di{\mathrm{d}} \newcommand\D{\mathrm{D}} \newcommand\Hess{\mathrm{H}} $$ I do not have much experience with matrix ...
5
votes
2answers
1k views

What is a consequence of an ill-conditioned Hessian matrix?

In this publication I found an explanation of the Hessian matrix, along with what it means for it to be ill-conditioned. In the paper, there is this link given between the error surface and the ...
2
votes
2answers
2k views

Standard error from Hessian matrix when likelihood is used (rather than Ln L)

I understand that at MLE point, the inverse of the Hessian matrix can be used as approximation of V-Cov matrix: ...
2
votes
0answers
904 views

Obtaining Standard Errors in Optim() in R [duplicate]

I'm using a maximum likelihood estimation and I'm using the optim() function in R in a similar way as follows: ...
2
votes
1answer
1k views

Computing the Hessian of maximum log likelihood function

I am trying to find the Hessian matrix for the maximum log likelihood function given training data ${(xi, yi)}$ for $i=1:N$ with $yi ∈ \left\{+1, −1\right\}$ for each $i = 1,\dots, N$ for the function:...
1
vote
0answers
234 views

Inverting Hessian using Generalized Inverse for Inference

I am estimating a survival model with MLE. I use optim to maximize the likelihood function, and I intend to use the Hessian matrix returned by optim to get the standard errors (which lie on the ...
3
votes
1answer
226 views

Explanation of generalization of Newton's Method for multiple dimensions

I've been following the CS 229 lecture videos for machine learning, and in lecture 4 (~14:00), Ng explains Newton's Method for optimization to maximize an objective function ($f$), but doesn't clearly ...
3
votes
0answers
2k views

How the Hessian matrix is used in optimization if you can't invert it

I've seen quite a lot of work to do with approximating the Hessian such as the Hessian Vector Product but I'm not entirely sure how knowing the Hessian helps us evaluate the gradient step to take. ...
1
vote
0answers
248 views

Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network

I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem ...
3
votes
0answers
103 views

Which optimizer use for laplace approximation

I have been trying to estimate the marginal posterior for D variable using Laplace approximation: $p(\theta_i) \approx \left[\frac{\det{H}}{2\pi\det{H(\theta_i)}}\right]^{1/2} \exp\left[-L(\theta_i, \...
7
votes
1answer
3k views

gradient descent and local maximum

I read that gradient descent converge always to a local minimum while other methods as Newton's method this is not guaranteed (if the Hessian is not definite positive); but if the start point in GD is ...
1
vote
1answer
122 views

Question about port of R code from the library “rethinking” to PyMC3

A very generous human named Osvaldo Martin did us the favor of porting all the R sample code in Richard McElreath's superb book Statistical Rethinking to PyMC3. I'm hugely grateful, but I've already ...
0
votes
1answer
58 views

Newton-Raphson Error

According to Agresti(2013) pg 364-365, iterative methods such as Newton-Raphson methods, $ \begin{aligned} \beta^\text{new} &= \beta^\text{old} + (X^{T}WX)^{-1}X^{T}(V) \end{aligned} $ help to ...
3
votes
0answers
40 views

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian. For the time being, I obtain the derivative as $\frac{\...
1
vote
3answers
134 views

Hessian of Log of Matrix-t distribution

I am trying to calculate the hessian of the log of the matrix-t distribution. I know that the log of the matrix-t distribution can be written: $$\log T_{N\times P}(X| \nu, M, \Sigma, \Omega) \propto -\...
2
votes
0answers
465 views

Multiclass: I want to develop a customized objective function with weights given by both label and prediction, for Xgboost

I want to develop a customized objective function with weights given by both label and prediction, for Xgboost. Example, let's say you have 2 classes I want to assign a penalties according to this ...
0
votes
1answer
192 views

Cramér–Rao bound to multiple parameters

I was reading Cramér–Rao bound to multiple parameters from Wikipedia page, but I could not follow this line in the article: Let $\displaystyle {\boldsymbol {T}}(X)$ be an estimator of any ...
28
votes
6answers
3k views

Why not use the third derivative for numerical optimization?

If Hessians are so good for optimization (see e.g. Newton's method), why stop there? Let's use the third, fourth, fifth, and sixth derivatives? Why not?
30
votes
1answer
22k views

Explanation of min_child_weight in xgboost algorithm

The definition of the min_child_weight parameter in xgboost is given as the: minimum sum of instance weight (hessian) needed in a child. If the tree partition step results in a leaf node with the ...
4
votes
1answer
2k views

Gradient and hessian of the MAPE

I want to use MAPE(Mean Absolute Percentage Error) as my loss function. ...
4
votes
3answers
2k views

How does the second derivative inform an update step in Gradient Descent?

I was reading the deep learning book by Begnio, Goodfellow and Courville and there was one section where they explain the second derivative that I don't understand (section 4.31): The second ...
1
vote
0answers
56 views

Non-linear Likelihood function, large estimated standard errors

I have a highly non-linear (lots of jumps) likelihood function with K parameters (For example, a marked Hawkes Process used in seismology study). I implemented the L-BFGS-B optimization routine and it ...
0
votes
1answer
85 views

Pearlmutter's method for Hessian multiplication

I am trying to understand the abstract below from Pearlmutter's paper. Can someone clarify to me why $R_{\bf{v}}\{\bf{w}\}=\bf{v}$? Thanks a lot!
1
vote
0answers
101 views

Fast multiplication by the Hessian in Neural networks

I have question about the $R\{.\}$ function in Bishop's book on page 254 (see snippet below). My questions are as follows: I assume $R\{\bf w\}$ in (5.97) is the premultiplication of $\bf{v}^{T}$ ...
1
vote
0answers
27 views

Inferrence for peaked likelihoods

Suppose I have the likelihood $f(X|\theta)$ of some rich model, where $\theta\in\mathbb{R}^n$, and I have been able to find its maximum, $\hat\theta$. Suppose further that for some $i$, the plot of $...
2
votes
0answers
331 views

Hessian matrix of log marginal likelihood of Gaussian Process

I'm trying to compute the exact second derivatives of log marginal likelihood of Gaussian Process for learning hyperparameters. The log marginal likelihood and its partial derivative are given in 5 ...