Skip to main content

Questions tagged [hessian]

For on-topic questions involving the Hessian matrix, a square matrix generalizing the second derivative. Please include also a statistical methods tag. For purely mathemathical questions about the Hessian it is better to ask on math.SE at https://math.stackexchange.com/.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Calculate HC3 robust vcov matrix when knowing only of Gradient and Hessian matrices of a GLM model

Suppose that a Generalized Linear Model is fitted using Maximum-Likelihood Estimation, but we only have access to two results from it: the gradient matrix $G$ is a $n \times p$ matrix where each row $...
Parlare's user avatar
2 votes
1 answer
33 views

Representation of Hessian in Augmented State-Space Filtering Problems

Question Let $z_t := (x_t, x_{t-1}, x_t^2, x_{t-1}x_t,x_{t-1}x_t,x_{t-1}^2)$, and $R(x_t) := \sum_{(i,j) \in \{\{0,1,2,3,4\} \otimes \{0,1,2,3,4\} : i+j \le 4\}} w_{i,j} x_t^i x_{t-1}^j$, so $R(\cdot)$...
hipHopMetropolisHastings's user avatar
5 votes
1 answer
93 views

difference between GLM covariance matrix from MLE vs. IRLS for non-canonical link

Someone asked a question on Stack Overflow where they noted a difference between Minitab and R (glm) results for the variance-covariance matrix of the parameters, ...
Ben Bolker's user avatar
  • 44.5k
7 votes
1 answer
69 views

If I set a Gaussian prior in the parameter space, can I use the variance as diagonal of the Hessian?

Say I have a neural network, where I put a Gaussian prior over the parameters $\theta_i \sim N(\mu_i, \sigma_i^2)$ and that I learn both $\mu$ and $\sigma$ via the reparameterization trick $f = \mu + \...
Alberto's user avatar
  • 1,217
1 vote
0 answers
49 views

Metropolis Hastings Proposal with Gradient and Hessian Information

I need to sample a high-dimensional parameter vector from a distribution where the gradient, the Hessian and the inverse of the Hessian of the log-likelihood are very cheap to compute. Are there any ...
yrx1702's user avatar
  • 700
0 votes
0 answers
340 views

How to calculate covariance matrix in nonlinear least squares

I am fitting a nonlinear model to observations by using least squares to estimated the model parameters. Theoretically, the covariance matrix of the parameters can be estimated by inverting the ...
GreatJourney's user avatar
3 votes
1 answer
39 views

Computational instability when computing covariance in `glm` or `optim`

When we estimate parameters with the maximum likelihood method, then we can estimate the standard error with the Hessian. The code below tries to estimate this for the case of the question: How to ...
Sextus Empiricus's user avatar
2 votes
1 answer
938 views

scipy minimize gives a hess_inv that is completely different from inv(statmodel.approx_hess)

I'm fitting a model with MLE using scipy.minimize (method BFGS). I want to have the hessian to compute its inverse and retrieve the standard error of each parameter....
Jerem Lachkar's user avatar
2 votes
0 answers
41 views

Coefficient standard error for "GLM" not in exponential family

For GLMs in the exponential family, we can obtain the standard errors for the regression coefficients as a function of the diagonal of the fisher information matrix. Does this still hold if the ...
David Wang's user avatar
0 votes
0 answers
70 views

Large values in the inverse of a Hessian

I'm implementing Newton's method for a simple logistic regression model but I keep getting very large values for the inverse of the Hessian matrix. I am using the standard formulas found in books... I'...
nowtcggm's user avatar
1 vote
0 answers
56 views

What is the difference between $\partial^2/\partial{\boldsymbol\theta}^2$ and $\partial^2/\partial{\boldsymbol\theta}\partial{\boldsymbol\theta}^T$?

I am reading the paper Efficient Computation of the Fisher Information Matrix in the EM Algorithm (Meng & Spall) and am unsure what the difference (if any) there is between the notation $\partial^...
epsilonz3ro's user avatar
4 votes
1 answer
2k views

Is the inverse Hessian the Covariance Matrix? [duplicate]

I am currently working on determining the errors associated with a least-squares fit for powder X-ray diffraction data analysis. I am utilizing the numpy.linalg.lstsq function to accomplish this task. ...
Pawel's user avatar
  • 41
0 votes
1 answer
213 views

What does mean some eigenvalues of the Hessian are positive? How to fix it?

I prepared a research design with the discrete choice method. I have 2 labeled alternatives, 5 attributes for an alternative, and 6 attributes for an alternative. I want to analyze my data with ...
Halilka's user avatar
4 votes
1 answer
604 views

Gradient and Hessian of loss function

I'm trying to clear up the calculation of the gradient and Hessian of a loss function in an article that I am currently reading. The loss function is given by $$\ell(\beta)=\sum_{i=1}^{N} e^{-y_{i}{{x}...
ADAM's user avatar
  • 741
10 votes
1 answer
232 views

"Unbiased" (at least ballpark) Estimate of Condition Number of True Covariance Matrix being Estimated & other Symmetric Matrices (e.g.,Hessian)

Are there any known ways of getting an unbiased estimate of the condition number of the true covariance matrix being estimated, or at least correct within a small number of orders of magnitude? For ...
Mark L. Stone's user avatar
1 vote
0 answers
331 views

The Hessian from my optim() output in R contains mostly 0's. How can I interpret this?

The following is the Hessian matrix given to me from R's optim() function: 10230 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 How can I interpret this? Is the first ...
Ron Snow's user avatar
  • 2,093
4 votes
2 answers
479 views

Why is $\mathbf\Phi^{\top}\mathbf\Phi$ a positive definite matrix?

I had this question when reading section 3.5.3 on page 170 of "Pattern Recognition and Machine Learning" written by Christopher M. Bishop: Here $\mathbf\Phi$ represents the design matrix ...
zzzhhh's user avatar
  • 333
1 vote
0 answers
73 views

Please help me to understand the Taylor’s theorem when transiting from Gradient Boost to XGboost

I am reading this article, which explains how the algorithm replaces the actual loss function with so-called 2nd order Taylor expansion. I can understand til Step 4, and can't understand step 5. I ...
yts61's user avatar
  • 109
0 votes
0 answers
216 views

Interpretation of the elements of the error matrix as inverse of hessian matrix [duplicate]

In a report I am reading at work, the error matrix is calculated as the inverse of the hessian matrix and used to calculate the error ellipse angle and axes with a not theoretically correct formula. ...
cicciodevoto's user avatar
0 votes
1 answer
155 views

Is Hessian of neural nets with NLL loss positive semi-definite?

I learned that expected Hessian of negative log likelihood is the same as Fisher information matrix, which is known to be positive semi-definite $$ \begin{aligned} F(\theta) &= E_{x \sim p_\theta}...
Hohyun Kim's user avatar
2 votes
1 answer
134 views

Explanation of Equation 5.80 in Pattern Recognition and Machine Learning - Bishop

How the equation 5.80 in _Pattern Recognition and Machine Learning_ by Bishop is derived?
ironhide012's user avatar
2 votes
1 answer
242 views

Hessian not listed in GAM output of some models. Does this matter for significance of smooths and comparison with AIC?

Would really appreciate some help as I don't know how to proceed given my GAM results. I haven't had much luck getting answers to my GAM questions (not sure if there is something wrong with the way I ...
Jade's user avatar
  • 361
2 votes
1 answer
441 views

How to compute the probability of trajectories term in Stochastic Gradient Meta Reinforcement Learning

This paper introduces Stochastic Gradient Meta RL (SGMLR). My question is specifically about the computation. One needs to calculate the Hessian, $\nabla^2_\theta J_i(\theta)$ which is given by the ...
Schach21's user avatar
  • 145
0 votes
0 answers
136 views

What is the relationship between the residuals of an objective function and the uncertainties of the minimizer values?

Consider I have some optimization problem and an objective function $f(x, y, z)$. $f$ is defined using the sum of squared residuals, i.e. for some function $g$, we have $f(x, y, z) = \sum[g(x, y, z) - ...
Will's user avatar
  • 1
4 votes
1 answer
2k views

How to Determine Gradient and Hessian for Custom Xgboost Functions? [closed]

I'm trying to tackle a regression problem in which I want to predict data that sometimes has extreme values. The current machine learning algorithm I'm using is xgboost, specifically the python ...
Gadavi1's user avatar
  • 91
3 votes
1 answer
223 views

How does the approximate Hessian update in LBFGS work?

Looking at the wikipedia page for BFGS... Wikipedia It looks like a rearranging of Newton's method, but I can't really explain why the update to the approximate Hessian would be given by the following ...
Joff's user avatar
  • 942
3 votes
0 answers
75 views

What can a p-value (& sign) tell me about the marginal posterior distribution of a model parameter, and when?

EDIT: The tl;dr here would broadly be: given that both frequentist standard errors and a quadratic approximation of a Bayesian joint posterior can be obtained from the square root of the diagonal ...
Nikolai Gates Vetr's user avatar
2 votes
0 answers
378 views

Confusion about the Hessian approximation

There are two popular forms of the neural network Hessian approximation in the literature: $$ H \simeq \sum_i \left(\frac{\partial y}{\partial w_i}\right) \left(\frac{\partial y}{\partial w_i}\right)^...
spiridon_the_sun_rotator's user avatar
1 vote
0 answers
341 views

Random Fourier Features approximating a kernel inverse?

There is a method I have been studying called Spectral Normalised Neural Gaussian Processes which leaves me with a question I cannot answer. In this method, they utilize Random Fourier Features but ...
Joff's user avatar
  • 942
6 votes
1 answer
1k views

Parameter uncertainity in least squares optimization: rescaling Hessian

Given a least squares optimization problem of the form: $$ C(\lambda) = \sum_i ||y_i - f(x_i, \lambda)||^2$$ I have found in multiple questions/answers (e.g. here) that an estimate for the covariance ...
Daniel Arteaga's user avatar
1 vote
1 answer
4k views

What is the Hessian of the Gaussian likelihood

I am trying to learn the fine differences between different methods of Kronecker factoring for approximate curvature (like [1], and [2]) which require taking the Hessian of the pre-activations of the ...
Joff's user avatar
  • 942
1 vote
0 answers
200 views

Fisher information matrix for model based design of experiments

I'm trying to use the Fisher information matrix and D-optimal design for model-based experiment selection, but I'm not sure if I'm implementing or interpreting it correctly. I begin with the values of ...
auf-wiedersehen-yall's user avatar
0 votes
0 answers
351 views

Why is the Hessian in the Laplace approximation negative

The Laplace approximation builds from the Taylor expansion of the MAP estimate, where the first derivative is 0. The second order Taylor series goes... $$ f(a) + \frac{f'(a)}{1!}(x - a) + \frac{f''(a)}...
Joff's user avatar
  • 942
0 votes
0 answers
71 views

Is it possible to estimate the Hessian as the covariance of primal and cotangent?

Let's say we have a function $$f: \mathbb R^n \to \mathbb R.$$ Can we numerically approximate the Hessian $f''(x)$ as $$\textrm{Var}(a)^{-1} \textrm{Cov}(a, f'(a))$$ where $$E(a) = x?$$
Neil G's user avatar
  • 15.4k
5 votes
1 answer
5k views

Derivation of Hessian for multinomial logistic regression in Böhning (1992)

This question is basically about row/column notation of derivatives and some basic rules. However, I couldn't figure out where I'm wrong. For multinomial logistic regression, I'm trying to get the ...
groove's user avatar
  • 503
3 votes
0 answers
430 views

Information Matrix for Conditional Likelihood

I am studying the MLE theory on my own and I am confused by the difference between the fisher information matrix for the full sample and for one observation, when it comes to conditional likelihood. ...
Count's user avatar
  • 1,389
0 votes
0 answers
128 views

Interpolation of Hessian matrix

I have a model where hessian matrices are calculated along a path. Since the calculation is done using finite differences, this is very time consuming. I have tried to calculate only every second ...
Johny Dow's user avatar
2 votes
1 answer
118 views

$\frac\partial{\partial\beta}\left(\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i\right)$

How do you take the derivative of the function $$s(\beta)=\displaystyle\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i?$$ Attempt: $$H(\beta)=\frac\partial{\partial\...
user avatar
1 vote
0 answers
499 views

GLMMadaptive - Hessian matrix problem Hurdle Beta Model

Data: I have a percentage (or proportion see paragraph below) outcome dataset with a high number of zero's. I have therefore attempted to run a hurdle beta model using the GLMMadaptive package in R. I ...
RmyjuloR's user avatar
  • 143
1 vote
0 answers
414 views

How to obtain standard error of parameters in maximum likelihood when reparametrized/transformed?

Sometimes parameters in the maximum likelihood estimation process are reparametrized for numerical convenience. As an example if I'm fitting maximum likelihood estimation (MLE) to a data that is ...
forecaster's user avatar
  • 8,445
1 vote
0 answers
351 views

Linear Mixed Effects - degenerate Hessian with 1 negative eigenvalue

I am using linear mixed effects to look at how the variable Neuro (measured at baseline only) predicts change over time in the variable Score (measured at baseline, visit 2 and for some participants ...
Monika Grigorova's user avatar
1 vote
0 answers
207 views

How can I obtain group (factor) level Hessian and gradient from GLMM in lme4? [closed]

Summary I am working on a problem where I want to determine group level contributions to the maximum likelihood in a GLMM. To do so, I need to access the Hessian and gradients at the group level. This ...
Jan van den Brand's user avatar
2 votes
1 answer
754 views

Hessian optimization (Newton method) using the direction given by the gradient to make the next iteration step of the parameters

Reading Deep Learning Book (page 86) I am having trouble understanding the reasons behind using the gradient ($g$) as the direction of the step of the parameters ($x$). I understand that the Newton ...
Javier TG's user avatar
  • 1,220
3 votes
2 answers
317 views

Laplace approximation in high-dimensions

Obviously computing the inverse Hessian is hard when a probability distribution is fitted on high-dimensional datapoints. One idea to reduce computational cost would be to approximate the distribution ...
Dion's user avatar
  • 954
2 votes
1 answer
348 views

MLE: Standard error of function of parameter

Let's assume I'm fitting some arbitrary model via maximum likelihood. For simplicity let's assume I have only one parameter of interest, $\beta$. Let's choose a probit model to illustrate, with log-...
statsplease's user avatar
  • 2,842
0 votes
1 answer
28 views

Simplification of expression involving least squares estimators

Consider the sum of squared residuals of a linear regression given by $$ S_i(a,b) = \sum_{i=1}^n (y_i-a-bx_i)^2$$ I have to show that the optimal values of $a$ and $b$ found using the first-order ...
Pedro Cunha's user avatar
3 votes
1 answer
2k views

Negative entries on the diagonal of the variance covariance matrix after MLE estimation of a Pearson Type 4 distribution

I am trying to estimate the parameters of a Pearson Type 4 distribution using maximum likelihood. At the estimated values some of the diagonal entries of the variance-covariance matrix are not ...
Andrew's user avatar
  • 213
2 votes
0 answers
286 views

Why does the dimension of gradient and Hessian matrix not conform for this function?

The function is $f(\mathbf{x}) = e^{-\frac{1}{2}\mathbf{x^TAx}}$, where $\mathbf{A}$ is a square symmetric matrix, and $\mathbf{x}$ is an n-vector. What I found were: $$ \begin{align*} \nabla f ...
Yuchen Zhang's user avatar
2 votes
1 answer
1k views

Hessian of Ridge Regression

I have a ridge regression problem $$f(W) = \frac{1}{2} ||XW - Y||_2^2 + \lambda W^TW $$ I want to find the smoothness parameter of the function (the $L$ such that $f$ is $L$-smooth). For quadratic ...
sedrick's user avatar
  • 135
1 vote
2 answers
620 views

When do we need Hessian Vector products vs Hessians (in meta-learning or Deep Learning)?

I was reading the paper MAML and they say: This approximation removes the need for computing Hessian-vector products in an additional backward pass So when doing the derivative of the derivative ...
Charlie Parker's user avatar