Questions tagged [hessian]
For on-topic questions involving the Hessian matrix, a square matrix generalizing the second derivative. Please include also a statistical methods tag. For purely mathemathical questions about the Hessian it is better to ask on math.SE at https://math.stackexchange.com/.
110
questions
1
vote
0
answers
19
views
Metropolis Hastings Proposal with Gradient and Hessian Information
I need to sample a high-dimensional parameter vector from a distribution where the gradient, the Hessian and the inverse of the Hessian of the log-likelihood are very cheap to compute. Are there any ...
- 690
0
votes
0
answers
113
views
How to calculate covariance matrix in nonlinear least squares
I am fitting a nonlinear model to observations by using least squares to estimated the model parameters.
Theoretically, the covariance matrix of the parameters can be estimated by inverting the ...
- 31
3
votes
1
answer
25
views
Computational instability when computing covariance in `glm` or `optim`
When we estimate parameters with the maximum likelihood method, then we can estimate the standard error with the Hessian.
The code below tries to estimate this for the case of the question: How to ...
- 71.6k
2
votes
1
answer
434
views
scipy minimize gives a hess_inv that is completely different from inv(statmodel.approx_hess)
I'm fitting a model with MLE using scipy.minimize (method BFGS). I want to have the hessian to compute its inverse and retrieve the standard error of each parameter....
- 240
2
votes
0
answers
26
views
Coefficient standard error for "GLM" not in exponential family
For GLMs in the exponential family, we can obtain the standard errors for the regression coefficients as a function of the diagonal of the fisher information matrix. Does this still hold if the ...
- 97
0
votes
0
answers
52
views
Large values in the inverse of a Hessian
I'm implementing Newton's method for a simple logistic regression model but I keep getting very large values for the inverse of the Hessian matrix. I am using the standard formulas found in books... I'...
- 1
1
vote
0
answers
51
views
What is the difference between $\partial^2/\partial{\boldsymbol\theta}^2$ and $\partial^2/\partial{\boldsymbol\theta}\partial{\boldsymbol\theta}^T$?
I am reading the paper Efficient Computation of the Fisher Information Matrix in the EM Algorithm (Meng & Spall) and am unsure what the difference (if any) there is between the notation $\partial^...
- 111
4
votes
1
answer
1k
views
Is the inverse Hessian the Covariance Matrix? [duplicate]
I am currently working on determining the errors associated with a least-squares fit for powder X-ray diffraction data analysis. I am utilizing the numpy.linalg.lstsq function to accomplish this task. ...
- 41
0
votes
1
answer
158
views
What does mean some eigenvalues of the Hessian are positive? How to fix it?
I prepared a research design with the discrete choice method. I have 2 labeled alternatives, 5 attributes for an alternative, and 6 attributes for an alternative. I want to analyze my data with ...
- 1
3
votes
1
answer
381
views
Gradient and Hessian of loss function
I'm trying to clear up the calculation of the gradient and Hessian of a loss function in an article that I am currently reading. The loss function is given by
$$\ell(\beta)=\sum_{i=1}^{N} e^{-y_{i}{{x}...
- 129
9
votes
1
answer
201
views
"Unbiased" (at least ballpark) Estimate of Condition Number of True Covariance Matrix being Estimated & other Symmetric Matrices (e.g.,Hessian)
Are there any known ways of getting an unbiased estimate of the condition number of the true covariance matrix being estimated, or at least correct within a small number of orders of magnitude? For ...
- 13k
1
vote
0
answers
232
views
The Hessian from my optim() output in R contains mostly 0's. How can I interpret this?
The following is the Hessian matrix given to me from R's optim() function:
10230 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
How can I interpret this? Is the first ...
- 1,829
4
votes
2
answers
369
views
Why is $\mathbf\Phi^{\top}\mathbf\Phi$ a positive definite matrix?
I had this question when reading section 3.5.3 on page 170 of "Pattern Recognition and Machine Learning" written by Christopher M. Bishop:
Here $\mathbf\Phi$ represents the design matrix ...
- 303
1
vote
0
answers
58
views
Please help me to understand the Taylor’s theorem when transiting from Gradient Boost to XGboost
I am reading this article, which explains how the algorithm replaces the actual loss function with so-called 2nd order Taylor expansion. I can understand til Step 4, and can't understand step 5.
I ...
- 109
0
votes
0
answers
185
views
Interpretation of the elements of the error matrix as inverse of hessian matrix [duplicate]
In a report I am reading at work, the error matrix is calculated as the inverse of the hessian matrix and used to calculate the error ellipse angle and axes with a not theoretically correct formula.
...
- 101
0
votes
1
answer
115
views
Is Hessian of neural nets with NLL loss positive semi-definite?
I learned that expected Hessian of negative log likelihood is the same as Fisher information matrix, which is known to be positive semi-definite
$$
\begin{aligned}
F(\theta)
&= E_{x \sim p_\theta}...
2
votes
1
answer
114
views
Explanation of Equation 5.80 in Pattern Recognition and Machine Learning - Bishop
How the equation 5.80 in _Pattern Recognition and Machine Learning_ by Bishop is derived?
- 93
2
votes
1
answer
192
views
Hessian not listed in GAM output of some models. Does this matter for significance of smooths and comparison with AIC?
Would really appreciate some help as I don't know how to proceed given my GAM results. I haven't had much luck getting answers to my GAM questions (not sure if there is something wrong with the way I ...
- 371
2
votes
1
answer
296
views
How to compute the probability of trajectories term in Stochastic Gradient Meta Reinforcement Learning
This paper introduces Stochastic Gradient Meta RL (SGMLR). My question is specifically about the computation. One needs to calculate the Hessian, $\nabla^2_\theta J_i(\theta)$ which is given by the ...
- 145
0
votes
0
answers
115
views
What is the relationship between the residuals of an objective function and the uncertainties of the minimizer values?
Consider I have some optimization problem and an objective function $f(x, y, z)$.
$f$ is defined using the sum of squared residuals, i.e. for some function $g$, we have $f(x, y, z) = \sum[g(x, y, z) - ...
- 1
2
votes
1
answer
2k
views
How to Determine Gradient and Hessian for Custom Xgboost Functions?
I'm trying to tackle a regression problem in which I want to predict data that sometimes has extreme values. The current machine learning algorithm I'm using is xgboost, specifically the python ...
- 71
3
votes
1
answer
184
views
How does the approximate Hessian update in LBFGS work?
Looking at the wikipedia page for BFGS... Wikipedia
It looks like a rearranging of Newton's method, but I can't really explain why the update to the approximate Hessian would be given by the following ...
- 812
3
votes
0
answers
75
views
What can a p-value (& sign) tell me about the marginal posterior distribution of a model parameter, and when?
EDIT: The tl;dr here would broadly be: given that both frequentist standard errors and a quadratic approximation of a Bayesian joint posterior can be obtained from the square root of the diagonal ...
2
votes
0
answers
235
views
Confusion about the Hessian approximation
There are two popular forms of the neural network Hessian approximation in the literature:
$$
H \simeq \sum_i \left(\frac{\partial y}{\partial w_i}\right)
\left(\frac{\partial y}{\partial w_i}\right)^...
1
vote
0
answers
304
views
Random Fourier Features approximating a kernel inverse?
There is a method I have been studying called Spectral Normalised Neural Gaussian Processes which leaves me with a question I cannot answer. In this method, they utilize Random Fourier Features but ...
- 812
6
votes
1
answer
1k
views
Parameter uncertainity in least squares optimization: rescaling Hessian
Given a least squares optimization problem of the form:
$$ C(\lambda) = \sum_i ||y_i - f(x_i, \lambda)||^2$$
I have found in multiple questions/answers (e.g. here) that an estimate for the covariance ...
- 111
1
vote
1
answer
3k
views
What is the Hessian of the Gaussian likelihood
I am trying to learn the fine differences between different methods of Kronecker factoring for approximate curvature (like [1], and [2]) which require taking the Hessian of the pre-activations of the ...
- 812
1
vote
0
answers
166
views
Fisher information matrix for model based design of experiments
I'm trying to use the Fisher information matrix and D-optimal design for model-based experiment selection, but I'm not sure if I'm implementing or interpreting it correctly.
I begin with the values of ...
0
votes
0
answers
295
views
Why is the Hessian in the Laplace approximation negative
The Laplace approximation builds from the Taylor expansion of the MAP estimate, where the first derivative is 0. The second order Taylor series goes...
$$
f(a) + \frac{f'(a)}{1!}(x - a) + \frac{f''(a)}...
- 812
0
votes
0
answers
53
views
Is it possible to estimate the Hessian as the covariance of primal and cotangent?
Let's say we have a function
$$f: \mathbb R^n \to \mathbb R.$$
Can we numerically approximate the Hessian $f''(x)$ as
$$\textrm{Var}(a)^{-1} \textrm{Cov}(a, f'(a))$$
where
$$E(a) = x?$$
- 14.8k
4
votes
1
answer
4k
views
Derivation of Hessian for multinomial logistic regression in Böhning (1992)
This question is basically about row/column notation of derivatives and some basic rules. However, I couldn't figure out where I'm wrong.
For multinomial logistic regression, I'm trying to get the ...
- 473
3
votes
0
answers
397
views
Information Matrix for Conditional Likelihood
I am studying the MLE theory on my own and I am confused by the difference between the fisher information matrix for the full sample and for one observation, when it comes to conditional likelihood. ...
- 1,369
0
votes
0
answers
118
views
Interpolation of Hessian matrix
I have a model where hessian matrices are calculated along a path. Since the calculation is done using finite differences, this is very time consuming.
I have tried to calculate only every second ...
- 35
2
votes
1
answer
116
views
$\frac\partial{\partial\beta}\left(\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i\right)$
How do you take the derivative of the function
$$s(\beta)=\displaystyle\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i?$$
Attempt:
$$H(\beta)=\frac\partial{\partial\...
user308286
1
vote
0
answers
415
views
GLMMadaptive - Hessian matrix problem Hurdle Beta Model
Data:
I have a percentage (or proportion see paragraph below) outcome dataset with a high number of zero's. I have therefore attempted to run a hurdle beta model using the GLMMadaptive package in R. I ...
- 133
1
vote
0
answers
366
views
How to obtain standard error of parameters in maximum likelihood when reparametrized/transformed?
Sometimes parameters in the maximum likelihood estimation process are reparametrized for numerical convenience. As an example if I'm fitting maximum likelihood estimation (MLE) to a data that is ...
- 8,043
1
vote
0
answers
326
views
Linear Mixed Effects - degenerate Hessian with 1 negative eigenvalue
I am using linear mixed effects to look at how the variable Neuro (measured at baseline only) predicts change over time in the variable Score (measured at baseline, visit 2 and for some participants ...
1
vote
0
answers
184
views
How can I obtain group (factor) level Hessian and gradient from GLMM in lme4? [closed]
Summary
I am working on a problem where I want to determine group level contributions to the maximum likelihood in a GLMM. To do so, I need to access the Hessian and gradients at the group level. This ...
2
votes
1
answer
655
views
Hessian optimization (Newton method) using the direction given by the gradient to make the next iteration step of the parameters
Reading Deep Learning Book (page 86) I am having trouble understanding the reasons behind using the gradient ($g$) as the direction of the step of the parameters ($x$).
I understand that the Newton ...
- 1,180
3
votes
2
answers
284
views
Laplace approximation in high-dimensions
Obviously computing the inverse Hessian is hard when a probability distribution is fitted on high-dimensional datapoints. One idea to reduce computational cost would be to approximate the distribution ...
- 944
2
votes
1
answer
318
views
MLE: Standard error of function of parameter
Let's assume I'm fitting some arbitrary model via maximum likelihood. For simplicity let's assume I have only one parameter of interest, $\beta$. Let's choose a probit model to illustrate, with log-...
- 2,750
0
votes
1
answer
28
views
Simplification of expression involving least squares estimators
Consider the sum of squared residuals of a linear regression given by
$$ S_i(a,b) = \sum_{i=1}^n (y_i-a-bx_i)^2$$
I have to show that the optimal values of $a$ and $b$ found using the first-order ...
- 265
3
votes
1
answer
2k
views
Negative entries on the diagonal of the variance covariance matrix after MLE estimation of a Pearson Type 4 distribution
I am trying to estimate the parameters of a Pearson Type 4 distribution using maximum likelihood. At the estimated values some of the diagonal entries of the variance-covariance matrix are not ...
- 213
2
votes
0
answers
276
views
Why does the dimension of gradient and Hessian matrix not conform for this function?
The function is $f(\mathbf{x}) = e^{-\frac{1}{2}\mathbf{x^TAx}}$, where $\mathbf{A}$ is a square symmetric matrix, and $\mathbf{x}$ is an n-vector.
What I found were:
$$
\begin{align*}
\nabla f ...
- 121
2
votes
1
answer
1k
views
Hessian of Ridge Regression
I have a ridge regression problem $$f(W) = \frac{1}{2} ||XW - Y||_2^2 + \lambda W^TW $$
I want to find the smoothness parameter of the function (the $L$ such that $f$ is $L$-smooth). For quadratic ...
- 135
1
vote
2
answers
553
views
When do we need Hessian Vector products vs Hessians (in meta-learning or Deep Learning)?
I was reading the paper MAML and they say:
This approximation removes the need for computing Hessian-vector products in an additional backward pass
So when doing the derivative of the derivative ...
- 6,586
0
votes
0
answers
137
views
What is the difference between standard errors using the inverse of hessian and calculated using the the inverse of hessian and Fisher information?
In one of R packages for advanced survival analysis, the frailtypack, the output contains standard errors calculated in two ways, named: H (using the inverse of Hessian) and HIH (...
- 41
2
votes
0
answers
95
views
Can the likelihood ratio estimate multivariate confidence levels?
Wilks' theorem describes the log-ratio between the highest likelihood of a distribution $\mathcal{L}$ (aka the dominant mode, given at $\vec{x}_{m}$) and the likelihood of a distribution at a given ...
- 21
1
vote
0
answers
112
views
Standard errors for Composite Marginal Likelihood
I am estimating a multivariate ordered probit model using a composite marginal likelihood (CML) approach. In other words, I replace the full likelihood function by a surrogate likelihood constructed ...
- 53
2
votes
0
answers
123
views
Square root of an almost diagonal matrix
Is there an efficient way to compute square root of an almost diagonal symmetric Hessian matrix, which is diagonal with the exception of the last two columns and last two rows?
Could the efficient ...
- 6,133