Questions tagged [hessian]

For on-topic questions involving the Hessian matrix, a square matrix generalizing the second derivative. Please include also a statistical methods tag. For purely mathemathical questions about the Hessian it is better to ask on math.SE at https://math.stackexchange.com/.

Filter by
Sorted by
Tagged with
1 vote
1 answer
52 views

How to compute the probability of trajectories term in Stochastic Gradient Meta Reinforcement Learning

This paper introduces Stochastic Gradient Meta RL (SGMLR). My question is specifically about the computation. One needs to calculate the Hessian, $\nabla^2_\theta J_i(\theta)$ which is given by the ...
user avatar
  • 155
0 votes
0 answers
19 views

What is the relationship between the residuals of an objective function and the uncertainties of the minimizer values?

Consider I have some optimization problem and an objective function $f(x, y, z)$. $f$ is defined using the sum of squared residuals, i.e. for some function $g$, we have $f(x, y, z) = \sum[g(x, y, z) - ...
user avatar
  • 1
0 votes
0 answers
73 views

How to Determine Gradient and Hessian for Custom Xgboost Functions

I'm trying to tackle a regression problem in which I want to predict data that sometimes has extreme values. The current machine learning algorithm I'm using is xgboost, specifically the python ...
user avatar
  • 51
0 votes
0 answers
17 views

Hessian of log-linear models and intuition

Im considering the log-linear model defined as $p(y|x,\theta) = \frac{1}{Z(\theta)}\exp(\theta f(x,y))$ where $Z(\theta) = \sum_{y'} \exp(\theta f(x,y))$ is the normalizer and $f(\cdot)$ is a feature ...
user avatar
  • 113
3 votes
1 answer
38 views

How does the approximate Hessian update in LBFGS work?

Looking at the wikipedia page for BFGS... Wikipedia It looks like a rearranging of Newton's method, but I can't really explain why the update to the approximate Hessian would be given by the following ...
user avatar
  • 619
3 votes
0 answers
71 views

What can a p-value (& sign) tell me about the marginal posterior distribution of a model parameter, and when?

EDIT: The tl;dr here would broadly be: given that both frequentist standard errors and a quadratic approximation of a Bayesian joint posterior can be obtained from the square root of the diagonal ...
user avatar
1 vote
0 answers
38 views

Confusion about the Hessian approximation

There are two popular forms of the neural network Hessian approximation in the literature: $$ H \simeq \sum_i \left(\frac{\partial y}{\partial w_i}\right) \left(\frac{\partial y}{\partial w_i}\right)^...
user avatar
1 vote
0 answers
111 views

Random Fourier Features approximating a kernel inverse?

There is a method I have been studying called Spectral Normalised Neural Gaussian Processes which leaves me with a question I cannot answer. In this method, they utilize Random Fourier Features but ...
user avatar
  • 619
5 votes
1 answer
735 views

Parameter uncertainity in least squares optimization: rescaling Hessian

Given a least squares optimization problem of the form: $$ C(\lambda) = \sum_i ||y_i - f(x_i, \lambda)||^2$$ I have found in multiple questions/answers (e.g. here) that an estimate for the covariance ...
user avatar
1 vote
1 answer
899 views

What is the Hessian of the Gaussian likelihood

I am trying to learn the fine differences between different methods of Kronecker factoring for approximate curvature (like [1], and [2]) which require taking the Hessian of the pre-activations of the ...
user avatar
  • 619
1 vote
0 answers
68 views

Fisher information matrix for model based design of experiments

I'm trying to use the Fisher information matrix and D-optimal design for model-based experiment selection, but I'm not sure if I'm implementing or interpreting it correctly. I begin with the values of ...
user avatar
0 votes
0 answers
68 views

Why is the Hessian in the Laplace approximation negative

The Laplace approximation builds from the Taylor expansion of the MAP estimate, where the first derivative is 0. The second order Taylor series goes... $$ f(a) + \frac{f'(a)}{1!}(x - a) + \frac{f''(a)}...
user avatar
  • 619
0 votes
0 answers
28 views

Is it possible to estimate the Hessian as the covariance of primal and cotangent?

Let's say we have a function $$f: \mathbb R^n \to \mathbb R.$$ Can we numerically approximate the Hessian $f''(x)$ as $$\textrm{Var}(a)^{-1} \textrm{Cov}(a, f'(a))$$ where $$E(a) = x?$$
user avatar
  • 13.7k
2 votes
1 answer
2k views

Derivation of Hessian for multinomial logistic regression in Böhning (1992)

This question is basically about row/column notation of derivatives and some basic rules. However, I couldn't figure out where I'm wrong. For multinomial logistic regression, I'm trying to get the ...
user avatar
  • 351
3 votes
0 answers
191 views

Information Matrix for Conditional Likelihood

I am studying the MLE theory on my own and I am confused by the difference between the fisher information matrix for the full sample and for one observation, when it comes to conditional likelihood. ...
user avatar
  • 1,103
0 votes
0 answers
76 views

Interpolation of Hessian matrix

I have a model where hessian matrices are calculated along a path. Since the calculation is done using finite differences, this is very time consuming. I have tried to calculate only every second ...
user avatar
1 vote
1 answer
109 views

$\frac\partial{\partial\beta}\left(\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i\right)$

How do you take the derivative of the function $$s(\beta)=\displaystyle\sum\frac{Y_i}{X_i^\intercal\beta}X_i-\sum\frac{1-Y_i}{1-X_i^\intercal\beta}X_i?$$ Attempt: $$H(\beta)=\frac\partial{\partial\...
user avatar
  • 428
1 vote
0 answers
241 views

GLMMadaptive - Hessian matrix problem Hurdle Beta Model

Data: I have a percentage (or proportion see paragraph below) outcome dataset with a high number of zero's. I have therefore attempted to run a hurdle beta model using the GLMMadaptive package in R. I ...
user avatar
1 vote
0 answers
144 views

How to obtain standard error of parameters in maximum likelihood when reparametrized/transformed?

Sometimes parameters in the maximum likelihood estimation process are reparametrized for numerical convenience. As an example if I'm fitting maximum likelihood estimation (MLE) to a data that is ...
user avatar
  • 7,399
1 vote
0 answers
266 views

Linear Mixed Effects - degenerate Hessian with 1 negative eigenvalue

I am using linear mixed effects to look at how the variable Neuro (measured at baseline only) predicts change over time in the variable Score (measured at baseline, visit 2 and for some participants ...
user avatar
1 vote
0 answers
73 views

How can I obtain group (factor) level Hessian and gradient from GLMM in lme4? [closed]

Summary I am working on a problem where I want to determine group level contributions to the maximum likelihood in a GLMM. To do so, I need to access the Hessian and gradients at the group level. This ...
user avatar
1 vote
1 answer
249 views

Hessian optimization (Newton method) using the direction given by the gradient to make the next iteration step of the parameters

Reading Deep Learning Book (page 86) I am having trouble understanding the reasons behind using the gradient ($g$) as the direction of the step of the parameters ($x$). I understand that the Newton ...
user avatar
  • 1,078
3 votes
2 answers
141 views

Laplace approximation in high-dimensions

Obviously computing the inverse Hessian is hard when a probability distribution is fitted on high-dimensional datapoints. One idea to reduce computational cost would be to approximate the distribution ...
user avatar
  • 884
1 vote
1 answer
215 views

MLE: Standard error of function of parameter

Let's assume I'm fitting some arbitrary model via maximum likelihood. For simplicity let's assume I have only one parameter of interest, $\beta$. Let's choose a probit model to illustrate, with log-...
user avatar
  • 2,390
0 votes
1 answer
25 views

Simplification of expression involving least squares estimators

Consider the sum of squared residuals of a linear regression given by $$ S_i(a,b) = \sum_{i=1}^n (y_i-a-bx_i)^2$$ I have to show that the optimal values of $a$ and $b$ found using the first-order ...
user avatar
2 votes
1 answer
834 views

Negative entries on the diagonal of the variance covariance matrix after MLE estimation of a Pearson Type 4 distribution

I am trying to estimate the parameters of a Pearson Type 4 distribution using maximum likelihood. At the estimated values some of the diagonal entries of the variance-covariance matrix are not ...
user avatar
  • 183
2 votes
0 answers
199 views

Why does the dimension of gradient and Hessian matrix not conform for this function?

The function is $f(\mathbf{x}) = e^{-\frac{1}{2}\mathbf{x^TAx}}$, where $\mathbf{A}$ is a square symmetric matrix, and $\mathbf{x}$ is an n-vector. What I found were: $$ \begin{align*} \nabla f ...
user avatar
1 vote
1 answer
666 views

Hessian of Ridge Regression

I have a ridge regression problem $$f(W) = \frac{1}{2} ||XW - Y||_2^2 + \lambda W^TW $$ I want to find the smoothness parameter of the function (the $L$ such that $f$ is $L$-smooth). For quadratic ...
user avatar
  • 105
0 votes
1 answer
265 views

When do we need Hessian Vector products vs Hessians (in meta-learning or Deep Learning)?

I was reading the paper MAML and they say: This approximation removes the need for computing Hessian-vector products in an additional backward pass So when doing the derivative of the derivative ...
user avatar
0 votes
0 answers
65 views

What is the difference between standard errors using the inverse of hessian and calculated using the the inverse of hessian and Fisher information?

In one of R packages for advanced survival analysis, the frailtypack, the output contains standard errors calculated in two ways, named: H (using the inverse of Hessian) and HIH (...
user avatar
2 votes
0 answers
71 views

Can the likelihood ratio estimate multivariate confidence levels?

Wilks' theorem describes the log-ratio between the highest likelihood of a distribution $\mathcal{L}$ (aka the dominant mode, given at $\vec{x}_{m}$) and the likelihood of a distribution at a given ...
user avatar
  • 21
1 vote
0 answers
68 views

Standard errors for Composite Marginal Likelihood

I am estimating a multivariate ordered probit model using a composite marginal likelihood (CML) approach. In other words, I replace the full likelihood function by a surrogate likelihood constructed ...
user avatar
  • 43
2 votes
0 answers
100 views

Square root of an almost diagonal matrix

Is there an efficient way to compute square root of an almost diagonal symmetric Hessian matrix, which is diagonal with the exception of the last two columns and last two rows? Could the efficient ...
user avatar
  • 5,835
0 votes
0 answers
200 views

AutoGrad Hessian in custom loss function in XGboost takes very long time

I am using the autograd package and generating the hessian of my loss function using that package as part of my custom loss function XGB model. However, it takes an extremely long time to iterate ...
user avatar
  • 210
0 votes
1 answer
1k views

Calculate the uncertainty of a MLE

I have minimized the negative LL of a Poisson distribution to get an MLE of three parameters using scipy.minimize w/ Nelder-Mead. I want to calculate the uncertainty of the MLE. From reading, I ...
user avatar
1 vote
2 answers
58 views

Neural networks: why don't we use a multi-dimensional learning rate

I've searched a bit on the internet a have found the answer nowhere so I decided to post here. When confronted to an optimization problem, we know that the sanity of the problem can be characterized ...
user avatar
  • 111
3 votes
1 answer
594 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively repel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
user avatar
1 vote
0 answers
1k views

Hessian of logistic loss - when $y \in \{-1, 1\}$

Logistic Regression has two possible formulations depending on how we select the target variable: $y \in \{0,1\}$ or $y \in \{-1,1\}$. This question discusses the derivation of Hessian of the loss ...
user avatar
  • 184
0 votes
1 answer
44 views

Hessian Matrix Values

Its an easy question but still i cant seem to find the hessian matrix. I have the following function : $$-2x^2 + \sqrt{2}xy - \frac52y^2$$ Find the hessian matrix for this function. $$f_{11} = -4 \...
user avatar
1 vote
0 answers
315 views

How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles?

For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension. I am trying to ...
user avatar
2 votes
1 answer
336 views

Observed information matrix with multivariate normal distribution

$$ \DeclareMathOperator\tr{tr} \DeclareMathOperator\vecOP{vec} \newcommand\di{\mathrm{d}} \newcommand\D{\mathrm{D}} \newcommand\Hess{\mathrm{H}} $$ I do not have much experience with matrix ...
user avatar
5 votes
2 answers
3k views

What is a consequence of an ill-conditioned Hessian matrix?

In this publication I found an explanation of the Hessian matrix, along with what it means for it to be ill-conditioned. In the paper, there is this link given between the error surface and the ...
user avatar
2 votes
1 answer
293 views

why Newton's method is not sensitive to ill-conditioned Hessian?

On Jorge Nocedal's , Page 501, "This property alone is enough to make many unconstrained minimization algorithms such as quasi-Newton and conjugate gradient perform poorly. Newton’s method, on the ...
user avatar
2 votes
2 answers
5k views

Standard error from Hessian matrix when likelihood is used (rather than Ln L)

I understand that at MLE point, the inverse of the Hessian matrix can be used as approximation of V-Cov matrix: ...
user avatar
  • 103
2 votes
0 answers
2k views

Obtaining Standard Errors in Optim() in R [duplicate]

I'm using a maximum likelihood estimation and I'm using the optim() function in R in a similar way as follows: ...
user avatar
2 votes
1 answer
3k views

Computing the Hessian of maximum log likelihood function

I am trying to find the Hessian matrix for the maximum log likelihood function given training data ${(xi, yi)}$ for $i=1:N$ with $yi ∈ \left\{+1, −1\right\}$ for each $i = 1,\dots, N$ for the function:...
user avatar
  • 23
2 votes
1 answer
450 views

Inverting Hessian using Generalized Inverse for Inference

I am estimating a survival model with MLE. I use optim to maximize the likelihood function, and I intend to use the Hessian matrix returned by optim to get the standard errors (which lie on the ...
user avatar
3 votes
1 answer
678 views

Explanation of generalization of Newton's Method for multiple dimensions

I've been following the CS 229 lecture videos for machine learning, and in lecture 4 (~14:00), Ng explains Newton's Method for optimization to maximize an objective function ($f$), but doesn't clearly ...
user avatar
  • 131
5 votes
0 answers
3k views

How the Hessian matrix is used in optimization if you can't invert it

I've seen quite a lot of work to do with approximating the Hessian such as the Hessian Vector Product but I'm not entirely sure how knowing the Hessian helps us evaluate the gradient step to take. ...
user avatar
1 vote
0 answers
351 views

Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network

I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem ...
user avatar
  • 227