Questions tagged [hessian]

For on-topic questions involving the Hessian matrix, a square matrix generalizing the second derivative. Please include also a statistical methods tag. For purely mathemathical questions about the Hessian it is better to ask on math.SE at https://math.stackexchange.com/.

Filter by
Sorted by
Tagged with
132
votes
8answers
53k views

Why is Newton's method not widely used in machine learning?

This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes: After reviewing a set of lectures on convex optimization, Newton's method ...
29
votes
6answers
2k views

Why not use the third derivative for numerical optimization?

If Hessians are so good for optimization (see e.g. Newton's method), why stop there? Let's use the third, fourth, fifth, and sixth derivatives? Why not?
22
votes
1answer
11k views

Explanation of min_child_weight in xgboost algorithm

The definition of the min_child_weight parameter in xgboost is given as the: minimum sum of instance weight (hessian) needed in a child. If the tree partition step results in a leaf node with the ...
7
votes
2answers
3k views

Name for outer product of gradient approximation of Hessian

Is there a name for approximating the Hessian as the outer product of the gradient with itself? If one is approximating the Hessian of the log-loss, then the outer product of the gradient with itself ...
6
votes
1answer
1k views

gradient descent and local maximum

I read that gradient descent converge always to a local minimum while other methods as Newton's method this is not guaranteed (if the Hessian is not definite positive); but if the start point in GD is ...
6
votes
1answer
651 views

Interpretation of eigenvectors of Hessian inverse

I'm reading a paper in which they use the eigenvectors of the inverse Hessian of a continuous probability distribution to characterize dimensions along which the distribution is most and least ...
5
votes
1answer
2k views

Why is the Hessian of the log likelihood function in the logit model not negative *semi*definite?

The Hessian of the log likelihood function is $$\frac{\partial^2 \ln(\beta \mid x)}{\partial \beta \partial \beta'} = -\sum_{i=1}^n \underbrace{\Lambda(\beta'x_i)}_{\in(0,1)}\underbrace{\left[1-\...
4
votes
1answer
1k views

Gradient and hessian of the MAPE

I want to use MAPE(Mean Absolute Percentage Error) as my loss function. ...
4
votes
1answer
2k views

Why calculating standard error of an mle (and confidence intervals) from Hessian matrices?

I might not have fully understood these concepts, and I am confused about how standard error is calculated. Here are my understandings and confusions, let me know where went wrong. EDIT: I was taking ...
4
votes
1answer
240 views

Variance of maximum likelihood estimator in R

In different sources there is an algorithm how to calculate the variance of MLE in R. To keep it short: construct the negative log likelihood function. minimize it via nlm or optim with hessian=TRUE ...
4
votes
2answers
3k views

Intuition for the “information matrix equality” result?

I am trying to understand the intuition behind the "information matrix equality" condition in the Maximum Likelihood context (perhaps this is the only context?): $$ -E[H(\theta)] = E[s(\theta) s(\...
3
votes
1answer
118 views

Explanation of generalization of Newton's Method for multiple dimensions

I've been following the CS 229 lecture videos for machine learning, and in lecture 4 (~14:00), Ng explains Newton's Method for optimization to maximize an objective function ($f$), but doesn't clearly ...
3
votes
1answer
175 views

Show that the following optimization problem is convex

I have the following optimization problem \begin{equation} \label{logdual} \begin{array}{ll@{}ll} \text{minimize}_{\pmb\alpha \in \mathbb{R}^n} & \theta(\pmb\alpha) &\\ \text{subject to} &...
3
votes
1answer
176 views

Why does the determinant of the Hessian grow with n?

Context: I'm trying to understand BIC on a deeper level. I'm using BIC for model/structure selection for Bayesian networks. I'm confused because BIC is an approximation to the likelihood of a model, ...
3
votes
0answers
38 views

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian. For the time being, I obtain the derivative as $\frac{\...
2
votes
3answers
1k views

How does the second derivative inform an update step in Gradient Descent?

I was reading the deep learning book by Begnio, Goodfellow and Courville and there was one section where they explain the second derivative that I don't understand (section 4.31): The second ...
2
votes
2answers
219 views

What is a consequence of an ill-conditioned Hessian matrix?

In this publication I found an explanation of the Hessian matrix, along with what it means for it to be ill-conditioned. In the paper, there is this link given between the error surface and the ...
2
votes
2answers
581 views

Standard error from Hessian matrix when likelihood is used (rather than Ln L)

I understand that at MLE point, the inverse of the Hessian matrix can be used as approximation of V-Cov matrix: ...
2
votes
2answers
1k views

Poisson Regression and Hessian

I have been trying to estimate parameters of a poisson regression. I am using Newton Raphson method. This method requires that the inverse of Hessian be computed to obtain, updates to beta vector. The ...
2
votes
1answer
300 views

Hessian for Laplace Approximation in Uncertainty Propagation

This is possibly a silly conceptual question, ... but anyway: Imagine I have a function: $f = F(\mathbf{x}) = F(x_1,x_2) = ax_1^2 + bx_2^3,$ where $x_1,x_2 \sim N(0,1)$ for example. For a naive ...
2
votes
1answer
539 views

Computing the Hessian of maximum log likelihood function

I am trying to find the Hessian matrix for the maximum log likelihood function given training data {(xi, yi)} for i=1:N with yi ∈ {+1, −1} for each i = 1, . . . , N for the function: When I try to ...
2
votes
1answer
949 views

Second derivative of neural network cost function

This question is highly correlated with my previous one (I was asking about quadratic approximation of the cost function with Hessian matrix and didn't get any answer), but I think that I have the ...
2
votes
0answers
56 views

Square root of an almost diagonal matrix

Is there an efficient way to compute square root of an almost diagonal symmetric Hessian matrix, which is diagonal with the exception of the last two columns and last two rows? Could the efficient ...
2
votes
1answer
104 views

Observed information matrix with multivariate normal distribution

$$ \DeclareMathOperator\tr{tr} \DeclareMathOperator\vecOP{vec} \newcommand\di{\mathrm{d}} \newcommand\D{\mathrm{D}} \newcommand\Hess{\mathrm{H}} $$ I do not have much experience with matrix ...
2
votes
0answers
266 views

Obtaining Standard Errors in Optim() in R [duplicate]

I'm using a maximum likelihood estimation and I'm using the optim() function in R in a similar way as follows: ...
2
votes
0answers
995 views

How the Hessian matrix is used in optimization if you can't invert it

I've seen quite a lot of work to do with approximating the Hessian such as the Hessian Vector Product but I'm not entirely sure how knowing the Hessian helps us evaluate the gradient step to take. ...
2
votes
0answers
69 views

Which optimizer use for laplace approximation

I have been trying to estimate the marginal posterior for D variable using Laplace approximation: $p(\theta_i) \approx \left[\frac{\det{H}}{2\pi\det{H(\theta_i)}}\right]^{1/2} \exp\left[-L(\theta_i, \...
2
votes
0answers
320 views

Multiclass: I want to develop a customized objective function with weights given by both label and prediction, for Xgboost

I want to develop a customized objective function with weights given by both label and prediction, for Xgboost. Example, let's say you have 2 classes I want to assign a penalties according to this ...
2
votes
0answers
249 views

Quadratic approximation of the regularized cost function for neural network [closed]

I've been working on the topic of regularization for neural networks and in the textbook I'm following I found this quote: "We will further simplify the analysis by making a quadratic approximation ...
1
vote
2answers
48 views

Neural networks: why don't we use a multi-dimensional learning rate

I've searched a bit on the internet a have found the answer nowhere so I decided to post here. When confronted to an optimization problem, we know that the sanity of the problem can be characterized ...
1
vote
3answers
109 views

Hessian of Log of Matrix-t distribution

I am trying to calculate the hessian of the log of the matrix-t distribution. I know that the log of the matrix-t distribution can be written: $$\log T_{N\times P}(X| \nu, M, \Sigma, \Omega) \propto -\...
1
vote
1answer
4k views

How to deal with clmm warning: “hessian is numerically singular”?

I am using R's ordinal package to run a mixed regression model with an ordinal dependent variable. The data I am working with looks like this: ...
1
vote
1answer
87 views

Question about port of R code from the library “rethinking” to PyMC3

A very generous human named Osvaldo Martin did us the favor of porting all the R sample code in Richard McElreath's superb book Statistical Rethinking to PyMC3. I'm hugely grateful, but I've already ...
1
vote
0answers
67 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1
vote
0answers
72 views

How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles?

For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension. I am trying to ...
1
vote
0answers
114 views

Inverting Hessian using Generalized Inverse for Inference

I am estimating a survival model with MLE. I use optim to maximize the likelihood function, and I intend to use the Hessian matrix returned by optim to get the standard errors (which lie on the ...
1
vote
0answers
147 views

Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network

I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem ...
1
vote
0answers
91 views

Should Bayesian estimated error smaller than MLE?

I am dealing with a fitting problem. Specifically, I am fitting a Lorentzian profile to the power spectrum of an solar-like oscillating star. Three parameters in the Lorentzian profile characterize ...
1
vote
0answers
39 views

Non-linear Likelihood function, large estimated standard errors

I have a highly non-linear (lots of jumps) likelihood function with K parameters (For example, a marked Hawkes Process used in seismology study). I implemented the L-BFGS-B optimization routine and it ...
1
vote
0answers
77 views

Fast multiplication by the Hessian in Neural networks

I have question about the $R\{.\}$ function in Bishop's book on page 254 (see snippet below). My questions are as follows: I assume $R\{\bf w\}$ in (5.97) is the premultiplication of $\bf{v}^{T}$ ...
1
vote
0answers
23 views

Inferrence for peaked likelihoods

Suppose I have the likelihood $f(X|\theta)$ of some rich model, where $\theta\in\mathbb{R}^n$, and I have been able to find its maximum, $\hat\theta$. Suppose further that for some $i$, the plot of $...
1
vote
0answers
240 views

Hessian matrix of log marginal likelihood of Gaussian Process

I'm trying to compute the exact second derivatives of log marginal likelihood of Gaussian Process for learning hyperparameters. The log marginal likelihood and its partial derivative are given in 5 ...
1
vote
1answer
33 views

Uncertainty in collapsing several curves

I have a bunch of curves $f(x)$, and I have a parameter $\lambda$ that rescales $x$, such that $x \rightarrow x' = g(x, \lambda)$. I find the value of $\lambda$ that collapses all the curves on top of ...
1
vote
0answers
120 views

Positive definiteness of Hessians?

I'm reading the book "Convex Optimization" by Boyd and Vandenbherge. On the second paragraph of page 71, the authors seem to state that in order to check if the Hessian (H) is positve semidefinite (...
1
vote
0answers
219 views

Maximum likelihood estimation, how to derive the hessian

I am reading a paper and trying to understand how the authors estimated the standard errors of a set of parameter estimates $[\delta \ \ \phi \ \ \Sigma]$. Below is the loglikelihood function (sorry I ...
1
vote
0answers
42 views

Why does glmer break when I remove a subject?

I'm working with the epilepsy data set from Applied Longitudinal Analysis by Fitzmaurice et al. (http://www.hsph.harvard.edu/fitzmaur/ala/epilepsy.txt). In this trial, 59 patients are split into a ...
1
vote
1answer
392 views

SPSS: GLMM and(adjusted) odds ratio

I am performing a retrospective study and the relative statistic analysis. I am studying the the risk factors for the occurrence of complications during medical procedures. I have 50 subjects ...
1
vote
0answers
45 views

MLE: Does the scale of predictor variables affect whether the hessian is positive definite?

I am trying to fit a regression via maximum likelihood estimation, one of the regression terms involves $\beta_0e^{(\beta t)}$ where $t$ is measured in hours and has a range of 0 to 90 days. The ...
0
votes
1answer
31 views

Hessian Matrix Values

Its an easy question but still i cant seem to find the hessian matrix. I have the following function : $$-2x^2 + \sqrt{2}xy - \frac52y^2$$ Find the hessian matrix for this function. $$f_{11} = -4 \...
0
votes
1answer
156 views

Cramér–Rao bound to multiple parameters

I was reading Cramér–Rao bound to multiple parameters from Wikipedia page, but I could not follow this line in the article: Let $\displaystyle {\boldsymbol {T}}(X)$ be an estimator of any ...