Questions tagged [hessian]

For on-topic questions involving the Hessian matrix, a square matrix generalizing the second derivative. Please include also a statistical methods tag. For purely mathemathical questions about the Hessian it is better to ask on math.SE at https://math.stackexchange.com/.

1
vote
1answer
360 views

SPSS: GLMM and(adjusted) odds ratio

I am performing a retrospective study and the relative statistic analysis. I am studying the the risk factors for the occurrence of complications during medical procedures. I have 50 subjects ...
1
vote
1answer
73 views

Observed information matrix with multivariate normal distribution

$$ \DeclareMathOperator\tr{tr} \DeclareMathOperator\vecOP{vec} \newcommand\di{\mathrm{d}} \newcommand\D{\mathrm{D}} \newcommand\Hess{\mathrm{H}} $$ I do not have much experience with matrix ...
28
votes
6answers
2k views

Why not use the third derivative for numerical optimization?

If Hessians are so good for optimization (see e.g. Newton's method), why stop there? Let's use the third, fourth, fifth, and sixth derivatives? Why not?
118
votes
8answers
47k views

Why is Newton's method not widely used in machine learning?

This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes: After reviewing a set of lectures on convex optimization, Newton's method ...
20
votes
1answer
8k views

Explanation of min_child_weight in xgboost algorithm

The definition of the min_child_weight parameter in xgboost is given as the: minimum sum of instance weight (hessian) needed in a child. If the tree partition step results in a leaf node with the ...
1
vote
0answers
35 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
0
votes
0answers
95 views

Hessian of logistic loss - when $y \in \{-1, 1\}$

Logistic Regression has two possible formulations depending on how we select the target variable: $y \in \{0,1\}$ or $y \in \{-1,1\}$. This question discusses the derivation of Hessian of the loss ...
0
votes
1answer
25 views

Hessian Matrix Values

Its an easy question but still i cant seem to find the hessian matrix. I have the following function : $$-2x^2 + \sqrt{2}xy - \frac52y^2$$ Find the hessian matrix for this function. $$f_{11} = -4 \...
1
vote
0answers
55 views

How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles?

For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension. I am trying to ...
2
votes
2answers
133 views

What is a consequence of an ill-conditioned Hessian matrix?

In this publication I found an explanation of the Hessian matrix, along with what it means for it to be ill-conditioned. In the paper, there is this link given between the error surface and the ...
3
votes
0answers
34 views

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian. For the time being, I obtain the derivative as $\frac{\...
3
votes
1answer
155 views

Why does the determinant of the Hessian grow with n?

Context: I'm trying to understand BIC on a deeper level. I'm using BIC for model/structure selection for Bayesian networks. I'm confused because BIC is an approximation to the likelihood of a model, ...
2
votes
2answers
384 views

Standard error from Hessian matrix when likelihood is used (rather than Ln L)

I understand that at MLE point, the inverse of the Hessian matrix can be used as approximation of V-Cov matrix: ...
2
votes
0answers
151 views

Obtaining Standard Errors in Optim() in R [duplicate]

I'm using a maximum likelihood estimation and I'm using the optim() function in R in a similar way as follows: ...
1
vote
1answer
409 views

Computing the Hessian of maximum log likelihood function

I am trying to find the Hessian matrix for the maximum log likelihood function given training data {(xi, yi)} for i=1:N with yi ∈ {+1, −1} for each i = 1, . . . , N for the function: When I try to ...
0
votes
0answers
19 views

Can negative of empirical second derivative of the log likelihood with respect to the parameters not be semi-positive definite?

This is the empirical Fischer Information. Also consider the outer product with itself of the first derivative of the log likelihood with respect to the parameters. This will always be semi-negative ...
1
vote
0answers
89 views

Inverting Hessian using Generalized Inverse for Inference

I am estimating a survival model with MLE. I use optim to maximize the likelihood function, and I intend to use the Hessian matrix returned by optim to get the standard errors (which lie on the ...
1
vote
3answers
101 views

Hessian of Log of Matrix-t distribution

I am trying to calculate the hessian of the log of the matrix-t distribution. I know that the log of the matrix-t distribution can be written: $$\log T_{N\times P}(X| \nu, M, \Sigma, \Omega) \propto -\...
3
votes
1answer
108 views

Explanation of generalization of Newton's Method for multiple dimensions

I've been following the CS 229 lecture videos for machine learning, and in lecture 4 (~14:00), Ng explains Newton's Method for optimization to maximize an objective function ($f$), but doesn't clearly ...
1
vote
0answers
700 views

How the Hessian matrix is used in optimization if you can't invert it

I've seen quite a lot of work to do with approximating the Hessian such as the Hessian Vector Product but I'm not entirely sure how knowing the Hessian helps us evaluate the gradient step to take. ...
1
vote
3answers
1k views

How does the second derivative inform an update step in Gradient Descent?

I was reading the deep learning book by Begnio, Goodfellow and Courville and there was one section where they explain the second derivative that I don't understand (section 4.31): The second ...
1
vote
0answers
117 views

Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network

I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem ...
2
votes
0answers
59 views

Which optimizer use for laplace approximation

I have been trying to estimate the marginal posterior for D variable using Laplace approximation: $p(\theta_i) \approx \left[\frac{\det{H}}{2\pi\det{H(\theta_i)}}\right]^{1/2} \exp\left[-L(\theta_i, \...
0
votes
0answers
535 views

Which Hessian to use to compute standard errors

Let that I have a data vector $\textbf{x} = (x_1,x_2,x_3....x_n)$ Say these are realizations of IID random variables having a common density $f_\theta$ Likelihood computed using $i^\text{th}$ ...
6
votes
1answer
1k views

gradient descent and local maximum

I read that gradient descent converge always to a local minimum while other methods as Newton's method this is not guaranteed (if the Hessian is not definite positive); but if the start point in GD is ...
1
vote
1answer
78 views

Question about port of R code from the library “rethinking” to PyMC3

A very generous human named Osvaldo Martin did us the favor of porting all the R sample code in Richard McElreath's superb book Statistical Rethinking to PyMC3. I'm hugely grateful, but I've already ...
0
votes
1answer
572 views

Outer product approximation of the Hessian

On p251 of Bishop's machine learning book, the Hessian for least squares is derived (as a preliminary step to the outer product approximation): $ E = \frac{1}{2} \sum_{n=1}^{N} \left(y_n - t_n\...
0
votes
1answer
48 views

Newton-Raphson Error

According to Agresti(2013) pg 364-365, iterative methods such as Newton-Raphson methods, $ \begin{aligned} \beta^\text{new} &= \beta^\text{old} + (X^{T}WX)^{-1}X^{T}(V) \end{aligned} $ help to ...
1
vote
0answers
88 views

Should Bayesian estimated error smaller than MLE?

I am dealing with a fitting problem. Specifically, I am fitting a Lorentzian profile to the power spectrum of an solar-like oscillating star. Three parameters in the Lorentzian profile characterize ...
2
votes
0answers
274 views

Multiclass: I want to develop a customized objective function with weights given by both label and prediction, for Xgboost

I want to develop a customized objective function with weights given by both label and prediction, for Xgboost. Example, let's say you have 2 classes I want to assign a penalties according to this ...
0
votes
1answer
140 views

Cramér–Rao bound to multiple parameters

I was reading Cramér–Rao bound to multiple parameters from Wikipedia page, but I could not follow this line in the article: Let $\displaystyle {\boldsymbol {T}}(X)$ be an estimator of any ...
2
votes
2answers
1k views

Poisson Regression and Hessian

I have been trying to estimate parameters of a poisson regression. I am using Newton Raphson method. This method requires that the inverse of Hessian be computed to obtain, updates to beta vector. The ...
4
votes
1answer
894 views

Gradient and hessian of the MAPE

I want to use MAPE(Mean Absolute Percentage Error) as my loss function. ...
3
votes
2answers
3k views

Intuition for the “information matrix equality” result?

I am trying to understand the intuition behind the "information matrix equality" condition in the Maximum Likelihood context (perhaps this is the only context?): $$ -E[H(\theta)] = E[s(\theta) s(\...
1
vote
0answers
38 views

Non-linear Likelihood function, large estimated standard errors

I have a highly non-linear (lots of jumps) likelihood function with K parameters (For example, a marked Hawkes Process used in seismology study). I implemented the L-BFGS-B optimization routine and it ...
0
votes
1answer
67 views

Pearlmutter's method for Hessian multiplication

I am trying to understand the abstract below from Pearlmutter's paper. Can someone clarify to me why $R_{\bf{v}}\{\bf{w}\}=\bf{v}$? Thanks a lot!
1
vote
0answers
68 views

Fast multiplication by the Hessian in Neural networks

I have question about the $R\{.\}$ function in Bishop's book on page 254 (see snippet below). My questions are as follows: I assume $R\{\bf w\}$ in (5.97) is the premultiplication of $\bf{v}^{T}$ ...
1
vote
0answers
23 views

Inferrence for peaked likelihoods

Suppose I have the likelihood $f(X|\theta)$ of some rich model, where $\theta\in\mathbb{R}^n$, and I have been able to find its maximum, $\hat\theta$. Suppose further that for some $i$, the plot of $...
1
vote
0answers
195 views

Hessian matrix of log marginal likelihood of Gaussian Process

I'm trying to compute the exact second derivatives of log marginal likelihood of Gaussian Process for learning hyperparameters. The log marginal likelihood and its partial derivative are given in 5 ...
2
votes
1answer
292 views

Hessian for Laplace Approximation in Uncertainty Propagation

This is possibly a silly conceptual question, ... but anyway: Imagine I have a function: $f = F(\mathbf{x}) = F(x_1,x_2) = ax_1^2 + bx_2^3,$ where $x_1,x_2 \sim N(0,1)$ for example. For a naive ...
1
vote
1answer
33 views

Uncertainty in collapsing several curves

I have a bunch of curves $f(x)$, and I have a parameter $\lambda$ that rescales $x$, such that $x \rightarrow x' = g(x, \lambda)$. I find the value of $\lambda$ that collapses all the curves on top of ...
0
votes
1answer
2k views

Derivation of the Hessian of average empirical loss for Logistic Regression

How does the y term vanish/get cancelled ? Shouldn't it be this instead, h here is the Sigmoid function.
3
votes
1answer
168 views

Show that the following optimization problem is convex

I have the following optimization problem \begin{equation} \label{logdual} \begin{array}{ll@{}ll} \text{minimize}_{\pmb\alpha \in \mathbb{R}^n} & \theta(\pmb\alpha) &\\ \text{subject to} &...
2
votes
1answer
843 views

Second derivative of neural network cost function

This question is highly correlated with my previous one (I was asking about quadratic approximation of the cost function with Hessian matrix and didn't get any answer), but I think that I have the ...
2
votes
0answers
228 views

Quadratic approximation of the regularized cost function for neural network [closed]

I've been working on the topic of regularization for neural networks and in the textbook I'm following I found this quote: "We will further simplify the analysis by making a quadratic approximation ...
0
votes
1answer
67 views

Relation Between the Hessian Matrix at the max of loglikelihood and the Hessian matrix at the minimum of -loglikelihood

I would like to now what is the relation between the Hessian Matrix calculated at the maximum point of the log(likelihood) function, $H_{f}$ and the Hessian Matrix calculated at the minimum point of ...
1
vote
0answers
95 views

Positive definiteness of Hessians?

I'm reading the book "Convex Optimization" by Boyd and Vandenbherge. On the second paragraph of page 71, the authors seem to state that in order to check if the Hessian (H) is positve semidefinite (...
4
votes
1answer
189 views

Variance of maximum likelihood estimator in R

In different sources there is an algorithm how to calculate the variance of MLE in R. To keep it short: construct the negative log likelihood function. minimize it via nlm or optim with hessian=TRUE ...
1
vote
0answers
185 views

Maximum likelihood estimation, how to derive the hessian

I am reading a paper and trying to understand how the authors estimated the standard errors of a set of parameter estimates $[\delta \ \ \phi \ \ \Sigma]$. Below is the loglikelihood function (sorry I ...
1
vote
0answers
41 views

Why does glmer break when I remove a subject?

I'm working with the epilepsy data set from Applied Longitudinal Analysis by Fitzmaurice et al. (http://www.hsph.harvard.edu/fitzmaur/ala/epilepsy.txt). In this trial, 59 patients are split into a ...