Questions tagged [hessian]

For on-topic questions involving the Hessian matrix, a square matrix generalizing the second derivative. Please include also a statistical methods tag. For purely mathemathical questions about the Hessian it is better to ask on math.SE at https://math.stackexchange.com/.

24 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
36 views

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian

what is the hessian matrix of $f(W) = \sqrt{Tr(W A_0 W A_1)}$? Here, $W$, $A_0$ and $A_1$ are positive semidefinite matrices and hermitian. For the time being, I obtain the derivative as $\frac{\...
2
votes
0answers
60 views

Which optimizer use for laplace approximation

I have been trying to estimate the marginal posterior for D variable using Laplace approximation: $p(\theta_i) \approx \left[\frac{\det{H}}{2\pi\det{H(\theta_i)}}\right]^{1/2} \exp\left[-L(\theta_i, \...
2
votes
0answers
278 views

Multiclass: I want to develop a customized objective function with weights given by both label and prediction, for Xgboost

I want to develop a customized objective function with weights given by both label and prediction, for Xgboost. Example, let's say you have 2 classes I want to assign a penalties according to this ...
1
vote
0answers
44 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1
vote
0answers
60 views

How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles?

For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension. I am trying to ...
1
vote
0answers
94 views

Inverting Hessian using Generalized Inverse for Inference

I am estimating a survival model with MLE. I use optim to maximize the likelihood function, and I intend to use the Hessian matrix returned by optim to get the standard errors (which lie on the ...
1
vote
0answers
734 views

How the Hessian matrix is used in optimization if you can't invert it

I've seen quite a lot of work to do with approximating the Hessian such as the Hessian Vector Product but I'm not entirely sure how knowing the Hessian helps us evaluate the gradient step to take. ...
1
vote
0answers
122 views

Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network

I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem ...
1
vote
0answers
89 views

Should Bayesian estimated error smaller than MLE?

I am dealing with a fitting problem. Specifically, I am fitting a Lorentzian profile to the power spectrum of an solar-like oscillating star. Three parameters in the Lorentzian profile characterize ...
1
vote
0answers
38 views

Non-linear Likelihood function, large estimated standard errors

I have a highly non-linear (lots of jumps) likelihood function with K parameters (For example, a marked Hawkes Process used in seismology study). I implemented the L-BFGS-B optimization routine and it ...
1
vote
0answers
69 views

Fast multiplication by the Hessian in Neural networks

I have question about the $R\{.\}$ function in Bishop's book on page 254 (see snippet below). My questions are as follows: I assume $R\{\bf w\}$ in (5.97) is the premultiplication of $\bf{v}^{T}$ ...
1
vote
0answers
23 views

Inferrence for peaked likelihoods

Suppose I have the likelihood $f(X|\theta)$ of some rich model, where $\theta\in\mathbb{R}^n$, and I have been able to find its maximum, $\hat\theta$. Suppose further that for some $i$, the plot of $...
1
vote
0answers
201 views

Hessian matrix of log marginal likelihood of Gaussian Process

I'm trying to compute the exact second derivatives of log marginal likelihood of Gaussian Process for learning hyperparameters. The log marginal likelihood and its partial derivative are given in 5 ...
1
vote
0answers
96 views

Positive definiteness of Hessians?

I'm reading the book "Convex Optimization" by Boyd and Vandenbherge. On the second paragraph of page 71, the authors seem to state that in order to check if the Hessian (H) is positve semidefinite (...
1
vote
0answers
192 views

Maximum likelihood estimation, how to derive the hessian

I am reading a paper and trying to understand how the authors estimated the standard errors of a set of parameter estimates $[\delta \ \ \phi \ \ \Sigma]$. Below is the loglikelihood function (sorry I ...
1
vote
0answers
41 views

Why does glmer break when I remove a subject?

I'm working with the epilepsy data set from Applied Longitudinal Analysis by Fitzmaurice et al. (http://www.hsph.harvard.edu/fitzmaur/ala/epilepsy.txt). In this trial, 59 patients are split into a ...
1
vote
1answer
367 views

SPSS: GLMM and(adjusted) odds ratio

I am performing a retrospective study and the relative statistic analysis. I am studying the the risk factors for the occurrence of complications during medical procedures. I have 50 subjects ...
1
vote
0answers
43 views

MLE: Does the scale of predictor variables affect whether the hessian is positive definite?

I am trying to fit a regression via maximum likelihood estimation, one of the regression terms involves $\beta_0e^{(\beta t)}$ where $t$ is measured in hours and has a range of 0 to 90 days. The ...
0
votes
0answers
104 views

Hessian of logistic loss - when $y \in \{-1, 1\}$

Logistic Regression has two possible formulations depending on how we select the target variable: $y \in \{0,1\}$ or $y \in \{-1,1\}$. This question discusses the derivation of Hessian of the loss ...
0
votes
0answers
19 views

Can negative of empirical second derivative of the log likelihood with respect to the parameters not be semi-positive definite?

This is the empirical Fischer Information. Also consider the outer product with itself of the first derivative of the log likelihood with respect to the parameters. This will always be semi-negative ...
0
votes
1answer
67 views

Pearlmutter's method for Hessian multiplication

I am trying to understand the abstract below from Pearlmutter's paper. Can someone clarify to me why $R_{\bf{v}}\{\bf{w}\}=\bf{v}$? Thanks a lot!
0
votes
0answers
586 views

Variance estimation of GLM coefficients

I'm having trouble understanding the relationship between the variance of the GLM coefficients and the estimated observed Hessian. In the textbook I'm using it's stated "An obvious and suitable choice ...
0
votes
0answers
1k views

Singular Hessian/Observed information Matrix at optimal solution

I am trying to estimate the standard errors of an maximum likelihood estimate (multidimensional) in R'sfunction optim. I want to ...
0
votes
0answers
538 views

Which Hessian to use to compute standard errors

Let that I have a data vector $\textbf{x} = (x_1,x_2,x_3....x_n)$ Say these are realizations of IID random variables having a common density $f_\theta$ Likelihood computed using $i^\text{th}$ ...