Questions tagged [hessian]

For on-topic questions involving the Hessian matrix, a square matrix generalizing the second derivative. Please include also a statistical methods tag. For purely mathemathical questions about the Hessian it is better to ask on math.SE at https://math.stackexchange.com/.

63 questions
Filter by
Sorted by
Tagged with
53k views

Why is Newton's method not widely used in machine learning?

This is something that has been bugging me for a while, and I couldn't find any satisfactory answers online, so here goes: After reviewing a set of lectures on convex optimization, Newton's method ...
2k views

Why not use the third derivative for numerical optimization?

If Hessians are so good for optimization (see e.g. Newton's method), why stop there? Let's use the third, fourth, fifth, and sixth derivatives? Why not?
11k views

Explanation of min_child_weight in xgboost algorithm

The definition of the min_child_weight parameter in xgboost is given as the: minimum sum of instance weight (hessian) needed in a child. If the tree partition step results in a leaf node with the ...
3k views

Name for outer product of gradient approximation of Hessian

Is there a name for approximating the Hessian as the outer product of the gradient with itself? If one is approximating the Hessian of the log-loss, then the outer product of the gradient with itself ...
1k views

gradient descent and local maximum

I read that gradient descent converge always to a local minimum while other methods as Newton's method this is not guaranteed (if the Hessian is not definite positive); but if the start point in GD is ...
651 views

Interpretation of eigenvectors of Hessian inverse

I'm reading a paper in which they use the eigenvectors of the inverse Hessian of a continuous probability distribution to characterize dimensions along which the distribution is most and least ...
2k views

118 views

Explanation of generalization of Newton's Method for multiple dimensions

I've been following the CS 229 lecture videos for machine learning, and in lecture 4 (~14:00), Ng explains Newton's Method for optimization to maximize an objective function ($f$), but doesn't clearly ...
175 views

Show that the following optimization problem is convex

I have the following optimization problem \begin{equation} \label{logdual} \begin{array}{ll@{}ll} \text{minimize}_{\pmb\alpha \in \mathbb{R}^n} & \theta(\pmb\alpha) &\\ \text{subject to} &...
176 views

Why does the determinant of the Hessian grow with n?

Context: I'm trying to understand BIC on a deeper level. I'm using BIC for model/structure selection for Bayesian networks. I'm confused because BIC is an approximation to the likelihood of a model, ...
38 views

320 views

Multiclass: I want to develop a customized objective function with weights given by both label and prediction, for Xgboost

I want to develop a customized objective function with weights given by both label and prediction, for Xgboost. Example, let's say you have 2 classes I want to assign a penalties according to this ...
249 views

Quadratic approximation of the regularized cost function for neural network [closed]

I've been working on the topic of regularization for neural networks and in the textbook I'm following I found this quote: "We will further simplify the analysis by making a quadratic approximation ...
48 views

Neural networks: why don't we use a multi-dimensional learning rate

I've searched a bit on the internet a have found the answer nowhere so I decided to post here. When confronted to an optimization problem, we know that the sanity of the problem can be characterized ...
I am trying to calculate the hessian of the log of the matrix-t distribution. I know that the log of the matrix-t distribution can be written: $$\log T_{N\times P}(X| \nu, M, \Sigma, \Omega) \propto -\... 1answer 4k views How to deal with clmm warning: “hessian is numerically singular”? I am using R's ordinal package to run a mixed regression model with an ordinal dependent variable. The data I am working with looks like this: ... 1answer 87 views Question about port of R code from the library “rethinking” to PyMC3 A very generous human named Osvaldo Martin did us the favor of porting all the R sample code in Richard McElreath's superb book Statistical Rethinking to PyMC3. I'm hugely grateful, but I've already ... 0answers 67 views Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them? While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ... 0answers 72 views How positive definite Hessian approximations for SGD (e.g. Gauss-Newton) handle saddles? For example due to symmetry of parameters, functions optimized in machine learning usually have huge number of local minima and saddles - growing exponentially with dimension. I am trying to ... 0answers 114 views Inverting Hessian using Generalized Inverse for Inference I am estimating a survival model with MLE. I use optim to maximize the likelihood function, and I intend to use the Hessian matrix returned by optim to get the standard errors (which lie on the ... 0answers 147 views Computing the Hessian Matrix Diagonal of a multi-layered Feed Forward Neural Network I am working on using a Feedforward multi-layered perceptron as a function approximator for the pressure distribution of a groundwater system. I am essentially trying to solve a boundary value problem ... 0answers 91 views Should Bayesian estimated error smaller than MLE? I am dealing with a fitting problem. Specifically, I am fitting a Lorentzian profile to the power spectrum of an solar-like oscillating star. Three parameters in the Lorentzian profile characterize ... 0answers 39 views Non-linear Likelihood function, large estimated standard errors I have a highly non-linear (lots of jumps) likelihood function with K parameters (For example, a marked Hawkes Process used in seismology study). I implemented the L-BFGS-B optimization routine and it ... 0answers 77 views Fast multiplication by the Hessian in Neural networks I have question about the R\{.\} function in Bishop's book on page 254 (see snippet below). My questions are as follows: I assume R\{\bf w\} in (5.97) is the premultiplication of \bf{v}^{T} ... 0answers 23 views Inferrence for peaked likelihoods Suppose I have the likelihood f(X|\theta) of some rich model, where \theta\in\mathbb{R}^n, and I have been able to find its maximum, \hat\theta. Suppose further that for some i, the plot of ... 0answers 240 views Hessian matrix of log marginal likelihood of Gaussian Process I'm trying to compute the exact second derivatives of log marginal likelihood of Gaussian Process for learning hyperparameters. The log marginal likelihood and its partial derivative are given in 5 ... 1answer 33 views Uncertainty in collapsing several curves I have a bunch of curves f(x), and I have a parameter \lambda that rescales x, such that x \rightarrow x' = g(x, \lambda). I find the value of \lambda that collapses all the curves on top of ... 0answers 120 views Positive definiteness of Hessians? I'm reading the book "Convex Optimization" by Boyd and Vandenbherge. On the second paragraph of page 71, the authors seem to state that in order to check if the Hessian (H) is positve semidefinite (... 0answers 219 views Maximum likelihood estimation, how to derive the hessian I am reading a paper and trying to understand how the authors estimated the standard errors of a set of parameter estimates [\delta \ \ \phi \ \ \Sigma]. Below is the loglikelihood function (sorry I ... 0answers 42 views Why does glmer break when I remove a subject? I'm working with the epilepsy data set from Applied Longitudinal Analysis by Fitzmaurice et al. (http://www.hsph.harvard.edu/fitzmaur/ala/epilepsy.txt). In this trial, 59 patients are split into a ... 1answer 392 views SPSS: GLMM and(adjusted) odds ratio I am performing a retrospective study and the relative statistic analysis. I am studying the the risk factors for the occurrence of complications during medical procedures. I have 50 subjects ... 0answers 45 views MLE: Does the scale of predictor variables affect whether the hessian is positive definite? I am trying to fit a regression via maximum likelihood estimation, one of the regression terms involves \beta_0e^{(\beta t)} where t is measured in hours and has a range of 0 to 90 days. The ... 1answer 31 views Hessian Matrix Values Its an easy question but still i cant seem to find the hessian matrix. I have the following function :$$-2x^2 + \sqrt{2}xy - \frac52y^2$$Find the hessian matrix for this function.$$f_{11} = -4 \...
I was reading Cramér–Rao bound to multiple parameters from Wikipedia page, but I could not follow this line in the article: Let $\displaystyle {\boldsymbol {T}}(X)$ be an estimator of any ...