While studying the slides [Link] (https://www.dropbox.com/s/rdwzjjah9f2mb2j/Logistic%20Regression%20to%20ILRS.pdf?dl=0) on logistic regression, I faced a question.

In slide 15 and 16 it is stated that "To maximize L(β), we set the first order partial derivatives of L(β) to zero".

When the derivative of L(β) is zero, the corresponding data point can either maximize or minimize the value of L(β). More precisely, the derivative to be zero is only a necessary, but not sufficient, condition for a data point to maximize the value of L(β) . How can we guarantee that they have the maximal instead of the minimal value?

I will be grateful if you can provide me with some guidance.

  • $\begingroup$ This is standard way of maximizing/minimizing in calculus. If the first derivative of a curve is 0 at a point, it means that slope of the curve is 0, which means that curve attains either max or min value at this point (or it is a point of inflection). If the 2nd derivative is -ve at this point, then it is max, & if 2nd derivative is +ve, then it is min. The same concept is extended to functions of more than one variables, we you 1 can use partial derivatives. This link explains it very well math.oregonstate.edu/home/programs/undergrad/… $\endgroup$ – Gaurav Singhal Jul 15 '16 at 10:05
  • $\begingroup$ @GauravSinghal: thank you very much. How can we guarantee that they have the maximal instead of the minimal value? is L(β) convex? $\endgroup$ – master thesis Jul 15 '16 at 10:56
  • $\begingroup$ Not sure about convexity of L(β). However if it is convex, then the second derivative test given at the link does work. $\endgroup$ – Gaurav Singhal Jul 15 '16 at 11:46
  • $\begingroup$ See my comment at stats.stackexchange.com/questions/191020/… . $\endgroup$ – Mark L. Stone Jul 15 '16 at 13:51

This is related to hessians. Gaurav's answer summarizes the gritty details, but here's a way to rederive and interpret them. By Taylor's theorem a nice function $f$ can be written as:

$$f(x+h)=f(x)+(\nabla f(x))h+h^THh+o(h^Th)$$

Where the last term is small for $h$ small. Thus around the critical point this captures the essential behavior of $f$.

When $x$ is a critical point the gradient vanishes and the next largest term involves the Hessian. Since $H$ is a symmetric matrix, It's now quite obvious that if $H$ is strictly positive definite, then you're at a minimum. If it's strictly negative definite you're at a maximum.

To verify positive definiteness you just need to verify all eigenvalues are positive. One condition is that the determinant is positive. Since the determinant is the product of eigenvalues you now need to make sure at least one of them is positive which you can double check is satisfied when $f_{xx}>0$. You can generalize this to negative definite.

You also now know how to extend this reasoning to more than 2 dimensions, although there it might be easier to just explicitly compute the eigenvalues of $H$.

| cite | improve this answer | |
  • $\begingroup$ Thanks Alex, never thought it could be explained so precisely. Kudos to your explanation $\endgroup$ – Gaurav Singhal Jul 18 '16 at 8:23

Here is the answer from Wikipedia. Suppose that $f(x, y)$ is a differentiable real function of two variables whose second partial derivatives exist. The Hessian matrix $H$ of $f$ is the 2 × 2 matrix of partial derivatives of $f$:

$H(x,y)=$ \begin{pmatrix}f_{xx}(x,y)&f_{xy}(x,y)\\f_{yx}(x,y)&f_{yy}(x,y)\end{pmatrix}

Define $D(x, y)$ to be the determinant

$ D(x,y)=det(H(x,y))=f_{xx}(x,y)f_{yy}(x,y)-\left(f_{xy}(x,y)\right)^{2}$,

of $H$. Finally, suppose that $(a, b)$ is a critical point of $f$ (that is, $f_x(a, b) = f_y(a, b) = 0$). Then the second partial derivative test asserts the following:

If $ D(a,b)>0 $ and $ f_{xx}(a,b)>0$ then $(a,b)$ is a local minimum of $f$.

If $ D(a,b)>0 $ and $ f_{xx}(a,b)<0$ then $(a,b)$ is a local maximum of $f$.

If $ D(a,b)<0 $ then $(a,b)$ is asaddle point of f.

If $ D(a,b)=0 $ then the second derivative test is inconclusive, and the point $(a, b)$ could be any of a minimum, maximum or saddle point.

Functions of many variables

For a function f of more than two variables, there is a generalization of the rule above. In this context, instead of examining the determinant of the Hessian matrix, one must look at the eigenvalues of the Hessian matrix at the critical point. The following test can be applied at any critical point (a, b, ...) for which the Hessian matrix is invertible:

If the Hessian is positive definite (equivalently, has all eigenvalues positive) at (a, b, ...), then f attains a local minimum at (a, b, ...). If the Hessian is negative definite (equivalently, has all eigenvalues negative) at (a, b, ...), then f attains a local maximum at (a, b, ...). If the Hessian has both positive and negative eigenvalues then (a, b, ...) is a saddle point for f (and in fact this is true even if (a, b, ...) is degenerate). In those cases not listed above, the test is inconclusive

| cite | improve this answer | |
  • $\begingroup$ I think it's always worth mentioning where those mysterious conditions come from, you are simply checking if the hessian is positive definite, negative definite, or niether. $\endgroup$ – Matthew Drury Jul 15 '16 at 15:43
  • $\begingroup$ Thanks Matthew for the suggestion, I thought it would take a lot of effort to explain here, and it was explained graphically in my first comment. Never thought it could be explained so easily and briefly for functions of many variables like Alex did. $\endgroup$ – Gaurav Singhal Jul 18 '16 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.