What is an example of such a function in 2D and in high dimension space? this image

This image is from CS231n stanford course.


Build a quadratic form using a diagonal matrix as following:

$$ {x}^{T} A x + {b}^{T} x + c $$

Where the matrix is given by:

$$ A = \begin{bmatrix} 1000 & 0 \\ 0 & 0.0001 \end{bmatrix} $$

Then the condition number is:

$$ \operatorname{cond} \left( A \right) = \frac{ {A}_{11} } { {A}_{22} } = 1e6 $$

You can even get it higher if you want.

  • $\begingroup$ thank you but i am not good at math, is A the weights in the linear model? and what is b? $\endgroup$ – floyd Sep 11 '17 at 14:56
  • $\begingroup$ You should read about Quadratic Form Function. Basically it is a generalization of the Quadratic Function from High School - $ a {x}^{2} + b x + c $. $\endgroup$ – Royi Sep 11 '17 at 20:12

I always use the sillies functions when visualizing the multivariable concepts to build intuition. So, my favorite is $f(x,z)=x^2+az^2$. In this case it also roughly corresponds to the picture in your question if you take it that the horizontal axis is x, and the vertical is z. Here's the contour plot for a=2, which is similar to your picture:

enter image description here

Let's take its Hessian: $$H_{xx}=\partial^2 f/\partial x^2=2$$ $$H_{xz}=\partial^2 f/\partial x\partial z=0$$ $$H_{zx}=\partial^2 f/\partial x\partial z=0$$ $$H_{zz}=\partial^2 f/\partial z^2=2a$$

Now your condition number is whatever you want it to be: $$H_{zz}/H_{xx}=a$$

here's the plots for a=1 and a=1000:

enter image description here

enter image description here

This should give you an idea why it's so difficult to deal with high condition numbers. In a=1000 case the minimum looks more like a valley, it's hard to find its lowest point; while at a=1 it's a hole, and the bottom is easy to spot.

Imagine that you toss a steel ball inside these cavities. It'll settle at the bottom much faster in a=1. That's what drawing in your question tries to illustrate.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.