In case of Deep NN, why is gradient big in direction in which we want to travel a small distance? I was going through the intuition behind Momentum and RMSprop technique applied in gradient descent. I read the following statement (from the CE NN lecture notes by G. Hinton) which is not clear to me 
• Going downhill reduces the error, but the direction of steepest descent does not point at the minimum unless the ellipse is a circle.


*

*The gradient is big in the direction in which we only want to travel a small distance.

*The gradient is small in the direction in which we want to travel a large distance.
Can somebody please elaborate on this. Because my understanding is that gradient is in the direction of greatest-ascent. If the objective is to minimize go in opposite direction. I understand that going in opposite direction to gradient will move the cursor to next lowest point in the nbd of the current position. But how does it indicate point (1) and (2). How can I understand this with intuition? 
 A: The diagram (on the page you referred) shows what's the problem:

The gradient will be pointing in the direction of the arrow, because the slope of the surface is steeper in South East direction, rather than North East where the bottom is.
One way to see this is by looking at a hyperboloid $z=x^2+100y^2$
Let's take two partials:
$$\frac \partial {\partial x}z=2x\\
\frac \partial {\partial y}z=200y\\$$
Let's see which direction the gradient is in the point (1,1). If it's what we want then it has to be along the line that goes through the origin, i.e. it must be (c,c), where c is a constant. This line is on the 45 degrees line, and goes through the origin.
Here's what we got for our curve:
$$\nabla z = (2,200) $$
It clearly point to a different direction, definitely not the 45 degree line. Actually, it's pointing almost vertically, because out curve is the steepest along y-axis. So, with a naive implementation of the steepest descent you can easily end up in a slow convergence, where your algorithm will be zig-zagging up and down while very slowly moving horizontally towards the minimum if the initial step size is too big. That's why at the very least you want to pay attention to the size of the step.
Just for the sake of completeness the gradient at the same point for a circular contours such as $z=x^2+y^2$ is $\nabla z(1,1)=(2,2)$, i.e. the steepest descent direction goes through the origin.
