I have been taking Andrew Ng's Machine Learning course, and in the lesson on feature scaling's effect on gradient descent, I just can't understand how because of the different scales on the features cause an elliptical contour plot of the cost function. I made the inference from the contour plot that the cost function increases and decreases very rapidly with an increase of decrease in theta 1, and it increases or decreases very slowly with an increase or decrease in theta 2. (X1=Sq feet of the house. X2=Number of bedrooms.) But I still cant understand how having different ranges fuels that. Please help
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
We can use two functions here as examples to help to understand the concept of derivative.
$$f_1 (x_1, x_2) =x_1^2+x_2^2$$
$$f_2 (x_1, x_2) =x_1^2+10x_2^2$$
If we plot the contour we will find $f_2$ is elliptical. The reason is that along the $x_2$ direction, we can reduce to cost more quickly than $x_1$ direction.
For example, if we are at point $(1,1)$, $f_1(1,1)=2$. For next point, $f_1(0.9,1)=f_1(1,0.9)=1.81$, i.e., they have the same value.
But for $f_2$, it is a different story. $f_2(1,1)=11$, $f_2(0.9,1)=10.81$ and $f_2(1,0.9)=9.1$. So, we can see, if we reduce $x_2$ it is more effective to reduce the cost.
This is why it make make the elliptical contour.
answered May 12, 2020 at 8:27
-
$\begingroup$ Ok, so according to you, if x2 is of large value, then we can reduce to cost function more faster, but it makes the elliptical contour? $\endgroup$ May 12, 2020 at 10:56
-
$\begingroup$ But my professor says that θ will descend quickly on small ranges and slowly on large ranges, which is the opposite of what you said, please clarify. $\endgroup$ May 12, 2020 at 10:57