For a machine learning classifier, an initial theta of zeros is valid for logistic regression (but not neural networks). I don't understand why matrix multiplying an array of zeros with a non zero feature matrix is valid. Wouldn't the zeros cancel out whatever the feature values are and turn the sum to zero?
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
Initialization is what the gradient descent optimization technique (in most cases) starts with, not what you think might be a good model.
You might be right, that in the first iteration, the output is zero (depending on the classifier), however the gradient won't be, and as long as there is a non-zero gradient, the gradient descent method will start and a local optimum can be found.
-
$\begingroup$ But doesn't the cost/error affect what the gradient during optimization? $\endgroup$– CharCommented Aug 30, 2017 at 22:48
-
1$\begingroup$ I'm assuming when you say $\theta$, you mean the parameters. So yes, the optimization tries to minimize the error, but the gradient gives you a "good direction" for the next step. The only "really bad way" to initialize your optimization, is to initialize it somewhere where the gradient is close to 0, everywhere else the process will start going $\endgroup$– SamCommented Aug 30, 2017 at 22:59
$\begingroup$
$\endgroup$
2
The zeros will not minimize your cost function, which is ($θ^T$X - $y)^2$, the cost function in first iteration will be $y^2$, since $θ^T$X will be zero because all the initial parameters are zeros - refer https://www.youtube.com/watch?v=Q4GNLhRtZNc and https://www.youtube.com/watch?v=Q6Pd82GsNuU for detailed explanation.
The next iteration would try to optimize your cost function with new set of parameters. That's why the initial selection of zeros or any set of values as parameter values has no effect on your final result.
-
1$\begingroup$ Could you fix your algebra & explain what $\theta$TX is? $\endgroup$ Commented Nov 28, 2019 at 3:45
-
$\begingroup$ Hi Michael, made the change. Sorry, new to this whole thing, not very familiar with formatting rules :-) $\endgroup$– DSGyanCommented Nov 30, 2019 at 5:58