0
$\begingroup$

In this deck about the perceptron, there's a pseudo-code for the batch version on page 12. During each iteration of the inner loop, the dot product $y_ix_i\theta$ is compared to 0, however since the weights ($\theta$) are initialized to $\vec{0}$, and only updated in the outer loop, this product will stay the same throughout all first N iterations of the inner loop. Thus, wouldn't it be better to initialize the weights to the first instance or a random vector?

$\endgroup$
1
$\begingroup$

You're right, but it only happens in the first iteration of the outer loop, so you only waste one iteration and the rest of the iterations will shift the weights in the assumingly right direction so it's not that big a deal. However, as you suggested, you can initialize the weights to small random numbers in the region of [-eps,eps].

$\endgroup$
  • $\begingroup$ Thanks,what would be a wise choice for eps, and how does it compare to taking the first instance? $\endgroup$ – dimid Oct 16 '16 at 14:15
  • 1
    $\begingroup$ It doesn't really matter but I'd choose eps=1e-4. Its practical influence as a number is small but the sign make sure that a certain direction won't be favorable, meaning it will make sure that approximately 50% of the weights will shift towards one direction (due to the comparison to 0) and the other half will shift towards the other direction. I guess that in the general case it will converge a bit faster than taking the first instance. $\endgroup$ – Gabizon Oct 17 '16 at 17:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.