1
$\begingroup$

Question 1

Why do we divide by the number of data points (N)? I think it's done to minimize the error being back-propagated, but can't we just don't do that and instead decrease the learning rate to decrease the "learning" instead?

enter image description here

Question 2

For classification tasks, would it work to apply a softmax activation function to the last (output) layer and then calculate the error using the Mean Squared Error formula? (For the sake of simplicity, cross-entropy is harder compared to MSE)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In Question 1, yes, you can do that. It's equivalent if you use $\frac{\alpha}{N}$ as you stated.

In Question 2, also yes, you can choose to have softmax activation together with MSE. Then, you'll be minimizing MSE and of course your convergence path will change. Cross-entropy and softmax layer get along with each other very well in gradient calculation, which is one of the many reasons the two are so commonly used.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.