1.Across different epochs, which of the following is/are updated?

initial weights (initial ConvNet filter matrices, initial fully connected weights)

hyper parameters: number of ConvNet filters, size of ConvNet filters, number of layers...

2.The lost function calculated from the last epoch appears to be the initial value of the lost function for the current epoch. Why?

  • 2
    $\begingroup$ Welcome to Cross Validated! Please don't edit questions once they're answered (except for minor things of course). Else the answer may become irrelevant to the new version of the question. Best to leave the question & answer as they are (they may well help other people), & ask a new question, linking to this one if it provides some context. $\endgroup$ – Scortchi - Reinstate Monica Mar 13 '19 at 14:50
  • $\begingroup$ as i thought my original question misled a bit. i tried to clarify $\endgroup$ – feynman Mar 14 '19 at 3:23
  • $\begingroup$ Of course, no problem. $\endgroup$ – Scortchi - Reinstate Monica Mar 14 '19 at 9:17
  1. Weights and biases are updated using the back-propagation algorithm. If you're using batch norm, those have parameters which are also updated, but they are not updated as part of the back-propagation.

Model hyperparameters such as the number of weights, layer sizes and so on are not updated. These are all fixed by the researcher when the model is created.

  1. Descending a loss surface is a lot like hiking down a mountain. Where you make camp at the end of one day (epoch) is where you wake up at the beginning of the next day (epoch).

Likewise, one epoch ends with a particular configuration of parameters; that configuration of parameters corresponds to a specific loss value. When you start a new epoch with that parameter configuration, the loss won't change. (The only "catch" is that your estimate of the loss might change because you're using mini-batching; but it probably won't be different by a large value.)

  • $\begingroup$ great thx! but if the hyperparameters don't change at any time, and the weights were ALREADY trained to minimize the loss function during the last epoch, why will the loss function ever move during the current epoch? $\endgroup$ – feynman Mar 12 '19 at 2:44
  • $\begingroup$ Minimization is the goal, but it usually takes more than 1 epoch to find the minimum, or even get near it. $\endgroup$ – Sycorax says Reinstate Monica Mar 12 '19 at 2:45
  • $\begingroup$ why does it take more than 1 epoch to minimize? over the course of each epoch, the training finishes only after the loss function is minimized. am i correct? $\endgroup$ – feynman Mar 12 '19 at 2:46
  • $\begingroup$ No. An epoch is a single pass over the training data. A single pass over the training data is typically not enough to find a minimum. Minimizing the loss can require multiple passes over the training data (multiple epochs). $\endgroup$ – Sycorax says Reinstate Monica Mar 12 '19 at 2:47
  • $\begingroup$ SO, r u saying that 1 epoch only goes thru the gradient descent equation by a tiny bit of the weights? i thought, during each epoch, gradient descent finishes doing its job. across different epochs, it's the validation set's job to calculate the errors to somehow modify the model $\endgroup$ – feynman Mar 12 '19 at 2:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.