Neural networks are usually trained by calculating gradients using backprop and then performing gradient descent. I am not sure what are difficulties in training a neural network using alternating minimization?

Also, what is the order of computing gradient using backprop vs. naive gradient computation in neural networks?

Edit: It seems that the main idea in neural networks is that there are many parameters, and updates can be performed in parallel, thus speeding up the process. In this process, we lose the theoretical guarantees of alternating minimization, but we gain on speed. Is that right? What is the difference from a theoretical and optimization point of view of updating everything at once and serially?

Alternating minimization- In this optimization strategy, the idea is to fix everything except one variable and perform gradient descent for it, and do this procedure alternatively for all variables till some convergence criteria is met. Reference: https://www.mit.edu/~rakhlin/6.883/lectures/lecture07.pdf

  • 1
    $\begingroup$ By "alternating minimization" are you referring to an optimization strategy that fixes all but 1 of the parameters under optimization, performs an update for the 1 parameter, and then repeats for each parameter? Or something else? $\endgroup$
    – Sycorax
    Mar 7 at 21:05
  • 1
    $\begingroup$ @newbie I'll be honest, I have no idea what you mean by "alternating" loss. Is there any article or paper that you got this idea from? I've never heard of it before. $\endgroup$
    – Sean
    Mar 9 at 0:27
  • 1
    $\begingroup$ This is an interesting question. Examples of alternating gradient descent can be found in several machine-learning applications, e.g., see these papers and presentations: cs.cmu.edu/~ggordon/singh-gordon-kdd-factorization.pdf#page=3 (page 3), cs.ubc.ca/~schmidtm/Courses/340-F16/T10.pdf#page=6 (page 6-12), tensorlab.cms.caltech.edu/users/anima/pubs/… (page 7). It is a common technique when optimization involves multiple variables. But I have not seen these in deep learning literature. I wonder if there's a reason. $\endgroup$
    – vbip
    Mar 9 at 17:31
  • 1
    $\begingroup$ How would you imagine updating 175 billion parameters (GPT-3) one-by-one? $\endgroup$
    – Tim
    Mar 9 at 21:47
  • 1
    $\begingroup$ It's an interesting question, but could you please edit it a bit to at least include your definition of "alternating minimization" and at least one relevant reference that explains it in more detail ? $\endgroup$ Mar 9 at 22:26

1 Answer 1


I assume that by alternating minimization you mean optimizing over one layer while keeping the others fixed, and then repeating this process. This is also called block coordinate descent (coordinate descent being the setting in which we minimize over each variable one at a time).

A critical flaw of coordinate descent methods is that they need not converge to a stationary point. See the limitations section of this Wikipedia article on Coordinate Descent. This is worse that the situation for stochastic gradient descent, where you converge to a local minima. This is one reason why coordinate descent isn't used to train neural networks--even in small examples. It only works in special cases, and the gnarly objective functions produced when training neural networks isn't one of those cases.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.