# Constrained optimization with gradient descent

Suppose I want to maximize the likelihood $$L(\theta_1, \theta_2)$$ for some constraint for example $$\theta_1 + \theta_2 = 1$$ and no other constraints

1. Can I just replace $$\theta_2$$ by $$1 - \theta_1$$ in the likelihood then do gradient descent on $$\theta_1$$. If I can, or cannot, why?

2. Can I set up an objective function with Lagrange multiplier $$\mathcal{L} = L(\theta_1, \theta_2) + \lambda (\theta_1 + \theta_2 - 1)$$ and do a gradient descent algorithm on $$\mathcal{L}$$? If I can, or cannot, why?

3. Do I only can rely on projected gradient descent if I want to solve this constrained optimization problem using gradient descent?

EDIT: I tried all 3 options and maybe my likelihood function is not "regular" and only option 3 works :'( I would like to know why and when options 1 and 2 work.

Thank you very much in advance for all the help.

• The first option is fine. The second option, you'll need an absolute value in in there. Jun 18, 2021 at 2:31
• @AryaMcCarthy Thanks for your comment. I wonder where I need an absolute value. Btw I tried all 3 options and maybe my likelihood function is not "regular" and only option 3 works :'( I would like to know why and when options 1 and 2 work.
– wut
Jun 18, 2021 at 2:45
• You can try reparameterizing ,see vene.ro/blog/mirror-descent.html
– a06e
May 8, 2022 at 16:40

The first option is still constrained as $$\theta_1$$ still has to lie between $$(0,1)$$
Let $$\log \theta_1 = \alpha_1 - \log (e^{\alpha_1}+e^{\alpha_2})$$ and $$\log \theta_2 = \alpha_2 - \log (e^{\alpha_1}+e^{\alpha_2})$$. As you can notice, that this reparametrization still preserves the constraint as $$\theta_1 = \frac{e^{\alpha_1}}{e^{\alpha_1}+e^{\alpha_2}}$$ and $$\theta_2 = \frac{e^{\alpha_2}}{e^{\alpha_1}+e^{\alpha_2}}$$. Here, $$\alpha_1,\alpha_2$$ have no constraints.
• The problem formulation didn't say that $\theta_1$ has to be greater than 0. Jun 18, 2021 at 2:54