I have been working on a manual implementation of ARMA GARCH (1,1) with: $$\sigma_{t}^2 = \omega + \alpha\epsilon_{t-1}^2 + \beta\sigma_{t-1}^2$$
and estimating parameters through MLE. However, my constant term in GARCH, $\omega$, seems to grow without bound as the optimization proceeds. Is there any sort of constraint on the GARCH parameters other than they must all be non-negative and that $\alpha + \beta < 1$?
In order to ensure that the conditional variance is always non-negative you have the restrictions $\omega>0$, $\alpha \geq 0$ and $\beta\geq 0$. Also for a weakly stationary process you need have $\alpha+\beta<1$. There are other constrains in order to ensure that higher moments exist. For example, assuming normal distributed innovations, you need the parameter restriction $3\alpha^2+2\alpha\beta+\beta^2<1$ for a finite fourth moment. However, this should not effect your estimation process, since for many GARCH models higher moments do not exist, when these models are estimated using real financial data. I assume there may be an error in your code, but this is difficult to answer without without seeing the code.
Regards
