Is it true that standard gradient descent algorithm (be it batch or mini batch or stochastic) cannot be used when we have certain constraint on parameters? If yes, why is it so? Is it because gradient descent just determines gradient of the cost function with respect to the parameter, changes parameter value based on it, and hence has no mechanism to handle the constraints?
Could anyone please help me understand this concept?
Yes, you are right, this is indeed the problem.
What can be done? You could either go for an optimization method other than gradient descent, or you try to find an equivalent problem where your search space is again an open set so that you can still use gradient descent. E.g., if your objective function is $g$ and the parameters are $x$ and your constraint is of the form: $$ f(x) = 0, $$ you could try to fix this by parameterizing the submanifold $M$ given by $f(x)=0$, consider a chart $\xi:U\to M$ from an open subset $U\subset \mathbb{R}^d$ to $M$ and then apply gradient descent to $g\circ \xi$.
Of course, there are many different possible constraints and many other approaches to dealing with them.
