Answer to Question 1:
The maximization step can be conducted by numerical optimization. In fact, it is possible to incorporate constraints (if any) on the original problem into the numerical optimization by use of constrained numerical optimization. One possible use of such constraints is to steer the whole EM optimization away from "bad" regions; but they can also be used to incorporate hard constraints on parameter values.
In fact, the maximum need not even be found during the maximum step. In a variant called Generalized Expectation Maximization, the M step may consist of only one or some number of optimization algorithm iterations, with the M step being terminated prior to a maximum being found. The key is to get an improvement in the objective function in each M step.
If numerical optimization of the M step isn't easy or effective enough, you may be better off using a traditional numerical optimizer on the full problem. All sorts of hybrid approaches are possible.
Answer to Question 2:
Example of use of numerical maximization to perform M step: Slides 16-21 of http://www.control.isy.liu.se/student/graduate/MachineLearning/Lectures/le6.pdf show an example of nonlinear system identification, in which the objective function for the M step does not have a closed form maximum. However, a formula for the gradient of the objective function is derived. The gradient can be used in an off-the-shelf gradient-based numerical optimizer, for example, a Quasi-Newton method such as BFGS (if the gradient were not available, some other numerical maximization method could be used). I have no idea whether this EM algorithm is better or faster in this case than applying Quasi-Newton (BFGS) to the original problem.