When a set of equations cannot be solved analytically, then we can use a gradient descent algorithm. But it seems that there is also the method of Monte Carlo simulation that can be used to solve problems that do not have analytical solutions.
How to tell when to use gradient descent and when to use Monte Carlo? Or am I just plain confusing the term 'simulation' with 'optimization'?
Thank you very much!
These techniques do different things.
Gradient descent is an optimization technique, therefore it is common in any statistical method that requires maximization (MLE, MAP).
Monte Carlo simulation is for computing integrals by sampling from a distribution and evaluating some function on the samples. Therefore it is commonly used with techniques that require computation of expectations (Bayesian Inference, Bayesian Hypothesis Testing).
These are both huge families of algorithms, so it's difficult to give you a precise answer, but...
Gradient ascent (or descent) is useful when you want to find a maximum (or minimum). For example, you might be finding the mode of a probability distribution, or a combination of parameters that minimize some loss function. The "path" it takes to find these extrema can tell you a little bit about the overall shape of the function, but it's not intended to; in fact, the better it works, the less you'll know about everything but the extrema.
Monte Carlo methods are named after the Monte Carlo casino because they, like the casino, depend on randomization. It can be used in many different ways, but most of these focus on approximating distributions. Markov Chain Monte Carlo algorithms, for example, find ways to efficiently sample from complicated probability distributions. Other Monte Carlo simulations might generate distributions over possible outcomes.
This answer is partially wrong. You can indeed combine Monte Carlo methods with gradient descent. You can use Monte Carlo methods to estimate the gradient of a loss function, which is then used by gradient descent to update the parameters. A popular Monte Carlo method to estimate the gradient is the score gradient estimator, which can e.g. be used in reinforcement learning. See Monte Carlo Gradient Estimation in Machine Learning (2019) by Shakir Mohamed et al. for more info.
As explained by others, gradient descent/ascent performs optimisation, i.e. finds the maximum or minimum of a function. Monte Carlo is a method of stochastic simulation, i.e. approximates a cumulative distribution function via repeated random sampling. This is also called "Monte Carlo integration" because the c.d.f. of a continuous distribution is actually an integral.
What's common between gradient descent and Monte Carlo is that they're both particularly useful in problems where no closed-form solution exists. You may use simple differentiation to find the maximum or minimum point of any convex function whenever an analytical solution is feasible. When such a solution does not exist, you need to use an iterative method such as gradient descent. Is is the same for Monte Carlo simulation; you can basically use plain integration to calculate any c.d.f. analytically but there's no guarantee that such a closed form solution will always be possible. The problem becomes solvable again with Monte Carlo simulation.
Can you use gradient descent for simulation and Monte Carlo for optimisation? The simple answer is no. Monte Carlo needs a stochastic element (a distribution) to sample from and gradient descent has no means of handling stochastic information problems. You can, however, combine simulation with optimisation in order to produce more powerful stochastic optimisation algorithms that are able to solve very complex problems that simple gradient descent is unable to solve. An example of this would be Simulated Annealing Monte Carlo.
