From this blog post:
For any Optimization problem with respect to Machine Learning, there can be either a numerical approach or an analytical approach. The numerical problems are Deterministic, meaning that they have a closed form solution which doesn’t change. Hence it is also called time invariant problems. These closed form solutions are solvable analytically. But these are not optimization problems.
My interpretation of this text is that the author considers that there are two types of approaches for optimization problems under machine learning:
- Numerical approach, for which we can compute solutions directly because they have a closed form solution (e.g. linear regression with least squares)
- Optimization(?)/analytical approach, wherein we try to approximate a good solution (e.g. gradient descent)
This doesn't, however, seem correct. We analytically differentiate the sum of squares expression to obtain its closed form, right? The last sentence of the accepted answer in this post also seems to imply that gradient descent is a numerical method.
Question: Is the categorization above correct? If not, what would be the correct taxonomy for problems in machine learning?
Thank you in advance!