I am working on theoretical machine learning — on transfer learning, to be specific — for my Ph.D.
Out of curiosity, why should I take a course on convex optimization?
What take-aways from convex optimization can I use in my research on theoretical machine learning?
Machine learning algorithms use optimization all the time. We minimize loss, or error, or maximize some kind of score functions. Gradient descent is the "hello world" optimization algorithm covered on probably any machine learning course. It is obvious in the case of regression, or classification models, but even with tasks such as clustering we are looking for a solution that optimally fits our data (e.g. k-means minimizes the within-cluster sum of squares). So if you want to understand how the machine learning algorithms do work, learning more about optimization helps. Moreover, if you need to do things like hyperparameter tuning, then you are also directly using optimization.
One could argue that convex optimization shouldn't be that interesting for machine learning since instead of dealing with convex functions, we often encounter loss surfaces like the one below, that are far from convex.
(source: https://www.cs.umd.edu/~tomg/projects/landscapes/ and arXiv:1712.09913)
Nonetheless, as mentioned in other answers, convex optimization is faster, simpler, and less computationally intensive. For example, gradient descent and alike algorithms are commonly used in machine learning, especially for neural networks, because they "work", scale, and are widely implemented in different software, nonetheless, they are not the best that we can get and have their pitfalls, as discussed by Ali Rahimi's talk at NIPS 2017.
On another hand, non-convex optimization algorithms such as evolutionary algorithms seem to be gaining more and more recognition in the ML community, e.g. training neural networks by neuroevolution seems to be a recent research topic (see also arXiv:1712.07897).
I think there are two questions here.
I think @Tim has a good answer on why optimization. I strongly agree and would recommend anyone interested in machine learning to master continuous optimization. Because the optimization process / finding the better solution over time, is the learning process for a computer.
I want to talk more about why we are interested in convex functions. The reason is simple: convex optimizations are "easier to solve", and we have a lot of reliably algorithm to solve.
But is the world convex? No. Why obsessed with convexity? Check this metaphor
A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, "this is where the light is".
As hxd1011 said, convex problems are easier to solve, both theoretically and (typically) in practice. So, even for non-convex problems, many optimization algorithms start with "step 1. reduce the problem to a convex one" (possibly inside a while loop).
A similar thing happens with nonlinear rootfinding. Usually the solution (e.g., with Newton's method) goes "step 1. Reduce to a linear problem, because we know how to solve those".
The most important takeaway is that machine learning is applied to problems where there is no optimal solution available. The best you can do is find a good approximation.
In contrast, when you have an optimisation problem, there is an optimal solution, but it usually cannot be found in reasonable time or with reasonable processing power.
The tools and algorithms you use are fundamentally different. So while I would say that there is no immediate benefit of taking an optimisation class, it is always good to know a bit about related fields. If you can recognise an optimisation problem you'll know that you should not tackle it with machine learning algorithms but with optimisation algorithms instead. That alone is worth a lot I'd say.
If your interests lie in (convex) optimisation applied to deep learning (you mention transfer learning, which is widely used in practice with neural networks) applications, I strongly encourage you to consider reading chapter 8 (optimization for training deep neural networks) of http://www.deeplearningbook.org/
There is a discussion of convex optimisation, and why it has not yet been so successful when applied deep neural networks. Of course, perhaps you could do research in this area that will change the current consensus!
As I heard from Jerome H. Friedman methods developed in Machine Learning is in fact not belong to Machine Learning community by itself.
From my point of view Machine Learning is more like a collection of various methods from another fields.
From point of view of Statistical Learning the three main questions for regression and classification are:
What is function family from which you pull approximator
What is a criteria how you pull a function
What is a method to find the best function
To operate in some constructive way on (1) - it's not so obvious how use math optimizaion can help
To operate in some constructive way on (2) - it's obvious that objective is the goal. And math optimizaion can help on it.
To operate in some constructive way on (3) - you need math optimization.
There several parts of math optimization:
Also there are ways how to deal "stochasticity" in some way, even "Nobody know how to solve stochastic convex optimization"
NonConvex Optimization - usually people by this mean something which is continious objective, but curvature can vary. People in this planet don't know how to solve it precisely. And in fact all mehtods make leverage into (1)
Combinatorial optimizaion - it's even more wild then (2), now for parameters that you find you even can not apply minus operator. One example is "regions" in Decision Trees. So there are two way how to deal with it: a) Convexify problem and use methods from (1) b) Make brute force. Doesn't work for huge number of parameters. c) Make brute force but with some greedy steps. It's something that CART do.
So at least I think I convice you that:
I) Convex Optimization is central thing for most optimization problems.
II) "01:15 Optimization is in fact more bigger subject then ML or AI, but it's bigger subject in fact." (https://www.youtube.com/watch?v=uF3htLwUHn0&t=992s)
