I've been wondering, why is it so important to have principled/theoretical machine learning? From a personal perspective as a human, I can understand why principled Machine Learning would be important:
- humans like understanding what they are doing, we find beauty and satisfaction to understanding.
- from a theory point of view, mathematics is fun
- when there are principles that guide design of things, there is less time spent on random guessing, weird trial and error. If we understood, say, how neural nets really worked, maybe we could spend much better time designing them rather than the massive amounts of trial and error that goes into it right now.
- more recently, if the principles are clear and theory is clear too, then there should be (hopefully) more transparency to the system. This is good because if we understand what the system is working, then AI risks that lots of people hype about pretty much goes away immediately.
- principles seem to be a concise way to summarize the important structures the world might have and when to use a tool rather than another.
However, are these reasons strong enough really to justify an intense theoretical study of machine learning? One of the biggest criticism of theory is that because its so hard to do, they usually end up studying some very restricted case or the assumptions that have to be brought essentially make the results useless. I think I heard this once at a talk at MIT by the creator of Tor. That some of the criticism of Tor he has heard is the theoretical argument but essentially, people are never able to prove things about the real scenarios of real life because they are so complicated.
In this new era with so much computing power and data, we can test our models with real data sets and test sets. We can see if things work by using empiricism. If we can get instead achieve AGI or systems that work with engineering and empiricism, is it still worth pursuing principled and theoretical justification for machine learning, especially when the quantitate bounds are so difficult to achieve, but intuitions and qualitative answers are so much easier to achieve with a data driven approach? This approach was not available in classical statistics, which is why I think theory was so important in those times, because mathematics was the only way we could be sure things were correct or that they actually worked the way we thought they did.
I've personally always loved and thought theory and a principled approach was important. But with the power of just being able to try things out with real data and computing power has made me wonder if the high effort (and potentially low rewards) of theoretical pursue is still worth it.
Is theoretical and principled pursue of machine learning really that important?