I am currently in a linear regression class, but I can't shake the feeling that what I am learning is no longer relevant in either modern statistics or machine learning. Why is so much time spent on doing inference on simple or multiple linear regression when so many interesting datasets these days frequently violate many of the unrealistic assumptions of linear regression? Why not instead teach inference on more flexible, modern tools like regression using support vector machines or Gaussian process? Though more complicated than finding a hyperplane in a space, wouldn't this give students a much better background for which to tackle modern day problems?
It is true that the assumptions of linear regression aren't realistic. However, this is true of all statistical models. "All models are wrong, but some are useful."
I guess you're under the impression that there's no reason to use linear regression when you could use a more complex model. This isn't true, because in general, more complex models are more vulnerable to overfitting, and they use more computational resources, which are important if, e.g., you're trying to do statistics on an embedded processor or a web server. Simpler models are also easier to understand and interpret; by contrast, complex machine-learning models such as neural networks tend to end up as black boxes, more or less.
Even if linear regression someday becomes no longer practically useful (which seems extremely unlikely in the foreseeable future), it will still be theoretically important, because more complex models tend to build on linear regression as a foundation. For example, in order to understand a regularized mixed-effects logistic regression, you need to understand plain old linear regression first.
This isn't to say that more complex, newer, and shinier models aren't useful or important. Many of them are. But the simpler models are more widely applicable and hence more important, and clearly make sense to present first if you're going to present a variety of models. There are a lot of bad data analyses conducted these days by people who call themselves "data scientists" or something but don't even know the foundational stuff, like what a confidence interval really is. Don't be a statistic!
Linear regression in general is not obsolete. There are still people that are working on research around LASSO-related methods, and how they relate to multiple testing for example - you can google Emmanuel Candes and Malgorzata Bogdan.
If you're asking about OLS algorithm in particular, the answer why they teach this is that method is so simple that it has closed-form solution. Also it's just simpler than ridge regression or the version with lasso/elasticnet. You can build your intuition/proofs on the solution to simple linear regression and then enrich the model with additional constraints.
I don't think regression is old, it might be considered as trivial for some problems that are currently faced by data scientists, but still is the ABC of statistical analysis. How are you supposed to understand if SVM are working correctly if you don't know how the simplest model is working? Using such a simple tool teaches YOU how to look into the data before jumping into crazy complex models and understand deeply which tools can be used in further analysis and which cannot. Once having this conversation with a professor and colleague of mine she told me that her students where great in applying complex models but they could not understand what leverage is or read a simple qq-plot to understand what was wrong with the data. Often in the most simple and readable model stands the beauty.
The short answer is no. For example, if you try linear model with MNIST data, you will still get ~90% of the accuracy!
A long answer would be "depending on the domain", but linear model is widely used.
In certain fields, say, medical study, it is super expensive to get one data point. And the analysis work is still similar to many years ago: linear regression is still plays an very important role.
In morden machine learning, say, text classification, linear model is still very important, although there are other fancier models. This is because linear model is very "stable", it will have less like to over fit the data.
Finally, linear model is really the building blocks for most of the other models. Learning in well will benefit you in the future.
In practical terms, linear regression is useful even if you are also using a more complex model for your work. The key is that linear regression is easy to understand and therefore easy to use to conceptually understand what is happening in more complex models.
I can offer you a practical application example from my real live job as a statistical analyst. If you find yourself out in the wild, unsupervised, with a large dataset, and your boss asks you to run some analysis on it, where do you start? Well, if you are unfamiliar with the dataset and don't have a good idea of how the various features are expected to relate to each other, then a complex model like the ones you suggested is a bad place to start investigating.
Instead, the best place to start is simple old linear regression. Perform a regression analysis, look at coefficients and graph the residuals. Once you start to see what is going on with the data, then you can make some decisions as to what advanced methods you are going to try to apply.
I assert that if you just plugged your data into some advanced model black box like sklearn.svm (if you are into Python), then you will have very low confidence that your results will be meaningful.