I know that every model has assumptions and work best when those assumptions hold. So, theoretically there are problem domains where linear regression works best. My question is rather this: can you give a real life problem domain where linear regression is known to perform (has higher accuracy in prediction) better than more sophisticated methods like neural networks, support vector machines, or random forests.
Say that you are predicting sells of some products given their price and some other variables. Your data is noisy, since you have many different products and there are many factors that you are not able to account. You may assume that there is some kind of effect that may be approximated with linear function (better price leads to better sells). You need this model for doing future predictions of different products, sold in different part of the year, so possibly by different clients etc., so basically lot's of things may change.
Surely you could use many different methods for approaching this problem, but in many cases linear regression would be something that you would start and end with. There are many reasons for this, e.g.
- more complicated models would possibly overfit, simple model would be more robust, this is important if you care about out-of-sample errors (and you care),
- by design it would give you the results that are "on average" correct, if you have products that sell very well and very bad, then it possibly wouldn't give you the exactly correct results for them, but "in total" it should give you the balanced solution,
- regression will work out-of-the-box for many cases,
- it will work even for larger sets of the data and it would be fast,
- it is easily interpretable, so it would be easy to explain to the management what is your model and how did it predict what it did, this is something that you cannot say about many of the machine learning models, etc.
Finally, there is many different measures of "accuracy" for predictions, different models would aim at minimizing different loss functions. Regression minimizes the squared errors, but it may be so that you need to minimize something different and then it obviously would be sub-optimal.
If the underlying process is linear, linear regression will be superior. What comes to mind is estimating calories on food; the amount of calories will be strictly linear (i.e., they won't interact with each other). A neural network (etc) could be trained to do this but it would be more computationally demanding and prone to overfitting.
That said, I think that people will choose linear regression over machine learning less because of the problem domain and more because of their own goals. If you want an accurate prediction, then a well-trained model will be superior most of the time and linear regression wouldn't be considered seriously. In other cases, predictive accuracy isn't really interesting: social scientists will frequently publish regression models with accuracy (that is to say, $R^2$) well below .1, because the interest is in the contribution of individual factors to the outcome rather than predicting the value of Y given X. A neural network, rather, doesn't really care about the marginal effect of changing an input, it just runs the model again and returns its prediction.
Here's a real life example: a do it yourself (DIY) spring scale from Scientific American, where they explain how to make a spring scale, and how the underlying model is linear (within the elastic limit):
the relationship between weight and distance is linear—if you double the weight, the amount the spring stretches will double.
Furthermore, they explain how to estimate a spring constant:
Using Hooke's law and your data, can you calculate the spring constant, k, for your spring? Hint: k is the slope of a graph of force versus distance.
This is a basic setup for univariate OLS with length on y-axis and weight on x-axis. It'll beat any machine learning on every level. Additionally, you have an experimental design where the design matrix X is truly fixed, and there's undoubtly a causal relationship present. These are the ideal conditions for OLS.
When I want to know the magnitude of an independent variable's relationship with my dependent variable, I would want to use a regression.
This is one of the reasons that logistic regressions are still valuable statistical tools with neural networks. Though, if all you care about is being able to make a prediction, a neural network will outperform a traditional logistic regression model. Now, if you can make sense of all the weights and hidden layers and use that to tell me how independent variable X relates to dependent variable Y, my hat's off to you because that is no easy feat (untangling a neural net).
When we want to talk about relationships rather than just making predictions, regressions are usually the way to go. In other words, it depends on what you are asking.
If I wanted to predict whether or not a given student will pass a course in college, I would use a neural network model.
If I wanted to know the relationship between passing a course and study time, I would use a regression model.