7
$\begingroup$

tl;dr: linear model is better than ANN and decision tree in timeseries regression task, why is that?

I have a time series dataset of 151 observations each with 43 macroeconomic variables. Some of the variables are percentage change from year over year (e.g. GDP change), and some of them are stationary (I hope I use that word correctly e.g. employment rate). The frequency of data is quarterly. Variable I want to predict is CPI inflation and it is lagged appropriately t-1. I divide the data set to training and testing in many different proportions.

My problem is that multiple linear regression performs better ( as of MSE and R squared) than machine learning techniques like: artificial neural networks, decision trees with and without extreme gradient boosting. They of course both performed well but linear regression is always better MSE 0.2 vs 0.8 and r squared 87% and 82%. In the case of ANN I tried training them in almost every possible way, I guess. the most confusing thing is that ANN according to Universal approximation theorem should performed at least as good as linear regression. Maybe the linear regression I make is misleading as the following warning is being shown "prediction from a rank-deficient fit may be misleading". Also when I compare logistic regression and these ML techniques in classification task, they performed better

Anyone have experienced such an outcome? Is it normal? and if not how can I repair that?

I use R language and package neuralnets and xgBoost.

Improve this question
$\endgroup$
5
  • 13
    $\begingroup$ It is normal for linear regression to outperform other methods when the problem you have is well suited for that type of model. Universal approximation theorem doesn't mean that ANN will outperform any method on all problems. It means that ANN is flexible enough (given enough parameters) to approximate functions of arbitrary shape. If the dependencies in your data are truly linear then linear regression should outperform more complex models. Especially so on lower sample sizes. But I wouldn't ignore the warning you got, it basically says that you have more features than samples. $\endgroup$
    – Karolis Koncevičius
    Sep 5, 2018 at 17:24
  • $\begingroup$ Thanks for answer! it is rather frustrating but anyway i had to accept it haha. Althought do you think that even if my ANN will have single node and one hidden layer it could performe worse than linear regression? $\endgroup$
    – SquintRook
    Sep 5, 2018 at 17:33
  • 3
    $\begingroup$ Your dataset is tiny. Typically advanced ML techniques start to outperform simpler techniques when you have a lot more data. $\endgroup$
    – kbrose
    Sep 5, 2018 at 18:26
  • 2
    $\begingroup$ @KarolisKoncevičius - why not expand on that a little and convert it to an answer? $\endgroup$
    – jbowman
    Sep 5, 2018 at 21:27
  • 1
    $\begingroup$ You will need a quite smart validation strategy to compare methods. $\endgroup$
    – Michael M
    Sep 2, 2022 at 10:16

1 Answer 1

Reset to default
5
$\begingroup$

Assumptions give you power - when they are valid.

When the assumptions of a linear regression (or any other simple model) are fulfilled, they will outperform more complex and flexible models. Let's consider a model with a single predictor. If the relationship is linear, a linear regression will quite likely outperform a neural network. More complex/flexible models can be beneficial when important assumptions are violated.

But in order to gain the benefits of the flexible models, you need a lot more data in order to learn the complex pattern, or that flexibility can degrade performance relative to a simple model.

So in your case, it seems likely that this is because you have a small dataset, or because the relationships you are modelling are linear, or both.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ This is a low signal:noise ratio situation with too many candidate predictors for any method to work well. It requires 70 observations just to adequately estimate the residual variance by itself. To treat the variables as separate entities as being done here may require more than several hundred observations. The linear model is capitalizing on an additivity assumption the ML is not able to make, which reduces the number of parameters by a large factor. That likely explains most of the better performance from the statistical model. $\endgroup$
    – Frank Harrell
    Sep 2, 2022 at 11:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.