2
$\begingroup$

tl;dr: linear model is better than ANN and decision tree in timeseries regression task, why is that?

I have a time series dataset of 151 observations each with 43 macroeconomic variables. Some of the variables are percentage change from year over year (e.g. gdp change), and some of them are stationary (i hope i use that word correctly e.g. employement rate). The frequency of data is quarterly. Variable i want to predict is CPI inflation and it is lagged appropriately t-1. i divide the data set to training and testing in many diffrent proportions.

My problem is that mutlivariate linear regression performes better ( as of MSE and R squared) than machine learning techinques like: artificial neural networks, decision trees with and without extreme gradient boosting. They of course both performe well but linear regression is always better MSE 0.2 vs 0.8 and r squared 87% and 82%. In the case of ANN i tried training them in almost every possible way, i guess. the most confusing thing is that ANN according to Universal approximation theorem should performe at least as good as linear regression. Maybe the linear regression i make is misleading as the following warning is being shown"prediction from a rank-deficient fit may be misleading". Also when i compare logistic regression and these ML techniques in classification task, they performe better

Anyone have experienced such an outcome? Is it normal? and if not how can i repair that?

I use R language and package neuralnets and xgBoost. If anything is unclear i can provide aditional informations. thanks in advance!

|cite|improve this question
$\endgroup$
  • 9
    $\begingroup$ It is normal for linear regression to outperform other methods when the problem you have is well suited for that type of model. Universal approximation theorem doesn't mean that ANN will outperform any method on all problems. It means that ANN is flexible enough (given enough parameters) to approximate functions of arbitrary shape. If the dependencies in your data are truly linear then linear regression should outperform more complex models. Especially so on lower sample sizes. But I wouldn't ignore the warning you got, it basically says that you have more features than samples. $\endgroup$ – Karolis Koncevičius Sep 5 '18 at 17:24
  • $\begingroup$ Thanks for answer! it is rather frustrating but anyway i had to accept it haha. Althought do you think that even if my ANN will have single node and one hidden layer it could performe worse than linear regression? $\endgroup$ – SquintRook Sep 5 '18 at 17:33
  • 2
    $\begingroup$ Your dataset is tiny. Typically advanced ML techniques start to outperform simpler techniques when you have a lot more data. $\endgroup$ – kbrose Sep 5 '18 at 18:26
  • 2
    $\begingroup$ @KarolisKoncevičius - why not expand on that a little and convert it to an answer? $\endgroup$ – jbowman Sep 5 '18 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.