# Improving a linear regression: Add predictors or change model?

I am trying to model a time series variable $Y_{t}$ with $4$ physical predictor variables. I used the following linear regression:

$Y=\beta_{0}+\beta_{1}f_{1}(X_{1})+\beta_{2}f_{2}(X_{2})+\beta_{3}f_{3}(X_{3})+\beta_{4}f_{4}(X_{4})+\epsilon$ where $f_{i} \in \{\log, \tan, \sin, \cos,1/.,Id\}$.

My best model gives me an adjusted determination coefficient $R^2_\rm{adjusted}$ of $0.87$. But I know that this indicator is not completely reliable. So I would like to know if I can improve my model.

If yes, how can I know if I have to:

1. Change the model of regression: add some polynomials or other depedencies?
2. Look for more predictors $X_{i}$ that I may have not discovered yet?

Any ideas would be nice!

Edit: My true purpose

$Y$ is a value which is measured each day, not at the same hour. What I really want to do is studying the variation of $Y$ with time and "cleaned" from the other external variables $X_{i}$ which are "blurring" my data.

So thanks to my regression I want to find the part of $Y$ which depends on this external varying factors, and then remove this part. At the end, if my estimation $\hat{Y}$ is correct, $Y-\hat{Y}$ will correspond to my data "unblurred", and I will be free to sudy a "normalized" or "cleaned" version of $Y$ with time.

I tested my model for another set of data of $Y$ measured a year ago, and it also fits the data with a $R^2_{adjusted}$ around $0.8$. And I can see graphically that the model is not completely insane, but not perfect either.

-
What are the variables? Also, usually Y is the dependent variable and X the independent variable (there's no rule that it has to be so, but it's pretty common) – Peter Flom Mar 28 '14 at 10:28
Yes you are right. Edit is done. Most of the variables are physical parameters such as: temperature, pressure, speed... – Gandhi91 Mar 28 '14 at 10:34
To statistical people temperature, pressure, speed are variables, not parameters. – Nick Cox Mar 28 '14 at 11:20
If there is a dependence structure in time, it is usually a good idea to think about modelling it explicitly. Time series people are defined by their thinking there is no other way to do it. – Nick Cox Mar 28 '14 at 12:35
The R-squared measure indicates "fit", i.e. how well the regressors "explain" the available data set, namely, the past. With time series data, in many cases we are interested in the future, in acquiring a model that would forecast adequately future values of the dependent variable. Fit and forecasting usually don't improve together, and one should attempt to strike some sort of balance. What is your target here? – Alecos Papadopoulos Mar 28 '14 at 13:47

In my field (social science using cross sectional surveys), an adjusted R squared of .87 would be much too large. That would be a sure sign that you have done somehting meaningless like predict something with a second measure of itself. So whether or not you need to improve your model depends on the context, which you did not give us.

If you are looking for alternative transformations of your explanatory/right-hand-side/predictor-variables you could consider fractional polynomials:

Royston P, Altman DG. (1994): Regression using fractional polynomials of continuous covariates: parsimonious parametric modelling (with discussion). Applied Statistics, 43:429-467.

Royston P, Ambler G, Sauerbrei W. (1999): The use of fractional polynomials to model continuous risk variables in epidemiology. International Journal of Epidemiology, 28:964-974.

Royston P, Sauerbrei W. (2004): A new approach to modelling interactions between treatment and continuous covariates in clinical trials by using fractional polynomials. Statistics in Medicine, 23:2509-2525.

Royston P, Sauerbrei W. (2007): Improving the robustness of fractional polynomial models by preliminary covariate transformation: a pragmatic approach. Computational Statistics and Data Analysis, 51:4240-4253.

Royston P, Sauerbrei W (2008): Multivariable Model-Building - A pragmatic approach to regression analysis based on fractional polynomials for continuous variables. Wiley.

Sauerbrei W. (1999): The use of resampling methods to simplify regression models in medical statistics. Applied Statistics, 48, 313-329.

Sauerbrei W, Meier-Hirmer C, Benner A, Royston P. (2006): Multivariable regression model building by using fractional polynomials: Description of SAS, STATA and R programs. Computational Statistics & Data Analysis, 50:3464-3485.

Sauerbrei W, Royston P. (1999): Building multivariable prognostic and diagnostic models: transformation of the predictors by using fractional polynomials. Journal of the Royal Statistical Society A, 162:71-94.

Sauerbrei W, Royston P, Binder H (2007): Selection of important variables and determination of functional form for continuous predictors in multivariable model building. Statistics in Medicine, 26:5512-28.

-
I see what you mean, but why is it too high for you. It could be true that your model is fine, no? What I did is estimating a model on a first serie of data $Y_{i}$ and I applied this model to another serie of data $Z_{i}$(representing the same physical variable). Then I obtain this value of $R^2$. So I believe, I'm not in the case you described. – Gandhi91 Mar 28 '14 at 14:59
One of the strongest cross-sectional relationship in sociology is between the education of husband and wife, and this correlation is in the order of .60, so the $R^2$ of this bivariate relationship is about .36. I would consider any other relationship that has a realtionship that is as strong as that as very suspicious. We just don't know a lot about humans. As a researcher I find that frustrating, as a human I find that reasuring. Things are different with time-series models, which seems to be what you are dealing with. – Maarten Buis Mar 28 '14 at 16:13

It's very hard to tell. If your criterion is $R^2$ (even adjusted) it is usually possible to "improve" it in the ways you mention. Indeed a completely different non-linear specification may be better. But the risk is that your extra terms will not be important and/or just complicate the model. If you are already using a mix of functions, that could be a great idea or just bizarre adhockery.

How you can try to improve it is a matter of scientific as well as statistical reasoning. There should be a scientific story for each change of model, but you can learn a lot by looking at the residuals, e.g. on judging whether a quadratic term is needed as well as a linear. (For "scientific" read "economic" or something else if you don't think you are doing science too.)

There is also an art and a craft to modelling. Trying very hard to catch all the structure in a particular set of data can result in a model specification that works very badly with new data. This is part of what is warned about under headings like "overfitting".

$R^2$ may sound a universal criterion, but what it means depends very much on how much variation there is that for practical reasons if no other will remain inexplicable. There are fields where $0.1$ is a tremendous achievement and fields where anything less than $0.99$ is abject failure.

-
$\cdot$ I'm interested on what you said about residus? How could I learn something from them. I have already ploted them and indeed seen strange stuff such as a mean different from 0. How can I study residus in general? $\newline$ $\cdot$I see what you mean by "overfitting": The model that gaves me $R_{adjusted}^2=0.87$ was actually built from another set of data for which it gave $R_{adjusted}^2=0,95$. – Gandhi91 Mar 29 '14 at 14:25
Look at any decent regression text to get ideas on how use evidence from residuals. – Nick Cox Mar 29 '14 at 15:52
I read some texts about it and realised that I could indeed see if some informations about $Y$ would remain in the residus. They seem to be close to a gaussian, but not centered around $0$ and I don't knows how to interpret this strange results, since by construction of the regression there should be $E(\epsilon_{i})=0$. – Gandhi91 Apr 3 '14 at 18:07

I think instead of totally relying on R^2, you should try cross validating techniques.. there are chances that the model is fine and you are trying to improve it further which may not be possible.

-
• What you could also do is test your model for excess parameters (I don't think you need both sine and cosine functions). You could do an SVD of the structure matrix you use for fitting and make sure none of the singular values are zero (or too small).
• You could also improve your model by cleaning your data sample with filters prior to fitting (Wiener filter for example is pretty good against white noise). If you want to use filters you will have to make a discrete Fourier transformation of the data, apply Filter in frequency space and then make an inverse discrete Fourier transformation.
• If your data is periodic (or if the disturbances are periodic) you could perform autocorrelation of your data sample, and any periodicity will be enhanced and the rest will be suppressed. Then you can check the power spectrum of the autocorrelation, to see which frequencies you have which you can then in turn either keep, or cut out with a filter.
• You could also use a non-linear regression model and try to find a different minimum. It could be, that the $R^2$ minimum you found is only local, whereas you want to have a global one. If the $R^2$ doesn't change too wildly (is smooth enough), you could try to fit your data using a simplex amoeba method.
• If you insist on using a linear model, why not $\chi^2$?
-
Sine and cosine are often (usually) both needed to give phase as well. The point is that $\sin(\theta + \phi)$ is in practice much more easily fitted using $\sin \theta$ and $\cos \theta$; the coefficients of those terms give estimates of $\phi$. Otherwise it's a nonlinear problem. – Nick Cox Mar 28 '14 at 18:32
Yes, you're absolutely right. I was solving some problems recently, where the phase was zero, so I thought maybe in this case phase could be deduced as well, without the necessity of fitting it. However, if that is not the case, one must use both sine and cosine, just as you suggested. – jozze Mar 28 '14 at 18:39
$\cdot$ My data is not exactly a signal subject to noise like in signal processing. I don't understand you idea of applying a filter. There is no "noise", the strong variations of my data is due to the strong variations of my predictors i.e my external variables which influence completely my data. $\cdot$ What do you mean by $\chi^2$? – Gandhi91 Mar 29 '14 at 14:50
Sorry, when you said your data was blurred, I thought that you were counting the other external variables as noise. I'm sorry for jumping to conclusions too soon, since I don't know the exact nature of your problem. By $\chi^2$ I meant the $\chi^2$ test which you can use to determine the "goodness of fit". But definitely, try to do an SVD of the structure matrix. Small singular values mean there are excess parameters in the model, which in turn will give bad results to the fit. – jozze Mar 29 '14 at 17:37
@jozze What do you mean by "structure matrix"? I have done a principal component analysis for the matrix $[X_{1} X_{2} X_{3} X_{4}]$ and a canonical correlation analysis. Are you talking about one of those? – Gandhi91 Apr 3 '14 at 12:56