# Using OLS for Model Selection and Prediction - Heteroscedasticity Issue

I am new to regression and having problem in solving Heteroscedasticity in OLS. Have done lots of homework and test before seeking your advice. Sharing the background and what I have done to solve the problem. Hope you can share your thoughts if my approach was correct.

Objectives:

1. To find the relationship (model) between an explanatory variable (x) and an explained variable (y) using OLS regression.
2. if a model (relationship) is found, its usefulness and accuracy of prediction will be studied.

Dataset (Cross-sectional):

1. Have 4 datasets, with each 350 sample size.
2. Each dataset obtained using different intensity of experiment and this is already captured by the explanatory variable in x.
3. Due to the heterogenity of data, not possible to lump all into a single dataset.

Requirement:

One common and statistically acceptable model for all the 4 datasets using OLS

Steps Followed:

1. Explanatory Analysis: Found Non-linear relationship

2. As intending to use OLS, did 3 transformations of variables in attempt to have linearity: a) ln(x) ~ ln(y); b) ln(x) ~ y; c) x ~ ln(y). Note: Kept d) x ~ y as benchmark

3. Did heteroscedasticity test using Breusch-Pagan (BP) test in R for 2(a)-(d) for all the datasets in attempt to find valid model(s). On the best case i.e 2b), only 2 out of 4 datasets passed the BP test (p-value>0.05)

4. As the aim is to have one common model for all the 4 datasets, another variable transformation is done using Tukey's Ladder of Transformation in attempt to have homoscedasticity: a) ? ? {-2,-1,-0.5, 0.5, 1, 2} is used for x/y/x and y for each of the models in 2(a)-(d). Have total of 64 models (16 x 4) to consider. X and Y refer to the transformed x and y; b) Now have 2 models passed BP test for 3 out of 4 datasets in the best case; c) The one that failed has p-value <2.20E-16.

5. [deadlock unable to find one valid model that passes all the 4 datasets]

6. Proceeded to take the two valid models in Step 4 and done inference Test: a) the p-values for t-test and F-test are below 0.05 for all the 4 datasets; b) R-square are above 0.9402 for all the 4 datasets.

7. Did cross validation and selected the best model using the smallest mean square error against the two "valid" models. Did back transformation on the original scale first before the selection is done so that its apple to apple data comparison. The mean average percentage error for the best model is below 10%

8. Now tried to use the best model for prediction: a) Selected 20 random x values which were not part of the dataset; b) Predicted y and compared it against Measured y; c) the mean average percentage error is below 8% and within the model's mean average percentage error i.e below 10%.

The problem:

With the steps above I am unable to get a model that passes the heteroscedasticity test all the 4 datasets. Have I done anything incorrectly or is there anything more can be done in Step 4?

Believe mis-specification issue has duly been attended. Not intending to use GLS as I need to use .OLS

I have used heteroscedasticity robust standard errors as a remedy of heteroscedasticity on the one dataset that failed BP test per the Youtube below. Refer - https://www.youtube.com/watch?v=hFoDDwTF4KY

The standard error increased and t-value decreased for Y for the HC3 corrected dataset. But the Y= a + b X model remain the same.

Is it sufficient to show the p-value for t-test and F-test for the corrected dataset are still below 0.05 hence its ok to use the same Y= a+bX though it failed the BP test earlier?

Hope you can share your thoughts as I am new to regression.

Using many reference books to learn such as

1. Introduction to Econometrics by Wooldridge
2. Basic Econometrics by Gujerati
3. Regression Analysis by Example by Chatterjee

Original:

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)          -0.612116   0.009006  -68.76   <2e-16 ***
Y                     5.955984   0.039653  145.65   <2e-16 ***
---

Residual standard error: 0.04138 on 348 degrees of freedom
Multiple R-squared:  0.9832,    Adjusted R-squared:  0.9831
F-statistic: 2.092e+04 on 1 and 348 DF,  p-value: < 2.2e-16


Heteroskedasticity Robust Standard Errors corrected using HC3:

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)          -0.61212    0.01767  -33.77   <2e-16 ***
Y                     5.95598    0.08432   69.12   <2e-16 ***
---

Residual standard error: 0.04138 on 348 degrees of freedom
Multiple R-squared:  0.9832,    Adjusted R-squared:  0.9831
F-statistic:  4640 on 1 and 348 DF,  p-value: < 2.2e-16

Note: Heteroscedasticity-consistent standard errors using adjustment hc3


Thanks

• Is the data a time series. If so I have done extensive work in detecting variance changes leading to GLS. Your econometric references suggest that it is time series data. Please post one of your data sets and I will try to illustrate the art of the possible. – IrishStat Jul 26 '14 at 16:11
• Wait, there's something I'm not quite understanding: If you're dealing with non-homogeneous data sets, then why are you trying to find a single model for all four of them? – Steve S Jul 27 '14 at 1:08
• Also, I'd be concerned about that R^2 value if I were you because greater than 0.94 is extremely high. Definitely go back and make sure that you haven't misspecified the model somewhere. Lastly, consider jubo's second bullet point (it may help a lot since right now your approach seems pretty ad hoc)... – Steve S Jul 27 '14 at 1:19
• Why do you want to discard GLS? And what is the purpose of your analysis. It is clear from your question that you've already done a lot, but it is still not clear to me, why are you doing this. – mpiktas Jan 21 '15 at 7:16

It would help to know what your final goal is, and specifically whether you want a good predictive model (eg. capable of outputting good $\hat y$ on data other than your 350 observations) or good parameter estimates? From your econometric vocabulary it would seem that you want parameter estimates, but your search for a specification in step 2 would indicate the contrary. (But of course these two questions are linked by the difficult problem of correct specification.)
• if you want a good predictive model, and have no a priori information about the form of the function linking $x$ to $y$, then you don't really want OLS but rather a more explorative method (lasso, splines, gradient boosting...)