Using OLS for Model Selection and Prediction - Heteroscedasticity Issue

Question

I am new to regression and having problem in solving Heteroscedasticity in OLS. Have done lots of homework and test before seeking your advice. Sharing the background and what I have done to solve the problem. Hope you can share your thoughts if my approach was correct.

Objectives:

1. To find the relationship (model) between an explanatory variable (x) and an explained variable (y) using OLS regression.
2. if a model (relationship) is found, its usefulness and accuracy of prediction will be studied.

Dataset (Cross-sectional):

1. Have 4 datasets, with each 350 sample size.
2. Each dataset obtained using different intensity of experiment and this is already captured by the explanatory variable in x.
3. Due to the heterogenity of data, not possible to lump all into a single dataset.

Requirement:

One common and statistically acceptable model for all the 4 datasets using OLS

Steps Followed:

1. Explanatory Analysis: Found Non-linear relationship

2. As intending to use OLS, did 3 transformations of variables in attempt to have linearity: a) ln(x) ~ ln(y); b) ln(x) ~ y; c) x ~ ln(y). Note: Kept d) x ~ y as benchmark

3. Did heteroscedasticity test using Breusch-Pagan (BP) test in R for 2(a)-(d) for all the datasets in attempt to find valid model(s). On the best case i.e 2b), only 2 out of 4 datasets passed the BP test (p-value>0.05)

4. As the aim is to have one common model for all the 4 datasets, another variable transformation is done using Tukey's Ladder of Transformation in attempt to have homoscedasticity: a) ? ? {-2,-1,-0.5, 0.5, 1, 2} is used for x/y/x and y for each of the models in 2(a)-(d). Have total of 64 models (16 x 4) to consider. X and Y refer to the transformed x and y; b) Now have 2 models passed BP test for 3 out of 4 datasets in the best case; c) The one that failed has p-value <2.20E-16.

5. [deadlock unable to find one valid model that passes all the 4 datasets]

6. Proceeded to take the two valid models in Step 4 and done inference Test: a) the p-values for t-test and F-test are below 0.05 for all the 4 datasets; b) R-square are above 0.9402 for all the 4 datasets.

7. Did cross validation and selected the best model using the smallest mean square error against the two "valid" models. Did back transformation on the original scale first before the selection is done so that its apple to apple data comparison. The mean average percentage error for the best model is below 10%

8. Now tried to use the best model for prediction: a) Selected 20 random x values which were not part of the dataset; b) Predicted y and compared it against Measured y; c) the mean average percentage error is below 8% and within the model's mean average percentage error i.e below 10%.

The problem:

With the steps above I am unable to get a model that passes the heteroscedasticity test all the 4 datasets. Have I done anything incorrectly or is there anything more can be done in Step 4?

Believe mis-specification issue has duly been attended. Not intending to use GLS as I need to use .OLS

I have used heteroscedasticity robust standard errors as a remedy of heteroscedasticity on the one dataset that failed BP test per the Youtube below. Refer - https://www.youtube.com/watch?v=hFoDDwTF4KY

The standard error increased and t-value decreased for Y for the HC3 corrected dataset. But the Y= a + b X model remain the same.

Is it sufficient to show the p-value for t-test and F-test for the corrected dataset are still below 0.05 hence its ok to use the same Y= a+bX though it failed the BP test earlier?

Hope you can share your thoughts as I am new to regression.

Using many reference books to learn such as

1. Introduction to Econometrics by Wooldridge
2. Basic Econometrics by Gujerati
3. Regression Analysis by Example by Chatterjee

Original:

Coefficients:
                      Estimate Std. Error t value Pr(>|t|)    
(Intercept)          -0.612116   0.009006  -68.76   <2e-16 ***
Y                     5.955984   0.039653  145.65   <2e-16 ***
---

Residual standard error: 0.04138 on 348 degrees of freedom
Multiple R-squared:  0.9832,    Adjusted R-squared:  0.9831 
F-statistic: 2.092e+04 on 1 and 348 DF,  p-value: < 2.2e-16

Heteroskedasticity Robust Standard Errors corrected using HC3:

Coefficients:
                     Estimate Std. Error t value Pr(>|t|)    
(Intercept)          -0.61212    0.01767  -33.77   <2e-16 ***
Y                     5.95598    0.08432   69.12   <2e-16 ***
---

Residual standard error: 0.04138 on 348 degrees of freedom
Multiple R-squared:  0.9832,    Adjusted R-squared:  0.9831 
F-statistic:  4640 on 1 and 348 DF,  p-value: < 2.2e-16

Note: Heteroscedasticity-consistent standard errors using adjustment hc3 

Thanks

Answer 1

It would help to know what your final goal is, and specifically whether you want a good predictive model (eg. capable of outputting good $\hat y$ on data other than your 350 observations) or good parameter estimates? From your econometric vocabulary it would seem that you want parameter estimates, but your search for a specification in step 2 would indicate the contrary. (But of course these two questions are linked by the difficult problem of correct specification.)

• if you want good parameter estimates (and you are confident in your functional form) the good news is that heteroskedasticity doesn't affect the convergence of OLS, just its efficiency. In this case your last table (Heteroskedasticity Robust Standard Errors corrected) is the way to go.

• if you want a good predictive model, and have no a priori information about the form of the function linking $x$ to $y$, then you don't really want OLS but rather a more explorative method (lasso, splines, gradient boosting...)