0
$\begingroup$

I generated 3000 observations (3000) and carrying out multiple linear regression. Prior to regression i randomized my observations five times and then selected 30 % of the observations for testing and 70 % for training.

I discovered that when i choose a different set of 30% of my observations i get slightly different standardized regression coefficients and sometimes different sign (positive or negative). Then i decided to do 5-fold cross validation. But when i do this cross validation, i get five different sets of standardized regression coefficients. I want to know the best way to decide which standardized regression coefficient i should use for each independent variable.

Is it good take the geometric mean of the coefficients? If so, how do i also decide on which sign to choose for the final coefficient especially when i get both positive and negative signs during the cross validation.

On a last note, is it better to standardize the independent variables before multiple linear regression or do i regress with the unstandardized independent variables but take the standardized regression coefficient given by SigmaPlot?

The goal of my analysis is to factor the influence of each independent variable (predictor) on the dependent variable (response).

Hope to get your favorable response.

Kind regards

Note: In my regression i used the natural logarithm of the independent variables.

$\endgroup$
  • 2
    $\begingroup$ Please explain what you mean by "data set": could you perhaps mean the same thing as observation or data point? (If not, then this question is practically incomprehensible.) In other words, you have a single data set with 3000 observations and 5+1=6 variables, comprising 18,000 numbers in toto. If you had 3000 data sets then you would need to perform 3000 separate regressions... . $\endgroup$ – whuber Apr 9 '17 at 20:33
  • $\begingroup$ What are you trying to achieve by using train/test splits on your data? I think it's fairly rare to see people use such methods on ordinary least squares regression because there aren't really any tunable parameters that you need to validate. Are you using it as part of variable selection? $\endgroup$ – Eumenedies Apr 9 '17 at 20:33
  • $\begingroup$ Hi @whuber, I have edited my question for more clarity. $\endgroup$ – chikezie nwaoha Apr 9 '17 at 22:24
  • $\begingroup$ Hi @Eumenedies, The goal of my analysis is to conduct a parametric analysis and to factor the influence of each independent variable (predictor) on the dependent variable (response). I also believe that the coefficient sign (negative and positive) can reveal the relationship between the predictor and response. Hope to get your comment again. $\endgroup$ – chikezie nwaoha Apr 9 '17 at 22:27
0
$\begingroup$

When you are fitting a model, it is good practice to perform some kind of model validation. In your case, you are trying holdout validation and cross validation. Note, however, that these methods are for validation. The idea is that, by making a prediction on some data that you have not used to train the model, you obtain an estimate of how well your model will perform on new data.

In the case of holdout validation, the model would be trained on your 70% of the data and tested on the 30% holdout. The goodness of fit metric you obtain (probably RMSE) on the 30% tells you how good the model will perform on unseen data.

In the case of cross validation, you get a much better generalization estimate because it both trains and tests on every point. If you do 5-fold cross validation then you will have 5 different estimates of the goodness of fit, i.e. 5 different RMSE values. Averaging these values gives you a good idea of the goodness of fit overall.

For either method, though, you want to fit your final model with all of the data.

The purpose of cross validation is to check the model, not to build it. The actual model building is a separate step that uses all of the data to provide the best possible estimate.

$\endgroup$
  • $\begingroup$ Hi @Eumenedies, many many thanks for your detailed response. It was very helpful. $\endgroup$ – chikezie nwaoha Apr 10 '17 at 16:52
  • $\begingroup$ Hi @Eumenedies, What i do the cross validation and some of the coefficient signs for a variable changes? For instance, an independent variable gets two negative signs and three positive signs after the five-fold cross validation. What should i do? $\endgroup$ – chikezie nwaoha Apr 10 '17 at 17:58
  • $\begingroup$ I am assuming here that the coefficients that change sign are pretty small - if they are large then your model is likely not very good. You would expect each model in cross validation to be different. If they weren't different then there would be no point in doing the test. In the cross validation step, however, you "only" need to look at how well the model fits the testing set. It's okay if small coefficients change sign because they won't have a very large effect on the error. When you fit the model on all of your data then you will have one, single model with one set of coefficients. $\endgroup$ – Eumenedies Apr 10 '17 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.