0
$\begingroup$

After making some variable transformations, I chose this model because I works well with the assumptions and gives the largest R-Squared and the smallest Overall MS in cross-validation between a set of linear models.

> gn<- lm(NA.~ I(PC^0.25) + I(((PI)^2)),data=DSET)
> summary(gn)

Call:
lm(formula = NA. ~ I(PC^0.25) + I(((PI)^2)), data = DSET)

Residuals:
    Min      1Q  Median      3Q     Max 
-425.22  -87.46   -2.30   79.11  396.14 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -2.047e+03  1.094e+02  -18.71  < 2e-16 ***
I(PC^0.25)   1.206e+03  4.231e+01   28.52  < 2e-16 ***
I(((PI)^2)) -5.242e-02  1.233e-02   -4.25 2.81e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 127.2 on 319 degrees of freedom
Multiple R-squared:  0.7475,    Adjusted R-squared:  0.746 
F-statistic: 472.3 on 2 and 319 DF,  p-value: < 2.2e-16

The output of the cross-validation is the following:

> a<-CVlm(df=DSET,form.lm = gn ,m=5)
Analysis of Variance Table

Response: NA.
             Df   Sum Sq  Mean Sq F value  Pr(>F)    
I(PC^0.25)    1 14979750 14979750   926.5 < 2e-16 ***
I(((PI)^2))   1   292051   292051    18.1 2.8e-05 ***
Residuals   319  5157489    16168                    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1



fold 1 
Observations in test set: 64 


Sum of squares = 1119289    Mean square = 17489    n = 64 

fold 2 
Observations in test set: 65 

Sum of squares = 1072687    Mean square = 16503    n = 65 

fold 3 
Observations in test set: 65 

Sum of squares = 865598    Mean square = 13317    n = 65 

fold 4 
Observations in test set: 64 


Sum of squares = 1178545    Mean square = 18415    n = 64 

fold 5 
Observations in test set: 64 

Sum of squares = 1042335    Mean square = 16286    n = 64 

Overall (Sum over all 64 folds) 
   ms 
16393 


> 

So, my question is, Which steps do you recommend to me after the cross-validation? Should I choose the training set applied to the fold 3(it gives the small mean square error)? Is there a way to improve the coefficients after the cross-validation?

$\endgroup$
2
$\begingroup$

The output seems to list the cross-validation results from a single model. I assume what you did was produce analogous results for the models you were selecting between and are only displaying the results from the best model. Having decided that this is the best model, the standard practice is then to fit that model to the full dataset outside of cross-validation, as it seems you did at the top of your question. Then you can just report the coefficients and standard errors from this model. Note that these standard errors are optimistic (smaller than they should be) in that they don't take into account the fact that a different sample of data might have lead to you selecting a different "best" model.

$\endgroup$
  • $\begingroup$ I actually used the full data set to generate the model. With respect to new cases, I applied K-cross-validation. So. Is there a way to improve my coefficients after that? $\endgroup$ – CreamStat Jan 28 '14 at 21:00
  • $\begingroup$ What do you mean by "new cases" and what do you mean by "improve your coefficients"? Again the typical use of cross-validation is as follows:apply cross-validation to a number of candidate models, select the best model by, for example, selecting the one that minimizes mean squared error, then refit that model to the full data $\endgroup$ – Jeremy Coyle Jan 28 '14 at 21:12
  • $\begingroup$ I used the full data set to generate the models. With "new cases", I just mean that I use cross-validation with the full data set again. With "improve my coefficients" , I mean refining the coefficients after the cross-validation to get closer to the real f which best describes the truth. $\endgroup$ – CreamStat Jan 28 '14 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.