6
$\begingroup$

I have split my data into two parts. I have used my 80% data to build a multiple regression linear model. Now I want to test it using my rest 20% data. What tools on Minitab do I have to make this checking process?

Edit: I think I can use PRESS statistic. My PRESS for 80% data was 6000 and now based on this model I have calculated the PRESS for 20% data, it is 1000. Now, should I compare 6000 with 1000 or 6000 with 5 times 1000, ie 5000?

$\endgroup$
1
  • 1
    $\begingroup$ Oh. I have one response with more than one predictors. $\endgroup$
    – ilhan
    Jun 9, 2013 at 22:56

2 Answers 2

8
+50
$\begingroup$

Note that the predicted residual sum of squares, PRESS, is got by jack-knifing the sample: there's no sense in calculating it for training & test sets. Calculate it for a model fitted to the whole sample (& compare it to the RSS to assess the amount of over-fitting). For ordinary least-squares regression there's an analytic solution:

$$\sum_i \left(\frac{e_i}{1-h_{ii}}\right)^2$$

where $e_i$ is the $i$th residual & $h_{ii}$ its leverage—from the diagonal of the hat matrix

$$H=X(X^\mathrm{T}X)^{-1}X^\mathrm{T}$$

(where $X$ is the design matrix).

In general cross-validation & bootstrap validation are preferable to splitting a sample into training & test sets: you don't lose precision in the estimates as when fitting on a smaller training set, & the performance measure on the test set will be less variable. How preferable depends on sample size.

$\endgroup$
6
  • $\begingroup$ so you are saying that it is possible to use 100% of the data on a number of models, and then use a bootstrap/cross-validate method to rate the best model using some sort of fit statistic. Is that right? $\endgroup$ Jun 15, 2014 at 14:52
  • $\begingroup$ @EngrStudent: I had in mind the simpler situation of the OP's question, where you fit one model (to the whole sample) & then estimate the predictive performance of the fitting procedure through cross-validation or bootstrap validation. You certainly can evaluate a model selection procedure in the same way, but need to make sure you include the whole selection procedure in each cross-validation fold or bootstrap iteration. So validation(selection + fitting), not selection + validation(fitting). $\endgroup$ Jun 16, 2014 at 8:22
  • $\begingroup$ That sounds great. I have always disliked the separation of a sample into training and validation because I felt that I was giving up quality of fit. Can you point to a paper or tutorial that is a good step-by-step walkthrough of this approach? $\endgroup$ Jun 16, 2014 at 16:19
  • 2
    $\begingroup$ Hastie et al. (2009), The Elements of Statistical Learning, Ch.7 for what it is; Harrell (2001), Regression Modelling Strategies for where it fits in. $\endgroup$ Jun 16, 2014 at 16:29
  • $\begingroup$ Recalculate the hat matrix for each sample .. ? Seems large amount of calculations. $\endgroup$ Dec 23, 2015 at 21:49
7
$\begingroup$

You may use Root Mean Squared Error (RMSE) which is a measurement of accuracy between two set of values.

Use your model of type $Y = \beta_0 + \beta_1X_1 + \beta_2X_2 + \dots + \beta_nX_n$ calibrated from your 80% dataset, on the independent variables (IV) of your another 20% dataset (validation dataset).

In R use rmse function from hydroGOF package.

Example:

# create an object with dependent variable (DV) values from the validation dataset.
dv_observed = c(1,2,3,4,5,6,7,8,9,10)

# use the multiple linear regression model (derived from the calibration dataset) to predict DV values as from validation dataset IV values. Then, create another object.
dv_predicted = c(1,3,3,4,5,6,6,8,9,10)

require(hydroGOF)
rmse(dv_observed,dv_predicted)
[1] 0.4472136

RMSE output measurement unit is the same of your data (e.g. if DV is weight in pounds, RMSE will be pounds too).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.