I split my data into "estimation" and "holdout" sample, using "Select Cases" (and select only the cases I want to use as the estimation sample). Then, I created a time series using Expert Modeler in Forecasting. Doing this already gave me predicted values for the holdout cases (since my holdout cases were included in the sheet as "active" rows and hence were considered "forecast period").

My question though is how do I get a separate error measurements (fit measures like RMSE, MAE, MASE, etc.) on just the holdout sample. My understanding is that the provided fit measures were for the estimation sample, but it didn't provide the same fit measures for the holdout sample. It only gave the predicted values during the "forecast period."

It's true that once I have the true values and the predicted values, I can manually calculate those fit measures. But I'm wondering whether there is a better way to do this?

More generally, given a set of observations and a set of predicted values, is there a function in SPSS that automatically calculates all those fit measures?

Is there a better way to do the model validation on the holdout sample?

Any advice would be appreciated!

  • $\begingroup$ Your question is off-topic because it is software-specific. But you open Syntax Reference doc. and find commant TFIT (if I remember its name correctly). $\endgroup$ – ttnphns Aug 23 '17 at 20:11
  • $\begingroup$ @ttnphns The last question--"is there a better way"--might be on topic here if it's asking for a statistical procedure rather than an SPSS command. Beth, which one is it? $\endgroup$ – whuber Aug 23 '17 at 21:03
  • $\begingroup$ @ttnphns: I'm sorry if I posted the question to a wrong topic. I'm very new to this community and didn't know the rule/expectation. Thanks for your response though. When I checked TFIT in SPSS, this is what I got "TFIT controls the default column widths of the pivot tables." It seems to have to do something else? Did I not get the right command? $\endgroup$ – Beth Aug 24 '17 at 1:28
  • 1
    $\begingroup$ @whuber: Thanks for your comment. I actually would like to know both the command (software-specific) and if there is a better way in the stats/forecasting sense. $\endgroup$ – Beth Aug 24 '17 at 1:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.