0
$\begingroup$

Let's say mtcars contain 3 independent datasets: cyl=4, cyl=6 and cyl=8.

I'd like to use two of the groups as my training set, and cyl=4 as the testing set. The aim is to check how well my model performs on the test data:

training <- mtcars[mtcars$cyl!=4,]
testing <- mtcars[mtcars$cyl==4,]

Model:

m <- lm(mpg ~ disp + wt, data=training)

pred.test <- predict(m, testing)
mean.err <- mean((pred.test - testing$mpg)^2)

But what does that tell me about generalizability? How can I determine if mean.err (or any other value) is a reasonable number? Or is what I'm doing in the example above not the way to test generalizability?

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ "How can I determine if mean.err (or any other value) is a reasonable number?" People will often compare it to the mean squared error on the training set, and see how much bigger it is. "Or is what I'm doing in the example above not the way to test generalizability?" Note that what you're doing is harder than what people usually mean when they say "Out of Sample" or "Test" error, since you're extrapolating outside of your training domain (which has no cyl=4). $\endgroup$
    – John Madden
    Commented Feb 16, 2023 at 17:42
  • 3
    $\begingroup$ This seems to require domain knowledge, because you are asking how accurately you need to predict mpg. It depends on your application -- not on any statistical theory. $\endgroup$
    – whuber ♦
    Commented Feb 16, 2023 at 17:51
  • $\begingroup$ Thanks @JohnMadden. The mse in my training set is 7.71. But how can I test if the difference between the mse of the training model is meaningfully larger than that of mean.err? $\endgroup$
    – locus
    Commented Feb 16, 2023 at 17:52
  • $\begingroup$ @whuber, thanks! Yes, I know this likely depends on the application. At this stage, I'm just trying to understand (using simple examples like in my question) what kind of tools are available that would allow me to answer the question of model generalizability. $\endgroup$
    – locus
    Commented Feb 16, 2023 at 18:01
  • $\begingroup$ for example, comparing mse as @JohnMadden suggested would be one way, although I'd still need to somehow meaningfully compare/interpret the mses. I'm not sure about the utility of cross validation and related techniques in my case... $\endgroup$
    – locus
    Commented Feb 16, 2023 at 18:06

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.