1
$\begingroup$

I've built a linear model with Matlab using fitlm to predict the next value in a series of doubles. Now I would like to test this model on a different dataset so I get accuracy, p-value etc.

I've used predict, but that just outputs a prediction curve and a confidence interval.

How do I get an accuracy for this new data? What would be the correct way to evaluate this model on the test set? Could I somehow specify the testset when I am building fitlm in the first place?

I know this can be done in packages such as Weka, so I was wondering about the options in Matlab.

$\endgroup$

1 Answer 1

2
$\begingroup$

You could compute the mean squared error on the test set,

testmse = mean(Ytest - predict(lm,Xtest)).^2

and compare it with the mse that is computed from the training data in the original model.

This function does ordinary least squares. It uses classical statistical approaches to adjust mse for the number of parameters in the model, perform t and F tests, and compute an adjusted R-square. It doesn't do regularization and it takes the model as given. So it doesn't take a test set.

Your ideas are interesting and could help you build up the model that you specify to fitlm. For example, you could watch the relationship between the computed mse and testmse as you add more terms to the model. I haven't seen this done (of course I haven't seen everything), but it seems plausible.

$\endgroup$
1
  • $\begingroup$ Thanks, that's an interesting idea. It does seem like the extra data could be used to help build the model, have to think about that... $\endgroup$
    – dorien
    Commented Feb 2, 2016 at 22:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.