4
$\begingroup$

How can I, after running multivariate linear regression, compute the Generalization Error (also known as Test Error)? Are there many practises to do that?

EDIT: Can the following be considered as Generalization (or Test) Error?

y = data1; [M,N] = size(y);
X = [ones(M,N) data2];
y_train = y(M-30-483:M-30,:);
X_train = X(M-30-483:M-30,:);
y_test = y(M-29:M,:);
X_test = X(M-29:M,:);

[b,~,r,~,stats] = regress(y_train,X_train);
[M,N] = size(data2);
figure; plot(r);
axis([0,N+1,-Inf,Inf]);

y_estimated = X_test * b; y_expected = y_test;
generalization_error = y_expected - y_estimated;
figure; plot(generalization_error);
Improve this question
$\endgroup$
3
  • $\begingroup$ I take it that you're wondering how to do this in Matlab? $\endgroup$
    – MånsT
    Jul 22, 2012 at 8:37
  • $\begingroup$ @MånsT: Basically yes, but I would also appreciate if you could provide a few hints regarding the theoretical aspects of this concept... $\endgroup$
    – user2295350
    Jul 22, 2012 at 19:51
  • $\begingroup$ possible duplicate you can check this stats.stackexchange.com/questions/32748/… $\endgroup$
    – s.s.o
    Feb 8, 2013 at 1:43

2 Answers 2

Reset to default
2
$\begingroup$

What you are showing above is indeed an estimate of the model performance on unseen data (generalization error). However the variance of your test data set will presumably be non-negligible and so a better way to estimate the generalization error would be through cross-validation.

Try the crossval function in Matlab or use cvpartition to define and your cross-validation folds and iterate though using a loop.

Improve this answer
$\endgroup$
2
$\begingroup$

Generalzation error is usually further summarized to obtain one number for all samples:

From your differences, you can e.g. calculate

  • root mean squared error sqrt (mean ((yhat - y)^2))
  • mean absolute error mean (abs (yhat - y))
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.