How to compute the generalization error (test error) in a multivariate regression?

Question

How can I, after running multivariate linear regression, compute the Generalization Error (also known as Test Error)? Are there many practises to do that?

EDIT: Can the following be considered as Generalization (or Test) Error?

y = data1; [M,N] = size(y);
X = [ones(M,N) data2];
y_train = y(M-30-483:M-30,:);
X_train = X(M-30-483:M-30,:);
y_test = y(M-29:M,:);
X_test = X(M-29:M,:);

[b,~,r,~,stats] = regress(y_train,X_train);
[M,N] = size(data2);
figure; plot(r);
axis([0,N+1,-Inf,Inf]);

y_estimated = X_test * b; y_expected = y_test;
generalization_error = y_expected - y_estimated;
figure; plot(generalization_error);

Answer 1

What you are showing above is indeed an estimate of the model performance on unseen data (generalization error). However the variance of your test data set will presumably be non-negligible and so a better way to estimate the generalization error would be through cross-validation.

Try the crossval function in Matlab or use cvpartition to define and your cross-validation folds and iterate though using a loop.

Answer 2

Generalzation error is usually further summarized to obtain one number for all samples:

From your differences, you can e.g. calculate

• root mean squared error sqrt (mean ((yhat - y)^2))
• mean absolute error mean (abs (yhat - y))