1
$\begingroup$

I'm having a hard time understanding something. Let's say that I have 36 months of data (36 observations) regarding consumer behavior on a website. I constructed a model regressing $y$ on a number of predictors, and I get the desired coefficients. However, I'm interested in knowing how well my model did in predicting the response variable in month $x$. Given my model and the training set that it was run on, I want to be able to determine how well it did in predicting the response variable in a given month (say April or March). I have the traditional stats on the overall 'goodness of fit' of the model, but I really want to understand how well my model predict the data for month $x$.

Does that make sense? How would I go about getting that information? How would I perform this task in R?

What about predicting future months? The next three months?

I am running a glm in R (Poisson).

$\endgroup$

2 Answers 2

4
$\begingroup$

If I understand you correctly, something like leave one out cross-validation (LOOCV) would allow you to test "out of sample" predictions. The performance of prediction would measured by correlation between y and ypredicted (R^2 or ROC). How does that sound?

$\endgroup$
1
$\begingroup$

If you want to test its performance on data you don't have (like a missing month), something like a PRESS statistic (which is a leave-one-out type prediction sum of squares) adapted to the Poisson model's variance function might work okay (you don't need to refit, you can work it out from your GLM fit).

In the case of your three April values, you could calculate a predictive residual for each month and just take the root-mean-square (and the average error if you want to look at bias) of the April ones... but it will be a pretty noisy estimate; you'll mostly be chasing noise.

But from the second half of your question it sounds to me like you're actually interested in one step ahead predictive performance.

In which case, you might back up to (say) one year of data, and then start doing one-step-ahead predictions, re-estimating and adding in new data. You can do something similar with multi-period forecasts.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.