I'm trying to create a prediction model. I have about 10,000 samples comprising of about 100 predictor variables and the response variable. At the moment I'm using a method which is apparently known as principal component regression (PCR). It's working ok, but I'm wondering whether we can do better. But I'm not a statistician and I'm having trouble keeping afloat in the sea of acronyms. I don't care about what the model looks like, I just want good predictions. Can you help direct where I should be looking?

Some information about my data:

  • A scatter plot of each predictor against the response generally gives an ellipse of points (some long and thin, some almost circular). Does this imply that a linear model is the right choice?
  • The predictors are measuring wildly different things, and some are multi-correlated. This is why I've normalised them and thrown them through a PCA.
  • Some predictors are probably completely useless.
  • There are errors and outliers in the predictors and the response. I've tried to manually get rid of outliers, but not too aggressively.
  • 3
    $\begingroup$ This is an enormous subject. I can see that narrowing your question might be difficult, but any adequate answer would--and has--taken several large books. You might begin your research by linking through the model-selection tag I added; it will show you some threads in which these topics have been discussed. $\endgroup$ – whuber Jan 8 '14 at 23:56
  • $\begingroup$ Yeah, I was worried the question was too broad and fluffy. But I'd be happy with someone just saying "I'd try the elastic lasso vector extension support (elves) method first. Good luck!" $\endgroup$ – wxffles Jan 9 '14 at 1:17
  • 1
    $\begingroup$ Maybe you shouldn't be! As @whuber is implying, your quest for good predictions is much more complex than the matter of choosing the right model. You might want to spend a couple hours studying regression, if you haven't already. Your second question is pretty basic, but the answer may not be. Your information is all relevant to other, variably complex (or simple) issues that you may want to understand better. I can throw some jargon at you to help you narrow down the topics worth studying if you like, but this is just as likely to raise other questions as answer yours. Does that sound good? $\endgroup$ – Nick Stauner Jan 9 '14 at 1:57
  • 1
    $\begingroup$ Some of the measurements we are making are inherently correlated (e.g. the temperature in one spot is almost the same the temperature in a nearby spot, bigger things weight more, etc). We have done repeated measurements of a single object and how the predictors vary. But that's only for that one object, the next one will be different. And I wouldn't know what to do with the variation anyway. $\endgroup$ – wxffles Jan 9 '14 at 3:28
  • 1
    $\begingroup$ I rarely make this recommendation, because people come here out of self-sufficiency, but since you have a complex problem and don't have the time to learn how to analyze it, it seems like the best course of action is to collaborate with someone who has the knowledge and experience to help you. $\endgroup$ – whuber Jan 9 '14 at 15:21

I think we'd like the subject narrowed down a bit too; same concerns here about our time and effectiveness. Do edit in more detail if you can. That being said, here's a first attempt at narrowing things down for you somewhat. I hate to do this to you, but I'm still going to end up tossing jargon at you that you might have to read into a bit to understand. (Hovering your cursor over our tags may be enough, hopefully!)

With data that large that's all aimed at predicting one thing, overfitting () may pose one of the bigger problems for your predictive model, especially given high multicollinearity () among some of your predictors. Using regression (PCR) should be a good way to handle multicollinearity, assuming you or your software exclude(s) the principal components with trivially small relative to the total sum of eigenvalues. "What's trivial?" may be a difficult question, but if you're lucky, you'll find natural gaps. Rank order your principal components by eigenvalue, and look for sharp drops in the size of each eigenvalue compared to the next smallest. In a relatively simple scenario, you'd want to use all the principal components before the last big drop in eigenvalue. You'll probably have a lot of relatively high eigenvalues that drop off gradually, but if you're lucky, it'll look something like this:

which is a relatively clear case for retaining three factors (bifactor analysis aside). Note @Scortchi's comment on this answer though: you'll want to be careful about throwing out PCs that are doing some real predictive work for your model, even if they have really small eigenvalues.

is important in fitting a linear model, because data generally require regression. If, as it sounds, your DV is continuous, and your ellipses aren't curvilinear, discontinuous, pear-shaped, etc., but just smoothly elliptical like variably elongated footballs (in the American sense), you're probably right to go with a linear model, though I don't know that you can really just eyeball this sort of thing. Run some basic regression diagnostics if you know or can figure out how. If any of your predictors are , PCR probably won't know this, so you'll effectively be using them as approximations of continuous dimensions, which may not be safe, especially if they are , or there are less than five (approximate rule of thumb) categories, or you don't actually have any reason to expect a normal distribution underlies your system of ordinal categories.

You may want to throw out the relatively useless predictors, which are conventionally identified by $t$-testing the regression coefficients. If the coefficients don't differ significantly from zero, they may be adding more than about the DV to your model's predictions. Lots of better ideas on how exactly to test which predictors to retain when you've got so many can be found in the discussion, Is adjusting p-values in a multiple regression for multiple comparisons a good idea? @whuber's suggestion to hold out some data for model validation is particularly straightforward and convincing in ways I think you'll find appealing.

If you were to care enough about what your model looks like, particularly in terms of how those principal components of your predictors organize themselves, you might consider piecing together a structural equation model () of your own design. If you could model the latent factor structure of your set of predictors manually and accurately before using the latent factors to predict your DV, you could remove from the factors in advance of doing the predictive modeling with them, and probably gain a better understanding of your model in the process. This could also let you identify among your model's predictors, depending on how you organize it. I don't suppose you'd be inclined to care about that if you're only interested in prediction at the moment (and I don't mean to assume that you're wrong not to care), but if you ever find you need to explain how you're getting those predictions and why you think they're valid, you might have to revisit a lot of this when/if you do start caring. Therefore a little preemptive caring might be advisable, even if there's really no immediate reason! Then again, maybe you'll have more time later and be better able to afford starting over then, if necessary. Your call, your risk. Happy modeling, and may the trashy fiction be ever the result of someone else's work!

  • 2
    $\begingroup$ (1) Use of PCA to reduce the dimensionality of the predictor set in this way involves an assumption that the PCs explaining most of the variability in predictors relate most strongly to the response - an assumption which needs to be justified in any particular case. There's no general reason why the response shouldn't be strongly related to the principal component with the smallest eigenvalue and only weakly to the PCs with larger eigenvalues. $\endgroup$ – Scortchi - Reinstate Monica Jan 10 '14 at 13:35
  • 1
    $\begingroup$ (2) Removing "insignificant" predictors en masse is probably the worst way of doing model selection that ever gets seriously proposed, & particularly egregious when done without any assessment of whether model reduction is even necessary. See @chi's answer in the link you provided in the same paragraph for some better ideas. $\endgroup$ – Scortchi - Reinstate Monica Jan 10 '14 at 13:36
  • $\begingroup$ Good points! Leaving all the PCs in means not removing much multicollinearity though, doesn't it? Not that addressing multicollinearity is necessarily a top priority. +1 for the redirect to @chl's answer! $\endgroup$ – Nick Stauner Jan 10 '14 at 15:37
  • 1
    $\begingroup$ Yes; keeping all doesn't help, but keeping only a subset with the highest eigenvalues may lessen the model's predictive power. Not always - it seems quite reasonable when you've +ve correlations between all predictors & you'd expect each to have a +ve correlation with the response. Jolliffe (1982) gives examples & Jolliffe (2010) reviews other ways of selecting PCs $\endgroup$ – Scortchi - Reinstate Monica Jan 10 '14 at 19:34
  • $\begingroup$ Thanks for your points and citations. I've found them very useful, and incorporated them into another answer that expands on your point (1). $\endgroup$ – Nick Stauner Feb 19 '14 at 23:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.