# Improving a linear regression prediction

I'm trying to create a prediction model. I have about 10,000 samples comprising of about 100 predictor variables and the response variable. At the moment I'm using a method which is apparently known as principal component regression (PCR). It's working ok, but I'm wondering whether we can do better. But I'm not a statistician and I'm having trouble keeping afloat in the sea of acronyms. I don't care about what the model looks like, I just want good predictions. Can you help direct where I should be looking?

• A scatter plot of each predictor against the response generally gives an ellipse of points (some long and thin, some almost circular). Does this imply that a linear model is the right choice?
• The predictors are measuring wildly different things, and some are multi-correlated. This is why I've normalised them and thrown them through a PCA.
• Some predictors are probably completely useless.
• There are errors and outliers in the predictors and the response. I've tried to manually get rid of outliers, but not too aggressively.
• This is an enormous subject. I can see that narrowing your question might be difficult, but any adequate answer would--and has--taken several large books. You might begin your research by linking through the model-selection tag I added; it will show you some threads in which these topics have been discussed. – whuber Jan 8 '14 at 23:56
• Yeah, I was worried the question was too broad and fluffy. But I'd be happy with someone just saying "I'd try the elastic lasso vector extension support (elves) method first. Good luck!" – wxffles Jan 9 '14 at 1:17
• Maybe you shouldn't be! As @whuber is implying, your quest for good predictions is much more complex than the matter of choosing the right model. You might want to spend a couple hours studying regression, if you haven't already. Your second question is pretty basic, but the answer may not be. Your information is all relevant to other, variably complex (or simple) issues that you may want to understand better. I can throw some jargon at you to help you narrow down the topics worth studying if you like, but this is just as likely to raise other questions as answer yours. Does that sound good? – Nick Stauner Jan 9 '14 at 1:57
• Some of the measurements we are making are inherently correlated (e.g. the temperature in one spot is almost the same the temperature in a nearby spot, bigger things weight more, etc). We have done repeated measurements of a single object and how the predictors vary. But that's only for that one object, the next one will be different. And I wouldn't know what to do with the variation anyway. – wxffles Jan 9 '14 at 3:28
• I rarely make this recommendation, because people come here out of self-sufficiency, but since you have a complex problem and don't have the time to learn how to analyze it, it seems like the best course of action is to collaborate with someone who has the knowledge and experience to help you. – whuber Jan 9 '14 at 15:21

I think we'd like the subject narrowed down a bit too; same concerns here about our time and effectiveness. Do edit in more detail if you can. That being said, here's a first attempt at narrowing things down for you somewhat. I hate to do this to you, but I'm still going to end up tossing jargon at you that you might have to read into a bit to understand. (Hovering your cursor over our tags may be enough, hopefully!)

With data that large that's all aimed at predicting one thing, overfitting () may pose one of the bigger problems for your predictive model, especially given high multicollinearity () among some of your predictors. Using regression (PCR) should be a good way to handle multicollinearity, assuming you or your software exclude(s) the principal components with trivially small relative to the total sum of eigenvalues. "What's trivial?" may be a difficult question, but if you're lucky, you'll find natural gaps. Rank order your principal components by eigenvalue, and look for sharp drops in the size of each eigenvalue compared to the next smallest. In a relatively simple scenario, you'd want to use all the principal components before the last big drop in eigenvalue. You'll probably have a lot of relatively high eigenvalues that drop off gradually, but if you're lucky, it'll look something like this:

which is a relatively clear case for retaining three factors (bifactor analysis aside). Note @Scortchi's comment on this answer though: you'll want to be careful about throwing out PCs that are doing some real predictive work for your model, even if they have really small eigenvalues.

is important in fitting a linear model, because data generally require regression. If, as it sounds, your DV is continuous, and your ellipses aren't curvilinear, discontinuous, pear-shaped, etc., but just smoothly elliptical like variably elongated footballs (in the American sense), you're probably right to go with a linear model, though I don't know that you can really just eyeball this sort of thing. Run some basic regression diagnostics if you know or can figure out how. If any of your predictors are , PCR probably won't know this, so you'll effectively be using them as approximations of continuous dimensions, which may not be safe, especially if they are , or there are less than five (approximate rule of thumb) categories, or you don't actually have any reason to expect a normal distribution underlies your system of ordinal categories.

You may want to throw out the relatively useless predictors, which are conventionally identified by $t$-testing the regression coefficients. If the coefficients don't differ significantly from zero, they may be adding more than about the DV to your model's predictions. Lots of better ideas on how exactly to test which predictors to retain when you've got so many can be found in the discussion, Is adjusting p-values in a multiple regression for multiple comparisons a good idea? @whuber's suggestion to hold out some data for model validation is particularly straightforward and convincing in ways I think you'll find appealing.