Selecting variables using SAS and R - all effects are significant even when shuffling the data Dear all: I need to test which effects I should include in my model for genetic evaluation of cows. I was using the following code in R: 
model1 = lm(milk ~ factor(year) + factor(herd) + factor(season) + age + I(age^2), data=paula1)
anova(model1)

However, all my effects were highly significant (< 2.2e-16 ***). I treid using step(model1) to choose the best model, I tried to include other effects that I would never expect to be significant and they were. So I was thinking that I was doing something wrongly. Then I tried proc glm in SAS using the following code:
data paula1; set paula0;
proc glm;
class year herd season;
model milk= year herd season age age*age;
run;
And my results were very similar. I decided then to exclude some data and shuffle some variables and it is still signifficant. Now I have no doubts that my analysis are completely wrong, I just cant figure out what my mistake is. For some factors (e.g. herd I have more than 200 levels) and I have missing data as well (coded as NA in R and blank in SAS). The outputs look fine (sum of squares, degree of freedom, etc)
Any help would be very much appreciated. Thanks. Paula
 A: What you observe is to be expected with stepwise selection.
Stepwise model reduction has a long list of well-documented undesirable effects.
One of those is that p-values become artificially small. They're essentially meaningless; you don't know how big they should be, only that the number you have is far smaller than it ought to be.
Other effects include that parameter estimates are biased (the ones still in the model are biased away from 0); standard errors are much too small. Measures of 'fit' ($R^2$ and similar measures) are inflated. Confidence intervals don't achieve anything like their desired coverage. Out of sample performance is usually very bad. (These problems tend to occur with pretty much any kind of model selection - on any kind of model - that doesn't in some way split samples so that model selection, estimation and evaluation aren't done on the same data. But stepwise has some additional problems as well.)
Numerous answers on this site discuss these problems at length.
I highly recommend Chapter  4 of Frank Harrell's Regression Modeling Strategies for reading. I always ask any research students I have to familiarize themselves with it. (Well, read the rest of it, too - it's an excellent book, but Chapter 4 speaks to this issue)
