Two common approaches for selecting correlated variables are significance tests and cross validation. What problem does each try to solve and when would I prefer one over the other?
First, lets be explicit and put the question into the context of multiple linear regression where we regress a response variable, $y$, on several different variables $x_1, \ldots, x_p$ (correlated or not), with parameter vector $\beta = (\beta_0, \beta_1, \ldots, \beta_p)$ and regression function
The question is how to select a subset of the $\beta_i$'s to be non-zero, and, in particular, a comparison of significance testing versus cross validation.
To be crystal clear about the terminology, significance testing is a general concept, which is carried out differently in different contexts. It depends, for instance, on the choice of a test statistic. Cross validation is really an algorithm for estimation of the expected generalization error, which is the important general concept, and which depends on the choice of a loss function.
The expected generalization error is a little technical to define formally, but in words it is the expected loss of a fitted model when used for prediction on an independent data set, where expectation is over the data used for the estimation as well as the independent data set used for prediction.
To make a reasonable comparison lets focus on whether $\beta_1$ could be taken equal to 0 or not.
Using significance testing we are not directly concerned with the "performance" of the model under the null hypothesis versus other models, but we are concerned with documenting that the null is wrong. This makes most sense (to me) in a confirmatory setup where the main objective is to confirm and document an a priory well specified scientific hypothesis, which can be formulated as $\beta_1 \neq 0$.
The expected generalization error is, on the other hand, only concerned with average "performance" in terms of expected prediction loss, and concluding that it is best to allow $\beta_1$ to be different from 0 in terms of prediction is not an attempt to document that $\beta_1$ is "really" different from 0 $-$ whatever that means.
I have personally never worked on a problem where I formally needed significance testing, yet $p$-values find their way into my work and do provide sensible guides and first impressions for variable selection. I am, however, mostly using penalization methods like lasso in combination with the generalization error for any formal model selection, and I am slowly trying to suppress my inclination to even compute $p$-values.
For exploratory analysis I see no argument in favor of significance testing and $p$-values, and I will definitely recommend focusing on a concept like expected generalization error for variable selection. In other contexts where one might consider using a $p$-value for documenting that $\beta_1$ is not 0, I would say that it is almost always a better idea to report an estimate of $\beta_1$ and a confidence interval instead.
Simply using significance tests and a stepwise procedure to perform model selection can lead you to believe that you have a very strong model with significant predictors when you, in fact, do not; you may get strong correlations by chance and these correlations can seemingly be enhanced as you remove other unnecessary predictors.
The selection procedure, of course, keeps only those variables with the strongest correlations with the outcome and, as the stepwise procedure moves forward, the probability of committing a Type I error becomes larger than you would imagine. This is because the standard errors (and thus p-values) are not adjusted to take into account the fact that the variables were not selected for inclusion in the model randomly and multiple hypothesis tests were conducted to choose that set.
David Freedman has a cute paper in which he demonstrates these points called "A Note on Screening Regression Equations." The abstract:
One potential solution to this problem, as you mentioned, is using a variant of cross validation. When I don't have a good economic (my area of research) or statistical reason to believe my model, this is my preferred approach to selecting an appropriate model and performing inference.
Other respondents might mention that stepwise procedures using the AIC or BIC are asympotically equivalent to cross validation. This only works as the number of observations relative to the number of predictors gets large, however. In the context of having many variables relative to the number of observations (Freedman says 1 variable per 10 or fewer observations), selection in this manner can exhibit the poor properties discussed above.
In an age of powerful computers, I don't see any reason not to use cross validation as a model selection procedure over stepwise selection.