circular reasoning? At the moment I´m working on a huge dataset of genetic data.
It contains about 1500 variables. My goal is to classify a disease risk group (20% of subjects) based on this data.
At first I have run a series of Mann–Whitney U tests and identified 32 variables on a significant <0.01 level. If I'm not mistaken, 15 of these variables could be result of random processes.
But my goal was to train a neural network to make successful classifications. I have 400 subjects and it was not possible to successfully train a model with the full set of variables.
If I only take the 32 variables from the significant u-test I get a decent model. My question is. Is this some kind of circular reasoning? 
Thanks in advance!
 A: It's not terrible but it's not great, either. Ideally, we'd prefer that the network be able to sort out what is/isn't important given the presence of the other variables. On the other hand, a large number of irrelevant features can make that a challenging fitting & regularization task, so it's hard to fault you for taking a different approach.
A useful example of a prominent researcher using univariate tests in this way is described in Elements of Statistical Learning (2nd edition, Section 11.9), on Neal and Zhang's Bayesian neural nets. (Don't bother hunting down Neal and Zhang's write-up, published in an obscure text -- the discussion in ESL covers it very well.)
A: It's related to circular reasoning.  Variables chosen because they are associated with the outcome will likely continue to show (with an alternate approach) that they are associated with the outcome.  It's very important to cross-validate.  Choose predictors based on a subset ("training set") of the data, and test their predictive accuracy on a holdout set ("test set").  In fact, this needs to be done multiple times -- often hundreds or thousands of times, automated via code -- to yield a stable estimate of accuracy with an acceptable confidence interval.
A: Your ad hoc selection method is guaranteed to inflate false-positive error rate due to overfitting. 
Philosophically, I'm not sure whether this is formally a kind of "circular reasoning"--practically this is like debating the sex of angels, to quote Miguel Hernan. 
A 0.01 significance level with 1500 comparisons reason leads to an expected 15 false discoveries. This means nearly half of your 32 U-stat statistically significant features are questionable. Put differently, the probability of having 10 or more false discoveries is 88%. I do not think a 0.01 significance cut off is defensible here. You have a $1-0.99^{1500} = 0.9999997$ family wise error rate. You should better control for multiple comparisons, Bonferroni is imperfect but easy to do. Try your ad hoc method again using a $0.05/1400 = 0.00004$ cut off.
