huge sample sizes and null hypothesis testing I'm kinda brand new to statistics and I'm developing classification algorythm. My method is based on simple chi-square goodness of fit. I am counting the effect size of known cases to predict the future ones.
My problem is that I don't really know how to deal with large sample sizes, neither do I know when is a sample size "too big". Did research the answer, but did not find solution.
So what is the best approach to take when you have a double possible outcome and like 2000-3000 known cases where you know the outcome? Obviously it isn't safe for NHST as large sample sizes will yield false positives (as much as I know).
Can anyone suggest some good approach to take, or maybe a good article on the topic?
Thanks for help!
 A: The problem is not that you get "false positives". The problem is that small effect sizes will be significant.  But what is "small" varies from context to context and is something you  will be better able to answer than us.
Even after your discussion with @Glen_b in the comments, I am not sure exactly what you are trying to do, nor whether chi-square is best (you might read my blog post [how to ask a statistics question http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) for help with asking a question that can get a good answer) but the same issue comes up whenever you use NHST (as an aside, Patricia Cohen once said that her husband Jacob wanted to call it Statistical Hypothesis Inference Testing - for the acronym - but she convinced him not to do so). 
Let's suppose you have a 2x2 table (which I think is what you are saying you have). You can then look at various measures: Odds ratio, false positives, false negatives, specificity and others and decide which are meaningful in your context and how high or low a value is acceptable or interesting. A large sample size, will, other things being equal, allow you to more precisely estimate any of those measures. 
