I'm kinda brand new to statistics and I'm developing classification algorythm. My method is based on simple chi-square goodness of fit. I am counting the effect size of known cases to predict the future ones.

My problem is that I don't really know how to deal with large sample sizes, neither do I know when is a sample size "too big". Did research the answer, but did not find solution.

So what is the best approach to take when you have a double possible outcome and like 2000-3000 known cases where you know the outcome? Obviously it isn't safe for NHST as large sample sizes will yield false positives (as much as I know).

Can anyone suggest some good approach to take, or maybe a good article on the topic?

Thanks for help!

  • 3
    $\begingroup$ What makes you say that large sample sizes in hypothesis testing lead to false positives (i.e. type I errors)? Could you explain how you think one causes the other? $\endgroup$ – Glen_b -Reinstate Monica Jun 20 '14 at 2:49
  • $\begingroup$ I took one beginner course in Coursera, and the instructor stated a multiple times to generally be cautious with large samples in the denominator as they affect the outcome. Is it a good pratice to just plug it in straight no matter if 10000 or 6 samples I have? I know this must be some very beginner question, but everybody starts somewhere. Thanks for answer! $\endgroup$ – Netface Jun 20 '14 at 2:59
  • 1
    $\begingroup$ It depends on what you mean by good practice (I would tend to avoid NHST for most applications people seem happy to use them for). I simply don't see the justification for the part about 'false positives'; the type I error rate is set by the person doing the hypothesis test; how can the rate of false positives by affected by sample size unless you choose your significance level as a function of sample size? The problem many people have with NHST with large samples isn't false positives, it's true positives - that it will reject small effects with large samples... (ctd) $\endgroup$ – Glen_b -Reinstate Monica Jun 20 '14 at 3:05
  • 2
    $\begingroup$ (ctd)... the reason people have a problem with that isn't an issue with large samples, the thing they get bothered about is caused by the fact that NHST doesn't solve the problem they're trying to deal with, and they only notice that issue at large sample sizes - they're blaming the hammer for being bad at inserting screws. When the answers are problematic, it's because they must ask the right questions. $\endgroup$ – Glen_b -Reinstate Monica Jun 20 '14 at 3:07
  • 1
    $\begingroup$ Somewhat familiar with statistics, yes. Your application is out of my areas of expertise, sorry, but I'd say it's an application where the need for actual NHST-type hypothesis tests would be relatively rare. (Equivalence testing might sometimes be useful, perhaps, but mostly the questions where one might be tempted to use NHSTs would relate to estimation of effect sizes - in either case, 'are they different enough to matter'-type questions, rather than 'detectably more different than chance' type questions) $\endgroup$ – Glen_b -Reinstate Monica Jun 20 '14 at 3:34

The problem is not that you get "false positives". The problem is that small effect sizes will be significant. But what is "small" varies from context to context and is something you will be better able to answer than us.

Even after your discussion with @Glen_b in the comments, I am not sure exactly what you are trying to do, nor whether chi-square is best (you might read my blog post [how to ask a statistics question http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/) for help with asking a question that can get a good answer) but the same issue comes up whenever you use NHST (as an aside, Patricia Cohen once said that her husband Jacob wanted to call it Statistical Hypothesis Inference Testing - for the acronym - but she convinced him not to do so).

Let's suppose you have a 2x2 table (which I think is what you are saying you have). You can then look at various measures: Odds ratio, false positives, false negatives, specificity and others and decide which are meaningful in your context and how high or low a value is acceptable or interesting. A large sample size, will, other things being equal, allow you to more precisely estimate any of those measures.

  • 1
    $\begingroup$ You are absolutely right. I think we could get into a lengthy theoretic discussion: Asking the right questions yields the right answers: here, and generally. As a newcomer to a field sometimes it is the hardest to determine what is it that you are missing in your idea. I certainly feel that I am missing parts, just yet not sure what they exactly are. As you grow knowledgable questions get more precise, actually being able to ask better questions means getting better at something. Thanks for the guide, I'll keep that in mind! Hope we'll discuss more precise questions some day! $\endgroup$ – Netface Jun 20 '14 at 12:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.