There are a few discussions on CV about what happens to hypothesis tests in "large" data sets, e.g. here or here. These questions are very interesting, but I'm not sure how they affect my work.

We work with observations of a physical system and want do develop a model of it. To this end, we made measurements of the relevant variables and recorded them at 100Hz. In a typical case, we use a few 100'000 samples of, say, 4 to 10 covariates to develop a model.
In general, we have a reasonably good idea of what is going on in the system and what the main effects are, but it is not always clear which interactions need to be included. Deciding which interaction terms to include into the model is one of the trickier tasks for us and I thought that hypothesis testing and p-values could help with that.

From an engineering point of view, we would like to keep the model as simple as possible (but as complex as needed). But unfortunately, almost always the interaction terms appear to be significant (in R: p-value <2e-16), even though from a practical point of view it doesn't look like that.

My question is the following: Is my sample size too "large" for the p-values to be of practical use?

I would appreciate some practical tips to estimate what makes a sample large or if p-values are appropriate to use in my context.

  • $\begingroup$ Hi @raphael , I was curious if you eventually found an answer to your question, what technique did you to resolve the effect size impact? $\endgroup$ Jul 6 '21 at 2:38
  • $\begingroup$ Hi @avabhishiek, unfortunately, nobody had an answer to my question. Feel free to add one if you have an idea $\endgroup$
    – Raphael
    Aug 10 '21 at 12:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.