We work with observations of a physical system and want do develop a model of it. To this end, we made measurements of the relevant variables and recorded them at 100Hz. In a typical case, we use a few 100'000 samples of, say, 4 to 10 covariates to develop a model.
In general, we have a reasonably good idea of what is going on in the system and what the main effects are, but it is not always clear which interactions need to be included. Deciding which interaction terms to include into the model is one of the trickier tasks for us and I thought that hypothesis testing and p-values could help with that.
From an engineering point of view, we would like to keep the model as simple as possible (but as complex as needed). But unfortunately, almost always the interaction terms appear to be significant (in R:
p-value <2e-16), even though from a practical point of view it doesn't look like that.
My question is the following: Is my sample size too "large" for the p-values to be of practical use?
I would appreciate some practical tips to estimate what makes a sample large or if p-values are appropriate to use in my context.