I have encountered a statistician who is suggesting that for secondary analyses, she would not check model assumptions (e.g., linearity, normality). Sample sizes for each group are 26 and 28, and this is somewhat of a pilot study. I am of the persuasion that model assumptions should always be checked, and another model be used if needed, unless you have a large enough sample. Otherwise, what is the point of doing the secondary analyses if the model is incorrect? Am I wrong? Thank you in advance!
1
-
2$\begingroup$ If she only has summary statistyics for the secondary analysis, there is little possibility for checking assumptions ... $\endgroup$– kjetil b halvorsen ♦Commented Apr 7, 2015 at 21:38
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
I'm sure there is a lot of debate around how to treat pilot or feasibility data. I'll tell you my experience as a statistician/researcher at a federal agency - others that are more "pure" statisticians may have differing opinions, and they certainly may be more right than mine.
When we have done pilot studies that preamble bigger ones (which are contingent on the results of the pilots), we have approached the pilot analysis differently than the full scale analysis. The purposes of the pilot study are not the same as the full analysis - they are usually not meant to be generalizable, so things like assessing invariance are less important. Small sample sizes also preclude more complex analyses - you may not be able to do all the interactions you want to do, nonlinear regressions may be out of the question, etc. Really the analysis for a pilot study is to make sure that you can collect the data, the data are usable, the distributions are more or less what you expect, that any constructs you have hold up, that bivariate and multivariate (to the extent they are testable) are as expected, that respondents understand what they are doing and any logic checks you have built in to instruments stand up.
As far as checking model assumptions, I personally don't see why anyone would be opposed to doing it. If the assumptions hold, great, it supports the specific model you are using. If they do not, then you could argue that it isn't that big of a deal since this is pilot data, small sample size so there may be less stability, you'll have more data (hopefully!) for full data collection so you will be more robust to violations or at least more stable estimates.
answered Apr 8, 2015 at 1:14
-
1$\begingroup$ There is still a cost in terms of time, and I can see a statistician pushing back if the only possible outcome of spending their time is "Well, that muddied the waters a bit, but lets proceed anyway, it's only pilot data!" $\endgroup$– FomiteCommented Apr 8, 2015 at 5:07