I tested my main hypothesis with a multiple regression, which (unfortunately) turned out to be not significant at all. I am not sure in how to report the results now.
Which parameters should I include and how do I phrase the findings?
$\begingroup$
$\endgroup$
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
This is a profound question. At some time, it may be worth reading the American Statistical Association discussion on p-values: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5187603/
It gets more complex when you are modelling. With observational data, it is possible to try a vast combination of including / excluding predictors, adding interactions and so on. You can also have confounding whereby omitting predictors can mask an import effect. A lot of work is done in terms of model search, with techniques such as Lasso. However, you say you had a specific hypothesis (which you may have in an experimental setting). Failure to reject the null hypothesis here could be due to the null hypothesis being true, you having insufficient data, or any one of several technical assumptions failing. So I think you report it as saying you did not reject the null hypothesis. Especially if your sample size is small, I would report the summary statistics, and it really depends whether you think what you have could be important. If I did a study on poison, with two groups of three animals, I would not have a significant result. But I would have three dead animals in the treatment group. That is important, so I would report that. It is up to others to read this report and decide whether further investigation is appropriate.
I don't know what you mean by testing a main hypothesis. Are you looking at the F-value for the overall model fit, or at the Wald tests on the individual parameters?
$\begingroup$
$\endgroup$
Simply: you use the same language as you would to report a significant result, altering as necessary.
I had the honor of collaborating with a much regarded biostatistical mentor who wrote an entire manuscript prior to performing final data analysis, with just a placeholder for discussion, as that's truly the only place where discourse diverges depending on the result of the primary analysis.
The "penultimate" manuscript read something like the following"
"The multivariate relative ratio of caries for adolescent residents served with municipal fluoridated tap water was XX.X compared to adolescents not served with municipal fluoridated tap water, 95% CI XX.X - XX.X, p-value = x.xxx. These data (do not) provide statistically significant evidence that municipal fluoridation of tap water can reduce incidence of caries in an adolescent population."
Obviously, only at the point of discussion do we start to talk about any need for confirmation, follow-up, or even policy change.
For a "negative" result, you should devote some discussion to the following:
Was there adequate power according to preliminary power estimates? For instance, if you expected a 0.5 RR but the resulting RR is 0.7, post-hoc power is down for the same $n$. The meaningful discussion is whether you got the right $n$ in your sample to have a good test had the RR been 0.5.
Were there undetected interactions in the data? Are there alternative measures of effect to consider? Present a concise and relevant summary of sensitivity analyses.
Was the study meant to be hypothesis generating or hypothesis confirming? If the latter, and with adequate power, you might consider reporting the "negative" finding as a "positive" one. E.g. in the water case, "This study does not support the hypothesis that municipal fluoridation can reduce incidence of caries in minors."
$\begingroup$
$\endgroup$
There is a move, I do not know where it lies now these things tend to go back and forth among methods people, to simply ignore p values. The reason is, or at least one reason, that a variable can be statistically significant and have a very small effect size. Statistical power, and likely generalizability is another issue. It may be there was a significant effect and your power was to low to pick it up. At the least you should report your power.
