1
$\begingroup$

I have a question regarding results from a study and it's probably a very basic question (but I only have a very very basic understanding of epidemiology). I'm looking at what characteristics have a significant association with an outcome from multiple studies, so I have generally looked at results that were found to be significantly associated with the outcome through statistical analysis (regression analysis mostly and some studies used odds ratio). However, I found a study that just states percentages and says there is an association. I'm a little bit wary of taking those results. I've looked in the methods but it does not say anything except that it conducted an analysis using StatSoft Statistica 13.1. So I'm not sure if I should be considering this?

Edit: The study also mentions the number of cases examined and provides n (%) for the study. In case anyone is wondering what the study looks like, here is a link: https://doi.org/10.3390/vaccines9050475

Edit: My main issue is that the study uses summary statistics and there is no statistical significance to the results, and that is why I was wondering if I should even be considering these results because of that.

$\endgroup$
1
  • 3
    $\begingroup$ Does the study that only described percentages also state the number of cases that were examined? Please add that information to the question, as comments are easy to overlook and can be deleted. $\endgroup$
    – EdM
    Jun 30, 2022 at 16:55

3 Answers 3

2
$\begingroup$

It seems that the paper in question didn't report statistical significance of the results, but you can use the formula for the standard error of a binomial proportion estimate to help evaluate the results. From this page, for example, the standard error of a binomial proportion estimate $\hat p$ based on an overall sample size $n$ can be taken as $\sqrt{\hat p(1-\hat p)/n}$. With a sample size of 419, as reported by the paper, that would correspond to about 0.014 standard error in an estimated proportion of 0.1, to a maximum standard error of about 0.024 in an estimated proportion of 0.5.

There are several detailed approaches for how to use that relationship to evaluate signficance statistically; see this Wikipedia page for example.

$\endgroup$
1
$\begingroup$

The authors of the paper acknowledge the lack of accuracy in their study and that's why they did not compute confidence intervals (statistical significance would not make sense because the research is not testing a hypothesis)

The study’s main limitation was the nonrandom volunteer sample of survey participants, which made it impossible to calculate a participation rate or confidence limits of the observed proportions.

Therefore the study is not a very strong measurement. This is also mentioned by the researchers who consider the search as 'a preliminary description of attitudes'

The study is limited by its nonrandom sample of HCWs but provides a preliminary description of attitudes towards SARS-CoV-2 vaccination.


Does it matter that they did not state significance?

However, I found a study that just states percentages and says there is an association. I'm a little bit wary of taking those results.

Even if they would have computed confidence intervals or provided some other calculation, then you should still be wary.

The output from SPSS* is overemphasized, but it is just a number from a model. Too many people look just at the p-value and regard it as a proof that the research is correct, a sort of validation label, and don't look further anymore.

But, the p-values can be wrongly calculated when the model is not appropriate. The sampling or the experiment could be biased. Systematic errors may have occurred. Etc.

The value of this study is also not as a scientific rigorous measurement of attitudes. It is a preliminary study to investigate attitudes. It also seems more like a research as they would do in businesses. Based on those results you can build new studies in the case that one would investigate it further and do actual research related to understanding the attitudes.

*Or whatever other software that researchers regard as a black box that they don't understand. But if they put in the numbers in the way that they learned during their university course, then they will get some magic number. And if it doesn't satisfy then possibly they can change the test by clicking some other buttons in the software or redo the test untill they have something they can publish.

$\endgroup$
2
  • $\begingroup$ Would you suggest looking at statistically significant results in the first place for a literature review? $\endgroup$
    – Eems
    Jul 5, 2022 at 21:28
  • $\begingroup$ If you are performing a meta-study then significance is irrelevant for selection. A problem with this study is however how you are gonna wheigh the results in comparison to other studies. You can compute the significance yourself based on the presented results, but there may be bias in this study as well. Significance just relates to the statistical effect of the properties of a sample being different from the properties of the entire population due to randomness. $\endgroup$ Jul 6, 2022 at 6:06
1
$\begingroup$

By using summary based on their sample and not model based approach they failed the main rule of Causal Inference, "transferability". Their results cannot be transferred to the population and hence highly biased.

And exact details are mentioned by James Stanley below, which provides more details why results from non-model based approach should not be considered.

"(i.e. here your main concerns are that the estimates provided will effectively be crude/unadjusted results, and the proper consideration should take into account adjustment for confounding to get a robust causal estimate?)"

$\endgroup$
3
  • $\begingroup$ This is a good start to an answer -- it would be good to edit to explain your points a bit more (i.e. here your main concerns are that the estimates provided will effectively be crude/unadjusted results, and the proper consideration should take into account adjustment for confounding to get a robust causal estimate?) $\endgroup$ Jun 30, 2022 at 21:37
  • $\begingroup$ Yes those are exactly my concerns. I did not know how to word it, but I will add the edit. Thanks. $\endgroup$
    – Eems
    Jul 1, 2022 at 1:56
  • $\begingroup$ Could you please edit the answer to include the details mentioned in James Stanley's comment, so the answer is self-contained and doesn't rely on people reading the comment. $\endgroup$
    – Lynn
    Jul 1, 2022 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.