2
$\begingroup$

I have a multiple regression where one of the independent variable is a factor variable with many levels, some of those having just one observation (say, mutations of a gene with one rare mutation present in just one observation on 200 subject).

Sometimes it happens that those one observation factor are significant (Confidence intervals not crossing the null effect). This happens even more often when error intervals are computed using bootstrapping instead of classical analytic methods.

How do I interpretate and discuss these results? I suppose there's an high risk of type 1 error, because the strong effect associated to those factor levels could be instead due to some other not considered variable specific for the subject. What do you think?

Furthermore the fact that bootstrapping increases the frequency of significant singleton effects is a misbehave of bootstrap (type 1 error) or of the analytics methods (type 2 error)?

$\endgroup$
3
$\begingroup$

In my view, a "frequentist" would be limited by a single observation and forced to conclude that the standard error or confidence interval associated with any associated parameter was inestimable or undefined. So, while I can understand how bootstrapping would produce standard errors, I don't understand how you're able to obtain those metrics using traditional, "frequentist" models without it.

A Bayesian, on the other hand, would not be so constrained -- see chap 13 of Andrew Gelman's book Data Analysis Using Regression and Multilevel/ Hierarchical Models where he states that estimating these metrics can be done even with an n of 1. His point is that the Bayesian approach creates a wide set of information in the posterior distribution which draws on and shrinks information about these underidentified factors from the full set of data.

That said, the issue is less about the type 1 errors -- statistical significance -- than it is about power and type II errors. You can obtain a significant result in terms of a p-value while having little confidence that the finding is a real one. In other words, even with the potentially huge amount of empirical information available to the Bayesian posterior distribution, with an n of 1 your statistical power remains limited. Confirmation of this can be found in the observation that Bayesian approaches will give answers possessing a high degree of accuracy that can still be completely wrong.

Gathering data sufficient for large-scale genome mapping of rare mutations is a separate topic best dealt with by computer scientists since it can involve integrating confidential, private and encrypted information from many data sources as well as across national boundaries and continents. This is not a trivial challenge. Shafi Goldwasser, computer scientist at MIT, has a paper which won the ACM Turing Award for its development of a game theoretic approach to sharing data that solves these challenges and concerns.

http://www.eecs.berkeley.edu/~raluca/FE-TM.pdf

$\endgroup$
  • $\begingroup$ +1 for the bayesian/frequentist note. Incidentally I use Gelman bayesglm() function in R for most of my regression needs. Much more stable and reasonable estimates. Anyway, it's rare but it happens sometimes that I get these extremely significant effects with n=1 categories, especially when using poisson distributions. With bootstrap of course the significancy increase, since you don't have the enlarging effect of a low n on the SEs. But still my question remains; are this significant effect believable, especially when they are significant with the bootstrap and not otherwise? $\endgroup$ – Bakaburg Oct 22 '15 at 19:29
  • $\begingroup$ The trivial (gold standard) answer would be: "repeat the experiment and see for yourself", but I'd like to know from the theoretical point view how to interpretate and discuss such results. Furthermore I didn't understand your point on type two errors. I'd gladly rather accept a false negative with n=1 than risk to affirm a false positive effect! $\endgroup$ – Bakaburg Oct 22 '15 at 19:32
  • 1
    $\begingroup$ In my view, power is the real issue behind your question of the significance of a result. From that perspective, the issue of "false positives" is a red herring. Power is really a statement about whether or not you can believe in the significance of a result. To your point about Gelman, last week he posted an old, excellent paper by Carl Morris that makes exactly this point. See if you agree with him... andrewgelman.com/wp-content/uploads/2015/07/morris_example.pdf $\endgroup$ – Mike Hunter Oct 22 '15 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.