2
$\begingroup$

There are cases (arguably the vast majority) where the data distribution is unknown. Confidence intervals make sense for the class of "nicely" defined theoretical pdfs ie Gaussian. In this case, we know that additional samples help to derive the characteristics (statistical moments) of the pdf. However, there are other distributions in which additional sampling is not helping to converge to estimates but on the contrary diverge, for example, Cauchy distribution enter image description here

sources:

So with this context, my question is: Why do we need to add confidence intervals (or the equivalent reporting mean and variance) to model predictions in cases we don't know the true data distribution?

I would like to focus the question in the context of Machine learning inference evaluation scores a common example is to add +/- in the score interval usually by rerunning the model, with some different random seed conditions.

enter image description here

(note that this is distinct question from from k-fold method discussed here: https://datascience.stackexchange.com/questions/108792/why-is-the-k-fold-cross-validation-needed but it could be also a relevant method there if the data is violating some of Central limit theorem conditions such as being independently and identically distributed. )

$\endgroup$
7
  • 3
    $\begingroup$ Are you asking about confidence intervals or prediction intervals? There is a difference. I don't think many people give PIs for Cauchys, so it seems like you are looking at CIs... but then again, "mean and variance" does sound more like PIs. $\endgroup$ Nov 14, 2022 at 10:53
  • $\begingroup$ My question is more about ML model predictions. ie people report scores +/- for different instances of the model configured with the same hyperparameters but different random seeds to initialize weights and random states in various parts. $\endgroup$
    – partizanos
    Nov 14, 2022 at 19:08
  • $\begingroup$ That is yet a third possible meaning: a range of point predictions for different randomizations. This is neither a confidence interval nor a prediction interval. Is it this you are asking about? $\endgroup$ Nov 14, 2022 at 19:42
  • $\begingroup$ Yes and I see it in quite some papers I will put an example in the question to demonstrate the case. $\endgroup$
    – partizanos
    Nov 14, 2022 at 21:36
  • $\begingroup$ @StephanKolassa I added an example $\endgroup$
    – partizanos
    Nov 14, 2022 at 21:48

2 Answers 2

2
$\begingroup$

We only assume that the distribution is Gaussian for convenience, and mathematical results like the Central Limit Theorem tell us that assuming a Gaussian is a good approximation to the true distribution of uncertain quantities.

If we did not make simplifying assumptions then we would not be able to communicate any uncertainty in our predictions. Yes, our mean and variance estimates are wrong (or perhaps, imperfect would be a better word), but reporting only a point estimate ignores all uncertainty, which is clearly much worse than making some simplifying assumptions.

$\endgroup$
1
  • $\begingroup$ Thank you for the response, I understand that if we have a suspicion that a random variable converges CLT is super valid. However, it is not always the case and there are random Variables for which this assumption is strong. I would be very interesting in ways to assert convergence . Simplifying analogy filling the ocean with 1 drop or 2 drops doesn't make you more sure of that. I appreciate and value pdf as the theoretical circle (useful although it doesn't exist) but can you use a drop-by-drop for problems that you don't know if they are ocean big? $\endgroup$
    – partizanos
    Nov 14, 2022 at 14:39
1
$\begingroup$

Apart from simulated examples, we never know the true distribution of the quantities we are looking at, whether these are point predictions or summary statistics like the AUROC you give as an example. We don't even know whether they satisfy the conditions for the Central Limit Theorem so that we could assume asymptotic normality - and even if we could, we rarely know how good this asymptotic approximation is.

However, this does not matter. Rarely does reporting a mean plus/minus a standard deviation imply a confidence interval. Rather, it is simply a description of the variability of the quantity being reported. Thus, it is an example of descriptive statistics, rather than inferential statistics.

Of course, if someone claims a confidence interval with specific properties, e.g., based on a normal distribution with estimated means and variances, then your point certainly holds: insofar as we don't know the true distribution, we have to trust in the CLT that we are at least approximately right.

This makes more sense in certain applications than in others, and knowing when it does and when it does not is part of the domain knowledge of specific areas of application in statistics. For instance, since the AUROC is bounded, it is certainly not Cauchy, and I would have no qualms about accepting an asymptotically normal distribution of AUROCs, with corresponding confidence intervals, as long as the sample size is large enough.

$\endgroup$
3
  • $\begingroup$ Thank you for the response. Indeed AUROC cannot be Cauchy, and it cannot be also Gaussian (which is also unbounded ). The "evil part" of Cauchy distribution is the fact that increasing the number of samples makes our score (in AUROC case) estimate to diverge. This could be a characteristic of the whole family of pdfs - I might be wrong but I guess there can exist some bounded - non-converging pdfs also. For a complex model where the random seeds enter in many places - leading partially to a replication crisis - how can one know what sample size is sufficient and drops-in -ocean situation. $\endgroup$
    – partizanos
    Nov 15, 2022 at 15:08
  • 1
    $\begingroup$ Yes, AUROC cannot be Cauchy nor Gaussian. But that does not matter. What is relevant is that average AUROC, averaged over many runs, will be asymptotically Gaussian, since the preconditions of the CLT are (IMO) met. For a bounded random variable not to be subject to the CLT would require something extremely strange to go on. $\endgroup$ Nov 15, 2022 at 15:12
  • $\begingroup$ I really appreciate the answers and the links. Some thoughts: for f(X) = Y, with f a model, X a set of random seeds in N, and Y the codomain AUC in [0,1] the a bounded interval. We have a "causality" X-> Y, do the preconditions of CLT is that the samples of the variable under study are i.i.d. . In the problem we discuss do we need to X and/ or Y random variables to be i.i.d? If it's only X then yes CLT conditions are met considering the usage of a random generator. But maybe Y (AUC scores) are not totally independent (mode,data, hyper/parameters) $\endgroup$
    – partizanos
    Nov 16, 2022 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.