This may seem like a pretty simple question, but I want to make sure I am getting this right because it seems pretty foundational.

I'm reading this note on conformal prediction. In the very first section the scenario is this:

We collect i.i.d pairs $(X_i, Y_i)$, $i=1,\dots,n$ from distribution $P$ on $\boldsymbol{\mathcal{X}} \times \boldsymbol{\mathcal{Y}}$. Given a specified "error tolerance" $\alpha \in (0, 1)$, the goal is to find a prediction set

$$ C_\alpha(X) : \boldsymbol{\mathcal{X}} \to \{\text{subsets of }\boldsymbol{\mathcal{Y}}\} $$

with the property that for a new i.i.d pair ($X_{n+1}, Y_{n+1}$):

$$ \mathbb{P}(Y_{n+1} \in C_{\alpha}(X_{n+1})) \ge 1 - \alpha \tag{1} $$

where the probability is taken over all of our data $(X_1, Y_1) \dots (X_{n+1}, Y_{n+1})$.

This last line is where I want to focus on. This is my current interpretation, and please let me know if it is wildly incorrect or correct!

First let $x_i$ and $y_i$ denote a particular fixed realization of $X_i$ and $Y_i$; for example by going out and collecting data or measuring or observing. How I think about this line is the following: let's say I go out and collect the data $(x_1, y_1), \dots (x_n, y_n), (x_{n+1}, y_{n+1})$. I plug $x_{n+1}$ into my prediction set function $C_{\alpha}$ and get a prediction set. I then check if $y_{n+1}$ is in that prediction set. If it is I write down a check mark; if it is not I write down an x.

The property $(1)$ is saying that if I run this same exact experiment an infinite number of times (i.e. go collect the data, etc.), the proportion between check marks and x's will be $1-\alpha$.

Thanks in advance!


1 Answer 1


Your understanding is correct insofar as the probability refers to repeated realisations of the whole sequence $((X_1,Y_1),\ldots,(X_{n+1},Y_{n+1})$. You are probably aware though that property (1) will be fulfilled if data are generated from an idealised model ($P$); what actually happens if you repeat experiments in reality is anyone's guess.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.