# Is my interpretation of "the probability over data $X_1, ... X_N$ correct?

This may seem like a pretty simple question, but I want to make sure I am getting this right because it seems pretty foundational.

I'm reading this note on conformal prediction. In the very first section the scenario is this:

We collect i.i.d pairs $$(X_i, Y_i)$$, $$i=1,\dots,n$$ from distribution $$P$$ on $$\boldsymbol{\mathcal{X}} \times \boldsymbol{\mathcal{Y}}$$. Given a specified "error tolerance" $$\alpha \in (0, 1)$$, the goal is to find a prediction set

$$C_\alpha(X) : \boldsymbol{\mathcal{X}} \to \{\text{subsets of }\boldsymbol{\mathcal{Y}}\}$$

with the property that for a new i.i.d pair ($$X_{n+1}, Y_{n+1}$$):

$$\mathbb{P}(Y_{n+1} \in C_{\alpha}(X_{n+1})) \ge 1 - \alpha \tag{1}$$

where the probability is taken over all of our data $$(X_1, Y_1) \dots (X_{n+1}, Y_{n+1})$$.

This last line is where I want to focus on. This is my current interpretation, and please let me know if it is wildly incorrect or correct!

First let $$x_i$$ and $$y_i$$ denote a particular fixed realization of $$X_i$$ and $$Y_i$$; for example by going out and collecting data or measuring or observing. How I think about this line is the following: let's say I go out and collect the data $$(x_1, y_1), \dots (x_n, y_n), (x_{n+1}, y_{n+1})$$. I plug $$x_{n+1}$$ into my prediction set function $$C_{\alpha}$$ and get a prediction set. I then check if $$y_{n+1}$$ is in that prediction set. If it is I write down a check mark; if it is not I write down an x.

The property $$(1)$$ is saying that if I run this same exact experiment an infinite number of times (i.e. go collect the data, etc.), the proportion between check marks and x's will be $$1-\alpha$$.

Your understanding is correct insofar as the probability refers to repeated realisations of the whole sequence $$((X_1,Y_1),\ldots,(X_{n+1},Y_{n+1})$$. You are probably aware though that property (1) will be fulfilled if data are generated from an idealised model ($$P$$); what actually happens if you repeat experiments in reality is anyone's guess.