How much similarity between observations and an event is needed in order to count them for calculating the probability of event happening?

Question

Say my goal is to predict whether event $X$ will happen, and all I have is observation $O$. Is obervation $O$ applicable to calculate $\Pr(X)$? E.g. how similar observations in $O$ need to be to the event $X$ in order to use $O$ in the calculations?

Let me put this in an example: $X$ is the event that a coin toss will return head. $O=\{o_1, o_2, \ldots, o_n\}$ shows the outcomes of past $n$ coin tosses. I could say this: $$ \Pr_O(X) = \frac{\text{number of heads in $O$}}{\text{number of observations in $O$}} $$

In the limit as $n\to \infty$, most people would agree with me that $\Pr_O(X) = \Pr(X)$ as observations in $O$ are highly related/similar to the event $X$.

$\Pr_O(X) = \Pr(X)$ assumes that observations in $O$ exactly apply to $X$, which might seem somewhat fair, but is not perfectly fair:

• Observations in $O$ were made using an older coin than the coin used in $E$.
• Coin is probably changing as it is being used.
• Temperature differences between when coins were tossed in $O$ and the future where event $X$ is expected to happen.
• ...

In other words, it seems to me that we are implying a notion of similarity. E.g. $\Pr_O(X) = \Pr(X)$ assumes that observations in $O$ are pefectly similar to the future coin toss that will happen (event $X$).

But, on the other hand, imagine if my past observations is $D = \{d_1, d_2, \ldots, d_m\}$, which represents whether my neighbour's dog barked in the past days. Then if I do this:

$$ \Pr_D(X) = \frac{\text{number of barking days in $D$}}{\text{number of observations in $D$}} $$

and claim that $\Pr_D(X) = \Pr(X)$, most people would tell me that I am out of my mind, because observations in $D$ are totally not similar to the event $X$ that will happen in the future! E.g. $D$ is about barking dogs, while $E$ is about tossing coins!

My question is: how similar should observations be to an event in order to conider the observations predictive for the event?

Answer 1

To address the question in your comment about formalism, there is some formalism to address what It seems like you're asking, but I don't think it will be very satisfying.

---

It will be useful to back up and establish some foundational language for what you're trying to describe.

assumes that observations in O exactly apply to X

This would mean that the observations in the set $O$ are generated by the same probability distribution as $X$. Every observation in the set $O$ is a draw from a random variable.

So the first element of $O$, $o_1$, is a draw from the random variable $O_1$. When we use $o_1$ to make inferences about the distribution of $X$, we are assuming that $O_1$ and $X$ follow the same probability distribution.

When $O$ contains a sequence of coin tosses, this assumption is easy to accept because of what we know about coins: they are durable, cannot be reshaped, cannot be biased, etc.

In plain English, we are comfortable using $O$ to make inferences about $X$ because we believe they are described by the same data-generating process. Past coin tosses and future coin tosses ought to be generated by the same process, or a very similar one.

dog barks

We are not comfortable using $D$ to make inferences about $X$ because we do not believe the data generating processes for dog barks and coin tosses are related.

---

Note that I used the word "similar" without defining it. It sounds like you have in mind something like this:

The data generating process of $Y_1$ is similar enough to the data generating process of $Y_2$ that we can make inferences about the distribution of $Y_2$ from observations of $Y_1$.

What you are asking for sounds a lot like the Kullback-Leibler divergence between the distributions of $Y_1$ and $Y_1$. Consider an given observation $y$, and the probabilities $\Pr(Y_1=y)$ and $\Pr(Y_2=y)$. If these probabilities are similar across all possible $y$ values, then the Kullback-Leibler divergence is low, indicating that the distributions are similar. If KL divergence is zero, the distributions are identical.

If $Y_1$ and $Y_2$ are both coin toss occurrences, their KL divergence will be very low. But if $Y$ is your neighbor's dog bark occurrences, the KL divergence will be high, since the probability of $y$ heads is different in general from the probability of $y$ dog barks.

Returning to the notation of the previous section: if the KL divergence between any $O_i$ and $X$ is low, then our observation $o_i$ should be useful for making inferences about $X$.

---

As an aside, in econometric data the "distribution similarity" problem is actually quite serious. In an econometric model fitted to annual data, you might have very few observations at your disposal. If the underlying data generating process changes (known as a "structural break" in the time series literature), then your model is in trouble. There are specific hypothesis tests that attempt to measure whether and where such a break occurs, but they are specific to time series modeling, and they involve on comparing residuals before and after a suspected break.