How much similarity between observations and an event is needed in order to count them for calculating the probability of event happening?

Say my goal is to predict whether event $$X$$ will happen, and all I have is observation $$O$$. Is obervation $$O$$ applicable to calculate $$\Pr(X)$$? E.g. how similar observations in $$O$$ need to be to the event $$X$$ in order to use $$O$$ in the calculations?

Let me put this in an example: $$X$$ is the event that a coin toss will return head. $$O=\{o_1, o_2, \ldots, o_n\}$$ shows the outcomes of past $$n$$ coin tosses. I could say this: $$\Pr_O(X) = \frac{\text{number of heads in O}}{\text{number of observations in O}}$$

In the limit as $$n\to \infty$$, most people would agree with me that $$\Pr_O(X) = \Pr(X)$$ as observations in $$O$$ are highly related/similar to the event $$X$$.

$$\Pr_O(X) = \Pr(X)$$ assumes that observations in $$O$$ exactly apply to $$X$$, which might seem somewhat fair, but is not perfectly fair:

• Observations in $$O$$ were made using an older coin than the coin used in $$E$$.
• Coin is probably changing as it is being used.
• Temperature differences between when coins were tossed in $$O$$ and the future where event $$X$$ is expected to happen.
• ...

In other words, it seems to me that we are implying a notion of similarity. E.g. $$\Pr_O(X) = \Pr(X)$$ assumes that observations in $$O$$ are pefectly similar to the future coin toss that will happen (event $$X$$).

But, on the other hand, imagine if my past observations is $$D = \{d_1, d_2, \ldots, d_m\}$$, which represents whether my neighbour's dog barked in the past days. Then if I do this:

$$\Pr_D(X) = \frac{\text{number of barking days in D}}{\text{number of observations in D}}$$

and claim that $$\Pr_D(X) = \Pr(X)$$, most people would tell me that I am out of my mind, because observations in $$D$$ are totally not similar to the event $$X$$ that will happen in the future! E.g. $$D$$ is about barking dogs, while $$E$$ is about tossing coins!

My question is: how similar should observations be to an event in order to conider the observations predictive for the event?

• Could you clarify what you mean by "similarity" of observations?
– whuber
Commented Oct 22, 2018 at 13:45
• I am being very informal. Basically, it seems that if I wish to calculate the probability of a simple event like a future coin toss giving "heads", I can look at past coin tosses, and then calculate the probability of the future coin toss giving "heads". Most people are happy with this approach, because they think the past coin tosses are similar/related enough to the future coin toss. But I never heard any formalism around this similarity/relevance. My question is about this similarity/relevance aspect. Any formalism around this aspect? Commented Oct 22, 2018 at 23:10
• Should I rewrite my question :/ ? My question is simple: why is counting past coin toss outcomes used for predicting future coin tosses, but using dog barks isn't? Technically speaking, past coin tosses are not necessarily related to future coin tosses. But it seems that we consider certain jumps/assumptions legal. E.g. assuming past coin toss is the same. Is there any formalism around this? Commented Oct 24, 2018 at 10:33

To address the question in your comment about formalism, there is some formalism to address what It seems like you're asking, but I don't think it will be very satisfying.

It will be useful to back up and establish some foundational language for what you're trying to describe.

assumes that observations in O exactly apply to X

This would mean that the observations in the set $$O$$ are generated by the same probability distribution as $$X$$. Every observation in the set $$O$$ is a draw from a random variable.

So the first element of $$O$$, $$o_1$$, is a draw from the random variable $$O_1$$. When we use $$o_1$$ to make inferences about the distribution of $$X$$, we are assuming that $$O_1$$ and $$X$$ follow the same probability distribution.

When $$O$$ contains a sequence of coin tosses, this assumption is easy to accept because of what we know about coins: they are durable, cannot be reshaped, cannot be biased, etc.

In plain English, we are comfortable using $$O$$ to make inferences about $$X$$ because we believe they are described by the same data-generating process. Past coin tosses and future coin tosses ought to be generated by the same process, or a very similar one.

dog barks

We are not comfortable using $$D$$ to make inferences about $$X$$ because we do not believe the data generating processes for dog barks and coin tosses are related.

Note that I used the word "similar" without defining it. It sounds like you have in mind something like this:

The data generating process of $$Y_1$$ is similar enough to the data generating process of $$Y_2$$ that we can make inferences about the distribution of $$Y_2$$ from observations of $$Y_1$$.

What you are asking for sounds a lot like the Kullback-Leibler divergence between the distributions of $$Y_1$$ and $$Y_1$$. Consider an given observation $$y$$, and the probabilities $$\Pr(Y_1=y)$$ and $$\Pr(Y_2=y)$$. If these probabilities are similar across all possible $$y$$ values, then the Kullback-Leibler divergence is low, indicating that the distributions are similar. If KL divergence is zero, the distributions are identical.

If $$Y_1$$ and $$Y_2$$ are both coin toss occurrences, their KL divergence will be very low. But if $$Y$$ is your neighbor's dog bark occurrences, the KL divergence will be high, since the probability of $$y$$ heads is different in general from the probability of $$y$$ dog barks.

Returning to the notation of the previous section: if the KL divergence between any $$O_i$$ and $$X$$ is low, then our observation $$o_i$$ should be useful for making inferences about $$X$$.

As an aside, in econometric data the "distribution similarity" problem is actually quite serious. In an econometric model fitted to annual data, you might have very few observations at your disposal. If the underlying data generating process changes (known as a "structural break" in the time series literature), then your model is in trouble. There are specific hypothesis tests that attempt to measure whether and where such a break occurs, but they are specific to time series modeling, and they involve on comparing residuals before and after a suspected break.

• Thanks. About KL divergergennce, thing is we don't know X in my view. By the time we toss coins to measure X, we get Y. If we toss coins again to get X, we get yet something else, Z. Q: is there any better formalism? E.g. since the next coin toss is scheduled in 5 minutes, let us keep tossing the coin in 5 minutes intervals, and in doing so, we also model gradual outcome changes/shifts of the coin's tosses. Then, we create 2 things: 1. $\Pr(Y)$, and 2. a domain adaptation model that modifies $Y$ into $\hat Y$ so that (predict_toss_$\hat Y$() - test())^2 minimized? Commented Oct 25, 2018 at 1:13
• @caveman I'm not sure what you mean. If we already have measurements for $X$ then we don't need to toss any more coins to estimate $\Pr(X=z)$. If you don't have measurements for $X$ then $X$ is irrelevant anyway. I'm also not sure I understand what you mean by the domain adaptation model. Commented Oct 25, 2018 at 2:14
• I think what you're asking about is if there is any way to know if $X$ is suitable for making inferences about $Y$ before we have made any observations about $Y$. For that, there is no formalism. You have to make the assumption, and convince other people that your assumption makes sense. "All models are wrong but some are useful". Commented Oct 25, 2018 at 2:16