I've heard correlation for two variables or two random variables (strong or weak dependence). But do we have such concepts for events? Can two events, such as rainy weather and cloudy sky be correlated? Can they be weakly or strongly dependent? In each case how can one determine the degree of dependence?


Covariance and correlation that is just its normalized version, measure the linear relationship between two variables. A linear relationship is just one of the many possible relationships, so they cannot be thought of as a general measure of how strongly the two variables are related, though it's a popular proxy for measuring it.

If you have two events $X$ and $Y$, the most basic way of measuring how strongly they co-occur, is to look at their joint probability $P(X \cap Y)$, the probability of observing $X$ and $Y$ together. It's easy to interpret, since the probability is bounded between zero and one, so the closer to one, the stronger is the relationship.

If the events were independent of each other, the probability would be $P(X \cap Y) = P(X)P(Y)$ by definition. So joint probability higher than this means that the events are more likely to occur together than if they were independent, if lower, less likely as compared to case if they were independent. That’s why you can see the $\tfrac{P(X \cap Y) }{P(X)P(Y)}$ part in metrics like in mutual information or pointwise mutual information.

  • $\begingroup$ I found en.wikipedia.org/wiki/Pointwise_mutual_information another good measure! It compares the joint probability with the lower bound you mentioned (P(X)P(Y)) but since it can be negative, I think P(x)P(Y) then isn't a lower bound, not? $\endgroup$
    – Ahmad
    May 3 at 5:39
  • $\begingroup$ @Ahmad Wikipedia mentions the bounds for PMI. $\endgroup$
    – Tim
    May 3 at 5:54
  • $\begingroup$ I know that it says it's bound for PMI, but PMI = log P(x,y) / P(x)P(y)... in order that it be negative P(x,y) must be smaller than P(x)P(y), not? Then it violates what you said about the lower bound of P(x,y). $\endgroup$
    – Ahmad
    May 3 at 6:21
  • $\begingroup$ @Ahmad yes, it can be smaller if there is a smaller probability of X and Y appearing together as compared to the case if they are independent. In such case, the strength of the relationship between them obviously isn’t strong. I meant lower bound for strongly related ones, below this, the relationship is not strong & positive. $\endgroup$
    – Tim
    May 3 at 6:28
  • $\begingroup$ thank I thought that's the lower bound for joint probability. Anyways, I myself posted my understanding. $\endgroup$
    – Ahmad
    May 3 at 7:27

Pointwsei mutual information is a measure of association between two single events.

$$\operatorname{pmi}(x;y) \equiv \log\frac{p(x,y)}{p(x)p(y)} = \log\frac{p(x|y)}{p(x)} = \log\frac{p(y|x)}{p(y)}$$

As you can see, it compares the conditional probability of $P(x|y)$ to $P(x)$, the probability of occurring $x$. It also compares their joint probability with their marginalized probability under the independence assumption.

It can be positive, like pmi of rainy weather and cloudy sky, which implies that $P(rainy\ weather | cloudy\ sky)$ is more than $p(rainy\ weather)$ without any condition. It means that rainy weather tends to co-occur with cloudy sky. However pmi is symmetric, and it can be said about cloudy sky conditioned on rainy weather.

It can be zero, when they are independent, like rainy weather and the probability of my garbage being collected today ( I found the example online, don't we have better examples?)

And it can be even negative, when they tend not to co-occur like rainy weather and clear sky. The positive and negative dependence could be again weak or strong depends on how positive or negative they are.

  • 1
    $\begingroup$ Correct. But if you look at the formula, it’s just a log-transformed, scaled joint probability. In some cases it might be more useful than raw probabilities, but in other not necessarily. $\endgroup$
    – Tim
    May 3 at 7:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.