# How do I study the "correlation" between a continuous variable and a categorical variable?

What's a meaningful "correlation" measure to study the relation between the such two types of variables?

In R, how to do it?

• before you ask "how do you study", you should have the answer to "how do you define" :-) BTW, if you project the categorical variable to integer numbers, you can do correlation already. Commented May 30, 2012 at 16:40
• @Tomas, if you do that, the estimated strength of the relationship depends on how you've decided to label the points, which is kind of scary :) Commented May 30, 2012 at 16:47
• @Macro, you are right - another solid argument for having a good definition! Commented May 30, 2012 at 16:50
• @Macro Unless I have misunderstood your point, nope. Correlation is insensitive to linear transformations. So cor(X,Y) = cor(a+bX,Y) for finite a and b. The relabeling of a 0/1 as 1/11 does nothing to correlations using that var or its linear transformation. Commented May 24, 2018 at 3:45
• @Curious see my comment to Macro above. And note: (1) X <- sample(c(0,1),replace=TRUE,size=100) (2) Y <- X + rnorm(100,0.5) (3) corr(Y,X) (4) X <- 1 + 10*X (5) corr(X,Y) : same results for both correlations! Commented May 24, 2018 at 3:48

For a moment, let's ignore the continuous/discrete issue. Basically correlation measures the strength of the linear relationship between variables, and you seem to be asking for an alternative way to measure the strength of the relationship. You might be interested in looking at some ideas from information theory. Specifically I think you might want to look at mutual information. Mutual information essentially gives you a way to quantify how much knowing the state of one variable tells you about the other variable. I actually think this definition is closer to what most people mean when they think about correlation.

For two discrete variables X and Y, the calculation is as follows: $$I(X;Y) = \sum_{y \in Y} \sum_{x \in X} p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) }$$

For two continuous variables we integrate rather than taking the sum: $$I(X;Y) = \int_Y \int_X p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) } \; dx \,dy$$

Your particular use-case is for one discrete and one continuous. Rather than integrating over a sum or summing over an integral, I imagine it would be easier to convert one of the variables into the other type. A typical way to do that would be to discretize your continuous variable into discrete bins.

There are a number of ways to discretzie data (e.g. equal intervals), and I believe the entropy package should be helpful for the MI calculations if you want to use R.

• Thanks. But how high an MI is corresponding to the corr=1 and how low an MI corresponds to corr=0?
– Luna
Commented May 31, 2012 at 17:58
• MI has a minimum of 0, and MI = 0 if and only if the variables are independent. MI has no constant upper-bound though (the upper-bound is related to the entropies of the variables), so you might want to look at one of the normalized versions if that is important to you. Commented May 31, 2012 at 19:41

If the categorical variable is ordinal and you bin the continuous variable into a few frequency intervals you can use Gamma. Also available for paired data put into ordinal form are Kendal's tau, Stuart's tau and Somers D. These are all available in SAS using Proc Freq. I don't know how they are computed using R functions. Here is a link to a presentation that gives detailed information: http://faculty.unlv.edu/cstream/ppts/QM722/measuresofassociation.ppt#260,5,Measures of Association for Nominal and Ordinal Variables

A categorical variable is effectively just a set of indicator variable. It is a basic idea of measurement theory that such a variable is invariant to relabelling of the categories, so it does not make sense to use the numerical labelling of the categories in any measure of the relationship between another variable (e.g., 'correlation'). For this reason, and measure of the relationship between a continuous variable and a categorical variable should be based entirely on the indicator variables derived from the latter.

Given that you want a measure of 'correlation' between the two variables, it makes sense to look at the correlation between a continuous random variable $X$ and an indicator random variable $I$ derived from t a categorical variable. Letting $\phi \equiv \mathbb{P}(I=1)$ we have:

$$\mathbb{Cov}(I,X) = \mathbb{E}(IX) - \mathbb{E}(I) \mathbb{E}(X) = \phi \left[ \mathbb{E}(X|I=1) - \mathbb{E}(X) \right] ,$$

which gives:

$$\mathbb{Corr}(I,X) = \sqrt{\frac{\phi}{1-\phi}} \cdot \frac{\mathbb{E}(X|I=1) - \mathbb{E}(X)}{\mathbb{S}(X)} .$$

So the correlation between a continuous random variable $X$ and an indicator random variable $I$ is a fairly simple function of the indicator probability $\phi$ and the standardised gain in expected value of $X$ from conditioning on $I=1$. Note that this correlation does not require any discretization of the continuous random variable.

For a general categorical variable $C$ with range $1, ..., m$ you would then just extend this idea to have a vector of correlation values for each outcome of the categorical variable. For any outcome $C=k$ we can define the corresponding indicator $I_k \equiv \mathbb{I}(C=k)$ and we have:

$$\mathbb{Corr}(I_k,X) = \sqrt{\frac{\phi_k}{1-\phi_k}} \cdot \frac{\mathbb{E}(X|C=k) - \mathbb{E}(X)}{\mathbb{S}(X)} .$$

We can then define $\mathbb{Corr}(C,X) \equiv (\mathbb{Corr}(I_1,X), ..., \mathbb{Corr}(I_m,X))$ as the vector of correlation values for each category of the categorical random variable. This is really the only sense in which it makes sense to talk about 'correlation' for a categorical random variable.

(Note: It is trivial to show that $\sum_k \mathbb{Cov}(I_k,X) = 0$ and so the correlation vector for a categorical random variable is subject to this constraint. This means that given knowledge of the probability vector for the categorical random variable, and the standard deviation of $X$, you can derive the vector from any $m-1$ of its elements.)

The above exposition is for the true correlation values, but obviously these must be estimated in a given analysis. Estimating the indicator correlations from sample data is simple, and can be done by substitution of appropriate estimates for each of the parts. (You could use fancier estimation methods if you prefer.) Given sample data $(x_1, c_1), ..., (x_n, c_n)$ we can estimate the parts of the correlation equation as:

$$\hat{\phi}_k \equiv \frac{1}{n} \sum_{i=1}^n \mathbb{I}(c_i=k).$$

$$\hat{\mathbb{E}}(X) \equiv \bar{x} \equiv \frac{1}{n} \sum_{i=1}^n x_i.$$

$$\hat{\mathbb{E}}(X|C=k) \equiv \bar{x}_k \equiv \frac{1}{n} \sum_{i=1}^n x_i \mathbb{I}(c_i=k) \Bigg/ \hat{\phi}_k .$$

$$\hat{\mathbb{S}}(X) \equiv s_X \equiv \sqrt{\frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2}.$$

Substitution of these estimates would yield a basic estimate of the correlation vector. If you have parametric information on $X$ then you could estimate the correlation vector directly by maximum likelihood or some other technique.

• Thank you for your answer. I don't have strong statistics background, but is there any guarantee $\hat{\mathbb{E}}(X\vert C=k)\geq \hat{\mathbb{E}}(X)$ (which makes correlation unnegative)? I implemented your approach with some synthetic data, it turns out that some correlations are negative. I am not sure if my implementation is correct. Would it be possible a numerical example provided in your answer? Commented Sep 17, 2020 at 19:49
• There is no guarantee that correlation is non-negative, so don't worry if you are getting some negative values. For error-checking purposes, you should bear in mind that correlation is between $-1$ and $1$ (so if you are getting values outside that range then something has gone wrong).
– Ben
Commented Sep 17, 2020 at 22:36
• Thanks for your clarification. I mistaken correlation for $R^2$. Commented Sep 17, 2020 at 22:50
• Can you take a look at my another question on how to interpret the correlation value? Commented Sep 18, 2020 at 3:04

If $$X$$ is a continuous random variable and $$Y$$ is a categorical r.v., the observed correlation between $$X$$ and $$Y$$ can be measured by

1. the point-biserial correlation coefficient, if $$Y$$ is dichotomous;
2. the point-polyserial correlation coefficient, if $$Y$$ is polychotomous with ordinal categories.

It should be noted, though, that the point-polyserial correlation is just a generalization of the point-biserial.

For a broader view, here's a table from Olsson, Drasgow & Dorans (1982)[1].

[1]: Source: Olsson, U., Drasgow, F., & Dorans, N. J. (1982). The polyserial correlation coefficient. Psychometrika, 47(3), 337–347

R package mpmi has the ability to calculate mutual information for the mixed variable case, namely continuous and discrete. Although there are other statistical options like (point) biserial correlation coefficient to be useful here, it would be beneficial and highly recommended to calculate mutual information since it can detect associations other than linear and monotonic.