How do I study the "correlation" between a continuous variable and a categorical variable? - Cross Validated most recent 30 from stats.stackexchange.com 2019-06-26T08:14:21Z https://stats.stackexchange.com/feeds/question/29489 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/29489 18 How do I study the "correlation" between a continuous variable and a categorical variable? Luna https://stats.stackexchange.com/users/8783 2012-05-30T15:22:59Z 2018-10-24T15:35:30Z <p>What's a meaningful "correlation" measure to study the relation between the such two types of variables?</p> <p>In R, how to do it?</p> https://stats.stackexchange.com/questions/29489/-/29494#29494 6 Answer by Michael Chernick for How do I study the "correlation" between a continuous variable and a categorical variable? Michael Chernick https://stats.stackexchange.com/users/11032 2012-05-30T16:09:50Z 2012-05-30T16:09:50Z <p>If the categorical variable is ordinal and you bin the continuous variable into a few frequency intervals you can use Gamma. Also available for paired data put into ordinal form are Kendal's tau, Stuart's tau and Somers D. These are all available in SAS using Proc Freq. I don't know how they are computed using R routines. Here is a link to a presentation that gives detailed information: <a href="http://faculty.unlv.edu/cstream/ppts/QM722/measuresofassociation.ppt#260,5,Measures" rel="noreferrer">http://faculty.unlv.edu/cstream/ppts/QM722/measuresofassociation.ppt#260,5,Measures</a> of Association for Nominal and Ordinal Variables</p> https://stats.stackexchange.com/questions/29489/-/29505#29505 19 Answer by Michael McGowan for How do I study the "correlation" between a continuous variable and a categorical variable? Michael McGowan https://stats.stackexchange.com/users/2485 2012-05-30T19:14:38Z 2012-05-30T19:14:38Z <p>For a moment, let's ignore the continuous/discrete issue. Basically correlation measures the strength of the linear relationship between variables, and you seem to be asking for an alternative way to measure the strength of the relationship. You might be interested in looking at some ideas from <a href="http://en.wikipedia.org/wiki/Information_theory">information theory</a>. Specifically I think you might want to look at <a href="http://en.wikipedia.org/wiki/Mutual_information">mutual information</a>. Mutual information essentially gives you a way to quantify how much knowing the state of one variable tells you about the other variable. I actually think this definition is closer to what most people mean when they think about correlation.</p> <p>For two discrete variables X and Y, the calculation is as follows: $$I(X;Y) = \sum_{y \in Y} \sum_{x \in X} p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) }$$</p> <p>For two continuous variables we integrate rather than taking the sum: $$I(X;Y) = \int_Y \int_X p(x,y) \log{ \left(\frac{p(x,y)}{p(x)\,p(y)} \right) } \; dx \,dy$$</p> <p>Your particular use-case is for one discrete and one continuous. Rather than integrating over a sum or summing over an integral, I imagine it would be easier to convert one of the variables into the other type. A typical way to do that would be to <a href="http://en.wikipedia.org/wiki/Discretization_of_continuous_features">discretize</a> your continuous variable into discrete bins.</p> <p>There are a number of ways to discretzie data (e.g. equal intervals), and I believe <a href="http://cran.r-project.org/web/packages/entropy/index.html">the entropy package</a> should be helpful for the MI calculations if you want to use R.</p> https://stats.stackexchange.com/questions/29489/-/324950#324950 1 Answer by Ben for How do I study the "correlation" between a continuous variable and a categorical variable? Ben https://stats.stackexchange.com/users/173082 2018-01-25T02:15:39Z 2018-01-25T02:15:39Z <p>A categorical variable is effectively just a set of indicator variable. It is a basic idea of measurement theory that such a variable is invariant to relabelling of the categories, so it does not make sense to use the numerical labelling of the categories in any measure of the relationship between another variable (e.g., 'correlation'). For this reason, and measure of the relationship between a continuous variable and a categorical variable should be based entirely on the indicator variables derived from the latter.</p> <p>Given that you want a measure of 'correlation' between the two variables, it makes sense to look at the correlation between a continuous random variable $X$ and an indicator random variable $I$ derived from t a categorical variable. Letting $\phi \equiv \mathbb{P}(I=1)$ we have:</p> <p>$$\mathbb{Cov}(I,X) = \mathbb{E}(IX) - \mathbb{E}(I) \mathbb{E}(X) = \phi \left[ \mathbb{E}(X|I=1) - \mathbb{E}(X) \right] ,$$</p> <p>which gives:</p> <p>$$\mathbb{Corr}(I,X) = \sqrt{\frac{\phi}{1-\phi}} \cdot \frac{\mathbb{E}(X|I=1) - \mathbb{E}(X)}{\mathbb{S}(X)} .$$</p> <p>So the correlation between a continuous random variable $X$ and an indicator random variable $I$ is a fairly simple function of the indicator probability $\phi$ and the standardised gain in expected value of $X$ from conditioning on $I=1$. Note that this correlation does not require any discretization of the continuous random variable.</p> <hr> <p>For a general categorical variable $C$ with range $1, ..., m$ you would then just extend this idea to have a <em>vector</em> of correlation values for each outcome of the categorical variable. For any outcome $C=k$ we can define the corresponding indicator $I_k \equiv \mathbb{I}(C=k)$ and we have:</p> <p>$$\mathbb{Corr}(I_k,X) = \sqrt{\frac{\phi_k}{1-\phi_k}} \cdot \frac{\mathbb{E}(X|C=k) - \mathbb{E}(X)}{\mathbb{S}(X)} .$$</p> <p>We can then define $\mathbb{Corr}(C,X) \equiv (\mathbb{Corr}(I_1,X), ..., \mathbb{Corr}(I_m,X))$ as the vector of correlation values for each category of the categorical random variable. This is really the only sense in which it makes sense to talk about 'correlation' for a categorical random variable.</p> <p>(<strong>Note:</strong> It is trivial to show that $\sum_k \mathbb{Cov}(I_k,X) = 0$ and so the correlation vector for a categorical random variable is subject to this constraint. This means that given knowledge of the probability vector for the categorical random variable, and the standard deviation of $X$, you can derive the vector from any $m-1$ of its elements.)</p> <hr> <p>The above exposition is for the true correlation values, but obviously these must be estimated in a given analysis. Estimating the indicator correlations from sample data is simple, and can be done by substitution of appropriate estimates for each of the parts. (You could use fancier estimation methods if you prefer.) Given sample data $(x_1, c_1), ..., (x_n, c_n)$ we can estimate the parts of the correlation equation as:</p> <p>$$\hat{\phi}_k \equiv \frac{1}{n} \sum_{i=1}^n \mathbb{I}(c_i=k).$$</p> <p>$$\hat{\mathbb{E}}(X) \equiv \bar{x} \equiv \frac{1}{n} \sum_{i=1}^n x_i.$$</p> <p>$$\hat{\mathbb{E}}(X|C=k) \equiv \bar{x}_k \equiv \frac{1}{n} \sum_{i=1}^n x_i \mathbb{I}(c_i=k) \Bigg/ \hat{\phi}_k .$$</p> <p>$$\hat{\mathbb{S}}(X) \equiv s_X \equiv \sqrt{\frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2}.$$</p> <p>Substitution of these estimates would yield a basic estimate of the correlation vector. If you have parametric information on $X$ then you could estimate the correlation vector directly by maximum likelihood or some other technique.</p> https://stats.stackexchange.com/questions/29489/-/347924#347924 0 Answer by siyisoy for How do I study the "correlation" between a continuous variable and a categorical variable? siyisoy https://stats.stackexchange.com/users/200849 2018-05-24T03:32:45Z 2018-05-24T03:32:45Z <p>R package mpmi has the ability to calculate mutual information for mixed variable case, namely continuous and discrete. Although there are other statistical options like (point) biserial correlation coefficient to be useful here, it would be beneficial and highly recommended to calculate mutual information since it can detect associations other than linear and monotonic.</p> https://stats.stackexchange.com/questions/29489/-/373579#373579 0 Answer by Waldir Leoncio for How do I study the "correlation" between a continuous variable and a categorical variable? Waldir Leoncio https://stats.stackexchange.com/users/27433 2018-10-24T15:35:30Z 2018-10-24T15:35:30Z <p>If <span class="math-container">$X$</span> is a continuous random variable and <span class="math-container">$Y$</span> is a categorical r.v.. the observed correlation between <span class="math-container">$X$</span> and <span class="math-container">$Y$</span> can be measured by</p> <ol> <li>the <strong>point-biserial correlation</strong> coefficient, if <span class="math-container">$Y$</span> is dichotomous;</li> <li>the <strong>point-polyserial correlation</strong> coefficient, if <span class="math-container">$Y$</span> is polychotomous with ordinal categories.</li> </ol> <p>It should be noted, though, that the point-polyserial correlation is just a generalization of the point-biserial.</p> <p>For a broader view, here's a table from Olsson, Drasgow &amp; Dorans (1982).</p> <p><a href="https://i.stack.imgur.com/K1o25.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/K1o25.png" alt="correlation coefficients"></a></p> <p>: Source: Olsson, U., Drasgow, F., &amp; Dorans, N. J. (1982). The polyserial correlation coefficient. Psychometrika, 47(3), 337–347</p>