This is a little bit of a gut check, please do help me see if I'm misunderstanding this concept, and in what way.

I have a functional understanding of correlation but I'm feeling a little grasping-at-straws to really confidently explain the principles behind that functional understanding.

As I understand it, statistical correlation (as opposed to the more general usage of the term) is a way to understand two continuous variables and the way in which they do or do not tend to rise or fall in similar ways.

The reason you can't run correlations on, say, one continuous and one categorical variable is because it's not possible to calculate the covariance between the two, since the categorical variable by definition cannot yield a mean, and thus cannot even enter into the first steps of the statistical analysis.

Is that right?

  • 2
    $\begingroup$ Here are typed up lectures slides from a class I teach mostly dealing with population (not sample) correlation and covariance people.virginia.edu/~trb5me/3120_slides/5/5.2/5.2.pdf $\endgroup$ – Taylor Jan 15 '17 at 17:54
  • 3
    $\begingroup$ Simple reason, imagine that you ask people "what is your favorite color?" and they answer "red", "green", "blue", "orange", "yellow", ... , what is coded in your dataset as 1, 2, 3, ... Next, you calculate correlation coefficient between such variable with job satisfaction and get value 0.21. What does it mean? Could you provide any meaningful interpretation? $\endgroup$ – Tim Jan 15 '17 at 21:13
  • 2
    $\begingroup$ Closely related (perhaps even a duplicate?) - Correlation between a nominal (IV) and a continuous (DV) variable $\endgroup$ – Silverfish Jan 16 '17 at 2:49

Correlation is the standardized covariance, i.e the covariance of $x$ and $y$ divided by the standard deviation of $x$ and $y$. Let me illustrate that.

Loosely speaking, statistics can be summarized as fitting models to data and assessing how well the model describes those data points (Outcome = Model + Error). One way to do that is to calculate the sums of deviances, or residuals (res) from the model:

$res= \sum(x_{i}-\bar{x})$

Many statistical calculations are based on this, incl. the correlation coefficient (see below).

Here is an example dataset made in R (the residuals are indicated as red lines and their values added next to them):

X <- c(8,9,10,13,15)  
Y <- c(5,4,4,6,8)

enter image description here

By looking at each data point individually and subtracting its value from the model (e.g. the mean; in this case X=11 and Y=5.4), one could assess the accuracy of a model. One could say the model over-/ underestimated the actual value. However, when summing up all the deviances from the model, the total error tends to be zero, the values cancel each other out because there are positive values (the model underestimates a particular data point) and negative values (the model overestimates a particular data point). To solve this problem the sums of deviances are squared and now called sums of squares ($SS$):

$SS = \sum(x_i-\bar{x})(x_i-\bar{x}) = \sum(x_i-\bar{x})^2$

The sums of squares are a measure of deviation from the model (i.e. the mean or any other fitted line to a given dataset). Not very helpful for interpreting the deviance from the model (and comparing it with other models) since it is dependent on the number of observations. The more observations the higher the sums of squares. This can be taken care of by dividing the sums of square with $n-1$. The resulting sample variance ($s^2$) becomes the "average error" between the mean and the observations and is therefore a measure of how well the model fits (i.e. represents) the data:

$s^2 = \frac{SS}{n-1} = \frac{\sum(x_i-\bar{x})(x_i-\bar{x})}{n-1} = \frac{\sum(x_i-\bar{x})^2}{n-1}$

For convenience, the square root of the sample variance can be taken, which is known as the sample standard deviation:


Now, the covariance assesses whether two variables are related to each other. A positive value indicates that as one variable deviates from the mean, the other variable deviates in the same direction.

$cov_{x,y}= \frac{\sum(x_i-\bar{x})(y_i-\bar{y})}{n-1}$

By standardizing, we express the covariance per unit standard deviation, which is the Pearson correlation coefficient $r$. This allows comparing variables with each other that were measured in different units. The correlation coefficient is a measure of the strength of a relationship ranging from -1 (a perfect negative correlation) to 0 (no correlation) and +1 (a perfect positive correlation).

$r=\frac{cov_{x,y}}{s_x s_y} = \frac{\sum(x_1-\bar{x})(y_i-\bar{y})}{(n-1) s_x s_y}$

In this, case the Pearson correlation coefficient is $r=0.87$, which can be considered a strong correlation (although this is also relative depending on the field of study). To check this, here another plot with X on the x-axis and Y on the y axis:

enter image description here

So long story short, yes your feeling is right but I hope my answer can provide some context.

  • 1
    $\begingroup$ This is super helpful - in trying to deepen my own understanding, I figure if I can't sufficiently explain it to someone without a background in statistics, I don't understand it as well as I thought. $\endgroup$ – Toof Jan 17 '17 at 0:48

You are (nearly) right. Covariance (and therefore correlation too) can be computed only between numerical variables. That includes continuous variables but also discrete numerical variables.

Categorical variables could be used to compute correlation only given a useful numerical code for them, but this is not likely to get a practical advantage - maybe it could be useful for some two levels categorical variables, but other tools are likely to be more suitable.

  • $\begingroup$ To add to pere's point the Pearson product moment correlation coefficient represents the degree of a linear relationship between the two variables. Nonparametric measures like Spearman's rho or Kendall's tau characterize how much of a tendency there is for X and Y to increase or decrease together (behave to a degree like a monotonic relationship that need not necessarily be linear. $\endgroup$ – Michael Chernick Jan 15 '17 at 15:23

There is absolutely nothing wrong with computing correlations where one of the variables is categorical. A strong positive correlation would imply that turning your categorical variable on (or off depending on your convention) causes an increase in the response. For example this could happen when calculating a logistic regression where variables are categorical: predicting the chance of a heart attack given patient comorbidities like diabetes and bmi. In this case BMI would have would have a very strong correlation with heart attacks. Would you conclude that's not useful?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.