# Measure correlation for distance and binary outcome

I know enough about statistics to know that I don't know enough about statistics. This is therefore a humble question, and apologies in advance for asking a question that is likely to be missing key information or phrased badly.

I have two variables.

• One is a real-world metre distance between two points, and is therefore always >= 0. It doesn't seem to be normally distributed.
• The other is an outcome, with two values. It's not a 50:50 split.

I want to see if there is a correlation between these. Could someone advise me on a good way to do this?

A sample set of data is here. I'm providing this not out of laziness, but because it might be that there is some simple characteristic of this information which someone might want to know, and this might be quicker than asking me to provide it.

I would also be interested in the correct way to have phrased this question, or relevant tags.

First, note that your second column is a categorical variable, i.e. Yes/No, Up/Down in your case.

So, a first step will be to convert your non-binary data in the "Outcome" column to binary $0-1$ variables.

An Excel Implementation:

The next step will be to use the CORREL function in excel, to compute the correlation between your two types of variables - Distance and Outcome. To do so, apply =CORREL(A2:A3417,B2:B3417) in any empty cell (after you have converted your UP/Down into 1/0 respectively!) and you will obtain the correlation as desired.

R Implementation

If you try out the straightforward way in excel or by invoking cor(x,y) in R, you will find that there is almost no relation between the 2 variables, which kinda makes sense as one of them is a binary/categorical variable. ($cor(x,y) = -0.005750405$).

test <- read.csv("sample.csv", header = TRUE, sep = ',')
Outcome <- test$Outcome Distance <- test$Distance
cor(Outcome,Distance)
-----------------------Outputs------------------------------
 -0.005750405


From the result above, I don't think that's the kind of result you want as not much can be interpreted from it. I highly suggest you take a look at Correlation between a nominal (IV) and a continuous (DV) variable, where a detailed breakdown of a problem similar to yours has been given.

In any case, if you are interested in descriptive statistics, you can try the following:

If you call a general linear model, you obtain the following:

lm1 <- glm(test$Outcome ~ test$Distance, data = test)
summary(lm1)
-----------------------Outputs------------------------------
Call:
glm(formula = test$Outcome ~ test$ï..Distance, data = test)

Deviance Residuals:
Min       1Q   Median       3Q      Max
-0.8588   0.1413   0.1415   0.1417   0.1712

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)       8.588e-01  6.380e-03 134.606   <2e-16 ***
test$ï..Distance -4.717e-08 1.404e-07 -0.336 0.737 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 (Dispersion parameter for gaussian family taken to be 0.1218882) Null deviance: 416.14 on 3415 degrees of freedom Residual deviance: 416.13 on 3414 degrees of freedom AIC: 2508.7 Number of Fisher Scoring iterations: 2  Or if you call a linear model (lm): lm1 <- lm(test$Outcome ~ test$Distance, data = test) -----------------------Outputs------------------------------ Call: lm(formula = test$Outcome ~ test$ï..Distance, data = test) Residuals: Min 1Q Median 3Q Max -0.8588 0.1413 0.1415 0.1417 0.1712 Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 8.588e-01 6.380e-03 134.606 <2e-16 *** test$ï..Distance -4.717e-08  1.404e-07  -0.336    0.737
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.3491 on 3414 degrees of freedom
Multiple R-squared:  3.307e-05, Adjusted R-squared:  -0.0002598
F-statistic: 0.1129 on 1 and 3414 DF,  p-value: 0.7369


Also remember that correlation coefficients are used in statistics to measure how strong a relationship is between two variables, not to show causation. More information about interpreting correlation can be found at http://www.statisticshowto.com/probability-and-statistics/correlation-coefficient-formula/ and https://www.dummies.com/education/math/statistics/how-to-interpret-a-correlation-coefficient-r/.

• Thanks for the reply. I don't need to use Excel, I can code (or use libraries) in various languages. So I'm primarily concerned about the correct algorithm. I want to avoid reaching for the most convenient thing for lack of knowledge, though if the Excel function is the correct one to use then that is handy. I understand the point about correlation and causation. – Edward Hibbert Sep 2 '18 at 16:15
• I understand your concern. The correlation obtained is indeed almost zero, so not much interpretation can be done from it. Do check out the link I have added into the edited post above, which gives a more detailed breakdown of your problem and data. – Stoner Sep 2 '18 at 16:48

Welcome to crossvalidated. It looks like there is some "connection" between the two variables. Here are two histograms of the logarithm of the distance--one for each level of the Outcome variable. Observe that taking the log makes this variable look normally distributed. If you regress log distance on outcome, there seems to be something going on. I get a p-value of $0.0254$ on the t-test. Hopefully this will give you some ideas. And here's the code used to generate all of this:

df <- read.csv("~/Desktop/sample_data_xv.csv")

# plot 1
hist(log(df$Distance[df$Outcome=="Up"]), col = "red", xlab="log distance", main = "log distance by outcome (red = up, blue = down)")
hist(log(df$Distance[df$Outcome=="Down"]), col = "blue", add = T)

# plot 2
hist(log(df$Distance[df$Outcome=="Up"]), col = "red", xlab="log distance", main = "log distance by outcome (red = up, blue = down)", freq = F)
hist(log(df$Distance[df$Outcome=="Down"]), col = "blue", add = T, freq = F)

# regress log distance on outcome
is.factor(df\$Outcome)
mod <- lm(log(Distance+.000001) ~ Outcome, data = df)
summary(mod)


Edit: here's an image that plots density on the y-axis, instead of frequency, so it doesn't show the relative count as well. It does, however, show how the mean is shifted by the dummy variable in our regression. • Thanks for the reply. It's useful to know that there seems to be a connection! – Edward Hibbert Sep 5 '18 at 6:29