4
$\begingroup$

I have two variables: the number of employees who work from home in the firm and the number of independent contractors in the firm - and I have this data for 70 firms.

I am trying to determine what statistical measure of association I can use to determine whether there is a relationship between these two variables.

I assumed the best way to do this is to convert the two variables to percentages so that my hypothesis would be: there is a positive correlation between the percentage of telecommuters and percentage of independent contractors a firm has. However, I read that you cannot use correlation to compare two percentages.

I am not strong in statistics so I would be very grateful for any insight on whether what I am trying to do here seems logical, if converting to percentages to run the correlation makes sense, and if correlation is an acceptable means of determining association between two %s. If it is not, I would be very grateful for recommendations on statistical tests that would work best to determine the association between these two variables.

$\endgroup$
  • $\begingroup$ Related: Pearson's or Spearman's correlation with non-normal data. $\endgroup$ – gung Apr 24 '17 at 14:58
  • $\begingroup$ FYI. You didn't have to delete your initial question and then repost it. My comments on the initial question were intended to get the idea correctly. In fact, I had been working on a quick how-to simulation, and never got around to posting it. Erasing and reposting got your question bumped to the top, and gave it visibility this morning, though... :-) $\endgroup$ – Antoni Parellada Apr 24 '17 at 17:00
  • $\begingroup$ My sincerest apologies - I thought my initial question wasn't clear at all and that I was potentially confusing you with my responses (I was extremely grateful for your prompt response) - I deleted and tried to repost because I thought I had completely made a mess of my response to you. $\endgroup$ – SSE Apr 24 '17 at 17:03
  • 1
    $\begingroup$ I don't get which components are referred to which totals. What is problematic is correlating $p = P/T$ and $q = Q/T$ where the total $T$ includes $P + Q$ and possibly other components. The extreme case is binary classifications where one fraction (proportion) going up necessarily means the other going down by exactly the same amount and correlations of $-1$ are inevitable. Some inbuilt negative correlation is inherent with three or more categories; this is one of the first (negative) points about compositional data analysis; what to do instead is a much larger issue. $\endgroup$ – Nick Cox Apr 25 '17 at 12:56
  • 2
    $\begingroup$ Factors of 100 evidently don't affect the issue. (Incidentally, a percentage is exactly the same as a proportion insofar as % means /100 but maintaining that will confuse at least some audiences and most programs don't treat use of percents as purely a display issue. (I'm told that MS Excel does oblige.) $\endgroup$ – Nick Cox Apr 25 '17 at 12:58
8
$\begingroup$

One reason that someone might assert that "you cannot run correlation on percentages" is that percentages are bounded by [0, 1], and the underlying assumption of the Pearson r test is that values are normally distributed; these are manifestly incompatible.

If your percentages are concentrated in a not-too-wide band not-too-close to 0% or 100%, I wouldn't worry about this too much.

If your percentages are all over the [0, 1] interval, or are concentrated near 0 or near 1, I would do Spearman rho or Kendall tau correlation test instead. These tests to do not depend on any underlying assumption of normality.

$\endgroup$
  • $\begingroup$ Thank you very much for your reply. Would another way around this be to simply not convert the values to percentages? Would that be meaningful in the context of the data I have? For each firm, I have total number of independent contractors and the total number of telecommuting employees. I wonder if it is even necessary to convert to percentages (so that my hypotheses would be that there is a positive correlation between indepedent contractors and telecommuters). $\endgroup$ – SSE Apr 24 '17 at 18:22
  • 5
    $\begingroup$ I think the stress on normal distributions is exaggerated here. Pearson correlation measures linearity of association; what non-normality makes problematic is getting $P$-values or confidence intervals, but for almost 40 years we have had bootstrapping; for much longer we have known about transformations for both unbounded and bounded scales, not to mention other ways of doing inference. You didn't deny any of that, but this answer could be misread as implying: if nonnormality, then Pearson correlations inappropriate! which I don't think would be a fair statement. There are bigger problems. $\endgroup$ – Nick Cox Apr 25 '17 at 12:49
  • 2
    $\begingroup$ In the context of mentioning the pearson R test, David said "If your percentages are concentrated in a not-too-wide band not-too-close to 0% or 100%, I wouldn't worry about this too much," which is correct. If someone misread this as saying "if nonnormality, then Pearson correlations inappropriate!", then they barely read it at all. This answer is fine. $\endgroup$ – gammer Apr 25 '17 at 13:27
  • $\begingroup$ The range of the data and whether they are approximately normal are quite different issues. $\endgroup$ – Nick Cox Apr 25 '17 at 17:21
  • $\begingroup$ If lots of percents are near either bound then the double bound may not bite much either and transformation is always possible. $\endgroup$ – Nick Cox Apr 25 '17 at 17:34
4
$\begingroup$

The video you linked here in one of your comments makes reference to compositional data. This would be an issue if you tried to compare percentages adding up to $100\%$, but this is not the case in your question. Regardless of whether you express the variable independent contractors as counts or a percentage, it wouldn't constrain the range of possible values of the variable offsite workers.

In terms of the count (integer-valued) nature of independent contractors you could apply a Poisson, semi-Poisson or a negative binomial model:

I created a fictional dataset design to fit a Poisson regression model. I'll use R.

The Poisson regression model is of the form $\log (Y) = \beta_0 + \beta_1 X$. Hence, for each value of the $X=x$ variable the $\log(\mathbb E[Y\vert x])$ will be distributed as a Poisson variable with mean $\lambda = \exp(\beta_0 + \beta_1 x).$ We can therefore simulate a dataset suitable for a Poisson regression model as follows (please refer to this post to see the truncation trick to avoid to high or zero values for the number of independent contractors):

set.seed(0)     # Setting the seed value to make findings reproducible.
co = 70         # The number of companies (firms) you have data on.
n  =  1000      # Dirty trick to get "tons" of point to truncate later and end up with 70.
i  =  0         # Intercept chosen to be zero hoping to simplify things.
sl = .04        # The slope or beta1 in the equation in the previous paragraph.
# Assuming the % of off-site workers is varies uniformly from 0% to 100%:
offsite = runif(n, 0, 100) 
mu = exp(i + sl * offsite) # Getting the means 
# Generating the number of independent contractors:
indep   = rpois(n, mu)    
# Creating data set with offsite % and no. contractors in two columns:
dat     = as.data.frame(cbind(offsite, indep))  
# Truncating the data to obtain data points avoiding 0 contractors 
# ...and keeping max. to < 45 (sounds like a real-life plausible max):
dat     = dat[which(dat$indep > 0 & dat$indep < 45), ] 
# Selecting only 70 of these truncated data points:
dat     = dat[sample(nrow(dat),co), ]; rownames(dat) = NULL

We know that the fit of Poisson model will be perfect by design, and hence, we can apply it to proof the association between the number of subcontracting companies and the percentage of telecommuters; however, in a real-life situation this is not going to be the case. Therefore, models such a negative binomial regression that don't assume equal mean and variance will be more often use to deal with over-dispersion. In the basic statistical R package we can also deal with this problem running a quasi-Poisson regression:

> fit = glm(indep ~  offsite, family = "quasipoisson", data = dat)
> summary(fit)

Call:
glm(formula = indep ~ offsite, family = "quasipoisson", data = dat)

Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-1.96453  -0.61346  -0.08643   0.34988   2.16355  

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 0.238500   0.108206   2.204   0.0309 *  
offsite     0.036529   0.001404  26.009   <2e-16 ***

Not exactly the intercept and slope we had chosen initially, but the data has been truncated and we have only 70 points. And it shows the significant association between the percentage of off-site workers and the the number of independent contractors (by design, p ~ $ 0$): for each increase of $1\%$ in the number of off-site workers there will be a positive difference of $0.037$ in the log of the expected number of subcontracting firms. Or for a one percentage point increase in off-site workers there is a $e^{0.037}$ $(\times)$ increase in subcontractors.

These are the plots showing the relationship between the number of independent contractors and the percentage of off-site workers:

enter image description here


A different way to look at the relationship is through Spearman correlation (more in line with your original question). The results are very close to the default Pearson correlation as indicated in the correct answer above by David Wright:

> cor(dat$offsite, dat$indep, method="pearson")
[1] 0.90324
> cor(dat$offsite, log(dat$indep), method="spearman")
[1] 0.9510245

It is interesting to try a log transformation of the dependent variable given the plot to the left above, and running an OLS regression (note that this is not equivalent to a Poisson regression (or a negative binomial)). If we do so, we find that the slope:

> lm(log(indep) ~  offsite, data = dat)$coef[2]
   offsite 
0.03926557

is equivalent to

> cor(dat$offsite, log(dat$indep)) * sd(log(dat$indep)) / sd(dat$offsite)
[1] 0.03926557

because

$$r = \beta_1\sqrt{\frac{\sum_{i=1}^n(x_i-\bar x)^2}{\sum_{i=1}^n(y_i-\bar y)^2}}=\beta_1\frac{S_{xx}}{S_{yy}}$$

$\endgroup$
  • $\begingroup$ wow - thank you! This is EXTREMELY helpful and your explanation is very clear for someone like me who is quite weak in this stuff. Thank you very, very much for walking through this - for taking your time to provide such a comprehensive response. $\endgroup$ – SSE Apr 27 '17 at 2:17
  • $\begingroup$ You are very welcome! $\endgroup$ – Antoni Parellada Apr 27 '17 at 2:18
2
$\begingroup$

Where did you read that you cannot use correlation between percentages? I think that the meaning of the correlation coefficient will be preserved (e.g "If X increases, Y is likely to increase." or "High values of X are associated with high values of Y."), so correlation is fair game.

One thing to watch out for may be ecological correlation. But as long as you keep your interpretation in check, you should be ok.

Your question feels more opinion based - so please take my answer in aggregate with others :).

$\endgroup$
  • $\begingroup$ Thank you very much for your reply - this is extremely helpful. I found this video that says one should not use correlation with %s which is why I asked the forum: slideshare.net/AustralianBioinformatics/… . In short it says "Correlation is a very popular measure of association, but not one that you should use with proportions, percentages or parts per million" But perhaps I misinterpreted this video? $\endgroup$ – SSE Apr 24 '17 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.