I have a scatter plot which has sample size which is equal to the number of people on x axis and median salary on y axis, I am trying to find out if the sample size has any effect on the median salary.

This is the plot:

enter image description here

How do I interpret this plot ?

  • 3
    $\begingroup$ If you can, I'd suggest working with a transformation of both variables. If neither variable has exact zeros, take a look on the log-log scale $\endgroup$ – Glen_b Sep 5 '17 at 5:12
  • $\begingroup$ @Glen_b sorry, I am not familiar with the terms that you have stated, just by looking at the plot, can you make a relation between the two variables? what I can guess is for sample size upto 1000 there is no relation as for the same sample size values there are multiple median values. For values greater than 1000, median salary appears to decrease. What do you think ? $\endgroup$ – Sameed Sep 5 '17 at 5:17
  • $\begingroup$ I see no clear evidence for that, it looks pretty flat to me; if there's clear changes it's probably going on in the lower portion of sample size. Do you have the data, or only the image of the plot? $\endgroup$ – Glen_b Sep 5 '17 at 6:10
  • 4
    $\begingroup$ If you see the median as the median of n random variables, then it makes sense that the variation of the median decreases as the sample size increases. That would explain the large spread at the left side of the plot. $\endgroup$ – JAD Sep 5 '17 at 11:06
  • 2
    $\begingroup$ Your statement "for sample size upto 1000 there is no relation as for the same sample size values there are multiple median values" is incorrect. $\endgroup$ – Peter Flom Sep 5 '17 at 11:07

"Find out" indicates you are exploring the data. Formal tests would be superfluous and suspect. Instead, apply standard exploratory data analysis (EDA) techniques to reveal what may be in the data.

These standard techniques include re-expression, residual analysis, robust techniques (the "three R's" of EDA) and smoothing of the data as described by John Tukey in his classic book EDA (1977). How to conduct some of these are outlined in my post at Box-Cox like transformation for independent variables? and In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?, inter alia.

The upshot is that much can be seen by changing to log-log axes (effectively re-expressing both variables), smoothing the data not too aggressively, and examining residuals of the smooth to check what it might have missed, as I will illustrate.

Here are the data shown with a smooth that--after examining several smooths with varying degrees of fidelity to the data--seems like a good compromise between too much and too little smoothing. It uses Loess, a well-known robust method (it is not heavily influenced by vertically outlying points).

Log-log scatterplot

The vertical grid is in steps of 10,000. The smooth does suggest some variation of Grad_median with sample size: it seems to drop as sample sizes approach 1000. (The ends of the smooth are not trustworthy--especially for small samples, where sampling error is expected to be relatively large--so don't read too much into them.) This impression of a real drop is supported by the (very rough) confidence bands drawn by the software around the smooth: its "wiggles" are greater than the widths of the bands.

To see what this analysis might have missed, the next figure looks at the residuals. (These are differences of natural logarithms, directly measuring vertical discrepancies between data the preceding smooth. Because they are small numbers they can be interpreted as proportional differences; e.g., $-0.2$ reflects a data value that is about $20\%$ lower than the corresponding smoothed value.)

We are interested in (a) whether there are additional patterns of variation as sample size changes and (b) whether the conditional distributions of the response--the vertical distributions of point positions--are plausibly similar across all values of sample size, or whether some aspect of them (like their spread or symmetry) might change.

![Figure 2 Plot of residuals

This smooth tries to follow the datapoints even more closely than before. Nevertheless it is essentially horizontal (within the scope of the confidence bands, which always cover a y-value of $0.0$), suggesting no further variation can be detected. The slight increase in the vertical spread near the middle (sample sizes of 2000 to 3000) would not be significant if formally tested, and so it surely is unremarkable in this exploratory stage. There is no clear, systematic deviation from this overall behavior apparent in any of the separate categories (distinguished, not too well, by color--I analyzed them separately in figures not shown here).

Consequently, this simple summary:

median salary is about 10,000 lower for sample sizes near 1000

adequately captures the relationships appearing in the data and seems to hold uniformly across all major categories. Whether that is significant--that is, whether it would stand up when confronted with additional data--can only be assessed by collecting those additional data.

For those who would like to check this work or take it further, here is the R code.

# Read the data.
infile <- "https://raw.githubusercontent.com/fivethirtyeight/\
X <- as.data.table(read.csv(infile))
# Compute the residuals.
span <- 0.6 # Larger values will smooth more aggressively
X[, Log.residual := 
      residuals(loess(log(Grad_median) ~ I(log(Grad_sample_size)), X, span=span))]
# Plot the data on top of a smooth.
g <- ggplot(X, aes(Grad_sample_size, Grad_median)) + 
  geom_smooth(span=span) + 
  geom_point(aes(fill=Major_category), alpha=1/2, shape=21) + 
  scale_x_log10() + scale_y_log10(minor_breaks=seq(1e4, 5e5, by=1e4)) + 
  ggtitle("EDA of Median Salary vs. Sample Size",
          paste("Span of smooth is", signif(span, 2)))

span <- span * 2/3 # Look for a little more detail in the residuals
g.r <- ggplot(X, aes(Grad_sample_size, Log.residual)) + 
  geom_smooth(span=span) + 
  geom_point(aes(fill=Major_category), alpha=1/2, shape=21) + 
  scale_x_log10() + 
  ggtitle("EDA of Median Salary vs. Sample Size: Residuals",
          paste("Span of smooth is", signif(span, 2)))

Glen_b is suggesting you take the logarithm of sample_size and median salary to see if rescaling the data makes sense.

I don't know that I would agree with your belief that median salary decreases once the sample size rises above 1,000. I'd be more inclined to say there is no relationship at all. Does your theory predict that there should be a relationship?

Another way you could assess a possible relationship is to fit a regression line to the data. Alternatively, you could also use a lowess curve. Plot both lines to your data and see if anything can be teased out (I doubt there is anything overly substantive, however).

  • 3
    $\begingroup$ The scatterplot is very similar to a funnel plot used in meta-analyses. See a similar example. Plotting the funnel bands will more clearly show if there is any relationship, there might be a slightly positive one in this example. $\endgroup$ – Andy W Sep 5 '17 at 13:41

I also agree there's no relationship. I reproduced your original scatter plot (left) and made the log-log scatter plot suggested by glen_b (right).

enter image description here

Looks like there's no relationship to either. Correlation between log-transformed data is weak (Pearson R = -.13) and insignificant (p = .09). Depending on how much extra information you have there's maybe a reason to see some weak negative correlation, but that seems like a stretch. I'd guess that any apparent pattern you're seeing is the same effect seen here.

Edit: After looking at @famargar's plots I realized I plotted grad sample size vs non-grad median salary. I believe @sameed wanted sample size vs grad-median salary, although it's not totally clear. For the latter I reproduce @famargar's numbers, i.e. $R = 0.0022$ ($p = 0.98$) and our plots look identical.

  • $\begingroup$ Thanks for looking at the correlation between grad-median and grad-sample-size; I was deeply puzzled by the difference between the numbers! $\endgroup$ – famargar Sep 5 '17 at 21:36

Trying a linear regression will teach you something about this relation, as suggested in the first answer. Since It looks like you are using python plus matplotlib for this plot, you are one line of code away from the solution.

You could use seaborn jointplot, that will also display the linear regression line, the Pearson correlation coeffiecient, and its p-value:

sns.jointplot("Grad_sample_size", "Grad_median", data=df, kind="reg")

enter image description here

as you can see there is no correlation. Looking at this last plot, it seems log-transforming the x-variable would be useful. Let's try it:

df['log_size'] = np.log(df['Grad_sample_size'])
sns.jointplot("log_size", "Grad_median", data=df, kind="reg")

enter image description here

You can clearly see that - log-transformation or not - the correlation is small, and both the p-value and the confidence intervals say that it is not statistically meaningful.

  • 3
    $\begingroup$ The indications of strongly skewed conditional distributions suggest this is not a good approach. When you also observe that the skewness of the sample size distribution will cause the few largest sample sizes to control the appearance of a trend in the regression, you will see why others are recommending preliminary transformations of the data. $\endgroup$ – whuber Sep 5 '17 at 18:33
  • 1
    $\begingroup$ I am not guessing or speculating: the plot in the question clearly shows these characteristics. Also see the plots created by R Greg Stacey, which--by applying the suggested log-log transformations--demonstrates what they accomplish. $\endgroup$ – whuber Sep 5 '17 at 20:23
  • $\begingroup$ I just found the data and did the study myself - please see updated answer. $\endgroup$ – famargar Sep 5 '17 at 20:39
  • $\begingroup$ Your study has succumbed to the two problems I noted: the appearance of "no correlation" derives in no small part to the skewed conditional responses and the leverage for the high regressor values. In particular, neither the fitted line nor its error bands are trustworthy. $\endgroup$ – whuber Sep 5 '17 at 20:48
  • $\begingroup$ Please see the plot I just added; I hope I am not missing anything in this last iteration. $\endgroup$ – famargar Sep 5 '17 at 20:48

This plot works as a demonstration of the central limit theorem, where the variability between samples decreases as the sample size increases. It's also the shape that you would expect with a strongly skewed variable like salary.

  • 3
    $\begingroup$ These aren't independent samples from a common population. That makes the relevance of the CLT rather problematic. $\endgroup$ – whuber Sep 5 '17 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.