Is using deciles to find correlation a statistically valid approach?

I have a sample of 1,449 data points that are not correlated (r-squared 0.006).

When analyzing the data, I discovered that by splitting the independent variable values into positive and negative groups, there seems to be a significant difference in the average of dependent variable for each group.

Splitting the points into 10 bins (deciles) using the independent variable values, there seems to be a stronger correlation between the decile number and the average dependent variable values (r-squared 0.27).

I don't know much about statistics so here are a few questions:

1. Is this a valid statistical approach?
2. Is there a method to find the best number of bins?
3. What is the proper term for this approach so I can Google it?
5. What are some other approaches I can use to find relationships in this data?

Here is the decile data for reference: https://gist.github.com/georgeu2000/81a907dc5e3b7952bc90

EDIT: Here is an image of the data: Industry Momentum is the independent variable, Entry Point Quality is dependent

• Hopefully my answer (in particular responses 2-4) is understood in the sense it was intended. Jul 7 '14 at 5:57
• If your purpose is to explore a relationship form between the independent and the dependent, this is a fine exploratory technique. It may offend statisticians but is used in industry all the time (e.g. credit risk). If you are building a predictive model,then again feature engineering is OK - if it is done on a training set an properly validated. Jul 9 '14 at 0:56
• Can you provide any resources on how to make sure the result is "properly validated"? Jul 9 '14 at 2:44
• "not correlated (r-squared 0.006)" means that they are not linearly correlated. Perhaps there is some other correlation involved. Have you plotted the raw data (dependent vs independent)? Jul 10 '14 at 15:47

0. The correlation (0.0775) is small but (statistically) significantly different from 0. That is, it looks like there really is correlation, it's just very small/weak (equivalently, there's a lot of noise around the relationship).

1. What averaging within bins does is reduce the variation in the data (the $\sigma/\sqrt{n}$ effect for standard error of a mean), which means that you artificially inflate the weak correlation. Also see this (somewhat) related issue.

2. Sure, fewer bins means more data gets averaged, reducing noise, but the wider they are, the "fuzzier" the average becomes in each bin because the mean isn't quite constant - there's a trade-off. While one might derive a formula to optimize the correlation under an assumption of linearity and the distribution of the $x$'s, it wouldn't take full account of the somewhat exploitable effect of noise in the data. The easy way is to just try a whole variety of different bin boundaries until you get what you like. Don't forget to try varying the bin-widths and bin-origins. That strategy can occasionally prove surprisingly useful with densities, and that kind of occasional advantage can be carried over to functional relationships - perhaps enabling you to get exactly the result you hoped for.

4. This is a good place to start; it's a very popular book aimed at non-statisticians.

5. (more seriously:) I'd suggest smoothing (such as via local polynomial regression/kernel smoothing, say) as one way to investigate relationships. It depends on what you want, exactly, but this can be a valid approach when you don't know the form of a relationship, as long as you avoid the data-dredging issue.

There's a popular quote, whose originator appears to be Ronald Coase:

"If you torture the data enough, nature will always confess."

• Points 1 & 2 are good explanations of why the OP's approach is not a valid statistical approach, though it is a useful heuristic to convince regarding Point 0 (weak correlation). Jul 7 '14 at 7:28

Perhaps you would benefit from an exploratory tool. Splitting the data into deciles of the x coordinate appears to have been performed in that spirit. With modifications described below, it's a perfectly fine approach.

Many bivariate exploratory methods have been invented. A simple one proposed by John Tukey (EDA, Addison-Wesley 1977) is his "wandering schematic plot." You slice the x-coordinate into bins, erect a vertical boxplot of the corresponding y data at the median of each bin, and connect the key parts of the boxplots (medians, hinges, etc.) into curves (optionally smoothing them). These "wandering traces" provide a picture of the bivariate distribution of the data and allow immediate visual assessment of correlation, linearity of relationship, outliers, and marginal distributions, as well as robust estimation and goodness-of-fit evaluation of any nonlinear regression function.

To this idea Tukey added the thought, consistent with the boxplot idea, that a good way to probe the distribution of data is to start at the middle and work outwards, halving the amount of data as you go. That is, the bins to use need not be cut at equally-spaced quantiles, but instead should reflect the quantiles at the points $2^{-k}$ and $1-2^{-k}$ for $k=1, 2, 3, \ldots$.

To display the varying bin populations we can make each boxplot's width proportional to the amount of data it represents.

The resulting wandering schematic plot would look something like this. Data, as developed from the data summary, are shown as gray dots in the background. Over this the wandering schematic plot has been drawn, with the five traces in color and the boxplots (including any outliers shown) in black and white. The nature of the near-zero correlation becomes immediately clear: the data twist around. Near their center, ranging from $x=-4$ to $x=4$, they have a strong positive correlation. At extreme values, these data exhibit curvilinear relationships that tend on the whole to be negative. The net correlation coefficient (which happens to be $-0.074$ for these data) is close to zero. However, insisting on interpreting that as "nearly no correlation" or "significant but low correlation" would be the same error spoofed in the old joke about the statistician who was happy with her head in the oven and feet in the icebox because on average the temperature was comfortable. Sometimes a single number just won't do to describe the situation.

Alternative exploratory tools with similar purposes include robust smooths of windowed quantiles of the data and fits of quantile regressions using a range of quantiles. With the ready availability of software to perform these calculations they have perhaps become easier to execute than a wandering schematic trace, but they do not enjoy the same simplicity of construction, ease of interpretation, and broad applicability.

The following R code produced the figure and can be applied to the original data with little or no change. (Ignore the warnings produced by bplt (called by bxp): it complains when it has no outliers to draw.)

#
# Data
#
set.seed(17)
n <- 1449
x <- sort(rnorm(n, 0, 4))
s <- spline(quantile(x, seq(0,1,1/10)), c(0,.03,-.6,.5,-.1,.6,1.2,.7,1.4,.1,.6),
xout=x, method="natural")
#plot(s, type="l")
e <- rnorm(length(x), sd=1)
y <- s$y + e # ($ interferes with MathJax processing on SE)
#
# Calculations
#
q <- 2^(-(2:floor(log(n/10, 2))))
q <- c(rev(q), 1/2, 1-q)
n.bins <- length(q)+1
bins <- cut(x, quantile(x, probs = c(0,q,1)))
x.binmed <- by(x, bins, median)
x.bincount <- by(x, bins, length)
x.bincount.max <- max(x.bincount)
x.delta <- diff(range(x))
cor(x,y)
#
# Plot
#
par(mfrow=c(1,1))
b <- boxplot(y ~ bins, varwidth=TRUE, plot=FALSE)
plot(x,y, pch=19, col="#00000010",
main="Wandering schematic plot", xlab="X", ylab="Y")
for (i in 1:n.bins) {
invisible(bxp(list(stats=b$stats[,i, drop=FALSE], n=b$n[i],
conf=b$conf[,i, drop=FALSE], out=b$out[b$group==i], group=1, names=b$names[i]), add=TRUE,
boxwex=2*x.delta*x.bincount[i]/x.bincount.max/n.bins,
at=x.binmed[i]))
}

colors <- hsv(seq(2/6, 1, 1/6), 3/4, 5/6)
temp <- sapply(1:5, function(i) lines(spline(x.binmed, b$stats[i,], method="natural"), col=colors[i], lwd=2)) • I do not undertand where this vector is coming from c(0,.03,-.6,.5,-.1,.6,1.2,.7,1.4,.1,.6), is this being generated and data dependent (x)? You mention the 2^*(-k) but this is not related. Aug 14 '18 at 12:24 • @Max That vector generates this particular example. – whuber Aug 14 '18 at 12:33 • ok, but how did you procedure the figures? This is clearly wrong: k <- 1:11; ifelse(quantile(g, seq(0,1,1/10))>0, 2^(-k), 1-2^(-k)) Aug 14 '18 at 17:20 • @Max I cannot fathom what you might mean by "clearly wrong." You cannot be referring to my code: as a check, I re-ran it and it reproduced the figure in every detail. – whuber Aug 14 '18 at 20:10 • I‘m meant my approach is wrong...the ifelse statement. The question is that I do not know how to obtain the vector input into quantile function. Aug 14 '18 at 20:14 I do not believe that binning is a scientific approach to the problem. It is information losing and arbitrary. Rank (ordinal; semiparametric) methods are far better and do not lose information. Even if one were to settle on decile binning, the method is still arbitrary and non-reproducible by others, simply because of the large number of definitions that are used for quantiles in the case of ties in the data. And as alluded to in the nice data torture comment above, Howard Wainer has a nice paper showing how to find bins that can produce a positive association, and find bins that can produce a negative association, from the same dataset: @Article{wai06fin, author = {Wainer, Howard}, title = {Finding what is not there through the unfortunate binning of results: {The} {Mendel} effect}, journal = {Chance}, year = 2006, volume = 19, number = 1, pages = {49-56}, annote = {can find bins that yield either positive or negative association;especially pertinent when effects are small;With four parameters, I can fit an elephant; with five, I can make it wiggle its trunk.'' - John von Neumann} } • This is what I suspected. If you choose different numbers of bins, you can find the best correlation for to show whatever are looking for. But the results would not be reproducible or scientifically valid. Do you know of any introductory resources about ranking? Jul 8 '14 at 16:26 • This seems like an overly extreme position to take. Of course rank-based methods lose information too: they discard all the information about the actual values. So the question concerning whether information loss is a concern comes down to why is the analysis being performed? If it is for discovery and exploration, various guided forms of binning can do a good job whereas replacing everything with ranks would likely obscure and distort the relationships. If it is for confirmation or hypothesis testing then your remarks would be more generally applicable and defensible. – whuber Jul 8 '14 at 16:30 • I don't really agree with that. The type of information lost by rank-based methods is minimal (e.g.,$\frac{\pi - 3}{\pi}$in the Gaussian residuals case), and relative to more assumption-laden methods they can gain information. It is not hard to use a semiparametric model (e.g., proportional odds model) to estimate the mean and quantiles of$Y | X\$. Information loss is almost always a bad thing, whether during exploration (you might miss something important) or during formal analysis (power and precision loss and arbitrariness). Jul 8 '14 at 19:46
• Any reference to Rank (ordinal; semiparametric) methods please? And why are they better/dont lose information? Jan 7 '21 at 17:15
• Much material may be found in hbiostat.org/doc/bbr.pdf and hbiostat.org/rms Jan 8 '21 at 14:08

I found the localgauss package very useful for this. https://cran.r-project.org/web/packages/localgauss/index.html

The package contains

Computational routines for estimating and visualizing local Gaussian parameters. Local Gaussian parameters are useful for characterizing and testing for non-linear dependence within bivariate data.

Example:

library(localgauss)
x=rnorm(n=1000)
y=x^2 + rnorm(n=1000)
lgobj = localgauss(x,y)
plot(lgobj)

Result: Splitting the data into deciles based on the observed X ("Entry Point Quality") appears to be a generalization of an old method first proposed by Wald and later by others for situations wherein both X and Y are subject to error. (Wald split the data into two groups. Nair & Shrivastava and Bartlett split it into three.) It is described in section 5C of Understanding robust and Exploratory Data Analysis, edited by Hoaglin, Mosteller and Tukey (Wiley, 1983). However, a lot of work on such "Measurement Error" or "Error in Variables Models" has been done since then. The textbooks that I've looked at are Measurement Error: Models, Methods and Applications by John Buonaccorsi (CRC Press, 2010) and Measurement Error Models by Wayne Fuller (Wiley, 1987).

Your situation may be somewhat different because your scatterplot leads me to suspect that both observations are random variables and I don't know whether they each contain measurement error. What do the variables represent?

• Entry Point Quality is how much a stock rises or falls in the short term at a given point in time. Industry Momentum is a measure of "momentum" in the industry for the stock at the same point in time. The hypothesis is that there is a correlation between the industry momentum and the future price of the stock. Jul 10 '14 at 17:40
• We generally put the response on the vertical axis. Visual inspection suggest that even if there is a real correlation, variation in entry point quality makes it pretty useless. But since you're dealing with stock prices, things get even more complex because time series are involved. Jul 10 '14 at 17:59