11
$\begingroup$

I have a sample of 1,449 data points that are not correlated (r-squared 0.006).

When analyzing the data, I discovered that by splitting the independent variable values into positive and negative groups, there seems to be a significant difference in the average of dependent variable for each group.

Splitting the points into 10 bins (deciles) using the independent variable values, there seems to be a stronger correlation between the decile number and the average dependent variable values (r-squared 0.27).

I don't know much about statistics so here are a few questions:

  1. Is this a valid statistical approach?
  2. Is there a method to find the best number of bins?
  3. What is the proper term for this approach so I can Google it?
  4. What are some introductory resources to learn about this approach?
  5. What are some other approaches I can use to find relationships in this data?

Here is the decile data for reference: https://gist.github.com/georgeu2000/81a907dc5e3b7952bc90

EDIT: Here is an image of the data: Industry Momentum is the independent variable, Entry Point Quality is dependent

Industry Momentum is the independent variable, Entry Point Quality is dependent

$\endgroup$
4
  • $\begingroup$ Hopefully my answer (in particular responses 2-4) is understood in the sense it was intended. $\endgroup$
    – Glen_b
    Jul 7, 2014 at 5:57
  • $\begingroup$ If your purpose is to explore a relationship form between the independent and the dependent, this is a fine exploratory technique. It may offend statisticians but is used in industry all the time (e.g. credit risk). If you are building a predictive model,then again feature engineering is OK - if it is done on a training set an properly validated. $\endgroup$
    – B_Miner
    Jul 9, 2014 at 0:56
  • $\begingroup$ Can you provide any resources on how to make sure the result is "properly validated"? $\endgroup$
    – B Seven
    Jul 9, 2014 at 2:44
  • $\begingroup$ "not correlated (r-squared 0.006)" means that they are not linearly correlated. Perhaps there is some other correlation involved. Have you plotted the raw data (dependent vs independent)? $\endgroup$ Jul 10, 2014 at 15:47

5 Answers 5

9
$\begingroup$

0. The correlation (0.0775) is small but (statistically) significantly different from 0. That is, it looks like there really is correlation, it's just very small/weak (equivalently, there's a lot of noise around the relationship).

1. What averaging within bins does is reduce the variation in the data (the $\sigma/\sqrt{n}$ effect for standard error of a mean), which means that you artificially inflate the weak correlation. Also see this (somewhat) related issue.

2. Sure, fewer bins means more data gets averaged, reducing noise, but the wider they are, the "fuzzier" the average becomes in each bin because the mean isn't quite constant - there's a trade-off. While one might derive a formula to optimize the correlation under an assumption of linearity and the distribution of the $x$'s, it wouldn't take full account of the somewhat exploitable effect of noise in the data. The easy way is to just try a whole variety of different bin boundaries until you get what you like. Don't forget to try varying the bin-widths and bin-origins. That strategy can occasionally prove surprisingly useful with densities, and that kind of occasional advantage can be carried over to functional relationships - perhaps enabling you to get exactly the result you hoped for.

3. Yes. Possibly start with this search, then perhaps try synonyms.

4. This is a good place to start; it's a very popular book aimed at non-statisticians.

5. (more seriously:) I'd suggest smoothing (such as via local polynomial regression/kernel smoothing, say) as one way to investigate relationships. It depends on what you want, exactly, but this can be a valid approach when you don't know the form of a relationship, as long as you avoid the data-dredging issue.


There's a popular quote, whose originator appears to be Ronald Coase:

"If you torture the data enough, nature will always confess."

$\endgroup$
1
  • $\begingroup$ Points 1 & 2 are good explanations of why the OP's approach is not a valid statistical approach, though it is a useful heuristic to convince regarding Point 0 (weak correlation). $\endgroup$ Jul 7, 2014 at 7:28
12
$\begingroup$

Perhaps you would benefit from an exploratory tool. Splitting the data into deciles of the x coordinate appears to have been performed in that spirit. With modifications described below, it's a perfectly fine approach.

Many bivariate exploratory methods have been invented. A simple one proposed by John Tukey (EDA, Addison-Wesley 1977) is his "wandering schematic plot." You slice the x-coordinate into bins, erect a vertical boxplot of the corresponding y data at the median of each bin, and connect the key parts of the boxplots (medians, hinges, etc.) into curves (optionally smoothing them). These "wandering traces" provide a picture of the bivariate distribution of the data and allow immediate visual assessment of correlation, linearity of relationship, outliers, and marginal distributions, as well as robust estimation and goodness-of-fit evaluation of any nonlinear regression function.

To this idea Tukey added the thought, consistent with the boxplot idea, that a good way to probe the distribution of data is to start at the middle and work outwards, halving the amount of data as you go. That is, the bins to use need not be cut at equally-spaced quantiles, but instead should reflect the quantiles at the points $2^{-k}$ and $1-2^{-k}$ for $k=1, 2, 3, \ldots$.

To display the varying bin populations we can make each boxplot's width proportional to the amount of data it represents.

The resulting wandering schematic plot would look something like this. Data, as developed from the data summary, are shown as gray dots in the background. Over this the wandering schematic plot has been drawn, with the five traces in color and the boxplots (including any outliers shown) in black and white.

Figure

The nature of the near-zero correlation becomes immediately clear: the data twist around. Near their center, ranging from $x=-4$ to $x=4$, they have a strong positive correlation. At extreme values, these data exhibit curvilinear relationships that tend on the whole to be negative. The net correlation coefficient (which happens to be $-0.074$ for these data) is close to zero. However, insisting on interpreting that as "nearly no correlation" or "significant but low correlation" would be the same error spoofed in the old joke about the statistician who was happy with her head in the oven and feet in the icebox because on average the temperature was comfortable. Sometimes a single number just won't do to describe the situation.

Alternative exploratory tools with similar purposes include robust smooths of windowed quantiles of the data and fits of quantile regressions using a range of quantiles. With the ready availability of software to perform these calculations they have perhaps become easier to execute than a wandering schematic trace, but they do not enjoy the same simplicity of construction, ease of interpretation, and broad applicability.


The following R code produced the figure and can be applied to the original data with little or no change. (Ignore the warnings produced by bplt (called by bxp): it complains when it has no outliers to draw.)

#
# Data
#
set.seed(17)
n <- 1449
x <- sort(rnorm(n, 0, 4))
s <- spline(quantile(x, seq(0,1,1/10)), c(0,.03,-.6,.5,-.1,.6,1.2,.7,1.4,.1,.6),
            xout=x, method="natural")
#plot(s, type="l")
e <- rnorm(length(x), sd=1)
y <- s$y + e # ($ interferes with MathJax processing on SE)
#
# Calculations
#
q <- 2^(-(2:floor(log(n/10, 2))))
q <- c(rev(q), 1/2, 1-q)
n.bins <- length(q)+1
bins <- cut(x, quantile(x, probs = c(0,q,1)))
x.binmed <- by(x, bins, median)
x.bincount <- by(x, bins, length)
x.bincount.max <- max(x.bincount)
x.delta <- diff(range(x))
cor(x,y)
#
# Plot
#
par(mfrow=c(1,1))
b <- boxplot(y ~ bins, varwidth=TRUE, plot=FALSE)
plot(x,y, pch=19, col="#00000010", 
     main="Wandering schematic plot", xlab="X", ylab="Y")
for (i in 1:n.bins) {
  invisible(bxp(list(stats=b$stats[,i, drop=FALSE],
                     n=b$n[i],
                     conf=b$conf[,i, drop=FALSE],
                     out=b$out[b$group==i],
                     group=1,
                     names=b$names[i]), add=TRUE, 
                boxwex=2*x.delta*x.bincount[i]/x.bincount.max/n.bins, 
                at=x.binmed[i]))
}

colors <- hsv(seq(2/6, 1, 1/6), 3/4, 5/6)
temp <- sapply(1:5, function(i) lines(spline(x.binmed, b$stats[i,], 
                                             method="natural"), col=colors[i], lwd=2))
$\endgroup$
7
  • $\begingroup$ I do not undertand where this vector is coming from c(0,.03,-.6,.5,-.1,.6,1.2,.7,1.4,.1,.6), is this being generated and data dependent (x)? You mention the 2^*(-k) but this is not related. $\endgroup$
    – Maximilian
    Aug 14, 2018 at 12:24
  • $\begingroup$ @Max That vector generates this particular example. $\endgroup$
    – whuber
    Aug 14, 2018 at 12:33
  • $\begingroup$ ok, but how did you procedure the figures? This is clearly wrong: k <- 1:11; ifelse(quantile(g, seq(0,1,1/10))>0, 2^(-k), 1-2^(-k)) $\endgroup$
    – Maximilian
    Aug 14, 2018 at 17:20
  • $\begingroup$ @Max I cannot fathom what you might mean by "clearly wrong." You cannot be referring to my code: as a check, I re-ran it and it reproduced the figure in every detail. $\endgroup$
    – whuber
    Aug 14, 2018 at 20:10
  • $\begingroup$ I‘m meant my approach is wrong...the ifelse statement. The question is that I do not know how to obtain the vector input into quantile function. $\endgroup$
    – Maximilian
    Aug 14, 2018 at 20:14
6
$\begingroup$

I do not believe that binning is a scientific approach to the problem. It is information losing and arbitrary. Rank (ordinal; semiparametric) methods are far better and do not lose information. Even if one were to settle on decile binning, the method is still arbitrary and non-reproducible by others, simply because of the large number of definitions that are used for quantiles in the case of ties in the data. And as alluded to in the nice data torture comment above, Howard Wainer has a nice paper showing how to find bins that can produce a positive association, and find bins that can produce a negative association, from the same dataset:

 @Article{wai06fin,
   author =          {Wainer, Howard},
   title =       {Finding what is not there through the unfortunate
    binning of results: {The} {Mendel} effect},
   journal =     {Chance},
   year =        2006,
   volume =      19,
   number =      1,
   pages =       {49-56},
   annote =      {can find bins that yield either positive or negative
    association;especially pertinent when effects are small;``With four
    parameters, I can fit an elephant; with five, I can make it wiggle its
    trunk.'' - John von Neumann}
 }
$\endgroup$
5
  • $\begingroup$ This is what I suspected. If you choose different numbers of bins, you can find the best correlation for to show whatever are looking for. But the results would not be reproducible or scientifically valid. Do you know of any introductory resources about ranking? $\endgroup$
    – B Seven
    Jul 8, 2014 at 16:26
  • 3
    $\begingroup$ This seems like an overly extreme position to take. Of course rank-based methods lose information too: they discard all the information about the actual values. So the question concerning whether information loss is a concern comes down to why is the analysis being performed? If it is for discovery and exploration, various guided forms of binning can do a good job whereas replacing everything with ranks would likely obscure and distort the relationships. If it is for confirmation or hypothesis testing then your remarks would be more generally applicable and defensible. $\endgroup$
    – whuber
    Jul 8, 2014 at 16:30
  • 2
    $\begingroup$ I don't really agree with that. The type of information lost by rank-based methods is minimal (e.g., $\frac{\pi - 3}{\pi}$ in the Gaussian residuals case), and relative to more assumption-laden methods they can gain information. It is not hard to use a semiparametric model (e.g., proportional odds model) to estimate the mean and quantiles of $Y | X$. Information loss is almost always a bad thing, whether during exploration (you might miss something important) or during formal analysis (power and precision loss and arbitrariness). $\endgroup$ Jul 8, 2014 at 19:46
  • $\begingroup$ Any reference to Rank (ordinal; semiparametric) methods please? And why are they better/dont lose information? $\endgroup$
    – Millemila
    Jan 7, 2021 at 17:15
  • $\begingroup$ Much material may be found in hbiostat.org/doc/bbr.pdf and hbiostat.org/rms $\endgroup$ Jan 8, 2021 at 14:08
3
$\begingroup$

I found the localgauss package very useful for this. https://cran.r-project.org/web/packages/localgauss/index.html

The package contains

Computational routines for estimating and visualizing local Gaussian parameters. Local Gaussian parameters are useful for characterizing and testing for non-linear dependence within bivariate data.

Example:

    library(localgauss)
    x=rnorm(n=1000)
    y=x^2 + rnorm(n=1000)
    lgobj = localgauss(x,y)
    plot(lgobj)

Result:

enter image description here

$\endgroup$
0
2
$\begingroup$

Splitting the data into deciles based on the observed X ("Entry Point Quality") appears to be a generalization of an old method first proposed by Wald and later by others for situations wherein both X and Y are subject to error. (Wald split the data into two groups. Nair & Shrivastava and Bartlett split it into three.) It is described in section 5C of Understanding robust and Exploratory Data Analysis, edited by Hoaglin, Mosteller and Tukey (Wiley, 1983). However, a lot of work on such "Measurement Error" or "Error in Variables Models" has been done since then. The textbooks that I've looked at are Measurement Error: Models, Methods and Applications by John Buonaccorsi (CRC Press, 2010) and Measurement Error Models by Wayne Fuller (Wiley, 1987).

Your situation may be somewhat different because your scatterplot leads me to suspect that both observations are random variables and I don't know whether they each contain measurement error. What do the variables represent?

$\endgroup$
2
  • $\begingroup$ Entry Point Quality is how much a stock rises or falls in the short term at a given point in time. Industry Momentum is a measure of "momentum" in the industry for the stock at the same point in time. The hypothesis is that there is a correlation between the industry momentum and the future price of the stock. $\endgroup$
    – B Seven
    Jul 10, 2014 at 17:40
  • 1
    $\begingroup$ We generally put the response on the vertical axis. Visual inspection suggest that even if there is a real correlation, variation in entry point quality makes it pretty useless. But since you're dealing with stock prices, things get even more complex because time series are involved. $\endgroup$ Jul 10, 2014 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.