We have measured two variables, and the scatterplot seems to suggest multiple "linear" models. Is there a way to try to distill those models? Identifying other independent variables has turned out to be difficult.

Scatterplot of the two variables

Both variables are heavily left-skewed (towards the small numbers), this is an expected distribution in our domain. The intensity of the dot represents the amount of data points (on a $\log_{10}$ scale) at this $<x,y>$.

Alternatively, is there a way to cluster the points?

In our field, it is claimed that these two variables linearly correlate. We are trying to understand/explain why is it not the case in our data.

(note, we have 17M data points)

update: thank you for all the answers, here are some requested clarifications:

  • Both variables are integer only, which explains some of the patterns in the log scatterplot.
  • Luckily, by definition the minimal value of both variables is 1.
  • 7M points are at $<3,1>$ ("explained" by the left-skewness of the data)

Here are the requested plots:

log-log scatterplot: Scatterplot in log log

(the blanks are caused by the integer values)

log-log polar: polar coordinates $\theta = y$

Histogram of ratio: Histogram of ratio

The frequency is on a log scale, since the $1/3$ bar is 7M points, and would hide the other bars.

  • 2
    $\begingroup$ What does this plot look like in polar coordinates $(r,\theta)$? (It might be advisable first to take logarithms of $X$ and $Y$ (plus, if needed, a small starting offset to avoid zeros).) Since all lines appear to be radiating from the origin, then conceivably--especially if the $\theta$ variation around the lines appears homoscedastic--then all you need to do is cluster the points in the $\theta$ dimension. $\endgroup$ – whuber Oct 15 '14 at 22:47
  • $\begingroup$ Are there ratios involved in obtaining Y and X? Are variables that take only discrete values involved? How does it look as a log-log plot? $\endgroup$ – Glen_b Oct 16 '14 at 0:51
  • 1
    $\begingroup$ @whuber & Glen_b I've added plots with those transformations. $\endgroup$ – Davy Landman Oct 16 '14 at 11:03
  • $\begingroup$ Thank you, Davy. I should have been clearer about the point of using polar coordinates: by plotting $r$ on a horizontal axis and $\theta$ on a vertical axis, any radial lines on the original plot will emerge as perfectly horizontal lines. Not only can they be easily detected visually (our eyes have built-in processing to recognize horizontal linear features), once detected, they can be processed with a cluster analysis based solely on $\theta$. Your "log-log polar" plot, by applying nonlinear transformations to the coordinates (especially $\theta$), destroys these nice properties. $\endgroup$ – whuber Oct 16 '14 at 14:20
  • $\begingroup$ @whuber I've updated the plot, put the theta on the y, is this the lines you mean? $\endgroup$ – Davy Landman Oct 16 '14 at 16:06

You may have artefacts arising from restrictions on what is possible physically or on what is recorded (at the simplest, integers only). Completely anonymous $Y$ and $X$ don't suggest any confident guesses about how that arises, but it looks as if some $Y/X$ are favoured and I would certainly look at the distribution of that ratio. Also, if so it's not in my experience useful to look for separate models unless you really are mixing quite different situations. (For "physically" read "biologically" or whatever adverb makes sense.)

The more I look at this, the more I guess that lines such as $X/k$ or $kX$ are evident for integer $k$, because the values themselves are integers.

A different but possibly related point is that to me these data cry out for transformations. If they are all positive, logarithms are indicated. I fear that you have zeros, in which case what to do is open to discussion. For example, a line at $Y = 0$ may be guessed at from your graph. If there are zeros, some swear by $\log(Y + \text{constant})$ or cube root should help. Whatever helps you see patterns more clearly is defensible.

A point of terminology: skewness in statistics is described with reference to the tail that is more stretched out. You're free to regard this terminology as backwards. Here both variables are skewed to high values or positively or right-skewed.

UPDATE: Thanks for the extra graphs, which are most helpful. Almost all guesses appear confirmed. (The bottom line, so to speak, is $Y = 1$, not $Y = 0$.) The stripes are artefacts or secondary effects of using integers, which may well be the only, or at least the most practical, way of measuring what you are measuring (about which the question remains discreet). The log-log and other plots expose the discreteness. So despite the discretion, the discreteness is confirmed. There are pronounced modes (peaks in distribution) for the ratios 1/4, 1/2, 1/1 and 2/1.

As before, I wouldn't advise modelling different stripes differently without a scientific reason to distinguish them or treat them separately. You should just average over what you have. (There may be known methods with this kind of data to suppress the discreteness. If people in your field routinely measure millions of points for each plot, it is hard to believe that this has not been seen before.)

The correlation should certainly be positive. Apart from a formal significance test, which here would be utterly useless as minute correlations will qualify as significant with this sample size, whether it is declared strong is a matter of the expectations and standards in your field. Comparing your correlation quantitatively with others' results is a way to go.

Detail: The skewness is still described the wrong way round according to statistical convention. These variables are right-skewed; that jargon fits when looking at a histogram with horizontal magnitude axis and noting that skewness is named for the longer tail, not the concentration with more values.

  • $\begingroup$ I've added log-log plot, and tried to be more precise about skewness. $\endgroup$ – Davy Landman Oct 16 '14 at 11:03

The tool you want, I think, is called switching regression. The idea is that there are several regression lines, and each data point is assigned to one of them. For example, the equation of the first regression line would be: \begin{align} Y_i &= \alpha_1 + \beta_1X_i + \epsilon_i \end{align} The equation of the $m^{th}$ regression line would be: \begin{align} Y_i &= \alpha_m + \beta_mX_i + \epsilon_i \end{align} In total, there are $M$ different regression lines, say. For any given data point, we only get to see one of the regression lines. Thus, there has to be some mechanism for deciding which regression line we see for each point. The simplest mechanism is just the multinomial distribution. That is, we see the $m^{th}$ regression line with probability $p_m$, where $\sum_m p_m =1$.

The model is usually estimated by maximum likelihood. Assuming that the $\epsilon$ are distributed $N(0,\sigma^2)$, the likelihood function you would maximize would be: \begin{align} L(\alpha,\beta,\sigma) = \sum_{m=1}^M p_m\frac{1}{\sigma}\phi\left(\frac{Y_i-\alpha_1-\beta_1X_i}{\sigma}\right) \end{align} The function $\phi$ is the standard normal density. You maximize this in the $3M+1$ parameters, subject to the constraints $\sum_m p_m=1,\; p_m\ge0$. This is usually a somewhat cranky maximization problem if you are going to use quasi-Newton methods to solve it. You can't just start all the $\alpha$ and $\beta$ at zero and the $p_m$ at $\frac{1}{M}$, for example. You have to give distinct starting values to the $\alpha$ and $\beta$ so that the algorithm can "tell them apart."

There are a number of ways to make this more involved if you want to. Maybe you have a variable $Z_i$ which you think influences $p_m$, that is which influences which regression is chosen. Well, you can use a multinomial logit function to make $p_m$ be a function of $Z_i$: \begin{align} L(\alpha,\beta,\sigma) = \sum_{m=1}^M \left(\frac{exp(\delta_m+\gamma_mZ_i)}{\sum_{m'} exp(\delta_{m'}+\gamma_{m'}Z_i)}\right)\frac{1}{\sigma}\phi\left(\frac{Y_i-\alpha_1-\beta_1X_i}{\sigma}\right) \end{align}

Now there are $5M+1$ parameters. Actually, there are $5M-1$ parameters because there is a normalization required on the $\delta, \gamma$ --- read up on the multinomial logit for an explanation.

Another way to make it more involved is to use some method for choosing $M$, the number of regression lines. I'm pretty casual about this kind of choice in my own work, so maybe someone else can point you towards the best way to choose it.

  • 2
    $\begingroup$ This can be a natural model when there are a few different regimes and some independent rationale for why they exist. Here there are so many diagonal stripes -- and it may be guessed that more would be evident on logarithmic scale -- that the problem of choosing $M$ is paramount for this approach, not incidental, as seems to be implied here. $\endgroup$ – Nick Cox Oct 15 '14 at 13:36

I have observed similar behavior in some of my data sets. In my case the multiple-different lines were due to quantization error in one of my processing algorithms.

That is, we looking at scatter plots of processed data, and the processing algorithm had some quantization effects, that caused dependencies in the data that looked exactly like you have above.

Fixing the quantization effects, caused our output to look far smoother and less clumped.

As for your "linear correlation" comment. What you presented is insufficient to determine if this data is linear correlated or not. That is, in some fields, a correlation coefficient of > 0.7 is considered strong linear correlation. Given that most of your data is near the origin, it is quite conceivable that your data is linearly correlated relative to what "conventional wisdom" would say. Correlation tells you very little about a data set.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.