# Boundary or threshold test for regression-type scatter plot

I am looking for a way to test whether a boundary threshold exists in a physiological response – a sample of the data is plotted below. My hypothesis is that the X-variable imposes a physiological constraint on Y-values, thus producing a boundary 'ceiling' for maximum Y-values that decreases at higher X-values (indicated by the red line on figure). I assume any Y-values below the boundary are limited by some other factor not included in this model.

Essentially, my goal is to determine if the boundary exists and if so to derive a confidence interval for the boundary line model – similar to a linear regression model, but describing the upper bound of the Y-values, rather than the center of mass.

I'm sure something like this exists, but I haven't come across it before. Also, I would appreciate any suggestions on a better title or tags for this post – I assume there are more accurate terms for what I'm describing that would help folks find this post.

• Out of curiosity do you have a theory where the line should be beforehand? It is pretty simple to draw the hull on any plot, so finding the boundary ex-post facto beyond those points is difficult. Jul 8, 2014 at 17:38
• I don't have a theory about where the line should be--this is "discovery" work. We are trying to figure out if a change in an abiotic factor (x-axis) affects a plant's capacity to produce a chemical defense compound (y-axis). The hypothesis is that there is a biophysical limit that creates a 'ceiling' effect on the Y-axis attribute. Many other factors influence the Y-axis value, so the X-axis can be thought of as a necessary, but not sufficient condition. Consequently, a regression equation is pretty unhelpful to understanding the effect. Jul 8, 2014 at 19:23
• I have made some minor edits for spelling and punctuation but I also removed the phrase "varies inversely" because it seems clear from the diagram you want it to decrease linearly rather than to have the curved envelope $y_{max} \propto \frac{1}{x}$. Feel free to change it back if this was actually your intention. Feb 4, 2015 at 13:59

Such a pattern would often occur when no "boundary" actually exists.

Here I generate X and Y as independent right-skew random variates, yet such a pattern occurs:

The impression of any sense of a boundary in my plot is completely bogus, yet it looks very similar to yours. (There's an actual vertical boundary in this bivariate distribution at $x=80$, but I could generate very similar looking plots without any boundaries at all.)

Here's the code I used to generate the plot (in R):

x = rbeta(1000,1,10)*80
y = rbeta(1000,1,3)/1.5+.3
plot(x,y,ylim=c(0,1))


Trying it a few more times it looks like about a third of the time it gives a plot that seems to have such a slanting boundary.

No doubt a little fiddling with distributions could improve the proportion of times it occurs and at the same time make it look even more like your picture (this shifted/scaled beta(1,10)$\times$beta(1,3) was the very first counterexample I tried).

Given my picture doesn't actually have any boundary there, one should be careful of over-interpreting such a pattern. You'd need a characterization of what makes it a boundary that wouldn't generate lots of false positives on examples like the one I give.

You can use a permutation based test for such threshold.

Permutation-based test

It tests the hypothesis whether a "data-sparse" region above the threshold line is due to a random chance or not.

In brief:

The basic idea behind is to calculate the area of the "data-sparse" region and use it as a statistic. The next step is to randomly permute the X-coordinates of the scatter-plot and repeat the calculation of the area of "data-sparse" region.

Probability p is the proportion of times the calculated area exceeded the original area. If p is sufficiently small the "data-sparse" region deemed to be significant.

I would start by finding the "upper envelope" of your data and then representing the "envelope" as a straight line or piece-wise linear function.

For starters, you could estimate the "envelope" as a piece-wise constant function f(x) =max(yk, given abs(x-xk) is below delta), where delta is a parameter, say 3 and (xk, yk) are your data points. Drawing a straight line through points (xk, f(xk)) should be straightforward :)

My intention on how this problem might be solved is:

1. Calculate linear model to receive the regression line $r$.
2. Calculate the normal vector $v$ to the resulting regression line.
3. Shift $r$ by $v$ till all data points are under $r$.

To optimize $r$, you might rotate it by some angle $\alpha$ and stop for the best $\alpha$ you found, maybe using the Residual Sum of Squares as reference term.

Like I tried to show in this figure:

Another approach could be to use Support Vector machines. I don't know if this is possible with your data, but maybe you can produce some dummy points located above your data and split them from your original points using a SVM. This is just some idea I came up with. Though, I would prefer the first method.

This is potentially not the most robust solution. But you may be able to seriously improve the quality of the envelope using something along these lines:

1. break down your data into n intervals (where the number of intervals depends on the density of your data)

2. find the max of you data within each interval

3. Pass linear regression model though the selected maximum data points.

• Interesting thought, thanks for the idea. To clarify though--do you mean something like, classing the data into equal sized bins (i.e. 10 bins that each contain 10% of data points) and then looking at the maximum value (or average of top few, to avoid outlier effects) in each bin and doing linear regression on only those points? I've thought about doing something like that in similar circumstances, but I haven't seen anybody in my field use a technique like that. Are you aware of any published examples of something like that? Jul 4, 2016 at 17:46