Boundary or threshold test for regression-type scatter plot I am looking for a way to test whether a boundary threshold exists in a physiological response – a sample of the data is plotted below. My hypothesis is that the X-variable imposes a physiological constraint on Y-values, thus producing a boundary 'ceiling' for maximum Y-values that decreases at higher X-values (indicated by the red line on figure). I assume any Y-values below the boundary are limited by some other factor not included in this model.
Essentially, my goal is to determine if the boundary exists and if so to derive a confidence interval for the boundary line model – similar to a linear regression model, but describing the upper bound of the Y-values, rather than the center of mass.
I'm sure something like this exists, but I haven't come across it before. Also, I would appreciate any suggestions on a better title or tags for this post – I assume there are more accurate terms for what I'm describing that would help folks find this post.

 A: Such a pattern would often occur when no "boundary" actually exists.
Here I generate X and Y as independent right-skew random variates, yet such a pattern occurs:

The impression of any sense of a boundary in my plot is completely bogus, yet it looks very similar to yours. (There's an actual vertical boundary in this bivariate distribution at $x=80$, but I could generate very similar looking plots without any boundaries at all.)
Here's the code I used to generate the plot (in R):
x = rbeta(1000,1,10)*80
y = rbeta(1000,1,3)/1.5+.3
plot(x,y,ylim=c(0,1))

Trying it a few more times it looks like about a third of the time it gives a plot that seems to have such a slanting boundary.
No doubt a little fiddling with distributions could improve the proportion of times it occurs and at the same time make it look even more like your picture (this shifted/scaled beta(1,10)$\times$beta(1,3) was the very first counterexample I tried).
Given my picture doesn't actually have any boundary there, one should be careful of over-interpreting such a pattern. You'd need a characterization of what makes it a boundary that wouldn't generate lots of false positives on examples like the one I give.
A: You can use a permutation based test for such threshold. 
Permutation-based test
It tests the hypothesis whether a "data-sparse" region above the threshold line is due to a random chance or not.
In brief:
The basic idea behind is to calculate the area of the "data-sparse" region and use it as a statistic. The next step is to randomly permute the X-coordinates of the scatter-plot and repeat the calculation of the area of "data-sparse" region. 
Probability p is the proportion of times the calculated area exceeded the original area. If p is sufficiently small the "data-sparse" region deemed to be significant. 
A: I would start by finding the "upper envelope" of your data and then representing the "envelope"  as a straight line or piece-wise linear function. 
For starters,  you could estimate the "envelope"  as a piece-wise constant function f(x) =max(yk,  given abs(x-xk) is below delta),  where delta is a parameter,  say 3 and (xk,  yk)  are your data points. Drawing a straight line through points (xk,  f(xk))  should be straightforward :) 
A: My intention on how this problem might be solved is:


*

*Calculate linear model to receive the regression line $r$.

*Calculate the normal vector $v$ to the resulting regression line.

*Shift $r$ by $v$ till all data points are under $r$.


To optimize $r$, you might rotate it by some angle $\alpha$ and stop for the best $\alpha$ you found, maybe using the Residual Sum of Squares as reference term.
Like I tried to show in this figure:
 
Another approach could be to use Support Vector machines. I don't know if this is possible with your data, but maybe you can produce some dummy points located above your data and split them from your original points using a SVM. This is just some idea I came up with. Though, I would prefer the first method.
A: This is potentially not the most robust solution. But you may be able to seriously improve the quality of the envelope using something along these lines:


*

*break down your data into n intervals (where the number of intervals depends on the density of your data)

*find the max of you data within each interval

*Pass linear regression model though the selected maximum data points.
