How to guess when a number is "far away" from expected value I have a bunch of plots that look like the following:

Eyeballing this, it's clear that we have a bunch of values that follow a pattern which we can predict from a linear model, but then a couple that are "outliers". What are some accurate ways we can identify these outliers?
My thoughts
One way to go about this is to create a LM to fit this plot, and then look for points that are some threshold away from that LM. However, we see that there are a bunch of outliers that also follow a linear pattern; if there were enough of these, it could skew the LM so that it would be much more difficult to identify real outliers. Another idea is to assess the mean and SD of this dataset, and if a point is far enough away (determined by z-score) we say it's an outlier. Just a few thoughts; any ideas or thoughts would be helpful.
 A: Use a robust non-parametric fit, such as Loess, to smooth the points.  A boxplot of the residuals will separately indicate all those that are "far outliers." 
The code below computes those far outliers using a standard (robust) method very similar to the one popularized by John Tukey.  Assuming the robust fit captures the main trend of the points, up to 25% of the data can be high outliers and another 25% low outliers and still be detected.  An advantage of Loess is that it can be "tuned" to local parts of the plot or to follow the global trend; so if it seems to be affected by clumps of outliers, make it smooth more agressively (by modifying the f parameter in this implementation).

The Loess fit is drawn as a black curve.  The automatically identified outliers have been overplotted in red.
How well does it work?  Let's make the random errors fifty times greater and try again:

That's about its limit: if we increase the random errors still further, the "noise" around the main body of points will grow to absorb the outliers one by one.  Experiment by changing the sd parameter in the calculation of y.
#
# Generate sample data.
# Comment out `set.seed` to vary the data on subsequent runs.
#
set.seed(17)
x <- seq(250, 950, length.out=100)
y <- 120*(1 - exp((x-2000)/2000))
i <- runif(length(x)) < 1/8
y[i] <- y[i] + 20 * ceiling(rnorm(sum(i)))
y[!i] <- y[!i] + rnorm(sum(!i), sd=0.1)    # `sd=5` for the second figure
#
# Plot the data.
#
par(mfrow=c(1,1))
plot(x,y, ylim=c(0, max(y)), col="#404040", xlab="", ylab="")
#
# Fit the data with a robust method and plot the fit.
#
model <- lowess(y ~ x)
lines(model, lwd=2)
#
# Identify the extreme residuals and plot them.
#
residuals <- y - model$y
hinges <- quantile(residuals, c(1/4, 3/4))
step <- diff(hinges)
fences <- 2.5*step*c(-1,1) + hinges
far.out <- residuals < fences[1] | residuals > fences[2]
points(x[far.out], y[far.out], pch=19, col="Red")

