Automated detection of outliers in one dimensional data I have several datasets that I need to be able to fit (the goal is to find the outliers). The datasets were created by groups of images and the x is an index number of the image and y is a focus measure of an image. 
When I am plotting x vs y(focus measure) every set looks like the trend has a different shape (it could be 2 peaks (Gaussians or Lorentzians) or one peak (Gaussian) because of the structure of the image (these are complicated images and they have different structures which accounts for the differences). When I look at the plots there are several points that are very far away from the main trend, it is very easy to see them by eye, but what I want is to fit the data so that I can find the data points (using a program) that are further away than 3 standard deviations from the fit line. Because there are thousands of those datasets, I can not look at them one at a time and decide which curve shape fits the best, I need a more generic way to fit the data.  I am using Matlab to do that.
Is there a way to find the best curve fit programmatically without knowing the shape of the curve? I am not very familiar with Machine Learning but this might be something that I can do using ML but I am hoping of a suggestion of a specific method to do that.
 A: You will need to use some kind of non-parametric scatter plot smoothing to do this.  There are many available algorithms, loess, kernel smoothers, smoothing splines, etc.  I'll use smoothing splines because they are amenable to your second request for confidence bands.  It may be possible to use other smoothing algorithms though.
Note: I apologize that I cannot offer a solution in Matlab, as I do not have access to that software.  I will explain the procedure in R, and hopefully you can translate it into Matlab.
Here's the step by step:


*

*Fit a smoothing spline to your $x, y$ data, and let it choose the appropriate smoothing parameter by cross validation.  The matlab documentation for this kind of fit is here.  The R documentation is here.

*Calculate the confidence bands around the smoothing spline.

*Use these confidence bands to detect your outliers.


First, I created a signal that seems like what you're describing:
sig <- function(x) {x - 2*x^2 - 1*exp(-(x-.5)^2/.05)}


Next I generated some data. $95$ observations are tightly varied around the signal, $5$ are given much higher variance, these are the outliers I'm attempting to identify:
x <- runif(n=95, min=0, max=1)  # Tight data
y <- sig(x) + rnorm(95, 0, .1)
xn <- runif(n=5, min=0, max=1)  # Outlier data
yn <- sig(xn) + rnorm(5, 0, 1)

I concatenated these two sets to give an approximation of what you're seeing for your data sets:
x <- c(x, xn); y <- c(y, yn)


Next I fit a smooth.spline to the data.  This fit comes with a smoothing parameter $\lambda$ that controls how closely you want the spline to fit the observed data - between fitting a trend line at one extreme, to interpolating the data completely at the other extreme.  The default behavior of smooth.spline in R is to determine a best smoothing level $\lambda$ by cross validation, I'm just going to let it do that:
library(splines)
sp <- smooth.spline(x, y)


Now we need to determine the confidence bands.  The mathematical details of how to do this with smoothing splines takes some careful study.  Practically, I followed this answer:
res <- (sp$yin - sp$y)/(1-sp$lev)       # Get jacknife residual at each data point
sigma <- sqrt(var(res))                 # Calculate stddev of jackknife residuals
upper <- sp$y + 5.0*sigma*sqrt(sp$lev)  # Create confidence bands
lower <- sp$y - 5.0*sigma*sqrt(sp$lev)

You can see, I increased it to a $5\sigma$ band, you can tune this parameter to your liking.  Plotting the bands gives a a good indication we're on the right track:

Now I can attempt to detect my outliers by finding which data points lie outside of these confidence bands:
is_outlier <- sp$yin > upper | sp$yin < lower
xoutliers <- sp$x[is_outlier]
youtliers <- sp$yin[is_outlier]

Plotting these along with the fit and the confidence bands shows that we got it right:

