What is the best statistics way to identify the gap of a plot? For example, I have the following plot, where y-axis is the values and x-axis is the index of the data point. 

There are clearly two gaps in this plot, and I wonder what is the best way to identify these two gaps. 
I hope the method could be based on statistics without fancy machine learning methods with regularizers, because I won't have development data set to tune the weight of a regularizer. 
The one way I found can work is to calculate the mean and variance of the first fifty values (let's say we assume there are at least half of the values are in the first family), and test whether following points are five standard deviations away from this mean. This method works very well for to identify the second family, but the problem is that, since I don't know how many members will be roughly in the second family, it's hard to calculate the mean and variance for the second family. In addition, the number five seems to be very arbitrary. (A non-arbitrary choice might be 1.96, corresponding to the 5% of the Gaussian mass and 5% is more or less a standard choice of hypothesis testing, but 1.96 does not work.)
Are there any suggestions for the methods to identify such a gap?
 A: Since you are already onto a solution after @usεr11852's comment I'll just outline my suggestion instead of explaining a full solution.
This is similar to your idea of checking if the next point in the plot lies within 1.96 standard deviations.


*

*Begin by taking the first 3 points on your graph and do a linear regression for them

*Calculate a 95% prediction interval for the value of $y$ given the next value of $x$ on the graph. A 95% prediction interval for a given $x$ tells you the range of values where you will expect future observations (i.e. the points not used in the regression) Page 11 of these slides show the formula for a prediction interval.

*If the next point is outside the prediction interval then it's probably not part of the line, if it is within the prediction interval then repeat the regression with this point included and make the prediction interval for the next point

*Once you find a point which isn't in the prediction interval then that is where the gap starts


Once subtle point is that this repeated testing with the prediction interval isn't really giving you 95% certainty for the point where the gap starts. Since you are repeatedly testing with a 95% interval you should eventually find that a point is statistically significant even though it just a fluke.
To overcome this you should adjust the significance based on how many points you have already tested. If you already compared the prediction interval to $n$ points and you want 5% significance then use an adjusted significance of $1-(1-0.05)^{\frac{1}{1+n}}$. This comes from the Sidak correction
