1
$\begingroup$

I have 3 log scatter plots that I want to establish smooth maximum and minimum lines. What is the usual mathematical method to do that? (Image and Excel file links below.)

The black lines on the scatter plot images are hand drawn. The third scatter plot is especially tricky and not amenable to a moving average plus stddev because of the data clumping. Note: This is time series data so new data constantly comes in. In other words, I cannot just use the whole data population in one shot.

Any ideas would be greatly appreciated.

Excel File: https://dl.dropboxusercontent.com/u/44057708/Three%20Scatters.xls Image at: https://dl.dropboxusercontent.com/u/44057708/ThreeScatters.jpg

enter image description here

$\endgroup$

1 Answer 1

2
$\begingroup$

I don't know that there is a usual method to do this. If the data came all at once, I'd recommend quantile regression using a spline representation of time. But in your case, they do not.

So, one relatively simple thing is a moving quantile. The $k$th smallest or largest of the last $K$ gives a first stab at this. The closer $k/K$ is to 0.5, the closer you are to the middle. The larger $k$ and $K$ are, the less this will fluctuate (including jumps, as aberrant points join or leave windows). The result won't look as smooth as your lines without more work, e.g. updating the moving quantile more smoothly with some weights.

$\endgroup$
6
  • $\begingroup$ Thank you Nick for your answer! That makes sense and is pretty close to what I am doing now. I take the 25 largest/smallest of the last 100 elements and average that. I also add a stddev amount to the max. Then smooth by damping (multiplying the change by .1 and using it). I was hoping there was a more elegant/roboust/general solution as I have to tweak the stddev and damping factors for different data sets. At least it gives me some confidence that my random-stab-in-the-dark solution may actually be sound. :) $\endgroup$ Commented Mar 25, 2014 at 11:10
  • $\begingroup$ Your approach is similar to spirit to William S. Cleveland and Beat Kleiner. 1975. A graphical technique for enhancing scatterplots with moving statistics. Technometrics 17: 447-454 Adding an SD sounds rather ad hoc, however. $\endgroup$
    – Nick Cox
    Commented Mar 25, 2014 at 11:33
  • $\begingroup$ Wow! Thank you Nick. I will look into the article. Yes. Exactly. The SD on the maximum is ad hoc. The average pulls the max too low and could not think of a different way to pull it back up again. Any thoughts? $\endgroup$ Commented Mar 25, 2014 at 13:28
  • $\begingroup$ As I first posted, you don't have to take the average; you can just select a particular quantile. $\endgroup$
    – Nick Cox
    Commented Mar 25, 2014 at 13:30
  • $\begingroup$ Thank you Nick! That solves it. I've changed my quantile to the 10 percent. Unfortunately, all the versions of that Technometrics article are behind paywalls. It must be good. I appreciate you sharing your expertise, time and your bright display of kindness. I'm sure there is a wonderful place in heaven for generous mathematicians :) $\endgroup$ Commented Mar 27, 2014 at 2:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.