It looks like you mainly want to reject data at time 3000 and 28000. If you're going to believe all your data points, the dashed line looks reasonable. But you want to reject or replace the outliers, to achieve robust smoothing.
One specific approach to answer the question is to make a pass through the data in time order using a spike filter. That's very simple - see an explanation and pseudocode at http://gregstanleyandassociates.com/whitepapers/FaultDiagnosis/Filtering/Spike-Filter/spike-filter.htm That would reject the first obviously bad point,and at least flag the last point as questionable, depending on the maximum count parameter. That particular approach is a filter, not smoothing (at any point, only looking backwards, not ahead), so it wouldn't be optimal for your case, where at any time (except the beginning and end) you could look backwards or forwards. That spike filter would carry over only past values instead of considering both past and future, which would distort the curve slightly.
It would be better to either delete the outlier or replace it with a locally-interpolated value. You could still do this with one pass in time order. I'd modify the above filter to just mark the data until it was resolved as a spike or a longer-lasting change (resolved after the time period specified by the count parameter). When a spike is detected, then either delete those marked points or fill in with interpolated values rather than a previous value. The choice of deleting the marked points or filling in with a locally interpolated value is could be based on whether or not you need values at fixed intervals, or also whether you want to have one global function or something based on nearby points, as in Savitzky-Golay smoothing.
Savitzky-Golay smoothing is an extremely efficient implementation of least squares smoothing over a sliding time window (a convolution over that data) that comes down to just multiplying the data in each time window by fixed constants. You can fit values, derivatives, second derivatives, and higher. There's a good article in Wikipedia, although it's referred to as Savitzky-Golay filter.
Savitzky-Golay smoothing is specifically not for robust smoothing. It was derived based on a least squares fit of data within a time window of nearby points. Least squares answers are inherently not robust, because the squared objective function gives too much weight to outliers. But, in conjunction with the above spike detection logic, it might be used in local interpolation once the spikes are removed, and might be used for all data points after outlier removal.
Since your data doesn't appear to be regularly sampled (fixed sample time) anyway, you couldn't use Savitzky-Golay. So you would just delete the points marked as outliers. The above is included for others with similar questions with regularly-sampled data.