# How does lowess handle gaps in time series?

I have been using the lowess smoother to calculate trends for time series data for a while now but until now my data was always without gaps.

I now have to work with data where there are quite large gaps in time (~2h for minute sampled data so ~120 missing samples) and by reading the documentation and looking at the actual implementation of the lowess smoother in the statsmodels package I couldn't really find out how it handles missing data.

The produced trend for data is obviously not correct and the console output confirms that this somehow causes a problem since I get warnings regarding dividing by zero or another invalid number.

Any fraction of data over 0.03 results in a trend that reaches 1e26 while the data is closer to 0.4. Lower values produce a trend which does not follow the data as it should with such a low value. Does anyone know how exactly gaps are handled or if I have to divide my data into chunks without gaps?

EDIT: I just found the error, numpy and python integers have different sizes so for my data some timestamps were suddenly negative resulting in the spiky trend.

• From the documentation missing : Available options are ‘none’, ‘drop’, and ‘raise’. If ‘none’, no nan checking is done. If ‘drop’, any observations with nans are dropped. If ‘raise’, an error is raised. Default is ‘drop’. Commented Jun 21, 2023 at 9:17
• Yes, I read that too but that flag is only for NaN values in the time series while my series has a gap without values. The measurement stopped and was continued later so also the timestamps in between are missing. Commented Jun 21, 2023 at 9:19
• In that case it's not different than if you had some numeric values in the range 50-100 and 150-200 (example), with a gap between 100-150 of no values, it would just extrapolate some values which may or may not make sense. Commented Jun 21, 2023 at 9:22
• I thought the same, however it seems like the gaps also introduce errors in later parts of the trend where sufficient data is given. Additionally, the calculated trend also has the same amount of values as the original data, so if there was any kind of interpolation it would have been only internally which makes it way harder to understand what is actually happening. Commented Jun 21, 2023 at 9:33
• A little note for people who land in this question (probably not the OP's case): missing data in time series can be either "real" missing data or zeroes. I've seen lots of sales and demand forecasts done that considered missing data as real missing data, because no sales had been recorded. Missing data in sales (or demand) often means there were no sales, or that sales were zero. Treating those cases as "missing data" will throw a mathematically-correct erroneous forecast. Commented Jun 21, 2023 at 17:32

It doesn't do anything with missing data by itself. If there is some default handling of missing data in the implementation of the algorithm you are referring to, this should be described in the documentation.

LOWESS is a (local) regression model, so it predicts something as a function of something else. If I understand correctly, in your scenario, you are dealing with time-series, so you predict the values $$y_t$$ using their time index $$t$$. In such a case, as in any other regression model, it will interpolate the intermediate points. This is exactly what you see in the example from the R's documentation for lowess: there were no cars with speed equal to 21 and the regression line interpolates between what it predicted for the nearby points. In your case, the $$x$$-axis would be time.

If you are using some kind of errors, they are due to how you are using the software and what exactly is your data. You didn't give us much details on it, so it is hard to comment.

• My data consists of physical measurement values that are sampled each minute with corresponding timestamps. During the measurement, there was a 2h downtime during which no measurement was done resulting in a 2h gap without timestamps or measurement values. Since I am interested in whether a trend continues over the day my plan was to keep the gaps to retain any effects caused by it. Sadly I am not allowed to share plots or the data itself but it seems like the gaps have a lasting effect on later values of the lowess trend even with sufficient data given at that point in time. Commented Jun 21, 2023 at 9:29
• @Krautsultan I can't comment on the results you are seeing without knowing what they are, but the answer still applies: lowess would interpolate for the points that were not seen in the data. If lowess does not work for your data, it may be the case that you simply need to use something else.
– Tim
Commented Jun 21, 2023 at 9:32
• It uses linear interpolation Commented Jun 21, 2023 at 12:12

While OP found an error in the data, I found a workaround for lowess in R. I also had gaps in the data and the lowess returned randomly "trimmed" column, omitting more rows than there were actually missing. What I ended up doing is rearranging the data so each column of data had its own Year column without the years that had no data. This way there wasn't one Year column as rownames because this necessitated having NAs in the columns which had gaps. This way lowess run its smoothing on data without gaps, and then I joined all the data back together by rownames i.e. Year.

Not elegant, but worked.