6
$\begingroup$

I have a dataset of Total Sales from 2008-2015. I have an entry for each day, and so I have a created a pandas DataFrame with a DatetimeIndex and a column for sales. So it looks like thisenter image description here

The problem is that I am missing data for most of 2010. These missing values are currently represented by 0.0 so if I plot the DataFrame I get

enter image description here

I want to try forecast values for 2016, possibly using an ARIMA model, so the first step I took was to perform a decomposition of this time series

enter image description here

Obviously if I leave 2010 in the DataFrame any attempted prediction will be skewed by the apparent, albeit erroneous, drop in sales.

What is the recommended approach in this situation? I think I should just drop 2010 altogether, but then I don't know if my time series is valid going from 2009 to 2011. I don't want to fill the missing values, because I don't believe I can do so accurately.

If I just delete 2010, however, the plot 'fills in' 2010 which doesn't help me

sales = sales.drop(sales['2010'].index)

enter image description here

$\endgroup$
1
  • 2
    $\begingroup$ imputation is the statistical fields which attempts to make inferences in the presence of missing data. Perhaps that might be helpful here. $\endgroup$
    – Sycorax
    Aug 1, 2016 at 20:25

1 Answer 1

4
$\begingroup$

Keep the 2010 dates - but make the values 'nan' . If you have a timeseries Dataframe, then it's as simple as:

newDf = old.set_value('2010', 'Total Sales', float('nan').

if your data drop out isn't exactly 2010, you can replace 0s with nans:

new = old.replace([0], float('nan'))

This will cause a "pen lift" (if you can imagine an old pen plotter, nan caused a pen lift, therefore matlab, and consequently matplotlib, emulated that behavior).

If you do this, you need to make sure your analysis routines can handle nans properly. (particularly any time filtering, like MA, across the gap.

Finally, I would strongly suggest moving this kind of question to StackOverflow, since it's more of a Pandas question than an analytical question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.