I have a time-series dataset that has 120 missing rows due to consecutive network issues and I am trying to impute these values using MICE in Python. As the source of missingness is a total disconnection, all columns are missing from those 120 rows. First, I am trying to identify if it is MCAR or MAR. As missingness is happening due to an external factor, it seems to be MCAR. On the other hand, I can say that if feature 1 is missing, the same is true for features 2,3,4 and 5 as we have not received any data at that time. So I can say there is a relationship between missingness so it cannot be MCAR. I am confused about the type of missingness and how to choose a logical imputation method.
2
-
$\begingroup$ Do you have consecutive 120 missing rows? or random? $\endgroup$– Atul MishraMar 24, 2022 at 6:04
-
$\begingroup$ consecutive because there was a networks issue $\endgroup$– HannaMar 24, 2022 at 17:02
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
7
MICE is usually used for imputation in iid data. Say you have lots of iid measurements, where each should consist of $k$ attributes $(a_i)_{i=1}^k$, but in some of those measurements, the number of observed attributes is smaller than $k$. Then MICE helps you to impute the missing attributes from the present (not missing) attributes. But in your case, each measurement contains either all attributes or none, so you cannot deduce the missing attributes of an incomplete measurement from those still present.
The notions of MCAR and MAR are also usually referring to the situation of iid data: MCAR (missing completely at random) means that the fact that a certain attribute $a_m$ is missing is completely independent of the values of any of the attributes $(a_i)_{i=1}^k$, including the non-observed attribute $a_m$. And MAR (missing at random) means that the fact that a certain attribute $a_m$ is missing is independent of the values of the observed attributes. So, again, in your case, those notions are not really applicable.
But you say that your data is a time series. I.e. you don't have iid data, but some time-dependent series of vectors of attributes $(a_{tk})$, $k=1,\ldots,n$, $t\in\mathbb{Z}$, where there is some historical dependency, i.e. dependencies between attribute values of different times. So I would suggest you exploit this dependency, but not with the packages dealing with iid data, but with imputation software for time series.
There are many possibilities to do this, from the easiest which is just replacing the missing attributes with the mean over time of that attribute, or those doing simple interpolation, up to the complex ones which use (vector-valued) time series analysis or deep learning. Which is the best for you depends on your actual goal and your context. As a possible first approach, you might want to look into time series imputation with state-space models (using their smoothing capabilities), since they also deal with vector-valued time series and don't require special hardware like deep learning approaches.
Unevenly spaced time series
You can also look at the problem in a more general way, by considering unevenly spaced (or irregular) time series. For many time series analysis methods it is presumed, that they are evenly spaced (or regular), i.e. the time intervals between measurements are all of equal length. If this is not the case, you have mainly the following options:
- If the unevenness of your time series (TS) consists only of some missing events and the TS is otherwise regular, then you can try some kind of time series imputation, as mentioned above.
- Otherwise, if you know (using some expert knowledge) that you can shift the times of your events slightly, so as to obtain an evenly spaced TS, without changing the "characteristics" of your TS too much, this should be given a try.
- Finally, if the above is not applicable, i.e. you have some fundamentally irregular TS, you can try to apply some of the rarer TS methods designed for unevenly spaced TS.
For the last option, software is available in R, Julia, and Python, and references can be found e.g. at the above Wikipedia link.
-
$\begingroup$ +1 Thank you for addressing this in such a thoughtful, helpful way. $\endgroup$– whuber ♦Mar 24, 2022 at 14:38
-
$\begingroup$ Thank you for your answer. I want to explain more about how missing data is produced. I have good periods when there is no missing data and bad periods when all data is missing. These periods are generated using np.random.exponential and then imposed to the original dataset to simulate what might occur when we receive data from an unreliable network. Do you still think it is not iid data? $\endgroup$– HannaMar 24, 2022 at 17:11
-
1$\begingroup$ If two vectors $\mathbf{a}_t, \mathbf{a}_s, s\ne t$ are independent of each other for all pairs of times $s$ and $t$, then those vectors $\{\mathbf{a}_t\}$ are iid. But of course, you might still have lots of dependencies between the elements $a_{tk}$ of the vector (for fixed $t$). If the vectors $\{\mathbf{a}_t\}$ are not independent, i.e. there are some historic dependencies, one usually calls this a time series. When you said you have a time series dataset, I assumed you meant it in this way. $\endgroup$– frankMar 24, 2022 at 18:32
-
$\begingroup$ You're right. I have a time series dataset and timestamps for each row. I was thinking about an imputation method that can estimate missing values in real-time based on what we had received before the bad period. $\endgroup$– HannaMar 24, 2022 at 19:47
-
$\begingroup$ Yes, and for this imputation using earlier data, you should use the methods of time series analysis as described above. When you talk about a "bad period", how long is this "period" compared to the interval between two consecutive time stamps in a good period? $\endgroup$– frankMar 25, 2022 at 4:45