The question is: Can I delete all the rows of a certain id because of its missing data, before the data analysis? To be more specific I will give the context and the specific problem:
I have a daily time series dataframe from 1960 to 2018, where each observation is a measurement of precipitation, temperature, and level related to 133 rivers (the ids are the rivers). So, for each river, I will have all the measurements needed, for a given day, in a given year, between 1960 and 2018. I need to predict the river level.
I did this missing value plot to get some insights (the lower the percentage, the lower is the data for that flow in that year):
There are three insights from this plot that concern me in order to delete or not delete missing values (and when):
- There is an increasing number of distinct rivers per year since 1960.
- 2018 has a lot of missing data (which is because all the data stops at March of that year).
- You have some rivers without data for the last 35 years (83 to 87 rivers in the plot, for example). And you have some rivers without data for the last 10, 5, or 2 years (you can see a lot of rivers with 2017 and 2018 data missing in the graph).
Should I consider all the rivers data for prediction even of those rivers that don't have data for the last X years? Because it doesn't make sense to predict the next year without the data of the last X years (but I do have the data of 1980 to 2000, for example). If I should consider it, what kind of methodology should I use to predict, if is there any?. If I shouldn't consider it, should I delete it? If I should delete it, when should I delete it? Before or after the data analysis for prediction? For example, before or after analyzing the distributions? (in order to group the rivers to impute the missing values as they have different distributions, but I could find a way to group similar distributions). It makes sense to me to delete them before the analysis, as the noise for prediction will be lower as the data for the analysis will be consistent with the data for prediction.
However, if I don't delete them at all I will have more data if I want to see some overall analysis (i.e. the trend from 1960 to 2018 will be a more "representative trend"). Deleting after, in terms of prediction (and insights for feature selection) it will increase the noise as the data for analysis will be different to the data for prediction (as I will have deleted complete data for some rivers).
Besides, I'm thinking of making the assumption that if the river doesn't have data from the last 3 years, the station measuring the river is not operative. But again, if the station is not operative, should I consider its data for overall analysis/insights?.