2
$\begingroup$

Suppose that I got a dataset that measures different daily meteorological data from one active weather station for a span of 16 years.

The structure of the data is as follows: all columns are the different weather and climate data, and all rows are all of the days in the dataset (think 16 * 365 days).

It just so happens that there are missing values in this dataset, most of them close to the beginning of the period (2000 - 2016). Closer inspection of the data reveals that the missing data are often in spans of time and not individual days (i.e. a week of missing data, a month, a few months etc.)

My professor suggest that I just do manual replacement of missing days with data from the preceding day, but that proves much of a challenge since, to me replacing a week's worth of missing values with values from the preceding week sounds weird to me.

I have found techniques such as imputation and the accounts of missing data mechanisms, but I'm not sure if they would suffice.

From my research, the data to be predicted is not MNAR, i may just be wrong though.

My real question is: Is manual replacement of values even worth it with such large gaps in the dataset?

Thanks to everyone who replies.

Improve this question
$\endgroup$
1
  • $\begingroup$ I have run into trouble with missing meteorological data, because they can be absent for important reasons, such as discovery that a gauge had been failing or was destroyed in a storm and then was replaced. It therefore looks risky to assume you can impute the values as if these gap periods had been randomly chosen. Moreover, sometimes such gaps are the only warning you get that some important change has occurred in the data series. It can pay, therefore, to begin by comparing the data just before a gap to the data just after, whether or not you intend to impute any of the missing data. $\endgroup$
    – whuber ♦
    Commented Jun 1, 2017 at 21:46

3 Answers 3

Reset to default
1
$\begingroup$

You probably need to do something about it, because using the data without dealing with it will make extremely strong assumptions (depending on the analysis method that might be missing completely at random, or that these days had an average of the whole year, or of the last non-missing day, all of which is most likely quwstionable).

If you know why the data are missing that would help (e.g. no record created if the count was zero, someone randomly lost the notepad or spilled coffee over it etc.). It would also help, if you had data from another source for the missing days, you could e.g. do some form of multiple imputation (you may have to create your own) across multiple data sources and the mussing data would be much likely to be missing at random conditional on the other data.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Yeah, as far as I know, doing stuff like substituting values from the preceding day sounds like a lot of information loss and bias to me. As far the data in question goes, I have no other sources, I do not know of the reason data are missing (although personally I'd chalk it up to the stations not really reporting much at all, its a 3rd world country after all) Is it really alright to do some form of multiple imputation without having the missing data to begin with? $\endgroup$
    – ace_01S
    Commented May 7, 2017 at 10:59
  • 1
    $\begingroup$ On a side note, I really hate the examples on the internet that induce missing values artificially. It doesn't account for real world data and how missing data is almost always never around to confirm it with by error measurements. $\endgroup$
    – ace_01S
    Commented May 7, 2017 at 11:00
0
$\begingroup$

Similar trends exit in climate data and if missingness is reported for the early part of the dataset, I suggest you apply possibly a time series imputation scheme using the part of the data that is complete. You can train your model by having a training set and a test set. Then the missing values could be adequately imputed. Hope it helps.

Improve this answer
$\endgroup$
0
$\begingroup$

The method suggested by your professor is referred to as: "LOCF - Last observation carried forward"

As you already realized, this is actually one of the worst imputation methods you can choose. (especially for larger gaps)

Are always all 16 columns missing at once for some period of time? (if not imputation could be done using the correlations between these variables)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.