1
$\begingroup$

I am examining the use of underground pipe network information in the prediction of soil moisture in metropolitan areas; drinking water pipes leak water to the surrounding soil, storm sewer and wastewater pipes receive water from the surrounding soil. I am assuming certain leakage rates based on the size and age of the pipes - older pipes leak/receive more, larger pipes leak/receive more. I am missing information for certain pipes on their age and size.

Because this is a first pass at building a working model (set of mathematical equations predicting soil moisture), I don't need the model to be perfect - but I do need to replace my NODATA values with some estimate of age and size.

What is the best method for replacing my NODATA values? With the mean, median, or a random set of values that follow the distribution of the original data?

Total number of pipes: 10779, with 1034 = NODATA. Age range: 1858 - 1992.

Improve this question
$\endgroup$
2
  • $\begingroup$ The choice of imputation procedure depends fundamentally on why these data are missing. It is plausible that the missing information will more often be for older pipes and these will tend to leak more. In such a case neither the mean, median, nor a random choice will be accurate. $\endgroup$
    – whuber ♦
    Commented Oct 6, 2015 at 3:38
  • 1
    $\begingroup$ I understand that point, but at the same time, using no value in my mathematical model (which would be the same as saying a pipe wasn't present) for cells with a pipe age or size of NODATA will also be inaccurate. But, I guess there is no way to determine which leads to a greater inaccuracy. $\endgroup$
    – traggatmot
    Commented Oct 6, 2015 at 4:02

1 Answer 1

Reset to default
1
$\begingroup$

If you are interested in relations between variables then I would not use the mean or the median. You can see what that does below: here I imputed $x$ with the mean (in this case 0). The median won't make it any better.

enter image description here

A random draw from the distribution of $x$ would be even worse. In my experience you have to spend a lot of time and effort in order to get an imputation right -- or you just remove the observations with missing values. "Quick fixes" typically make the situation much worse.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I am not interested in relations between variables. But I am curious, why would a random distribution be even worse? The distribution wouldn't have to be perfectly random either - couldn't it be performed following the current distribution of values? And since your data is 2-dimensional, you'd have to randomly draw (or draw following the current distribution) from both the x and y ranges. $\endgroup$
    – traggatmot
    Commented Oct 6, 2015 at 17:42
  • $\begingroup$ You start your question with "I am examining the use of underground pipe network information in the prediction of soil moisture", so you are interested in the relationship between the characteristics (= variables) of pipes and soil moisture (= another variable). $\endgroup$
    – Maarten Buis
    Commented Oct 7, 2015 at 7:41
  • $\begingroup$ I assumed you wanted to draw from the random distribution of x alone. If your data is purely categorical and each cell in your k-dimensional table has a sufficient number of observations, then you can do what you want. It is called hot-deck imputation. Like most imputation models, they sound easy enough in theory, but in practice implementing them using real data is very tricky. So if you want to do that, you should reserve a significant amount of time to get it right. If you just want a quick fix for a preliminary analysis, then just removing missing values is by far your safest bet. $\endgroup$
    – Maarten Buis
    Commented Oct 7, 2015 at 7:48
  • $\begingroup$ I'd love to talk you about this in a chat room, but I'm not sure how to set one up. $\endgroup$
    – traggatmot
    Commented Oct 7, 2015 at 8:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.